timescale.info
Sign inSign up

How OpenChronology was built

Every merged pull request of openchronology/timeline-app, its first week of commits and its releases; and before them the 2020 PureScript editor, from snapshots of its deleted repositories in the Wayback Machine. Written to this server through the HTTP API.

327 moments · 2020-03-09T00:15:08Z CE – 2026-11-02T20:11:21Z CE · by @athan · @athan

#294 · The sitemap is an index of pages that covers every public timeline

  • timeline-app
  • pull request

Closes #287. `/sitemap.xml` used to be a single sitemap capped at the 45,000 most recently changed public timelines and 4,000 profiles, so a larger site lost its oldest timelines from it. It is now a **sitemap index**: - `/sitemap/pages.xml`: the site's own pages (dashboard, plugins, download, API reference, help and each help page, terms, privacy, legal). - `/sitemap/timelines- .xml`: every public timeline, 45,000 to a page, with `lastmod`. - `/sitemap/profiles- .xml`: every profile that owns a public timeline (not suspended), 45,000 to a page, with `lastmod`. - The index gives each page the latest change on it. `robots.txt` still names `/sitemap.xml`. **Queries:** a page starts at its first entry's ID and is read by an index range (`id >= start ORDER BY id LIMIT 45000`), never by `OFFSET`. The index numbers the public IDs once with `row_number()` and takes every 45,000th as a page start. At 200,000 public timelines: the index query takes 0.85 s, and a full page 0.27 s in SQL, 1.1 s over HTTP, 6.1 MB (the limit is 50,000 addresses and 50 MB). **Code:** `platform/lib/sitemap.js` (new) builds both, served by `app/sitemap.xml/route.js` and `app/sitemap/[file]/route.js`; `app/sitemap.js` is gone. Unknown pages and malformed IDs answer 404. `SITEMAP_PAGE_SIZE` (1–45,000) lowers the page size; the tests use it. **Tests:** - `test/sharing-postgres.mjs` runs with one entry per page, reads the index and then every page, and checks the timeline, the profile and `/privacy` are listed, that no address appears twice, that a private timeline never appears and that unknown pages are 404. - `test/next-postgres.mjs` follows the index to find its address. - Both pass against PostgreSQL 16 with pgmp, as do `npm run test:platform` and `npm run build`. `npm test` passes except the README-contents check, which only tripped on the worktree's `node_modules` symlink. **Docs:** `docs/sharing.md` describes the index and its pages, and how to submit `sitemap.xml` in Google Search Console.

Sources

Open #294 · The sitemap is an index of pages that covers every public timeline in the timeline