How OpenChronology was built

#401 · IndexNow: tell search engines which public pages changed

2026-10-31T04:50:55Z CE

• timeline-app
• pull request

Fixes #389 Adds [IndexNow](https://www.indexnow.org/documentation) so Bing, Yandex, Naver, Seznam and the other participating engines hear of new, changed and removed public pages within minutes. **Off by default.** ## How it works (`server/indexnow.mjs`) - **Setting:** `INDEXNOW_KEY`, 8–128 characters of `[A-Za-z0-9-]` (e.g. `openssl rand -hex 16`). An invalid key stops the server starting with a clear message, like the other site settings. Documented in `docs/administration.md#indexnow`, `.env.example`, `compose.yml`, `deploy/debian-server/openchronology.env`, `docs/docker.md` and `docs/sharing.md#search-engines`. - **Key file:** `platform/proxy.js` answers `/ .txt` with the key as `text/plain` (only that exact path, only when set). - **What is sent:** the platform starts a background job alongside retention and standing (`startIndexNow`, unref'd timers, never two runs at once). About once a minute it reads exactly what the sitemap lists (public timelines, and profiles of non-disabled owners of one) and diffs it with the previous read: - created or made public → its address and the owner's profile - saved (`updated_at` moved) → its address - new slug or owner renamed → old and new addresses, and the profiles - made private or deleted → its old (public) address, so engines recrawl and drop it, and the profile - a timeline that was never public is never sent; moment and duration pages are left to the sitemap. - **Submission:** `POST https://api.indexnow.org/indexnow` with `{host, key, keyLocation, urlList}`, up to 10,000 URLs per request, absolute on `APP_ORIGIN`. Each URL at most once an hour (a log-fed timeline saved all day is sent hourly). 10 s timeout. No answer / 429 / 5xx → retried after 1, 2, 4… up to 60 minutes; 400/403/422 → logged and dropped. Errors are logged, never thrown; requests are never held up. - The first read after a start is only a baseline (the sitemap covers anything changed while down). Nothing is sent when `APP_ORIGIN` is localhost/loopback, which also keeps every test server off the network. - This avoids hooking the ~60 places that write `oc_timelines` (saves, merges, settings, moderation, account deletion, admin…) and needs no schema change. It assumes one platform process holds the key (each process would otherwise submit its own copy). ## Tests - `test/indexnow.test.mjs` (in `npm test`): key validation, the key file path, the diff (save, rename, private/delete, new owner, account rename, reserved names, disabled accounts), the request shape, the hourly limit, batching at 10,000, retry/backoff and drops, DB failure logged, localhost origin never started. `globalThis.fetch` throws if anything tries the network. - `test/indexnow-postgres.mjs` (added to CI): through the real HTTP API, a private timeline created, saved and given a slug sends nothing; made public, saved, renamed, made private and deleted each send exactly the expected addresses to a stand-in endpoint. - `test/next-platform.mjs`: `/ .txt` serves the key; another key's file is 404. - Locally: `npm test`, `npm run typecheck`, `npm run build`, `npm run test:platform`, `test/indexnow-postgres.mjs`, `test/next-postgres.mjs`, `test/sharing-postgres.mjs` (sitemap and robots unchanged), prettier on changed files. ## To enable on a site Set `INDEXNOW_KEY= ` in the server's environment (with `APP_ORIGIN` already the public https origin) and restart; check `https:// / .txt`.

Open #401 · IndexNow: tell search engines which public pages changed in the timeline