How OpenChronology was built

#295 · Whole-timeline downloads are cached per revision

2026-10-25T22:35:11Z CE

• timeline-app
• pull request

Closes #204. The server now keeps each `.ochx`/`.och` file it makes for a timeline's revision and sends that file to the next download of the same revision, instead of building it from the database again. ## What's cached The editor's large-timeline export (`/api/timelines/{id}/export/{name}.ochx`), `.ochx` downloads with `?download=1` (a timeline's, or a saved revision's), and `.och` files (`/api/timelines/{id}/file`). Comparisons aren't cached. A save makes a new revision, so its next download is made fresh and kept in turn. Keys are ` :r : `, or `:s :` for a saved revision. ## How - **`server/export-cache.mjs`**, with entries in a new `oc_export_cache` table (idempotent `schema.sql`). - **First download:** sent as it's made, exactly as before, while it is written to a staged file beside it. The file is kept only if it is complete, under the limit, and of the revision asked for. A save that lands mid-build makes it newer, so it's sent but not kept. - **One build per key:** a build claims its row, and the claim works across servers. A download that arrives while this process is building waits for it, then gets the cached file. One arriving while another server is building sends its own copy without storing it. A claim abandoned for 15 minutes is taken over. - **Access:** checked on every download, cached or not, through `store.access`. A saved revision since forgotten (history turned off) is refused, via `store.revisionReadable`. - **Cached responses:** - `ETag` / `If-None-Match` answers 304, and single `Range` requests get 206 or 416. - `Content-Length` is set, with `Cache-Control: private, no-cache`. - This goes through both the Node server and the Next.js adapter, which now passes byte chunks through. - **Eviction:** least recently downloaded first, keeping the total within `EXPORT_CACHE_MAX_BYTES`. A file larger than the whole limit is sent but not kept. - **Removal:** - Deleting a timeline forgets its files at once. - A sweep (at the first download, then every 10 minutes on an unref'd timer) removes files of timelines deleted elsewhere (administration, account deletion), abandoned builds, and orphan folders on disk. - It also removes files handed out at the public address whose timeline is no longer public: made private, or restricted by moderators. Making a timeline private through its settings removes those files immediately. - **Public address:** with `EXPORT_CACHE_PUBLIC_URL` (a web server or CDN in front of the directory or bucket), public timelines' cached files redirect there (302). This applies to the export address and saved-revision downloads only, not `/file` or `document?download=1`, which scripts `fetch` and which would need CORS. Private timelines' files are never redirected. File names are 32 random hex digits plus the file name. - **S3:** a single PUT, a ranged GET and a DELETE, signed with Signature Version 4 in `node:crypto`, so there is no new dependency. The signer matches the AWS test suite's `get-vanilla` vector, and the backend works against a real S3 server (SeaweedFS). Path style is the default, with host style as an option. - **Also fixed:** `streamResponse` added a `close` listener on every backpressure wait and never removed it. Long downloads hit Node's MaxListeners warning. ## Configuration `.env.example`, the packaged env template, `compose.yml`, `docs/administration.md` ("Cached downloads") and `docs/docker.md`: | Setting | Meaning | | --- | --- | | `EXPORT_CACHE` | `disk`, `s3` or `off`. Unset, `disk` when a directory is known (`EXPORT_CACHE_DIR`, or systemd's `CACHE_DIRECTORY`), else off | | `EXPORT_CACHE_DIR` | the disk cache's directory (staging for S3) | | `EXPORT_CACHE_MAX_BYTES` | e.g. `10GB`; 2 GiB unset | | `EXPORT_CACHE_PUBLIC_URL` | a file server or CDN serving the cache's files | | `EXPORT_CACHE_S3_ENDPOINT`, `_REGION`, `_BUCKET`, `_PREFIX`, `_ACCESS_KEY_ID`, `_SECRET_ACCESS_KEY`, `_PATH_STYLE` | an S3-compatible bucket (AWS, R2, Spaces, MinIO…) | The Debian/RPM unit gains `Ca…

Open #295 · Whole-timeline downloads are cached per revision in the timeline