How OpenChronology was built

#146 · Answer large forks at once and copy their index afterwards

2026-10-14T04:30:29Z CE

• timeline-app
• pull request

Part of #145 (option B, as suggested there): a large fork or duplicate is answered at once, and its index is copied afterwards. ## What changes **Large copies are answered at once.** A fork or duplicate of 20,000 moments and durations or more commits straight away, with: - its history reading through to the forked revision (#138); - its first revision, which only records settings; - its storage charge; - its title and settings; - `oc_timelines.preparing_revision_id`, naming the revision to copy. Smaller copies, and copies whose history can't be shared, are copied or rebuilt in the request as before. **The index is copied in the background** (`store.prepare`). It runs after commit, in one transaction from one snapshot of the source: - **Bounded statements:** the index is copied a page of 50,000 rows per statement, keyed by each table's primary key, so no statement grows with the timeline. The in-request copy uses the same paging. - **A source that saved since the fork:** the copy is brought back to the forked revision by saving their differences into it in place (`revertCopy`, using `differences` and the incremental save path, without recording a revision). - **Rebuilds:** if those differences are too large to save in place, or the source was deleted, the copy is rebuilt from the forked revision's saved history. - **Afterwards:** statistics are refreshed (#143), then the catalogue text. **Concurrency and failures:** - The fork row is locked with `FOR UPDATE SKIP LOCKED`, so one server process prepares a copy at a time. - A crash rolls everything back, and `prepareStale` resumes unprepared copies when the platform starts. - Deleting a copy while it's being prepared is fine. **While preparing:** - `access` answers HTTP 409 ("This copy is still being prepared…") for the copy's index: reads, saves, syncs, forks of it, and pull requests from it. - Metadata, sharing, settings, members and deletion stay available, and the metadata reports `preparing: true`. - On the platform, the timeline's page shows "Preparing your copy" and checks every two seconds, opening the editor once the copy is ready. Its other pages (history, pull requests, moments) redirect there. ## Measured (200,000 benchmark moments, `node bench/fork-save.mjs`) | | Before (#144) | After | |---|--:|--:| | Fork answered | 23–33 s | 0.06 s | | Fork ready | (same as the answer) | about 33 s | | Saving 10 moments once ready | 0.32 s | 0.31 s | The work and storage are unchanged; removing them is option A of #145 (sharing index rows), which this doesn't attempt. ## Tests **New `test/fork-preparing-postgres.mjs`**, added to CI. It holds back the background job so each case is deterministic: - a small copy is ready when answered, and a large one is answered while preparing; - while preparing, reads, saves, forks and syncs get 409, and its metadata shows it as preparing; - once ready it holds exactly the forked revision, copied rather than rebuilt (a spy counts rebuilds), and saves normally; - a source that saved moments, durations, links and settings before the copy: brought back in place, with no rebuild; - a source rewritten in a whole-document save: rebuilt; - a source deleted before the copy: rebuilt from history; - a copy deleted before being prepared: nothing left to prepare; - copies left unprepared: prepared on start. **Updated tests:** - **`next-postgres`:** a preparing fork's page waits, its history page redirects, and the API reports it as preparing and refuses its document; once ready, the editor opens. - **`statistics-postgres`:** it now waits for the background job before checking statistics. Locally, all of these pass: - unit tests (257) and typecheck; - every PostgreSQL suite, including the platform's.

Open #146 · Answer large forks at once and copy their index afterwards in the timeline