How OpenChronology was built

#158 · Apply large patches in place, a slice at a time

2026-10-15T04:40:31Z CE

• timeline-app
• pull request

Part of step 5 of #135: large patches stop rebuilding the whole index. ## Why A patch changing more than an eighth of a timeline was rebuilt from a whole document in Node (`incremental` in `server/incremental.mjs`). Through the API that rarely mattered, since `/changes` takes at most 5,000 entries. But the server makes large patches itself: - **Retention** and **delete-range** delete up to 200,000 moments per save. - **Merges and syncs** can carry many changes. On a log timeline of a million moments, a retention pass therefore rebuilt everything. Since #154 made in-place saves follow the nodes they touch, in place is cheaper: | | In place | Rebuilt | |---|--:|--:| | Delete 30,000 of 200,000 moments | 26 s, 155 MB | 103 s, 689 MB | | Rebuild of a 1M-moment timeline | | ~10 min, over 1 GB | ## What changed **Threshold.** A patch rebuilds only when it changes more than half of a timeline under 200,000 moments (`REBUILD_UNDER`, `REBUILD_SHARE`). Any other patch is applied in place, however large. In place costs about 1 ms per change and a rebuild about 0.5 ms per moment, so break-even is around half. Past 200,000 moments, building a whole document in memory isn't worth it at all. Plugin changes still rebuild; #157 is about removing that. **Slices.** A large in-place patch is applied 5,000 changes at a time (`slicePatch`, `saveIncrementally`), moments first, then durations (which may name them), then relationships (which may link them). - Each slice is applied to the timeline as the previous one left it: its row is written between slices. - It's still one revision, with one history record over everything the slices touched. - No statement grows with the patch. At a million moments, with a fork reading the timeline so every deleted row is kept as a version, deleting 200,000 moments in one piece ran a 29 s statement, one second under the server's 30 s statement limit. In slices, the longest statement was 10 s, peak memory 259 MB instead of 510 MB, and the whole pass about 5 minutes. 20,000-change slices took about the same time, so the smaller slices keep memory lower. **Log importer:** batches follow the new threshold: at most `max(64, moments ÷ 2)` entries, up to 5,000. ## Tests **`test/incremental-postgres.mjs`**, the oracle comparing in-place saves with rebuilds: - every other random patch is now applied in slices of three changes; - a new case grows a 10-moment timeline by 300 moments, 40 durations, 30 relationships, a deletion and a move, applied in place in slices of 7. That moves the trees' roots repeatedly. It must match a rebuild of the patched document (queries, search, links, arcs), keep valid AVL trees, and read back as one revision. **Fault checks, each caught:** - not updating the timeline between slices (the new case catches it; the random patches alone didn't); - recording history for only the last slice's keys. **Other checks:** - all 36 database suites pass locally, apart from the two browser suites this VM can't run; - 258 unit tests pass (the importer's batch limit updated); - the build passes. ## Not changed - **Delete-range's API contract stays:** up to 200,000 moments per call, with `more: true` for the rest. - **Retention's single save holds the timeline's lock** for those minutes; splitting retention into several saves is a possible follow-up. - **Whole-document saves and merges of revisions** from before history was kept as rows still rebuild, by nature.

Open #158 · Apply large patches in place, a slice at a time in the timeline