How OpenChronology was built

#154 · Make saves follow the nodes they touch, and timeline creation linear

2026-10-15T00:29:07Z CE

• timeline-app
• pull request

Step 1 of #135: saves now scale with the nodes they touch, and a timeline is created in linear time. ## What changed **Tree path query** (`MOMENT_TREE.path` and `INTERVAL_TREE.path` in `server/avl.mjs`). Before a save changes anything, it reads the nodes it needs in one query. - **Before:** the query walked from the root once per key, deduplicated `(key, id)` pairs on every step, and parsed each key from text at every step. - **Now:** the keys are cast once and descend together. Each node splits the keys that reach it between its children (`ARRAY(... WHERE k n.time`), so each node on the paths is read once. - **Why it matters:** appended log moments share nearly their whole path. A 5,000-moment append now reads a few dozen nodes, instead of 5,000 × depth lookups. - **Comparisons:** tree comparisons parse each node's and key's time once. Parsing rationals was the largest CPU cost of a save. **Creating a timeline was quadratic.** Starting saved history (`copyKeys` in `history.mjs`, run when a timeline is created) copied each page by looking every entity up again by key. - Durations are looked up by an expression (`band->>'id'`), and the planner doesn't use its index before a new timeline's statistics count its rows. Every lookup read every duration. - At 200,000 moments that one statement took up to 144 s. At a million, creation took about 96 minutes. - Each page is now copied by the statement that reads it (`copyAll`), in key order through an index. **Adaptive batches in the log writers** (`TimelineWriter.send`, used by `och-import-logs` and the syslog receiver). - Batches are split to a size that adapts per timeline: - a save taking 8 s or more halves the next ones, down to 64 entries; - so does one the server stopped as too slow, which is sent again, halved; - quick full saves let them grow back, up to 5,000. - Moments go before durations, which may name them. Resending a part is harmless: saves are upserts by ID. - The server now answers a statement timeout (`57014`) with **503** and a clear message, instead of a generic 500. The writers already treat 503 as temporary. **Benchmark:** `bench/save-size.mjs [sizes] [runs]` times creation, then appends of 10, 1,000 and 5,000 moments and a 1,000-moment change spread through the timeline, at each size. ## Measurements 8-core VM, PostgreSQL 17, benchmark timelines (`bench/save-size.mjs`, median of 3): | | `main` | This PR | |---|--:|--:| | Create 200,000 moments | 283 s | 126 s | | Append 10 / 1,000 / 5,000 at 200,000 | 0.08 / 1.73 / 4.04 s | 0.11 / 0.91 / 2.49 s | | Change 1,000 spread at 200,000 | 0.79 s | 0.56 s | | Create 1,000,000 moments | ~96 min (earlier measurement) | 636 s | | Append 10 / 1,000 / 5,000 at 1,000,000 | 5,000: 21–25 s (in #135) | 0.24 / 1.35 / 5.22 s | | Change 1,000 spread at 1,000,000 | | 0.75 s | The largest batch at a million moments is now well under the 30 s statement timeout. Creation queries went from 45 s at 200k to 111 s at 400k, close to linear. ## Tests - **New `test/tree-paths-postgres.mjs`** (in CI): after one prefetch, walking to every key (and every child on the way) needs no further query. It covers moments and durations, on a timeline and on a fork reading through it, with: - existing and new times, shared times, and keys before and after everything; - appended keys, which must read under 60 nodes. Prefetching only fills a cache, so the other tests can't see a path query that misses nodes. Swapping the split in either tree's query fails this test; both fault checks were run. - **`test/syslog.test.mjs`:** the writer splits over-limit batches with durations last, halves after slow saves down to the minimum, grows after fast ones, keeps limits per timeline, and resends a 503'd save halved, failing at the minimum. - **Other checks:** - all 34 database suites pass locally, apart from the two browser suites this VM can't run; - 258 unit tests pass; - the build passes. ## Left for later in #135 - **Catalogue text:** `oc_event_text` still aggregat…

Open #154 · Make saves follow the nodes they touch, and timeline creation linear in the timeline