How OpenChronology was built

#155 · Build the catalogue text in order and stop once it is full

2026-10-15T01:44:57Z CE

• timeline-app
• pull request

Step 2 of #135: rebuilding the catalogue text now costs about the same at any size. ## What changed The catalogue text (`event_text`) is the first 1 MiB of a timeline's titles and notes, rebuilt in the database after saves. `oc_event_text` used to: - aggregate every duration's text in one statement; - gather every moment ID, sorted, into an array, then read moments a chunk at a time until the text was full. So each rebuild grew with the timeline, even though only the first 1 MiB is kept. Now it reads in order and returns as soon as the text is full: - **Durations** come in start order through their `(timeline_id, first_key)` index, with ties ordered by start time, then ID in "C" order. - **Moments** come a page of 1,000 at a time from `oc_events`, the existing moment-tree walk, with a `(time, id)` cursor. It returns them in time, then ID order, and already reads through for forks, so the fork variant no longer rewrites the moment part. **Ordering:** - Durations used to come in ID order; they now come in start order. That only decides which durations make the 1 MiB cut. - The text written from a whole document (`eventSearchText` in `replace`) now uses the same order: durations by start, then moments in time order even when the document lists them in another order. Before, it followed the document's order, so the next refresh rewrote the text, and the catalogue's GIN index, for no reason. **Upgrade statement:** an old statement in `schema.sql` filled `event_text` for timelines whose text was empty. It ran on every migration and re-read every moment of a timeline whose moments have no titles. It now reads at most 20,000. ## Measurements 1,000,000 benchmark moments (`oc_event_text` on its own, three runs): | | Before | After | |---|--:|--:| | Timeline | 5.8–6.6 s | 0.35–0.68 s | | Fork of it | 6.7–8.0 s | 0.62–0.69 s | **Limit:** a fork's durations are read through its levels, and that read can't stream in order: `oc_duration_nodes_at` is materialized, then sorted. So a fork's rebuild still grows with its durations, though not its moments. Log timelines rarely have durations, and forks that diverge are compacted. It's noted in the architecture doc and left for later. ## Tests **New `test/catalogue-text-postgres.mjs`** (in CI) compares the text exactly with a reference implementation, in four places: - the text written from a document; - the text built in the database; - a fork's; - the text refreshed after an incremental save. It covers: - durations that tie on start, and one anchored to a moment; - IDs whose "C" order differs from other collations, and shared times; - stacked entries; - parts over 4,096 characters; - 2,500 moments, paged with shared times across page ends; - the 1 MiB cap filled by moments, and filled by durations alone; - an empty timeline. **Fault checks:** - Ordering durations by ID fails the test. - Paging without the ID part of the cursor fails it. - Removing the last-page exit doesn't, correctly: the next, empty page ends the loop. **Other checks:** - all 35 database suites pass locally, apart from the two browser suites this VM can't run; - 258 unit tests pass; - the build passes; - the schema applies over `main`'s.

Open #155 · Build the catalogue text in order and stop once it is full in the timeline