timescale.info#149 · Forks read through to their source's index (#145, phase 2)
- timeline-app
- pull request
Phase 2 of #145: a fork or duplicate reads its source's index instead of copying it, so a fork of any size is ready at once. ## How it works **Forking.** Forking a timeline's head pins the source at its index clock (from #147) and records the levels the fork reads: `oc_timelines.index_bases` (nearest first) and `index_clocks`. That is the source, then whatever it reads through to, at most four levels. Nothing is copied. A fork one level past that copies its index, as before (prepared in the background when large). **A fork's index** is its own rows over each level's rows at that level's clock (live, or retired since). - What it changes becomes its own row. - What it removes from a level is hidden by a hidden row of its own with the same key (`hidden` column, `removeRows`). - A fork whose index is rewritten whole (a save rebuilt from a document) reads through to nothing from then on. **Server queries** read through with `indexReader` (`server/index-versions.mjs`), with the levels written into the SQL so it plans as written. It has three forms: - row-by-row checks of nearer levels; - hashed keys for scans (tags, search, snapshots, copies, history pages); - a key lookup that takes the first version found going down the levels. Saves, history, merges, retention and copies all read through. **Walks in the database** (overview, events, duration overview, recent times, catalogue text) hand forks to `_lineage` variants generated from the originals. Those variants rely on two invariants: - A node a fork changes becomes its own along with the whole path above it, and a moment it changes makes its node its own. - So a fork's own nodes only ever have its own nodes above them, and each step down starts at the level its parent was found at. That usually takes one lookup, as on the source. Within a level, a step tries the version (live or retired) its parent had first. **Deletion.** Index rows outlive their timeline while forks read them. `oc_release_index`, run by a trigger on delete, replaces the foreign keys: a deleted timeline's rows go once nothing reads through to it, along with deleted levels it read that nothing else reads. ## Gate: reads on a fork vs its source, 200,000 moments From `bench/fork-reads.mjs`, median of seven runs each: | Read | Just forked | Source moved 5,000 | Both moved 5,000 | |---|---|---|---| | Overview, whole | 1.37× | 1.43× | 1.27× | | Overview, 1% | 1.44× | 1.39× | 1.37× | | Overview, 0.001% in full | 1.32× | 1.35× | 1.22× | | Events, first 200 | 1.32× | 1.35× | 1.38× | | Events, 200 from the middle | 1.13× | 1.35× | 1.42× | | Durations in 1% | 1.42× | 1.50× | 1.65× | | Search | 1.13× | 1.09× | 1.17× | | Tags | 1.13× | 1.20× | 1.31× | | Moment by ID | 1.27× | 2.12× | 1.28× | - **Forking** takes 0.3 s at 200,000 moments; previously the copy was ready after about 30 s. - **Saves:** - A 10-moment save to the fork takes 0.15 s, the same as to the source. - A 2,500-moment save takes about 19 s on either. - **Outliers.** Durations in 1% after both moved reached 1.65× (+14 ms). Moment by ID after the source moved reached 2.12×, but that's 4.5 ms in absolute terms, and it read 1.3× in the other stages. - **Noise.** The machine is noisy: the same read varies by about ±0.2 between runs. ## What it took to get there The first prototype was 1.7× on fresh forks and 3–17× after the source changed. These were fixed in turn: - **Walks.** Level-hinted walks and retired-first hints. Range scans read a level's versions through mirror indexes on the `_retired` tables. - **Interval walks.** The overview's band and arc walks look a fork's nodes up by key instead of joining. The join read every node at every step and took 2 s per overview. - **JIT.** Off for fork transactions and walks. Read-through plans look costly enough to compile, and compiling took 0.5–2 s, far longer than running them. - **Stale statistics.** A save that retires 1,000 or more versions (counted by the triggers), or a large save to a fork, now refreshes statis…
Sources
Open #149 · Forks read through to their source's index (#145, phase 2) in the timeline