How OpenChronology was built

#156 · Stream exported timelines and saved revisions

2026-10-15T03:01:48Z CE

• timeline-app
• pull request

Step 3 of #135: exports and saved-revision reads stream instead of building whole documents in memory. ## What changed **Streaming documents** (`server/document-stream.mjs`). These two endpoints now stream: - `GET /api/timelines/:id/document` (`{timeline, document}`); - `GET /api/timelines/:id/history/:saved` (`{document}`). How they read: - **Cursors:** each reads the moments, durations and relationships a page at a time (2,000 rows) through cursors, in one read-only repeatable-read transaction. - **Output:** the JSON goes out about 256 KiB at a time. - **Errors:** access checks and errors such as 404, a 409 for a revision that isn't kept, or a copy still being prepared happen before the first byte, so they still answer normally. - **Download:** `?download=1` sends the document alone as an `.ochx` attachment. **Order.** Entities are read in orders their indexes give, since sorting a large timeline's moments by time in one statement would run into the 30 s statement timeout: - live moments by ID, through the primary key; - durations by start, through `(timeline_id, first_key)`; - relationships by their key columns; - a saved revision's entities through the history tables' primary keys. The entities are the same as before; only the order differs, and documents don't promise one. I tried walking the moment tree to keep time order, but that took 101 s at 1M, against 12 s by ID. **Next.js:** - **Adapter:** the route adapter (`next-handler.mjs`) collected the whole body. Responses that stream now get a `ReadableStream` that pulls chunks as the client reads. A client that stops reading cancels the stream, which closes the cursors and returns the connection. - **Revision page:** it used to pass the whole document to the browser to build the export. It now links to the streamed download, and shows the JSON inline only for timelines of up to 5,000 moments. **Determinism:** relationships read whole (`relationshipList`, `historyDocument`) are now ordered as `validateDocument` sorts them, rather than in whatever order the scan returned. **Bug fixed during review:** `return streamResponse(...)` inside the handler's `try` let a 409 escape as a 500. It's awaited now; the Next.js test caught it. ## Measurements 8-core VM, PostgreSQL 17, benchmark timelines. Peak memory is the server process's RSS growth while the document is read. | | Whole (before) | Streamed | |---|--:|--:| | Export, 200,000 moments | | 3.0 s, first part 0.15 s, ~0.1 GB | | Export, 1,000,000 moments | 23.6 s, 1.1 GB | 11.9 s, first part 0.17 s, ~0.1 GB | | Saved revision, 200,000 | | 3.4 s, ~0.1 GB | | Saved revision, 1,000,000 | 23.6 s, 1.15 GB | 13.8 s, first part 0.21 s, ~0.1 GB | | Fork of the 1M timeline | | 16.2 s, first part 3.7 s | Reading whole also failed outright past the server's entry limit (200,000 by default). The stream doesn't build a document, so it doesn't hit that limit. **Limit:** a fork's entities are read through its levels and sorted, so its first part waits for that sort (3.7 s at 1M). Documented. ## Tests - **New `test/document-stream-postgres.mjs`** (in CI): streams must give the same document as reading whole, with entities compared in one order, both wrapped and as a bare `.ochx`. It covers: - a 6,000-moment timeline (several cursor pages, several chunks), after edits; - a fork reading through it, an empty timeline and a comparison; - revisions kept as rows, a fork's revision, and a legacy revision stored as a document; - the head of a timeline that keeps no saved history. It also checks that a revision that isn't kept, or a missing timeline, fails before anything is written, and that a stream closed after one part gives its connection back. - **`test/next-postgres.mjs`:** through the Next.js route, `/document` and `?download=1` with the attachment header, a checkpoint download, the revision page's download link, and a guest refused. - **Fault checks, each caught:** dropping a row at page boundaries, and failing to reset the buffer after yieldi…

Open #156 · Stream exported timelines and saved revisions in the timeline