timescale.info#203 · .och conversions hold less memory, and their file limit is a setting (#202)
- timeline-app
- pull request
Closes #202. ## Memory Measured at 200,000 moments, the converter's peak was split into: - 164 MB parsing the JSON; - another 164 MB from `save()` copying the whole document; - 56 MB building the index. Two changes: - **`save()` no longer copies the document.** Only a document with legacy duration links is copied, to convert them. - **`och-convert write` streams the moments.** Each moment's metadata stays raw JSON (`Box `, serde_json's `raw_value` feature) until the new `save_streamed` writes that moment. `save_streamed` runs the same checks as `validate`, moment by moment: settings first, each moment as it comes (legacy links converted), entities last. Any problem rolls the save back. `validate` is split into `validate_settings`, `event_problem` and `validate_entities`, so both paths share the checks. | `och-convert write` | Before | After | | --- | --- | --- | | 200,000 moments, one-word titles (13 MB JSON) | 404 MB | 92 MB | | 150,000 moments with notes and tags (32 MB JSON) | failed: `.och` over 32 MiB | 79 MB, 43 MB `.och` | Reading a file back (`och-convert read`) takes about 4.5 times the file's size, which is why the memory limit scales with the file limit (below). ## File limit: `TIMELINE_FILE_BYTES` As agreed: a size from 1 MiB to 1 GiB, 64 MiB when unset. An invalid value stops the server at startup. **What it sets:** - the upload limit; - the export limit; - the converter's input and output, passed to it as `OCH_FILE_BYTES`; - the converter's memory limit: 512 MiB, or eight times the file limit when that is more. **Where it appears:** the session reports `fileLimit`, which the editor checks before uploading. It's documented in `docs/administration.md`, the Debian environment file, `compose.yml` and the OpenAPI document. An `.och` is usually somewhat larger than its JSON. For example, 61 MB of JSON (200,000 moments with a title and a 150-character note) makes a file just over 64 MiB. ## Which limit stopped a conversion These used to be "Invalid or unsupported timeline file." (400). Now: | Stopped by | Status | Message | | --- | --- | --- | | Time (Node's timeout, SIGXCPU) | 422 | "…exceeded its time limit." | | File size (SIGXFSZ, refused input, output past `maxBuffer`) | 413 | "SQLite export exceeds 64 MiB." or "Timeline exceeds the 64 MiB file limit." | | Memory (a failed allocation aborts) | 413 | "…exceeded its memory limit." | ## Testing - **`npm test` (295):** includes `TIMELINE_FILE_BYTES` parsing and the session's `fileLimit`. - **`cargo test` for native-store (25):** a new test checks that a streamed save matches a whole one, legacy links included. It also checks that duplicate IDs, a failing moment, a dangling relationship and more than 200,000 moments roll the save back. - **`test/files.mjs`, against the built converter:** with a 1 MiB limit, an upload over it is refused. So is an export whose JSON is over the limit, and one whose JSON is under it but whose file is over (SIGXFSZ). Each failure kind maps to its message. Uploads over 64 MiB get 413 over HTTP.