How OpenChronology was built

#92 · Import log files into timelines with och-import-logs

2026-10-11T22:16:04Z CE

• timeline-app
• pull request
• logs

Closes #65: a log file importer with pluggable formats and metadata mapping. ## Summary `och-import-logs` (`server/import-logs.mjs`) imports historic logs into timelines, **one moment per log entry**, through the API with an API key, beside the syslog receiver. Full documentation: [`docs/log-import.md`](docs/log-import.md). ```sh och-import-logs --format nginx-combined --timeline /var/log/nginx/access.log* och-import-logs --format syslog --timezone +02:00 --dry-run /var/log/syslog.2.gz journalctl -o json --since yesterday | och-import-logs --format journald --timeline - ``` ### Formats (`log-formats.mjs`, `log-time.mjs`) - **Built in:** `syslog` (rsyslog ISO lines, classic lines, and lines with a priority), `rfc5424`, `apache-common`, `apache-combined`, `nginx-combined`, `apache-error`, `nginx-error`, `jsonl`, `journald`. - **Custom (configuration):** a regex with named groups, or JSON; a time reader (`iso`, `apache`, `ctime`, `nginx`, `syslog`, `epoch-s/ms/us/ns`); typed fields; `extends` a built-in format; multi-line `start`/`continuation` patterns. - **Times:** exact Unix seconds, fractions kept. Zones are fixed offsets or IANA names (DST-aware). - **Classic syslog years:** the first line takes the latest year at or before the file's modification time (or `--year`), then lines step across New Year. ### Moments (`log-mapping.mjs`) - **Text:** title, notes and tags are templates. A tag whose fields are empty is dropped, and tags are made valid. - **Metadata:** every field is kept typed under `metadata. `, and a JSON metadata template adds more (`$field`, `$template`, `$const`, typed, nested, empty values dropped), e.g. keys plugins read. - **Severity:** from a field with a value map, or from text rules. - **Durations:** optional, from a numeric field, e.g. an access log's request time. - **Validation:** each moment is checked locally, so one bad entry is refused alone instead of failing its batch. - **IDs:** from each entry's text plus its occurrence among identical entries at the same time, so re-imports overwrite instead of duplicating. ### Routing and filters (`log-routes.mjs`) - The receiver's routes, generalized to any field: globs, numeric comparisons (`">=500"`), severity thresholds, `continue`, `default`. - `--only status>=400`, `--since`, `--until`. - The receiver keeps its exact rules and error messages, via a wrapper. ### Delivery (`timeline-writer.mjs`, now shared with the receiver) - **No drops:** per-timeline batches, and reading waits for each save. Temporary failures (429, 5xx, unreachable) are retried with backoff, honouring `Retry-After`; a 409 fetches the new revision and resends. - **Batch size:** at most `max(64, timeline moments ÷ 8)` entries, up to 5,000 or 12 MiB. Larger saves aren't applied in place: the server rebuilds the timeline and stores a **full snapshot per save**. With full-size batches, the test's 12.5k-entry import cost 60 MB of storage; with this rule it costs 43 MB, and a million-line import avoids ~100 full copies. The limit uses the server's own `event_count`, which a re-import doesn't grow. - **Quota:** when a quota applies, a counting pass estimates the storage the import adds by simulating the server's patches and snapshots. The PostgreSQL test measures 43.10 MB estimated against 43.21 MB charged. If the import doesn't fit, nothing is written (exit 3). - **`--dry-run`:** JSON with counts per timeline, skipped/filtered/unrouted/refused counts, the estimate, the account's usage and whether it fits, the first problems, and sample moments. - **Progress bar:** on stderr; redrawn in place on a TTY, a line every 10 s elsewhere, off with `--quiet`; a final summary always. - **`--help` and exit codes:** 0 imported, 1 partial, 2 usage or configuration error, 3 doesn't fit the quota. ### Server and receiver changes - **`GET /api/timelines/usage`:** the account's or API key owner's used bytes and quota (`null` when none applies), so the importer can check before writing. It's under `/api/timeli…

Open #92 · Import log files into timelines with och-import-logs in the timeline