How OpenChronology was built

#64 · Syslog receiver and end-to-end realtime timeline tests

2026-10-10T18:49:50Z CE

• timeline-app
• pull request
• logs

Fixes #49. Fixes #50. ## #49: syslog to the OpenChronology API `server/syslog-receiver.mjs` (`npm run syslog`, or `docker compose --profile syslog`): - **Receives** UDP and TCP on port 5514 by default. It parses RFC 5424 and BSD RFC 3164, accepts octet-counted or newline TCP framing (RFC 6587), and takes messages up to 64 KiB. - **Moments:** - time is the exact Unix seconds, keeping RFC 5424 fractions and offsets; BSD times use the year nearest to arrival, in UTC; - the title is the first line, and the full text goes in the notes; - tags are `syslog`, `severity:`, `facility:`, `host:` and `app:`; - the parsed fields and structured data go under `metadata.syslog`. - **Routing** (`syslog-routes.json`): - routes match on `host`, `app` and `msgid` globs, `facility` and a minimum `severity`; - the first matching route wins, `continue: true` fans out to later routes, and unmatched messages go to `default`; - unknown settings are rejected at startup. - **Delivery:** - one queue per timeline, batched every `SYSLOG_FLUSH_MS` (default 1 s, or at 500 messages), up to 5,000 per `PUT /changes`; - each save sends the current revision and the timeline's settings unchanged, and a 409 refetches the revision; - outages, 429 and 5xx keep the batch and retry with backoff up to 30 s; - the bounded queue (`SYSLOG_QUEUE_LIMIT`, 10k) drops the oldest messages when full; - IDs are fixed on receipt, so retries never duplicate; - permanent refusals (403/404/413) are logged and dropped; - counts are logged every minute, and SIGTERM saves what is queued (up to 10 s). - **Compose service:** the same image, bound to host loopback, with `SYSLOG_API_KEY` and `SYSLOG_ROUTES_FILE` in `.env.example`. - **Docs:** `docs/syslog.md` covers setup, routing, the mapping, delivery and the trust model (no TLS; relay through a syslog daemon that has it). ## #50: realtime updated timeline data `test/syslog-live-postgres.mjs` runs the real stack: PostgreSQL, the production Next.js server, the receiver, and an anonymous Chromium viewer of a public timeline. - **Live:** UDP and TCP messages appear in the open editor without a reload, and other apps route to the default timeline. - **Follow latest:** an idle viewer moves to a message far outside its view. - **Restart:** the server is killed mid-stream. The receiver keeps its queue and retries, and the viewer reconnects and catches up, with all moments present. - **Soak:** a steady rate, half UDP and half TCP. It asserts every message is stored exactly once and the viewer never reloaded, and reports viewer latency. If it falls behind, it reports which stage (stored, shown, or receiver counts). - **In CI:** 20/s for 10 s. The PostgreSQL job now installs Chromium for it. `npm run bench:realtime` runs 100/s for 60 s. Local results (Ryzen 7 7730U, 200 ms save interval), also in `docs/benchmarks.md`: | Rate | Messages | p50 | p95 | max | |---|---:|---:|---:|---:| | 20/s | 200 | 125 ms | 225 ms | 234 ms | | 100/s | 6,000 | 128 ms | 220 ms | 254 ms | | 500/s | 15,000 | 151 ms | 242 ms | 360 ms | | 1,000/s | 20,000 | 177 ms | 482 ms | 801 ms | Every message was stored exactly once at every rate. ## Notes - **Shutdown:** the Next.js server doesn't exit on SIGTERM while live streams are open. The test kills it after 5 s, as `docker stop` does after its grace period. Closing live streams on shutdown could be a follow-up. - **Tests:** `test/syslog.test.mjs` covers the parser, framing, routing validation and fan-out, the moment mapping, and ingest against a fake API (batching, conflicts, 503/429 retries, refusals, queue limit, drain). Unit tests (199) pass, and the end-to-end test passes locally.

Open #64 · Syslog receiver and end-to-end realtime timeline tests in the timeline