timescale.info#173 · Follow the systemd journal into timelines; name journald imports by their cursor
- timeline-app
- pull request
- logs
Closes #151. ## The question in the issue: would syslog give the same data as journalctl? Mostly not. Forwarding the journal to syslog (`ForwardToSyslog=`, or rsyslog's `imjournal`) carries the message text, its time (to the second for RFC 3164), severity, facility, host, program and PID. It loses: - the systemd unit (`forgejo.service`); - microsecond times; - everything else the journal keeps: user, command line, executable, boot ID, source file and line, and the program's own structured fields. Its moments also aren't the ones `och-import-logs --format journald` makes, so a range covered by both would appear twice. So this adds a live journald follower. It gives the same data as the historic `journalctl -o json` import, as the same moments. `docs/journald.md` has the full comparison table. ## The journald follower `server/journald-receiver.mjs` (`npm run journald`; the `openchronology-journald` service in the Debian and RPM packages): - **Reads** `journalctl --follow --output=json` on this machine, or on another over SSH (`JOURNALD_HOST`, `JOURNALD_SSH_KEY`, with a `known_hosts` kept beside its cursor). Arguments are quoted for the remote shell, so cursors containing `;` survive. - **Narrows** what it reads with `JOURNALD_UNITS` and `JOURNALD_MATCHES`. - **Routes** entries with `JOURNALD_TIMELINE`, or with `JOURNALD_ROUTES` rules like the importer's, matching on `unit`, `app`, `host`, `severity` or any journal field. - **Makes the same moments as the importer**, using the journald format and mapping from `log-formats.mjs` and `log-mapping.mjs`: - title: the message; - tags: `journald`, `severity:`, `host:`, `app:`, `unit:`; - metadata: every field. - **Resumes after restarts.** Once everything read has been sent, it keeps the last entry's cursor; on restart it runs `journalctl --after-cursor`. Nothing written while it was down is missed and nothing is sent twice; a crash at worst re-reads entries with the same IDs, which only overwrite. - **Survives journalctl stopping** (rotation, an SSH drop): it restarts journalctl after the last entry read, backing off up to a minute on repeated quick failures. - **Shares delivery with the syslog receiver.** The queues, batching, retries and draining move from `SyslogIngest` into `server/moment-ingest.mjs`, and both receivers now extend it. Syslog behaviour is unchanged (its tests and the live syslog test pass). ## A fix to the journald importer While testing this I found that **journalctl prints an entry's fields in a different order on every run**: the same entry gave three different lines in three runs. The importer named moments by a hash of the line, so re-importing an overlapping journald range (for example Forgejo history) **added duplicates** instead of overwriting. Journald entries are now named by their `__CURSOR`, through a new `identity` on formats (`compileFormat(...).identity`); every other format keeps hashing the line. One limit: moments imported before this fix used the old names, so re-importing a range imported before this release would add those entries once more. ## Packaging **Debian and RPM** install: - `openchronology-journald.service`: runs as `openchronology` with `SupplementaryGroups=systemd-journal` and `StateDirectory=openchronology-journald` for its cursor; hardened like the syslog unit; - `journald.env`, from `/usr/share/openchronology/journald.env.example`; - prerm and spec stop/disable entries. **Not packaged elsewhere:** FreeBSD, macOS and Windows have no journald, so they don't get it. **Package smoke tests:** `server-deb` and `server-rpm` check that the unit and configuration example are installed, and that the follower loads with the bundled Node.js. ## Tests - **New:** `test/journald.test.mjs`, with a fake API and a stand-in journalctl, covers: - the command, including SSH quoting and key options; - moments and routing by unit; - the same ID for an entry whose fields are reordered; - the cursor kept and resumed from; - restarting journalctl after th…