How OpenChronology was built

#368 · Server package builds retry a stalled mirror and fail in minutes (#367)

2026-10-28T13:01:41Z CE

• timeline-app
• pull request
• packages

Fixes #367. The `v0.12.1` CI run hung in **Server / debian13 → Build the package** for over an hour, held only by the job's 90-minute limit. The same release's own run shows the stall is real and not rare: its fedora44 build took 60 minutes and its ubuntu24.04 build 35, against a normal 2–3. - **apt retries and times out.** Every Dockerfile stage that runs apt first writes `/etc/apt/apt.conf.d/80-retries` (`Acquire::Retries "3"`, `Acquire::http::Timeout "30"`, `Acquire::https::Timeout "30"`): `deploy/debian-server`, `deploy/rpm-server` (its Node.js stage), `deploy/platform`, `deploy/Dockerfile.app` and `deploy/Dockerfile.postgres`. The .deb and .rpm Dockerfiles gain a `base` stage that the native and package stages share, so it's written once per distribution; `Dockerfile.postgres` writes it in its existing `postgres-base`. Nothing else changes; the Compose images keep the file, which only affects retries and timeouts. - **dnf too.** The .rpm build's `base` stage sets `retries=3` and `timeout=30` under `[main]` in `/etc/dnf/dnf.conf` (checked on Fedora 44's dnf5, which reads them). The Rocky desktop job (`desktop.yml`) does the same. - **curl too.** This PR's first CI run caught a stall: fedora44's download of Node.js from nodejs.org failed after 240 s with `curl: (92) HTTP/2 stream 1 was not closed cleanly`. On the rerun it stalled at the same `RUN curl` until the new 15-minute limit cut it off, and `--progress=plain` showed exactly where. So the .deb and .rpm builds' `base` stage also writes `/root/.curlrc` (`retry = 3`, `retry-all-errors`, `connect-timeout = 30`, `speed-limit = 1024`, `speed-time = 30`, `max-time = 180`). A later run stalled at the same download again, this time trickling above the 1 kB/s floor, so each attempt is now also capped at 3 minutes, and four attempts still fit in the 15-minute step. Locally the same download takes under a second. It covers the Rust installer, pgmp and Node.js downloads. `Dockerfile.postgres` passes the same options to its one curl. Only fedora44 has shown this stall so far, three times today. - **The install tests** (`test/server-deb.mjs`, `test/server-rpm.mjs`) configure apt and dnf the same way in their fresh container before installing. A rocky10 install step once took 9 minutes. - **Step time limits** in `server-packages.yml`, based on the last 15 successful CI runs and 5 release runs: | Step | Normal (min–max) | Slow but passed | Limit | | --- | --- | --- | --- | | Build the package (deb, rpm) | 102–192 s | 253–721 s (rocky10, ubuntu24.04) | 15 min | | Install, run, upgrade and remove it in a fresh container | 25–63 s | 90–548 s (rocky10) | 10 min | | Build the platform | 61–82 s | | 10 min | | Build / install on FreeBSD (VM) | 73–191 s | 278–374 s | 15 min | | macOS: Install it as users do / Set it up | 52–97 s / 31–45 s | | 10 / 5 min | | Windows: Build the installer / Install, run… | 161–254 s / 71–91 s | | 15 / 10 min | The limits are a few times the normal time and still above the slow runs that passed. A real hang now fails within 15 minutes, and a rerun can follow. I left Windows' *Native dependencies* alone: how long it takes depends on the vcpkg cache. - **Visible progress.** `scripts/server-package.mjs` runs `docker buildx build --progress=plain`, so the log shows each step's output as it runs, including where a hang happened. - **Tests.** `test/ci-workflows.test.mjs` checks that every Dockerfile stage under `deploy/` that runs apt has the retries and both timeouts, either in the stage or in a stage it starts from, before its first apt call. It checks the same for dnf, and that every curl in a Dockerfile has a retry and a stall limit. It also checks that the two server-package steps have a `timeout-minutes` of at most 20, and that `--progress=plain` is used. The docs (`deploy/README.md`, `.github/workflows/README.md`, `scripts/README.md`, `test/README.md`) describe the changes. **Tested locally:** `npm test` passes (358/358). `node scripts/server-package.mjs debia…

Open #368 · Server package builds retry a stalled mirror and fail in minutes (#367) in the timeline