Repository navigation
fix(docker): pin Bun to 1.3.13; gate release on built image authenticating to PostgreSQL - #987
rhonda-rodododo wants to merge 1 commit into
Conversation
…nticating to PostgreSQL
The freshly built app image (oven/bun:1.4.2) failed SCRAM-SHA-256 against
postgres:17-alpine ('Malformed proof in client-final-message') on a clean
host, so migrations never ran. The same host ran 1.3.13 for months.
- Dockerfile: pin all three oven/bun stages to 1.3.13 (digest-pinned), with a
comment recording why.
- scripts/verify-image-db-auth.sh: run the BUILT image's migration runner
against the compose file's postgres image over SCRAM.
- ci.yml (image-db-auth, in ci-status) and release.yml (image-db-auth, needed
by release -> docker-stable): run it before anything is published.
- docker-entrypoint.sh: bound the migration run so an unreachable/unauth-able
database exits non-zero instead of sitting unhealthy at 165% CPU.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPaW1Gu33GBWCKhEcNa2vz
|
Superseded by #988. This PR came from the first worker I dispatched, whose brief said to pin back to a known-good Bun. The operator's requirement is that we stay on the latest Bun SQL, so that worker was stopped and re-dispatched with a corrected brief — #988 ( The pin-back is also no longer warranted on the evidence: stock Bun 1.4.2 authenticates to PostgreSQL 17 correctly on any host exposing AVX2/SSE4.2. The SCRAM failure I originally reported reproduced only because my test VM ran without The useful half of this PR — gating a release on the built image actually authenticating to PostgreSQL — is in scope for #988, which addresses the genuine narrow bug (the fallback path) without pinning back. 🤖 Generated with Claude Code |
…ase on a PostgreSQL smoke on a kvm64 guest (#988) ## Summary **Root cause:** the 0.19.16 host's VM CPU model hides SSE4.2 in CPUID. On such a CPU, Bun ≥ 1.4's bundled simdutf 9.0.0 picks a **no-op stub** for all base64/UTF-8 work. Neither Bun nor PostgreSQL is misconfigured, and nothing is wrong with channel binding or the password. **Fix:** stays on **latest Bun (1.4.2)** by pinning simdutf's SSE4.2 kernel in the amd64 image. A fail-loud runtime self-test guards the pin, and the image is now smoked against PostgreSQL 17 on a **CPUID-masked KVM guest** before every release publish. Supersedes #987. That PR pins Bun to 1.3.13, which was rejected, and 1.3.13 **cannot start at all** on the affected CPU class (evidence below). ### Mechanism (verified in Bun 1.4.2 + simdutf 9.0.0 source, then reproduced) - Bun 1.4 moved its Postgres client to Rust (`src/sql_jsc/postgres/SASL.rs`). The client-final `p=` proof is base64 through `simdutf::binary_to_base64`, and Bun 1.4.0 bumped simdutf from 7.5.0 to 9.0.0. - simdutf chooses a kernel from **CPUID**: `icelake` (AVX-512), then `haswell` (AVX2), then `westmere` (SSE4.2). Bun's x64 binary compiles **no scalar fallback kernel**; I checked the kernel-name strings in the binary. If CPUID reports no SSE4.2, simdutf selects `unsupported_implementation`, whose `binary_to_base64` returns `0`. - On a KVM guest with `-cpu kvm64`/`qemu64`, still a common default CPU model on VPS / Proxmox / OpenStack hosts, CPUID hides SSE4.2 but the hardware still **executes** it. So Bun starts, and simdutf silently falls to the stub. Result: - an empty nonce (`r=`) and an **empty proof** (`p=`). PostgreSQL's `pg_b64_dec` yields 0 bytes instead of 32, hence `Malformed proof in client-final-message` (a *wrong-but-well-formed* proof would give "password authentication failed", which is why it isn't the password); - base64 encodes that spin forever, which is the entrypoint hanging at `Running database migrations...` with a spinning CPU; - **silent text corruption** in every bun process: `Buffer.byteLength(non-ASCII)` → 0, `btoa("abcd")` → `"://"`, and console writes that hang. SCRAM is just where it surfaced first. - Channel binding is not involved: Bun only offers `SCRAM-SHA-256` with gs2 `n,,`. ### Reproduction (real KVM guests on this machine, stock `postgres:17-alpine@sha256:979c43…`) Debian 13 cloud image booted under `qemu-system-x86_64 -enable-kvm -cpu <model>`, running official Bun release binaries (baseline builds, the ones `oven/bun` images ship): | Guest CPU | Bun 1.3.13 / 1.3.14 | Bun 1.4.0 / 1.4.1 / 1.4.2 | 1.4.2 + `SIMDUTF_FORCE_IMPLEMENTATION=westmere` | |---|---|---|---| | `-cpu host` | OK | OK | OK | | `-cpu kvm64` ("Common KVM processor", no `sse4_2`) | **killed at startup** (137) | **hang; server logs 40× `FATAL: malformed SCRAM message / DETAIL: Malformed proof in client-final-message`** | **OK** | The exact first-bad release is **1.4.0**, and no newer release fixes it (1.4.2 is latest). Before 1.4.0, Bun could not run on this CPU class at all. The 1.3.13 image's four healthy months were on a host with an honest CPU model. Ruled out on the way: every emulated CPU with honest CPUID passes on both versions. That covers QEMU TCG Nehalem, Westmere, SandyBridge, IvyBridge, Haswell ± pclmul/bmi2/avx/xsave, capped CPUID levels, EPYC/Rome/Milan/Genoa, Denverton, SierraForest and Dhyana, plus Intel SDE icx/skx/spr/gnr (AVX-512 kernels) and arm64 (NEON). The real app image under production hardening (`read_only`, tmpfs, `no-new-privileges`, 1 CPU) also passes on honest hardware. ### Changes - **`deploy/docker/Dockerfile`**: the `app` target is now an alias for a per-arch stage. `app-amd64` sets `ENV SIMDUTF_FORCE_IMPLEMENTATION=westmere`, and arm64 is unchanged. - **Why it's safe:** Bun 1.4's own x64 build executes SSE4.2 at startup, so any CPU that runs Bun can run this kernel. It's image ENV, so the HEALTHCHECK and `docker exec … bun` get it too. - **Cost:** SSE4.2 instead of AVX2 for text/base64, which is negligible here. - **Exit condition** (documented inline): remove the pin once upstream Bun ships a fallback kernel or stops selecting the stub, which is proven when `scripts/image-smoke.sh --cpu kvm64` passes with the pin removed. - **`scripts/verify-runtime.ts` + `docker-entrypoint.sh`**: startup self-test under `timeout`. It fails in about 3s with a named check plus CPU diagnostics instead of hanging. If a future Bun ever renames the `westmere` kernel, this makes the pin fail loudly, not silently. - **`scripts/image-smoke.sh`**: runs the built image's real entrypoint with production hardening against the compose-pinned `postgres:17-alpine`. It asserts SCRAM auth, all migrations applied, and schema present. `--cpu kvm64` runs the same inside a KVM guest from a checksum-pinned Debian 13 image, and asserts the guest really hid SSE4.2, so it can't pass vacuously. - **`release.yml` `docker-stable`**: **blocks the publish** on both smokes; the push rebuilds from the tested build's cache. **`image-smoke.yml`**: the same pair at PR time for Dockerfile/entrypoint/migration/lockfile changes, e.g. Dependabot `oven/bun` bumps. ### The test gap, precisely CI's E2E jobs already build this Dockerfile on Bun 1.4.2 and run it against `postgres:17-alpine`, and they pass, because CI runners expose honest CPUID. What was missing was the deploy target's *hardware class*, and the new guard adds exactly that. ## Test plan - [x] Guard goes **red** on a known-bad image: the unfixed `main` image, Bun 1.4.2, under `scripts/image-smoke.sh --cpu kvm64`: ``` [image-smoke] guest: cpu= Common KVM processor cpuid_sse4_2=no [image-smoke] guest: [image-smoke] app: [entrypoint] Running database migrations... [image-smoke] guest: [image-smoke] FAIL: the entrypoint did not reach 'Starting application' within 240s (running=true) [image-smoke] FAIL on KVM guest -cpu kvm64 ``` - [x] Guard goes **green** on this branch's image, Bun 1.4.2, same guest: ``` [image-smoke] guest: [image-smoke] app: [verify-runtime] text layer OK (bun 1.4.2, x64, SIMDUTF_FORCE_IMPLEMENTATION=westmere) [image-smoke] guest: [image-smoke] app: [migrate] All migrations applied successfully [image-smoke] guest: [image-smoke] PASS: SCRAM auth + migrations complete (101 tables) [image-smoke] PASS on KVM guest -cpu kvm64 ``` - [x] Native smoke green on this branch's image (runner-class CPU, 101 tables) - [x] Self-test fails loudly (~3s, exit 1, named check) with the kernel pin broken (`SIMDUTF_FORCE_IMPLEMENTATION=bogus`) - [x] `actionlint` clean; `tests/orchestrator` rails 1342/1342 - [x] `bun run typecheck` ✔ (pre-commit hook), `bun run build` ✔, `bun run test:worker:unit` ✔ 4060/4060 (201 files) - [x] `bun run test:worker:integration` against an isolated `postgres:17-alpine`: 3/4 files, 25/25 tests. The 4th file (`response-conformance.test.ts`) fails to import `bun:ffi` under Node's vitest; that's pre-existing, and the tier isn't run in CI - [x] `bun run test:backend:bdd` against an isolated server + PG17 + RustFS built from this worktree: **874 passed, 2 failed, 182 skipped**. Both failures come from the isolated stack's environment: SIP-bridge health (`@integration @Telephony @sip`, sidecar not running) and the IVR provider-language rejection (needs telephony provider config). This PR changes no server code - [ ] `bun run lint`: 46 **pre-existing** `@typescript-eslint/no-unused-vars` errors in `apps/worker/__tests__/`; none in files touched here (lint doesn't cover `scripts/`, `deploy/`, workflows) Not changed (outside this PR's owned paths, flagged for the supervisor): `sip-bridge/` and `signal-notifier/` images run Bun 1.3.5, which on the evidence above likely can't start on a CPUID-masked host. The upstream Bun bug report (x64 build lacks simdutf's fallback kernel, so SSE4.2-hiding CPUID yields a silent no-op text layer) is not filed yet. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: llamenos-auto <333950803+llamenos-auto@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Summary
A freshly built app image (
oven/bun:1.4.2) could not authenticate to PostgreSQL on a clean host (PostgresError: malformed SCRAM message/Malformed proof in client-final-message.), soscripts/run-migrations.tsnever ran and the container sat unhealthy. This PR (1) repins the runtime to the version proven on that host, (2) adds the missing check — the built image must authenticate to a real PostgreSQL and migrate — and makes it block CI and release, (3) makes the entrypoint fail loudly instead of hanging.What I could and could not establish (read this)
I could not reproduce the SCRAM failure on my machine, so the exact version boundary is NOT bisected. Same
bun -equery /postgres:17-alpine, 20 connections each, SCRAM stored (rolpassword=SCRAM-SHA-256$…):apt-get upgradelayer, non-root,--read-onlyscram_iterations1…600000Failed to read data, different error)-cpu Nehalem/maxdeploy/docker/Dockerfileapp image built fromorigin/main(Bun 1.4.2) → new verify scriptOnly host-dependent difference found: the oven/bun 1.3.x x64 images ship the baseline build, 1.4.x ships the non-baseline build (
Bun v1.3.14 (Linux x64 baseline)vsBun v1.4.2 (Linux x64)). That is a hypothesis only. Note PostgreSQL emits "Malformed proof" when the proof does not base64-decode to 32 bytes — i.e. the client sent a malformed proof, not merely a wrong password — which points at the client's crypto/base64 path, not credentials. No newer Bun than 1.4.2 exists (latest release 1.4.2, 2026-09-05), so there is no fix to move forward to. The upstream SASLprep work (oven-sh/bun#41605) only affects non-ASCII passwords and is not this.Pin: all three
oven/bunstages →1.3.13(digest-pinned) — the version the reporter's A/B shows healthy on the failing host for months, and within the 1.3.x line CI already tests on (bun = 1.3.5). This is a rollback to known-good on that host, not a proven boundary. The Dockerfile comment records the error string and the conditions for lifting it.The gate (the more important half)
scripts/verify-image-db-auth.sh <image>: starts the compose file's pinnedpostgres:17-alpineon a private network with a random url-safe password (SCRAM asserted), runs the image'sscripts/run-migrations.tswith apostgresql://llamenos:…@postgres:5432/llamenosURL, non-root/--read-onlylike the compose service, then asserts exit 0 and that the schema exists.ci.ymljobimage-db-auth(runs whenbackendpaths change —deploy/docker/,drizzle/,apps/worker/,scripts/…) and is inci-status.release.ymljobimage-db-auth, andreleasenowneedsit — so no GitHub Release is published, no downstream workflow dispatched, anddocker-stable(needsrelease) does not push if the image can't migrate.deploy/docker/docker-entrypoint.sh: migration run is bounded (MIGRATION_TIMEOUT, default 180s) and exits 1 with a message. Previously a failed auth left the container looping silently.Verify-by-breaking
OK: image authenticated … 101 public tables, exit 0.bunwrapper corrupts the DB password, standing in for the SCRAM defect):PostgresError … ERR_POSTGRES_SERVER_ERROR→FAIL: migrations exited 1, script exit 1.Test plan
bun test tests/orchestrator/{ci,release*}.test.ts tests/orchestrator/fragments.test.ts(121 pass)actionlint/shellcheckon changed files (only pre-existing findings elsewhere in ci.yml)docker build --target app+scripts/verify-image-db-auth.shon pinned imageimage-db-authjob green in CINo issue number was given/found for this, so no
Closesline — please link one.🤖 Generated with Claude Code