Skip to content

fix(docker): pin Bun to 1.3.13; gate release on built image authenticating to PostgreSQL - #987

Closed
rhonda-rodododo wants to merge 1 commit into
mainfrom
ll-bun-scram-regression
Closed

rhonda-rodododo wants to merge 1 commit into
mainfrom
ll-bun-scram-regression

Conversation

@rhonda-rodododo

Copy link
Copy Markdown
Collaborator

Summary

A freshly built app image (oven/bun:1.4.2) could not authenticate to PostgreSQL on a clean host (PostgresError: malformed SCRAM message / Malformed proof in client-final-message.), so scripts/run-migrations.ts never ran and the container sat unhealthy. This PR (1) repins the runtime to the version proven on that host, (2) adds the missing check — the built image must authenticate to a real PostgreSQL and migrate — and makes it block CI and release, (3) makes the entrypoint fail loudly instead of hanging.

What I could and could not establish (read this)

I could not reproduce the SCRAM failure on my machine, so the exact version boundary is NOT bisected. Same bun -e query / postgres:17-alpine, 20 connections each, SCRAM stored (rolpassword = SCRAM-SHA-256$…):

Bun image result
1.3.5, 1.3.13, 1.4.0, 1.4.1, 1.4.2 all OK
1.4.2 image, apt-get upgrade layer, non-root, --read-only OK
1.4.2, passwords of length 20/43/63/64/65/100/128, url-safe charset OK
1.4.2 vs 1.3.13, server scram_iterations 1…600000 OK (except iter=1 on 1.4.2: Failed to read data, different error)
1.3.13 and 1.4.2 under qemu-user -cpu Nehalem / max OK
real deploy/docker/Dockerfile app image built from origin/main (Bun 1.4.2) → new verify script OK — passes on my host

Only host-dependent difference found: the oven/bun 1.3.x x64 images ship the baseline build, 1.4.x ships the non-baseline build (Bun v1.3.14 (Linux x64 baseline) vs Bun v1.4.2 (Linux x64)). That is a hypothesis only. Note PostgreSQL emits "Malformed proof" when the proof does not base64-decode to 32 bytes — i.e. the client sent a malformed proof, not merely a wrong password — which points at the client's crypto/base64 path, not credentials. No newer Bun than 1.4.2 exists (latest release 1.4.2, 2026-09-05), so there is no fix to move forward to. The upstream SASLprep work (oven-sh/bun#41605) only affects non-ASCII passwords and is not this.

Pin: all three oven/bun stages → 1.3.13 (digest-pinned) — the version the reporter's A/B shows healthy on the failing host for months, and within the 1.3.x line CI already tests on (bun = 1.3.5). This is a rollback to known-good on that host, not a proven boundary. The Dockerfile comment records the error string and the conditions for lifting it.

The gate (the more important half)

  • scripts/verify-image-db-auth.sh <image>: starts the compose file's pinned postgres:17-alpine on a private network with a random url-safe password (SCRAM asserted), runs the image's scripts/run-migrations.ts with a postgresql://llamenos:…@postgres:5432/llamenos URL, non-root/--read-only like the compose service, then asserts exit 0 and that the schema exists.
  • ci.yml job image-db-auth (runs when backend paths change — deploy/docker/, drizzle/, apps/worker/, scripts/…) and is in ci-status.
  • release.yml job image-db-auth, and release now needs it — so no GitHub Release is published, no downstream workflow dispatched, and docker-stable (needs release) does not push if the image can't migrate.
  • deploy/docker/docker-entrypoint.sh: migration run is bounded (MIGRATION_TIMEOUT, default 180s) and exits 1 with a message. Previously a failed auth left the container looping silently.

Verify-by-breaking

  • Real image built from this branch (Bun 1.3.13): OK: image authenticated … 101 public tables, exit 0.
  • Same script against a simulated broken client (image whose bun wrapper corrupts the DB password, standing in for the SCRAM defect): PostgresError … ERR_POSTGRES_SERVER_ERROR → FAIL: migrations exited 1, script exit 1.
  • I have no genuinely bad Bun image that fails on my hardware, so the honest statement is: the guard fails on an auth failure and passes on a good image; whether it would have caught 1.4.2 depends on the CI runner reproducing the fault. If the new CI job is green while a clean-host deploy still fails, the runner is not exposing the bug and a clean-host smoke test remains necessary.

Test plan

  • bun test tests/orchestrator/{ci,release*}.test.ts tests/orchestrator/fragments.test.ts (121 pass)
  • actionlint/shellcheck on changed files (only pre-existing findings elsewhere in ci.yml)
  • docker build --target app + scripts/verify-image-db-auth.sh on pinned image
  • image-db-auth job green in CI
  • Deploy the rebuilt image to a clean host and confirm migrations run

No issue number was given/found for this, so no Closes line — please link one.

🤖 Generated with Claude Code

…nticating to PostgreSQL

The freshly built app image (oven/bun:1.4.2) failed SCRAM-SHA-256 against
postgres:17-alpine ('Malformed proof in client-final-message') on a clean
host, so migrations never ran. The same host ran 1.3.13 for months.

- Dockerfile: pin all three oven/bun stages to 1.3.13 (digest-pinned), with a
  comment recording why.
- scripts/verify-image-db-auth.sh: run the BUILT image's migration runner
  against the compose file's postgres image over SCRAM.
- ci.yml (image-db-auth, in ci-status) and release.yml (image-db-auth, needed
  by release -> docker-stable): run it before anything is published.
- docker-entrypoint.sh: bound the migration run so an unreachable/unauth-able
  database exits non-zero instead of sitting unhealthy at 165% CPU.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPaW1Gu33GBWCKhEcNa2vz
@rhonda-rodododo

Copy link
Copy Markdown
Collaborator Author

Superseded by #988.

This PR came from the first worker I dispatched, whose brief said to pin back to a known-good Bun. The operator's requirement is that we stay on the latest Bun SQL, so that worker was stopped and re-dispatched with a corrected brief — #988 (keep latest Bun working on CPUID-masked hosts) is the result.

The pin-back is also no longer warranted on the evidence: stock Bun 1.4.2 authenticates to PostgreSQL 17 correctly on any host exposing AVX2/SSE4.2. The SCRAM failure I originally reported reproduced only because my test VM ran without -cpu host, giving it qemu's default qemu64 model and forcing Bun's simdutf fallback path. A real VPS of this class exposes avx avx2 sse4_2.

The useful half of this PR — gating a release on the built image actually authenticating to PostgreSQL — is in scope for #988, which addresses the genuine narrow bug (the fallback path) without pinning back.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PPaW1Gu33GBWCKhEcNa2vz

rhonda-rodododo added a commit that referenced this pull request Sep 26, 2026
…ase on a PostgreSQL smoke on a kvm64 guest (#988)

## Summary

**Root cause:** the 0.19.16 host's VM CPU model hides SSE4.2 in CPUID.
On such a CPU, Bun ≥ 1.4's bundled simdutf 9.0.0 picks a **no-op stub**
for all base64/UTF-8 work. Neither Bun nor PostgreSQL is misconfigured,
and nothing is wrong with channel binding or the password.

**Fix:** stays on **latest Bun (1.4.2)** by pinning simdutf's SSE4.2
kernel in the amd64 image. A fail-loud runtime self-test guards the pin,
and the image is now smoked against PostgreSQL 17 on a **CPUID-masked
KVM guest** before every release publish.

Supersedes #987. That PR pins Bun to 1.3.13, which was rejected, and
1.3.13 **cannot start at all** on the affected CPU class (evidence
below).

### Mechanism (verified in Bun 1.4.2 + simdutf 9.0.0 source, then
reproduced)
- Bun 1.4 moved its Postgres client to Rust
(`src/sql_jsc/postgres/SASL.rs`). The client-final `p=` proof is base64
through `simdutf::binary_to_base64`, and Bun 1.4.0 bumped simdutf from
7.5.0 to 9.0.0.
- simdutf chooses a kernel from **CPUID**: `icelake` (AVX-512), then
`haswell` (AVX2), then `westmere` (SSE4.2). Bun's x64 binary compiles
**no scalar fallback kernel**; I checked the kernel-name strings in the
binary. If CPUID reports no SSE4.2, simdutf selects
`unsupported_implementation`, whose `binary_to_base64` returns `0`.
- On a KVM guest with `-cpu kvm64`/`qemu64`, still a common default CPU
model on VPS / Proxmox / OpenStack hosts, CPUID hides SSE4.2 but the
hardware still **executes** it. So Bun starts, and simdutf silently
falls to the stub. Result:
- an empty nonce (`r=`) and an **empty proof** (`p=`). PostgreSQL's
`pg_b64_dec` yields 0 bytes instead of 32, hence `Malformed proof in
client-final-message` (a *wrong-but-well-formed* proof would give
"password authentication failed", which is why it isn't the password);
- base64 encodes that spin forever, which is the entrypoint hanging at
`Running database migrations...` with a spinning CPU;
- **silent text corruption** in every bun process:
`Buffer.byteLength(non-ASCII)` → 0, `btoa("abcd")` → `"://"`, and
console writes that hang. SCRAM is just where it surfaced first.
- Channel binding is not involved: Bun only offers `SCRAM-SHA-256` with
gs2 `n,,`.

### Reproduction (real KVM guests on this machine, stock
`postgres:17-alpine@sha256:979c43…`)
Debian 13 cloud image booted under `qemu-system-x86_64 -enable-kvm -cpu
<model>`, running official Bun release binaries (baseline builds, the
ones `oven/bun` images ship):

| Guest CPU | Bun 1.3.13 / 1.3.14 | Bun 1.4.0 / 1.4.1 / 1.4.2 | 1.4.2 +
`SIMDUTF_FORCE_IMPLEMENTATION=westmere` |
|---|---|---|---|
| `-cpu host` | OK | OK | OK |
| `-cpu kvm64` ("Common KVM processor", no `sse4_2`) | **killed at
startup** (137) | **hang; server logs 40× `FATAL: malformed SCRAM
message / DETAIL: Malformed proof in client-final-message`** | **OK** |

The exact first-bad release is **1.4.0**, and no newer release fixes it
(1.4.2 is latest). Before 1.4.0, Bun could not run on this CPU class at
all. The 1.3.13 image's four healthy months were on a host with an
honest CPU model.

Ruled out on the way: every emulated CPU with honest CPUID passes on
both versions. That covers QEMU TCG Nehalem, Westmere, SandyBridge,
IvyBridge, Haswell ± pclmul/bmi2/avx/xsave, capped CPUID levels,
EPYC/Rome/Milan/Genoa, Denverton, SierraForest and Dhyana, plus Intel
SDE icx/skx/spr/gnr (AVX-512 kernels) and arm64 (NEON). The real app
image under production hardening (`read_only`, tmpfs,
`no-new-privileges`, 1 CPU) also passes on honest hardware.

### Changes
- **`deploy/docker/Dockerfile`**: the `app` target is now an alias for a
per-arch stage. `app-amd64` sets `ENV
SIMDUTF_FORCE_IMPLEMENTATION=westmere`, and arm64 is unchanged.
- **Why it's safe:** Bun 1.4's own x64 build executes SSE4.2 at startup,
so any CPU that runs Bun can run this kernel. It's image ENV, so the
HEALTHCHECK and `docker exec … bun` get it too.
- **Cost:** SSE4.2 instead of AVX2 for text/base64, which is negligible
here.
- **Exit condition** (documented inline): remove the pin once upstream
Bun ships a fallback kernel or stops selecting the stub, which is proven
when `scripts/image-smoke.sh --cpu kvm64` passes with the pin removed.
- **`scripts/verify-runtime.ts` + `docker-entrypoint.sh`**: startup
self-test under `timeout`. It fails in about 3s with a named check plus
CPU diagnostics instead of hanging. If a future Bun ever renames the
`westmere` kernel, this makes the pin fail loudly, not silently.
- **`scripts/image-smoke.sh`**: runs the built image's real entrypoint
with production hardening against the compose-pinned
`postgres:17-alpine`. It asserts SCRAM auth, all migrations applied, and
schema present. `--cpu kvm64` runs the same inside a KVM guest from a
checksum-pinned Debian 13 image, and asserts the guest really hid
SSE4.2, so it can't pass vacuously.
- **`release.yml` `docker-stable`**: **blocks the publish** on both
smokes; the push rebuilds from the tested build's cache.
**`image-smoke.yml`**: the same pair at PR time for
Dockerfile/entrypoint/migration/lockfile changes, e.g. Dependabot
`oven/bun` bumps.

### The test gap, precisely
CI's E2E jobs already build this Dockerfile on Bun 1.4.2 and run it
against `postgres:17-alpine`, and they pass, because CI runners expose
honest CPUID. What was missing was the deploy target's *hardware class*,
and the new guard adds exactly that.

## Test plan
- [x] Guard goes **red** on a known-bad image: the unfixed `main` image,
Bun 1.4.2, under `scripts/image-smoke.sh --cpu kvm64`:
  ```
  [image-smoke] guest: cpu= Common KVM processor cpuid_sse4_2=no
[image-smoke] guest: [image-smoke] app: [entrypoint] Running database
migrations...
[image-smoke] guest: [image-smoke] FAIL: the entrypoint did not reach
'Starting application' within 240s (running=true)
  [image-smoke] FAIL on KVM guest -cpu kvm64
  ```
- [x] Guard goes **green** on this branch's image, Bun 1.4.2, same
guest:
  ```
[image-smoke] guest: [image-smoke] app: [verify-runtime] text layer OK
(bun 1.4.2, x64, SIMDUTF_FORCE_IMPLEMENTATION=westmere)
[image-smoke] guest: [image-smoke] app: [migrate] All migrations applied
successfully
[image-smoke] guest: [image-smoke] PASS: SCRAM auth + migrations
complete (101 tables)
  [image-smoke] PASS on KVM guest -cpu kvm64
  ```
- [x] Native smoke green on this branch's image (runner-class CPU, 101
tables)
- [x] Self-test fails loudly (~3s, exit 1, named check) with the kernel
pin broken (`SIMDUTF_FORCE_IMPLEMENTATION=bogus`)
- [x] `actionlint` clean; `tests/orchestrator` rails 1342/1342
- [x] `bun run typecheck` ✔ (pre-commit hook), `bun run build` ✔, `bun
run test:worker:unit` ✔ 4060/4060 (201 files)
- [x] `bun run test:worker:integration` against an isolated
`postgres:17-alpine`: 3/4 files, 25/25 tests. The 4th file
(`response-conformance.test.ts`) fails to import `bun:ffi` under Node's
vitest; that's pre-existing, and the tier isn't run in CI
- [x] `bun run test:backend:bdd` against an isolated server + PG17 +
RustFS built from this worktree: **874 passed, 2 failed, 182 skipped**.
Both failures come from the isolated stack's environment: SIP-bridge
health (`@integration @Telephony @sip`, sidecar not running) and the IVR
provider-language rejection (needs telephony provider config). This PR
changes no server code
- [ ] `bun run lint`: 46 **pre-existing**
`@typescript-eslint/no-unused-vars` errors in `apps/worker/__tests__/`;
none in files touched here (lint doesn't cover `scripts/`, `deploy/`,
workflows)

Not changed (outside this PR's owned paths, flagged for the supervisor):
`sip-bridge/` and `signal-notifier/` images run Bun 1.3.5, which on the
evidence above likely can't start on a CPUID-masked host. The upstream
Bun bug report (x64 build lacks simdutf's fallback kernel, so
SSE4.2-hiding CPUID yields a silent no-op text layer) is not filed yet.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: llamenos-auto <333950803+llamenos-auto@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants