Skip to content

fix(docker): drop arm64 Next.js build to 1 page-data worker, the 2-worker fix still OOMs - #11647

Closed
PauloFH wants to merge 3 commits into
cheaper-inference:release/v3.8.51from
PauloFH:fix/docker-arm64-oom
Closed

PauloFH wants to merge 3 commits into
cheaper-inference:release/v3.8.51from
PauloFH:fix/docker-arm64-oom

Conversation

@PauloFH

@PauloFH PauloFH commented Aug 26, 2026 •

Copy link
Copy Markdown
Contributor

Why

Build Docker (linux/arm64) has been red on every push to release/v3.8.51 (and release/v3.8.50) since ~2026-08-24 23:xx UTC — 60+ consecutive failures, including on draft #11442 — always the same signature:

ERROR: process "... npm run build ..." did not complete successfully: cannot allocate memory

right after Collecting page data using 2 workers, on the native ubuntu-24.04-arm GitHub-hosted runner.

What changed

The 2026-08-25 fix (OMNIROUTE_BUILD_WORKERS 8→3, i.e. 7→2 actual Next.js page-data workers, #10060) modeled 2 workers as fitting the 16 GB runner's budget (tests/unit/docker-build-memory-budget.test.ts: 6144 + 2×2560 = 11264 MB under the 12288 MB budget) — but that model's WORKER_PEAK_MB=2560 inference undercounted the real per-worker RSS, so the test kept passing a config that was red on every real run.

This PR:

  • Drops OMNIROUTE_BUILD_WORKERS 3 → 2 (Next derives workers = CIRCLE_NODE_TOTAL - 1, so this is 2 → 1 actual worker) — the floor the regression test still allows.
  • Raises WORKER_PEAK_MB 2560 → 4096 in the regression test so it now correctly rejects the previous 2-worker config (6144 + 2×4096 = 14336 MB > 12288 MB budget) instead of silently passing a red config.
  • New worst case: 6144 + 1×4096 = 10240 MB, ~2 GB under the 12288 MB budget (vs. the previous ~1 GB modeled margin that turned out to be negative in practice).
  • Documents the incident inline in Dockerfile, the regression test, and docs/guides/DOCKER_GUIDE.md (matching the existing comment convention for these ARGs — DOCKER_GUIDE.md was flagged as out of sync by Copilot review and is now fixed too).
  • Adds changelog.d/fixes/11647-docker-arm64-single-worker.md per changelog.d/README.md.

Verification

  • node --experimental-strip-types --test tests/unit/docker-build-memory-budget.test.ts → 3/3 pass with the new Dockerfile values.
  • Confirmed the updated WORKER_PEAK_MB would have failed the previous (2-worker) config, so this is a real regression guard, not just a number bump.
  • Not verified against a live ubuntu-24.04-arm build — the docker-publish.yml workflow only triggers on push to main/release/v* (not on PR from a fork), so this PR's checks can't exercise the actual arm64 build. This is the most defensible next step per the codebase's own documented playbook (same lever as the 8→3 fix), but treat a real push to release/v3.8.51 as the actual test.

Scope

Dockerfile, tests/unit/docker-build-memory-budget.test.ts, docs/guides/DOCKER_GUIDE.md, changelog.d/. Dockerfile.bun does not use OMNIROUTE_BUILD_WORKERS/OMNIROUTE_BUILD_MEMORY_MB or run this build step, so it's unaffected and untouched.

Known unrelated CI red on this PR

Docs Gates (fast-path), No new ESLint warnings, and 4× Unit Tests fast-path are currently failing on this PR, but none touch anything this diff changed:

  • Docs Gates: pre-existing provider-count drift (354 documented vs 356 live) in README.md/PROVIDER_REFERENCE.md/SVG diagrams — unrelated to Docker build args.
  • ESLint: There are suppressions left that do not occur anymore (stale eslint-suppressions.json entries) — unrelated to this diff.
  • Unit tests: failures are in chatCore/Codex-passthrough and live-ws-eaddrinuse-6324 tests — unrelated to the Dockerfile/build-memory-budget test touched here.

Consistent with the already-open release/v3.8.51 base-red issue (#11449). Not fixing these here to keep this PR scoped to the arm64 OOM.

…rker fix still OOMs

The 2026-08-25 fix (7 workers -> 2, cheaper-inference#10060) modeled 2 workers as safe
(6144 + 2x2560 = 11264 MB, under the 12288 MB budget) but did not hold:
60+ consecutive "Build Docker (linux/arm64)" runs from 2026-08-24
23:xx UTC through 2026-08-26 14:xx UTC (release/v3.8.50, release/v3.8.51,
PR cheaper-inference#11442) all died with the same `ResourceExhausted: ... cannot
allocate memory` right after "Collecting page data using 2 workers".

WORKER_PEAK_MB=2560 undercounted the real per-worker RSS. Drop
OMNIROUTE_BUILD_WORKERS to the test's floor (2 -> 1 actual worker) and
raise WORKER_PEAK_MB to 4096 so tests/unit/docker-build-memory-budget.test.ts
actually rejects the 2-worker config it previously passed by mistake.

New worst case: 6144 + 1x4096 = 10240 MB, ~2 GB under the 12288 MB budget.
@PauloFH
PauloFH marked this pull request as ready for review August 26, 2026 14:48
@PauloFH
PauloFH requested a review from diegosouzapw as a code owner August 26, 2026 14:48
Copilot AI lite review requested due to automatic review settings August 26, 2026 14:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The documented OMNIROUTE_BUILD_WORKERS default in docs/guides/DOCKER_GUIDE.md is now out of sync with the new Dockerfile default (and the PR description also needs updating to reflect the added changelog fragment).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Reduces memory pressure in the Docker next build step on GitHub’s 16 GB ubuntu-24.04-arm runners by forcing a single Next.js page-data worker, and tightens the unit-test memory model so it would have rejected the previously “passing” (but still OOMing) 2-worker configuration.

Changes:

  • Drop OMNIROUTE_BUILD_WORKERS default 3 → 2 (Next page-data workers 2 → 1) in Dockerfile.
  • Raise WORKER_PEAK_MB 2560 → 4096 in the regression test so the budget model rejects the prior 2-worker setup.
  • Add a changelog fragment documenting the operational fix.
File summaries
File Description
tests/unit/docker-build-memory-budget.test.ts Updates the inferred per-worker peak RSS used by the memory-budget regression guard.
Dockerfile Lowers the derived Next.js page-data worker count to 1 via CIRCLE_NODE_TOTAL.
changelog.d/fixes/11647-docker-arm64-single-worker.md Adds a changelog fragment describing the arm64 Docker build fix and the updated budget model.
Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread Dockerfile
Comment thread changelog.d/fixes/11647-docker-arm64-single-worker.md
…ault

Copilot review on cheaper-inference#11647 flagged the guide as out of sync with the
Dockerfile's new default (3 -> 2, i.e. 2 -> 1 actual page-data worker).
Also documents that 2 workers (the previous default) still OOM'd on
the arm64 runner, which is why the default sits at the test's floor.
@RaviTharuma

Copy link
Copy Markdown
Contributor

Cross-checked against the latest release Docker failures: both architectures compile and then terminate with BuildKit ResourceExhausted / cannot allocate memory. This PR is the smallest sufficient fix: it reduces actual Next page-data workers from 2 to 1 and leaves about 2 GB modeled headroom; the updated regression test passes 3/3. The overlapping #11663 makes the same runtime change with a larger diff. Recommend merging this PR, closing #11663 as duplicate, then treating the first successful real amd64+arm64 publish as the acceptance test. This blocks the versioned image tracked in #10296.

@RaviTharuma

Copy link
Copy Markdown
Contributor

Superseded by #11663, merged as 124f4cf7 into release/v3.8.51.

That merge contains the same operative fix as this PR: OMNIROUTE_BUILD_WORKERS=2 (one Next.js page-data worker), updates the Docker guide and memory-budget regression test, and adds the changelog fragment. This branch is now conflicting with the release branch, so keeping a second implementation open would only duplicate the merged change.

Closing as superseded. The post-merge Docker publish is being tracked separately because run 33024696933 still failed both architecture jobs.

@PauloFH PauloFH closed this Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants