Repository navigation
ci: build with webpack in the nightlies and in the Build job (Turbopack OOMs the .113) - #16870
Merged
Merged
Conversation
…ck OOMs the .113)
A Turbopack production build of this tree needs more memory than the
.113 can give it. Measured alone on the 32-vCPU / 47 GB box on
2026-10-16 (systemd-run cgroup cap, swap off, per-process sampler):
16.3.5 Turbopack, cap 36 GB ......... passes in 25 min, peak ~36 GB
16.3.6 Turbopack, cap 36 GB ......... passes in 24.5 min, peak 33.5 GB
16.3.8 Turbopack, cap 36 GB ......... passes in 25 min, peak 36 GB
16.4.2 Turbopack, cap 36 GB ......... OOM in the compile phase (next-build alone 33 GB),
with or without the FS build cache, the Rust React
Compiler, or a 4-CPU pin
16.3.8 webpack, cap 28 GB ........... passes in 1h23 under load, peak 14 GB
The ~36 GB come from the Rust compile (~21 GB) plus 31 static-generation
workers (17-21 GB). Next to the three omni-light runners, which went from
6-11 jobs/day to 34-81/day on 10-06, the 47 GB VM cannot spare that: every
Build on omniroute-113-5 died since 10-06 (~200 red runs, each
swap-thrashing the box for 17-90 min, guest OOM-killer at 16-41 GB), and
the backend-only nightlies (garak, promptfoo, schemathesis, a11y) have not
been green since 10-13 for the same reason. The Next version is not the
cause within 16.3.x; 16.4.2 is a real regression on top of it.
Every CI step that compiles the app now sets OMNIROUTE_USE_TURBOPACK=0
(the documented webpack escape hatch): the four nightly build:cli steps
and the ci.yml Build job, whose self-hosted branch used to opt into
Turbopack. The hosted Build was already on webpack, so nothing changes
there. Guarded by tests/unit/ci-build-bundler-webpack.test.ts; the
heavy-build-queue test now pins the new policy.
diegosouzapw
force-pushed
the
ci/webpack-for-ci-builds
branch
from
October 16, 2026 19:06
c78d064 to
2828ca0
Compare
diegosouzapw
changed the title
Contributor
CI Coverage Report
Coverage artifact was not available for this run. |
This was referenced Oct 17, 2026
diegosouzapw
added a commit
that referenced
this pull request
Oct 17, 2026
… finish (#16953) ci(nightly-schemathesis): give the webpack backend-only build time to finish The job's 30-minute cap predates the webpack switch (#16870). The run dispatched on 2026-10-17 (38982564979) compiled with webpack for 25 min on the busy .113 and was cancelled by the cap before the probes started, so the nightly stayed red for a reason unrelated to the contract fuzz. 120 minutes covers the backend-only build under load plus the probes. Guarded in tests/unit/ci-build-bundler-webpack.test.ts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
OMNIROUTE_USE_TURBOPACK: "0", the escape hatch documented indocs/reference/ENVIRONMENT.md): the four nightlybuild:clisteps (nightly-llm-securitypromptfoo-guard + garak,nightly-resiliencek6-soak,nightly-schemathesis) and theci.ymlBuildjob, whose self-hosted branch used to opt into Turbopack. The hosted Build was already on webpack, so PR builds do not change.tests/unit/ci-build-bundler-webpack.test.ts;tests/unit/build/heavy-build-queue.test.tsnow pins the new policy instead of the old "Turbopack on self-hosted" expression.Why
A Turbopack production build of this tree needs more memory than the
.113can give it. Measured alone on the 32-vCPU / 47 GB box on 2026-10-16 (runner slot paused,systemd-runcgroup with a hard cap and swap off, per-process sampler):release/v3.8.52@ 0e7e9e1a)turbopackFileSystemCacheForBuild: false, cap 36 GBnext-build25 GB + ~110 loader workers 13 GB)turbopackRustReactCompiler, cap 36 GBnext-buildalone reaches 33 GB in the compile phaseThe ~36 GB are the Rust compile (~21 GB) plus the 31 static-generation workers (17–21 GB). The
.113runs threeomni-lightrunners next to the build slot, and their load went from 6–11 jobs/day (10-03 → 10-05) to 34–81/day from 10-06 on — which is exactly when every Build onomniroute-113-5started dying (~200 red runs, each swap-thrashing the box for 17–90 min, guest OOM-killer at 16–41 GB) and when the backend-only nightlies (dashboard stubbed) stopped being green (10-13 onwards). So within 16.3.x the Next version is not the cause; 16.4.2 is a genuine regression on top of it (close to vercel/next.js#97802) and should not be picked up by the self-hosted build.Alternatives considered: more RAM for the VM (the build needs ~36 GB for itself, so ≥64 GB to coexist with the other runners), or
experimental.cpusto shrink the static-generation workers — both are follow-ups, not blockers; webpack (14 GB) is the fix that works on every runner today.Follow-ups (not in this PR): a webpack cache step for the hosted Build (the old one was removed with the Turbopack switch);
mainstill builds with Next 16.3.5 and is not touched.Related Issues
Target Branch
release/v3.8.xbranch (the highest openrelease/v*), notmainValidation
tests/unit/ci-build-bundler-webpack.test.ts(written first: 2 failures on the current workflows, then green),tests/unit/build/heavy-build-queue.test.ts,tests/unit/vps-build-runner-gate.test.ts,tests/unit/vps-runner-variable-scope.test.ts,tests/unit/check-workflows-provenance-runner.test.ts,tests/unit/quality-rail-gate-membership.test.ts— 24/24 pass.build/next/standaloneproduced)Tests Added Or Updated
tests/unit/ci-build-bundler-webpack.test.ts.tests/unit/build/heavy-build-queue.test.ts(policy change, see above).Reviewer Notes
build:cliruns are backend-only and will take longer on webpack than the Turbopack build took when it still fit; they were not finishing at all since 10-13.USE_VPS_BUILD_RUNNERstays unset (Build on hosted). If the.113slot is re-enabled, it now builds with webpack (14 GB) instead of dying.