Repository navigation
ci: make the .113 omni-build slot opt-in (USE_VPS_BUILD_RUNNER) - #16833
Merged
Merged
Conversation
Since 2026-10-06 every Build that landed on omniroute-113-5 died: the Turbopack `next-build` grew to 16-41 GB on the 32-core box and the guest OOM-killer took it (CONSTRAINT_NONE, the whole 47 GB VM). ~200 runs in a row, each swap-thrashing 17-90 min, while the same build passes in 8-12 min on the hosted runner (webpack, fork PRs). The slot is single (#12048), so same-repo PRs queued up to 12 h behind it only to fail. Every `omni-build` job now needs USE_VPS_RUNNER *and* USE_VPS_BUILD_RUNNER == 'true'; unset = hosted fallback, so a parked build box can no longer stall publishes. The ci.yml Turbopack switch mirrors the new condition, and the docker-publish preflight / nightly offline alert follow the same switch (a deliberately parked pool is not an outage). Re-enable the box with `gh variable set USE_VPS_BUILD_RUNNER --body true`.
diegosouzapw
added a commit
that referenced
this pull request
Oct 16, 2026
Same change as #16833 (release/v3.8.52), applied to main's own workflow files: main pushes still routed Docker publish, npm publish and Release-Green to the omni-build slot, which is parked. Since 2026-10-06 every Build that landed on omniroute-113-5 died: the Turbopack `next-build` grew to 16-41 GB on the 32-core box and the guest OOM-killer took it (CONSTRAINT_NONE, the whole 47 GB VM). ~200 runs in a row, each swap-thrashing 17-90 min, while the same build passes in 8-12 min on the hosted runner (webpack). Every `omni-build` job now needs USE_VPS_RUNNER *and* USE_VPS_BUILD_RUNNER == 'true'; unset = hosted fallback, so a parked build box can no longer stall publishes. Re-enable with `gh variable set USE_VPS_BUILD_RUNNER --body true`.
3 of 4 tasks
diegosouzapw
added a commit
that referenced
this pull request
Oct 16, 2026
…R) (#16835) ci: make the .113 omni-build slot opt-in on main (USE_VPS_BUILD_RUNNER) Same change as #16833 (release/v3.8.52), applied to main's own workflow files: main pushes still routed Docker publish, npm publish and Release-Green to the omni-build slot, which is parked. Since 2026-10-06 every Build that landed on omniroute-113-5 died: the Turbopack `next-build` grew to 16-41 GB on the 32-core box and the guest OOM-killer took it (CONSTRAINT_NONE, the whole 47 GB VM). ~200 runs in a row, each swap-thrashing 17-90 min, while the same build passes in 8-12 min on the hosted runner (webpack). Every `omni-build` job now needs USE_VPS_RUNNER *and* USE_VPS_BUILD_RUNNER == 'true'; unset = hosted fallback, so a parked build box can no longer stall publishes. Re-enable with `gh variable set USE_VPS_BUILD_RUNNER --body true`.
Contributor
CI Coverage Report
Coverage artifact was not available for this run. |
This was referenced Oct 16, 2026
Merged
arashashrafii
pushed a commit
to arashashrafii/OmniRoute
that referenced
this pull request
Oct 24, 2026
…R) (cheaper-inference#16835) ci: make the .113 omni-build slot opt-in on main (USE_VPS_BUILD_RUNNER) Same change as cheaper-inference#16833 (release/v3.8.52), applied to main's own workflow files: main pushes still routed Docker publish, npm publish and Release-Green to the omni-build slot, which is parked. Since 2026-10-06 every Build that landed on omniroute-113-5 died: the Turbopack `next-build` grew to 16-41 GB on the 32-core box and the guest OOM-killer took it (CONSTRAINT_NONE, the whole 47 GB VM). ~200 runs in a row, each swap-thrashing 17-90 min, while the same build passes in 8-12 min on the hosted runner (webpack). Every `omni-build` job now needs USE_VPS_RUNNER *and* USE_VPS_BUILD_RUNNER == 'true'; unset = hosted fallback, so a parked build box can no longer stall publishes. Re-enable with `gh variable set USE_VPS_BUILD_RUNNER --body true`. (cherry picked from commit e62b4b7)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.113omni-buildslot (ci.ymlBuild, docker-publish amd64build, npm-publishpublish, nightly-release-greenrelease-green/main-green, nightly-v4-buildbuild, preview-artifactbuild) now needs bothUSE_VPS_RUNNERand the new repo variableUSE_VPS_BUILD_RUNNERset to'true'. Unset (the default) = the hosted fallback each job already had.OMNIROUTE_USE_TURBOPACK) mirrors the new runner condition, so the hosted fallback keeps building with webpack.runner-preflightand the nightlyrunner-alertfollow the same switch: a deliberately parked build pool is not an outage and must not red the publish or open the offline issue.Why
Since 2026-10-06 no
Buildhas passed on omniroute-113-5 (~200 Failed/Canceled; last success 2026-10-05). The Turbopacknext-buildgrows to 16–41 GB on the 32-core box and the guest OOM-killer takes it (CONSTRAINT_NONE— the whole 47 GB VM, not the runner's MemoryMax), after 17–90 min of swap/page-cache thrash (~520 MB/s reads, queue depth 65) on the host NVMe. The same build passes in 8–12 min on the hosted runner (webpack, fork PRs). The slot is single (#12048), so same-repo PRs waited up to 12 h behind it only to fail.Re-enable the box after the build is fixed there:
gh variable set USE_VPS_BUILD_RUNNER --body true.Related Issues
omni-buildrunner stalls the publish silently — add a job timeout and a runner-availability alert #14560Target Branch
release/v3.8.xbranch (the highest openrelease/v*), notmainValidation
tests/unit/vps-build-runner-gate.test.ts,tests/unit/build/heavy-build-queue.test.ts,tests/unit/vps-runner-variable-scope.test.ts,tests/unit/docker-publish-runner-preflight-14560.test.ts,tests/unit/check-workflows-provenance-runner.test.ts,tests/unit/docker-publish-stale-digests-15184.test.ts,tests/unit/quality-rail-gate-membership.test.ts— 27/27 passcheck:workflows(actionlint + zizmor) — not installed locally, runs in this PR's CITests Added Or Updated
tests/unit/vps-build-runner-gate.test.ts(written first; failed listing the 7 ungated jobs, then green).Reviewer Notes
omni-buildwork to hosted runners immediately (the variable is unset). Watch the docker-publish amd64 leg on hosted: its comment citesResourceExhaustedon the old ~7 GB runners (ci: docker-publish OOM (ResourceExhausted) on every release/v3.8.51 push — :next stale since 08-22 and the v3.8.51 tag will not build on the hosted runner #11976); hosted ubuntu runners are 16 GB now and the webpack Next build fits there.omni-buildkeep that target and must be cancelled/re-run.