Repository navigation
docs(ops): the .113 heavy-build ceiling is one runner, not two - #12048
Merged
Merged
Conversation
Two concurrent next-builds (15.4 GB + 17.2 GB RSS) OOM-killed one on 2026-08-29 17:26 UTC; systemd booked the kill on the other runner's unit and its job died with the same "shutdown signal" text a hosted-runner OOM shows. omni-build now lives on omniroute-113-5 only; 113-6 keeps omni-release. The janitor ceiling counts every listener on the box (4 OmniRoute + OmniHeuris + OmniMind = 6). The second heavy slot returns when the Proxmox VM gets more RAM; the exact command is in the doc.
diegosouzapw
added a commit
that referenced
this pull request
Aug 29, 2026
Turbopack had 31 GB on omniroute-113-6 and still panicked (TurbopackInternalError: there must be a path to a root, run 33253576569). The same tree's arm64 webpack build on hosted ARM succeeded. Dockerfile already documents webpack as the Docker escape hatch. Keep amd64 on the one omni-build slot (#12048).
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…er-inference#12048) * docs(ops): the .113 heavy-build ceiling is one runner, not two Two concurrent next-builds (15.4 GB + 17.2 GB RSS) OOM-killed one on 2026-08-29 17:26 UTC; systemd booked the kill on the other runner's unit and its job died with the same "shutdown signal" text a hosted-runner OOM shows. omni-build now lives on omniroute-113-5 only; 113-6 keeps omni-release. The janitor ceiling counts every listener on the box (4 OmniRoute + OmniHeuris + OmniMind = 6). The second heavy slot returns when the Proxmox VM gets more RAM; the exact command is in the doc. * docs(ops): apply the single-heavy-slot text (previous commit only carried formatting)
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…ence#12050) Turbopack had 31 GB on omniroute-113-6 and still panicked (TurbopackInternalError: there must be a path to a root, run 33253576569). The same tree's arm64 webpack build on hosted ARM succeeded. Dockerfile already documents webpack as the Docker escape hatch. Keep amd64 on the one omni-build slot (cheaper-inference#12048).
diegosouzapw
added a commit
that referenced
this pull request
Oct 16, 2026
Since 2026-10-06 every Build that landed on omniroute-113-5 died: the Turbopack `next-build` grew to 16-41 GB on the 32-core box and the guest OOM-killer took it (CONSTRAINT_NONE, the whole 47 GB VM). ~200 runs in a row, each swap-thrashing 17-90 min, while the same build passes in 8-12 min on the hosted runner (webpack, fork PRs). The slot is single (#12048), so same-repo PRs queued up to 12 h behind it only to fail. Every `omni-build` job now needs USE_VPS_RUNNER *and* USE_VPS_BUILD_RUNNER == 'true'; unset = hosted fallback, so a parked build box can no longer stall publishes. The ci.yml Turbopack switch mirrors the new condition, and the docker-publish preflight / nightly offline alert follow the same switch (a deliberately parked pool is not an outage). Re-enable the box with `gh variable set USE_VPS_BUILD_RUNNER --body true`.
4 of 5 tasks
diegosouzapw
added a commit
that referenced
this pull request
Oct 16, 2026
ci: make the .113 omni-build slot opt-in (USE_VPS_BUILD_RUNNER) Since 2026-10-06 every Build that landed on omniroute-113-5 died: the Turbopack `next-build` grew to 16-41 GB on the 32-core box and the guest OOM-killer took it (CONSTRAINT_NONE, the whole 47 GB VM). ~200 runs in a row, each swap-thrashing 17-90 min, while the same build passes in 8-12 min on the hosted runner (webpack, fork PRs). The slot is single (#12048), so same-repo PRs queued up to 12 h behind it only to fail. Every `omni-build` job now needs USE_VPS_RUNNER *and* USE_VPS_BUILD_RUNNER == 'true'; unset = hosted fallback, so a parked build box can no longer stall publishes. The ci.yml Turbopack switch mirrors the new condition, and the docker-publish preflight / nightly offline alert follow the same switch (a deliberately parked pool is not an outage). Re-enable the box with `gh variable set USE_VPS_BUILD_RUNNER --body true`.
This was referenced Oct 16, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Só documentação (
docs/ops/RUNNER_BOX.md), refletindo o estado real da caixa desde 29/08 17:40Z:omni-buildsó noomniroute-113-5;113-6=omni-release): doisnext-buildsimultâneos (15,4 + 17,2 GB de RSS) levaram a caixa a 5 GB livres + 4 GB de swap e o kernel matou um deles — o systemd contabilizou o kill na unidade do outro runner e o job ali morreu com o mesmo "shutdown signal" dos runners hospedados. Comando de reversão (quando a VM Proxmox ganhar RAM) no doc.MAX_ACTIVE_RUNNERS=6(conta todos os listeners: 4 OmniRoute + OmniHeuris + OmniMind).docs-syncPASS; prettier limpo.