Unwedge PR Builds: drop dead amd64 ARC runner-image row - #1070
Unwedge PR Builds: drop dead amd64 ARC runner-image row#1070michaelneale wants to merge 1 commit into
Conversation
The mesh-llm-amd64 ARC scale set is not scheduling pods, so the Runner image / ARC amd64 job sits queued indefinitely. That job has no hosted fallback by design, so every PR Builds run is held open and cannot complete, blocking merges across all open PRs. Temporarily remove the amd64 matrix row while keeping the healthy mesh-llm-arm64 row so PR Builds can complete again. The mesh-llm-amd64 actionlint label is retained so the row can be restored without a lint change once the scale set is healthy.
📝 WalkthroughWalkthroughThe PR temporarily removes the ChangesARC runner contract
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@ci/ci.md`:
- Around line 292-297: Update the temporary amd64 disablement declarations in
ci/ci.md lines 292-297 and .github/workflows/pr_builds.yml lines 178-182 to
include a concrete owner and linked removal issue or expiry date, keeping the
same tracking metadata synchronized in both locations while preserving the
existing reason and re-enable condition.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: ad931bf5-7fda-4fc4-a6f5-fc24ad88f2fd
📒 Files selected for processing (2)
.github/workflows/pr_builds.ymlci/ci.md
| The matrix currently contains only the `mesh-llm-arm64` row. The | ||
| `mesh-llm-amd64` row is temporarily removed because that scale set is not | ||
| scheduling pods, which wedged every PR Builds run indefinitely (the job has no | ||
| hosted fallback). Restore the amd64 row once the `mesh-llm-amd64` scale set is | ||
| healthy again; the `mesh-llm-amd64` label remains in `.github/actionlint.yaml` | ||
| so the row can be re-added without a lint change. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Add owner and removal tracking for the temporary amd64 disablement.
Both declarations provide the reason but omit an owner and linked removal issue or expiry date, allowing the exception to persist indefinitely.
ci/ci.md#L292-L297: record the concrete owner and removal issue or expiry date in the authoritative CI documentation..github/workflows/pr_builds.yml#L178-L182: reference the same tracking metadata so the workflow comment remains synchronized.
As per coding guidelines, emergency exceptions must include a reason, owner, and linked removal issue or expiry date.
🧰 Tools
🪛 LanguageTool
[uncategorized] ~296-~296: The official name of this software platform is spelled with a capital “H”.
Context: ...; the mesh-llm-amd64 label remains in .github/actionlint.yaml so the row can be re-a...
(GITHUB)
📍 Affects 2 files
ci/ci.md#L292-L297(this comment).github/workflows/pr_builds.yml#L178-L182
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@ci/ci.md` around lines 292 - 297, Update the temporary amd64 disablement
declarations in ci/ci.md lines 292-297 and .github/workflows/pr_builds.yml lines
178-182 to include a concrete owner and linked removal issue or expiry date,
keeping the same tracking metadata synchronized in both locations while
preserving the existing reason and re-enable condition.
Source: Coding guidelines
|
Now that the other jobs are finished, I have retired the static runner. It was causing no AMD64 GPU pod to be scheduled - this immediately unwedged the builds, so this workaround is not necessary. Solved via infra configuration on my K3S cluster. Longer term - I think this check is relatively useless (ARC runner image), so I will follow up with a better setup. |
2 participants
What this does
PR Builds runs again. Right now every open PR's PR Builds run is stuck
queued(some for 8+ hours) and cannot complete, so nothing can merge. This removes the single job that is wedging the queue.Why
The
Runner image / ARC amd64job runs on the self-hostedmesh-llm-amd64ARC scale set, which is currently not scheduling any pods. That job has no hosted fallback by design, so it sitsqueuedforever and holds the entirePR Buildsrun open even though every other job (Linux/macOS builds, Rust tests, smoke, SDK) has already finished.Observed live:
Runner image / ARC arm64-> completes/succeeds on every run (ARC controller, K3s, and org registration are healthy).Runner image / ARC amd64->queuedon every run, 0-for-N for 8+ hours.So the fault is isolated to the
mesh-llm-amd64scale set (dead amd64 pool / node capacity / image digest), not the ARC setup as a whole. That is a cluster-side problem a mesh-llm PR cannot fix.Change
amd64row from thearc_runner_image_contractmatrix inpr_builds.yml. arm64 ARC coverage is kept -- this does not delete the gate, only the row that cannot be satisfied.ci/ci.mdtopology to match.mesh-llm-amd64label in.github/actionlint.yamlso the row can be restored later without a lint change.This is temporary -- restore amd64 when the pool is healthy
This intentionally reduces coverage: the amd64 ARC contract (native-arch check + self-hosted image validation + small Rust check on amd64) is what catches amd64 ARC / K3s / runner-image regressions before merge. Re-add the amd64 matrix row once the
mesh-llm-amd64scale set is scheduling pods again.Suggested follow-up for whoever has cluster/org access:
AutoScalingRunnerSetspec (image digest, node selector, resource requests,minRunners) since arm64 works and amd64 does not.Validation
actionlint -config-file .github/actionlint.yaml .github/workflows/pr_builds.yml-> exit 0git diff --check-> cleanneeds:arc_runner_image_contract, so removing the row has no downstream graph impact.Summary by CodeRabbit
Bug Fixes
Documentation