Repository navigation
chore(ci): self-heal corrupt runner checkout in scrub-refs step - #4920
Conversation
Shared self-hosted runners (srv1/srv2) persist the repo between jobs. A wedged checkout with a ref whose object is missing (observed on srv2-23: 'fatal: bad object refs/remotes/origin/session/<branch>') survives the existing dangling-ref scrub when update-ref -d cannot clear it (e.g. corrupt packed-refs) or when for-each-ref itself errors, and then reds the very next Checkout for ANY PR landing on that runner. Harden the scrub step (all 13 jobs): after the dangling-ref pass, if the ref store is unreadable OR any ref tip object is still missing, rm -rf the repo dir so actions/checkout fresh-clones. This self-heals the wedged runner on its next job, before checkout, removing the operator from the checkout-repair axis. The rm -rf is precise by construction -- it fires ONLY when `git for-each-ref` errors or a ref tip resolves to a missing object; a healthy or empty checkout is never removed. Verified by execution: healthy (with refs/remotes/origin/session/*) and fresh-no-refs repos are kept; a repo with a ref to a missing object is detected and removed; the normal case where update-ref -d clears the bad ref takes no rm -rf. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
The single failing check — This is the This PR is ci.yml-only and is otherwise healthy: — sent from swift-stag-552 |
Problem
Shared self-hosted runners (srv1/srv2) persist the repo checkout between jobs. When a runner's checkout gets a ref whose object is missing, every subsequent
Checkouton that runner dies — observed on srv2-23:This reds CI for any PR that lands on the wedged runner (recently hit #4918's
layering_importsand others). The existing "Scrub dangling remote refs" step (whichupdate-ref -ds dangling refs) does not clear it whenupdate-ref -dcan't remove the ref (e.g. corrupt packed-refs) or whengit for-each-refitself errors.Fix
Harden the scrub step in all 13 jobs: after the dangling-ref pass, if the ref store is unreadable or any ref tip object is still missing,
rm -rfthe repo dir soactions/checkoutfresh-clones. This self-heals the wedged runner on its next job, before checkout — taking the operator off the checkout-repair axis (no manual runner re-provision needed for this failure mode).Safety — the
rm -rfis precise by constructionIt fires only when
git for-each-referrors or a ref tip resolves to a missing object. A healthy or empty checkout is never removed.Verified by execution (local repro):
refs/remotes/origin/session/*refs) → keptupdate-ref -dclears the bad ref → norm -rf(last-resort only)Note
This PR will itself sit red on
v4_claim_witness_corpus_buntil the separateplacement_supply.dagNat-multiply main-red is fixed (snappy's lane) — that's the corpus_b axis, orthogonal to this checkout-axis fix. It can merge once main greens, or be merged as an infra fix.🤖 Generated with Claude Code