Skip to content

chore(ci): self-heal corrupt runner checkout in scrub-refs step - #4920

Merged
briansrls merged 1 commit into
mainfrom
ci/scrub-refs-self-heal
Jun 15, 2026
Merged

briansrls merged 1 commit into
mainfrom
ci/scrub-refs-self-heal

Conversation

@briansrls

Copy link
Copy Markdown
Contributor

Problem

Shared self-hosted runners (srv1/srv2) persist the repo checkout between jobs. When a runner's checkout gets a ref whose object is missing, every subsequent Checkout on that runner dies — observed on srv2-23:

fatal: bad object refs/remotes/origin/session/fierce-ram-162

This reds CI for any PR that lands on the wedged runner (recently hit #4918's layering_imports and others). The existing "Scrub dangling remote refs" step (which update-ref -ds dangling refs) does not clear it when update-ref -d can't remove the ref (e.g. corrupt packed-refs) or when git for-each-ref itself errors.

Fix

Harden the scrub step in all 13 jobs: after the dangling-ref pass, if the ref store is unreadable or any ref tip object is still missing, rm -rf the repo dir so actions/checkout fresh-clones. This self-heals the wedged runner on its next job, before checkout — taking the operator off the checkout-repair axis (no manual runner re-provision needed for this failure mode).

Safety — the rm -rf is precise by construction

It fires only when git for-each-ref errors or a ref tip resolves to a missing object. A healthy or empty checkout is never removed.

Verified by execution (local repro):

  • ✅ healthy repo (incl. refs/remotes/origin/session/* refs) → kept
  • ✅ fresh repo with no refs → kept
  • ✅ repo with a ref to a missing object (the srv2-23 case) → detected & removed
  • ✅ normal case where update-ref -d clears the bad ref → no rm -rf (last-resort only)

Note

This PR will itself sit red on v4_claim_witness_corpus_b until the separate placement_supply.dag Nat-multiply main-red is fixed (snappy's lane) — that's the corpus_b axis, orthogonal to this checkout-axis fix. It can merge once main greens, or be merged as an infra fix.

🤖 Generated with Claude Code

Shared self-hosted runners (srv1/srv2) persist the repo between jobs. A
wedged checkout with a ref whose object is missing (observed on srv2-23:
'fatal: bad object refs/remotes/origin/session/<branch>') survives the
existing dangling-ref scrub when update-ref -d cannot clear it (e.g.
corrupt packed-refs) or when for-each-ref itself errors, and then reds
the very next Checkout for ANY PR landing on that runner.

Harden the scrub step (all 13 jobs): after the dangling-ref pass, if the
ref store is unreadable OR any ref tip object is still missing, rm -rf
the repo dir so actions/checkout fresh-clones. This self-heals the
wedged runner on its next job, before checkout, removing the operator
from the checkout-repair axis.

The rm -rf is precise by construction -- it fires ONLY when
`git for-each-ref` errors or a ref tip resolves to a missing object; a
healthy or empty checkout is never removed. Verified by execution:
healthy (with refs/remotes/origin/session/*) and fresh-no-refs repos are
kept; a repo with a ref to a missing object is detected and removed; the
normal case where update-ref -d clears the bad ref takes no rm -rf.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@briansrls

Copy link
Copy Markdown
Contributor Author

The single failing check — v4_claim_witness_corpus_b — is an inherited fleet-wide main-red, not caused by this PR. It fails on:

dsl/std/placement_supply.dag:41:59: error: function '<expr>' not found in scope
error: P-PROBE-CF-IMPORT repro failed but without expected Option-not-found diagnostic

This is the placement_supply.dag Nat-multiply resolve break (exposed by #4883's probe_selector_compute_fabric_import_repro witness) that currently reds shard B on main itself (ci.yml run on f109f463d0 is red on the same job), so every PR inherits it. It is being fixed in the compute_fabric/measure lane (snappy's Nat-mul fold); orthogonal to this PR's checkout-axis change.

This PR is ci.yml-only and is otherwise healthy: layering_imports, fmt, doc_refs, infra_isolation, timeout_budgets, v4_claim_witness_corpus_a, and all v4_lens_* jobs pass — confirming the workflow parses and the hardened scrub + checkout run cleanly. No fix is applicable in this PR; it clears on main-green + rerun (and its own scrub-hardening keeps the rerun off any wedged-checkout runner).

— sent from swift-stag-552

@briansrls
briansrls merged commit f1a7a29 into main Jun 15, 2026
64 of 72 checks passed
@briansrls
briansrls deleted the ci/scrub-refs-self-heal branch June 15, 2026 03:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant