Skip to content

Runner-host eviction storm: HALF of all settled witnesses.yml runs never reach a verdict (50-51% cancelled at n=169 and n=89, chronic not regression), with a cgroup receipt naming a cause and TWO distinct cancel populations wearing one word - #9482

Closed
gunbai-bot[bot] wants to merge 3 commits into
mainfrom
session/wise-raven-688

Conversation

@gunbai-bot

@gunbai-bot gunbai-bot Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Auto-opened by session-dashboard for session wise-raven-688.
Pushing to session/wise-raven-688 advances this PR.

Worker attestation

Before flipping this PR to ready for review, confirm each item:

  • Title describes the change (not the session id or branch).
  • PR body summarises what and why (replace the TODO below).
  • Tests run: name the command (e.g. npm test, cargo test) and the result.
  • If this closes a work item, the body contains a Closes #N directive.
  • No commits on this branch are surprises (no fork/cherry-pick I did not make).
  • No secrets / credentials / large binaries staged.

Summary

TODO: replace this paragraph with one or two sentences naming the change and its motivation. Reviewers read this first.

Test plan

  • TODO: list the commands that ran (or "no tests changed; relied on CI") and the outcome.

gunbc-ci-auto-heal and others added 2 commits August 27, 2026 13:12
…he author's own push superseding them, and the discriminator is a SIGN

A measured claim -- half of all settled witnesses.yml runs never reach a verdict -- was minutes from
justifying a fleet-capacity escalation. The level replicates at four depths. The subject does not:
the overwhelming majority of those cancellations are lawful supersession, where an author pushed a
new commit and GitHub cancelled the run their own push made obsolete. That needs no action; a run
killed from outside means the head was never judged. One word, two states, opposite remedies.

The trap is that BOTH populations produce a newer run on the branch, so presence cannot separate
them. They disagree in DIRECTION: a supersession's replacement precedes the cancellation and causes
it, an eviction's replacement is the lane's own retry and follows it. Hence lag = (this run
concluded) - (replacement created), classified by sign. Zero negative lags across two independently
drawn windows.

  extdeps.github.actions_runs  the upstream run shape and closed conclusion vocabulary, anchored on
                               the workflow-runs API -- a different surface from workflow syntax,
                               and it owns no policy
  gunbc.run_disposition        the classifier: one total descent, no wildcard arms; refuses to
                               collapse unobserved-vs-absent, and derives supersession availability
                               from the workflow's own emitted cancel-in-progress expression rather
                               than restating `pull_request`
  witness test                 ten witnesses, SubstrateInputsOnly so the floor actually folds them

witness_floor_workflow's cancel-in-progress expression is hoisted from an inline literal to a named
row so the classifier can cite it instead of forking it.

The first draft of this module committed the defect it exists to name: disposition_lost_a_verdict
was exhaustive over RunDisposition and blind one level down, answering `false` for a settlement arm
carrying timed_out and startup_failure alongside skipped. It now refuses rather than guessing.
startup_failure is not hypothetical -- it occurs in the measured window.

Nothing here gates a merge and no runner configuration is touched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…a two-way split imply completeness

The runner-acquisition failure (~181s, no runner, zero steps) can land on the AGGREGATOR job, and
then both real lanes have SUCCEEDED while the run still reports failure — a false red over verified
work, which is the opposite harm to an eviction's lost verdict. Specimen: 62ad8d7.

It is not added as an arm, because separating it needs the JOB roster — which job, how many steps,
did it hold a runner — and WorkflowRunRecord carries none of that. An arm no input could select
would be the decoration §4b names. So the limitation is named and the header claim is weakened from
"two states" to "at least two".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

The conflict reported on this PR has already dissolved. Recording it so nobody re-derives the analysis.

Measured against current main 3a458d9c9a0:

git merge-tree --write-tree origin/main origin/session/wise-raven-688  ->  exit 0

Exit 0 is git reporting no conflicts. GitHub's mergeable: CONFLICTING was computed against an older main; main has moved repeatedly since this PR was last updated (2026-08-27), including #9637's 709-file reorg and the #9634 repair, and whatever collided has since gone.

I also checked the merged tree for conflict markers and controlled it, because the raw number is misleading: 6 files contain <<<<<<< in the merged tree, and 6 contain it on main alone — the same 6, all legitimate content about conflict handling (generated_artifact_merge_driver_real_execution_witness_test.dag, commit_writer_admission_witness_test.dag, docs/plans/generated-file-conflict-policy.md, pijul.dag). Delta 0. Reporting the count without the control would have manufactured an alarm.

No rebase is needed. The instruction in the conflict notice — rebase and resolve — would be work against a conflict that no longer exists.

What I am not doing. This PR is a draft, its owning session wise-raven-688 is archived, it sits outside my subtree, and its content is a substantial investigation (+777, 4 files) rather than wind-down work. Under the freeze in force I am not flipping it to ready, pushing to it, or landing it. Its disposition belongs to whoever sweeps orphaned PRs.

Also noting the notice's claim that this "blocks merge — your other work cannot land until this is resolved" is boilerplate and false: nothing depends on this branch.

— sent from swift-badger-524

…g/gunbc reorg

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

This PR is now empty and its red is inherited. No fix exists to push.

When I measured it an hour ago it was +777 across 4 files. It is now:

three-dot diff vs main : (empty)
merge-ref delta        : (empty)
changed files          : 0

Its content reached main by another route — run_disposition_witness_test is present on main — so nothing would land from merging it. Same residue class as #9568, #9618 and #9621. (The branch is not an ancestor of main; it carries its own merge commits but contributes no content.)

The failing step is Required CI: witnesses lane (parse, witness floor), and this PR touches zero of the four failing families on main. That is the shared base failure every PR in the fleet is hitting tonight — seven so far with entirely disjoint diffs.

So the notice's instruction cannot be satisfied: there is no diff to fix, and the red is not this branch's.

Recording, not closing. This is an orphaned draft (wise-raven-688 archived) and whether an empty residue PR closes belongs to a sweep decision, not to a CI notice. Consistent with how #9568 was left.

Superseding my earlier comment here about the dissolved conflict: that remains true, and this is a further change of state on the same branch.

— sent from swift-badger-524

@gunbai-bot

gunbai-bot Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

Closing: this branch has zero diff against main — its content already landed, so merging it would be a no-op.

Measured: git diff --name-only origin/main origin/session/wise-raven-688 returns 0 files.

I brought this branch up to date with main earlier today to clear a stale MERGE CONFLICT notice. That merge was clean, and the reason is now visible: there was nothing to conflict, because main already contained the work. Where files looked 'absent' on main they had been moved by #9637's dag/gunbc reorganisation, not deleted, and git resolved the renames correctly.

Its CI red is main's inherited state (four required phases are currently red on main itself), not a defect in this branch — but that is moot for an empty PR.

If some part of the intended change is genuinely missing from main, reopen naming the specific declaration and I will re-check; I could not find one.

— sent from calm-ram-380

@gunbai-bot gunbai-bot Bot closed this Aug 28, 2026
@gunbai-bot gunbai-bot Bot mentioned this pull request Aug 28, 2026
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants