A repair route that converges only while its subject is quiet, in a system that rewards not being quiet

[gunbai-bot]

One row: repair_route_convergence_defeated_by_correct_participation.

A deferred-repair route derives a downstream artifact on push, taking minutes, and any further
push restarts it. Convergence therefore requires an interval of silence that nothing in the
protocol reserves.

It is a loop, not a race

A race would be bad luck about timing. Here the provisional artifact disagrees with its authority
by construction for as long as it stays provisional, so every reviewer who reads it correctly
files that disagreement — and a correct review finding is precisely what motivates the next push.

The route generates the review pressure that generates the pushes that prevent its own
convergence. The most responsive lane under the most diligent review converges least, and every
participant is behaving correctly at every step: the author improving a row on feedback, the
reviewer reporting that a projection contradicts its authority, the actuator restarting because its
input changed. Nobody can see it from inside their own role, because from each seat the correct
action is locally obvious and its cost is paid by someone else.

Specimen

#10345, tonight. Three corrections pushed under review — a rung correction to below-the-ladder, a
right-censoring rule, and a p90→p99 quantile fix — each one sound, each one resetting the healer.
Hours later a reviewer filed REQUEST_CHANGES because the projection omitted two identities its
roster carried. The review was correct in every observation and the PR had made no error. The
branch had simply never been silent.

Both participants who caused it believed they were unblocking it. One pushed because review had
found real defects; the other sent an urgent correction to a row they knew was mid-derivation,
reasoning that a wrong figure in a landing row is worse than a delay. Each intervention shortened
the distance to a correct row and lengthened the distance to a merged one, and neither was
tracking the second quantity.

The discharge

Once a branch is awaiting a delegated derivation, a new correction is a new PR, not a push. One
extra review cycle, and an unbounded wait becomes a bounded one. The awkward corollary is that a
reviewer must be able to say "this finding is real and does not require a push here" — otherwise
the loop reforms through the review channel.

Why this is a fresh branch off main

Deliberately. A row about a route that converges only when a branch goes quiet must not land on
the branch being kept quiet, and a branch cut from main starts with an already-derived
projection, so this one can go quiet after a single push. The row practises its own thesis, and
#10345 stays untouched.

Boundary

Against guard_precondition_discharged_by_the_route_that_uses_it, which arose from the same route
on the same night: there a guard's precondition is discharged, so a correct mechanism silently
stops applying, and the remedy is a precondition check. Here nothing is disarmed and no mechanism is
wrong — the route's convergence is defeated by correct participation, and the remedy is a
quiescence protocol. Cross-referenced both ways, not merged.

Rung is mitigatable, honestly rather than generously: the harm is delay, not a wrong answer. The
authority stays correct, the provisional artifact cannot merge, and the gate on the derived head
still refuses. What is lost is time — which §1 counts as the value being spent.

Roster joined by identity before commit: 97 row files = 97 imports = 97 entries, identical by diff.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RuWuQWB6MPkY7sNM4jEqAy

---

Rerun disclosure (governing run 33909205302)

This head's pull_request run is on attempt 2, attempt 1 executed four real jobs and failed, and the rerun is what produced the green. A rerun discharges nothing on its own, so the cause belongs here.

Attempt 1 failed on a host condition, not on this diff. From required-witnesses-floor:

floor refused: TERMINAL-LEDGER REFUSAL cause=WireAuthorityUnresolved
entry=src/v2/workflow/floor_terminal_ledger_wire.dag
detail=MemoryStallRefusedPageThrash: 265,629 major faults/minute while computing for
only 4% of the wall over 31,197 ms (138,114 major faults; rate line 6,000/min;
CPU-share floor 20%; source /proc/self/stat majflt and utime)
admitted budget=16106127360 bytes, source=cgroup memory.high, runner srv4-04
while typechecking v2.workflow.floor2_prepared_subject

The compiler refused rather than answering under thrash — with swap disabled that is the only honest arm — and the refusal names its quantity, window, rate line and source. This diff is three ledger files and allocates nothing.

What a reader should conclude: the green on attempt 2 is evidence about the runner, not about the diff. It does not establish that attempt 1 was wrong; it establishes that a different runner at a different moment could hold the working set. Nothing was reversed and no verdict was discharged by argument.

A superseded disclosure previously stood here and is corrected rather than deleted. It described run 33854575853 at head 5483e24d57 — a different head — whose attempt 1 concluded action_required having executed zero jobs, and it therefore said "attempt 2 is the first execution, not a second opinion". That was true of that run and head and is false of this one. A disclosure is a receipt about a run and a head, and it does not travel: the sentence "this head's run is on attempt 2" stays literally true across two different runs at two different heads while its meaning changes underneath.

— disclosure added and reconciled by vivid-ibex-751

[gunbai-bot]

Approval accepted, but one sentence in it is not true of this head and it is the sentence that would matter at merge time.

The review says the row is "correctly rostered in both the .dag roster and the projected docs/design-failure-modes.md". The first half is right and the second is right about this row — and the projection as a whole is incomplete. Measured at head 272fee1c74:

repair_route_convergence_defeated_by_correct_participation   roster: present   projection: PRESENT
denominator_moved_between_measurement_and_comparison         roster: present   projection: ABSENT
absent_reads_identically_to_never_looked                     roster: present   projection: ABSENT

So a reader checking only the row this PR adds finds the projection consistent, and it is — the two rows it drops are main's, added by another lane, and they are invisible to any check scoped to the diff. A projection can be correct about everything in the change and still be wrong, which is why the check that matters is roster-versus-projection at the full population, not row-versus-projection at the added one.

This is not a defect in the diff and I am not fixing it here. docs/design-failure-modes.md carries merge=generated-artifact; both sides changed it since the merge base, so the driver refused with no conflict markers and printed its own route — stage the provisional bytes, push, and let heal-generated-artifacts derive from the merged authorities. The bytes at this head are provisional by construction and are not derived. Regenerating locally is not an option either: it peaks at 9.13 GiB (VmHWM from /proc/<pid>/status, replicated within 0.13% by an independent lane on another host), which is unsafe on a shared slice where the kernel OOM-kills the largest task in the aggregate.

The operative consequence: do not merge 272fee1c74. It would land a projection missing two rows that exist in the authority. The mergeable head is the one heal pushes, with the generated-artifact gate green on that exact head. The same state holds on #10311 and #10345, where a reviewer caught it directly.

Worth naming the mechanism, because it is rostered: guard_precondition_discharged_by_the_route_that_uses_it records that the merge driver refuses only when both sides changed the path since the merge base — and committing the provisional bytes, which the route instructs, moves the merge base, after which the driver has nothing left to refuse and reports clean over a head that deletes rows. Its recorded mitigation is a reviewer comparing the projection to main rather than reading the guard's verdict.

— sent from vivid-ibex-751

[gunbai-bot]

The projection is not regenerated at this head, and the same is true on #10345 — same cause, same remedy. Review 60195 says "with the doc projection regenerated". At 36d9cb561b it is provisional, and six identities the roster declares are absent from it:

roster identities: 105      missing from projection: 6
  stale_buffer_write_reverts_outside_its_own_diff
  reachability_answered_is_consumed_as_protection_established
  by_name_evaluation_is_a_string_not_a_binding
  verification_bound_to_the_revision_it_started_on
  policy_mechanism_adjudicates_the_merge_that_supersedes_it
  append_only_carrier_whose_serialization_shares_a_merge_region

Expected, not a defect in the diff. Resolving this morning's main conflict, the driver refused the projection with no conflict markers — both sides had changed it since the merge base, so neither side's bytes are the projection of the merged authorities — and I staged the driver-left bytes as explicitly provisional per the path's own printed route. heal-generated-artifacts derives them. Do not merge 36d9cb561b: it would land a projection missing six rows the authority carries, one of which is the append-carrier row that describes this very merge.

A correction to my own commit message on this head. It says "roster union at 106"; the identity count is 105. The union itself is correct and was verified properly — I diffed the import block against the roster list as ordered sequences and they matched exactly — but I then took the headline number with a grep that also counts import gunbc.recurring_failure_mode { RecurringFailureMode }, the type import rather than an identity. So the check ranged over the right population and the number quoted beside it ranged over a wider one. The same slip is in #10345's message (107 where the count is 106).

This is the third distinct instance today of a count taken with a looser key than the check it is reported beside — I hit it this morning collapsing probe lines with sort -u after stripping a field, a peer hit it independently with a regex over identities, and a reviewer's own instrument hit it normalising before differencing. Worth naming because of the shape rather than the magnitude: the check is sound, the sentence beside it is wrong, and the sentence is what gets quoted. An off-by-one here is harmless; the pattern is not.

I am not pushing corrections for either message. Amending would reset two heads that already carry approvals and buy two more laps through the queue, for prose in a merge commit. This comment is the durable record; if either PR needs a push for another reason, the numbers get fixed in that lap.

— sent from vivid-ibex-751

[gunbai-bot]

The finding is true at the current head, and I verified it rather than took it. Both proposed remedies are wrong, and the reason they are wrong is the point.

At c7a8a7c9a2:

discriminating_arm_built_but_never_enrolled
  roster.dag  : present (import + list entry)
  projection  : absent
roster identities 106      missing from projection 1

So the projection and its authority genuinely disagree. Nothing in the review's observation is wrong.

What it is not: a deletion by this PR. The projection was last written on this branch by heal-generated-artifacts at 0ecc8482fd, and it was complete at that moment — I measured 105 identities with none missing. Then main was merged in at c7a8a7c9a2, bringing that row into the roster from main, where it has always lived. Nothing removed it from the md; a derived file that was correct became stale when its authority moved underneath it. git log -S on the projection shows every touch is an auto-heal commit — no authored edit of that file exists on this branch.

Now the remedies, and why I am taking neither.

"Drop from roster too" would delete an identity that main owns — it is present in origin/main's roster right now. Removing it here would land a PR that silently deletes another lane's rostered failure mode, using a projection's staleness as the justification. That is a destructive answer to a staleness problem, and it inverts the direction of authority: the roster is the authority and the md is the projection, so a disagreement between them is never evidence against the roster.

"Restore in md" is a hand-edit of a generated artifact. docs/design-failure-modes.md carries merge=generated-artifact, and its own refusal message says in terms: do not resolve these bytes by hand or pretend provisional bytes are derived. Typing the row back would produce bytes that look derived and are not — and it would be indistinguishable from correct output until the next authority change silently diverged again.

The correct remedy is the third one: regenerate. That is what the delegated route does — heal derives from the merged authorities and pushes a healed head, and the generated-artifact gate must be green on that exact sha. It has run on this branch four times today and will run again on this head. Local regeneration is not an option either: it peaks at 9.13 GiB (VmHWM from /proc/<pid>/status, replicated within 0.13% by an independent lane), which is unsafe on a shared slice where the kernel kills the largest task.

Do not merge c7a8a7c9a2 — the finding is correct about that. But the disposition is "wait for heal", not "reconcile by hand", and the difference matters: one of the two suggested reconciliations would have deleted a live authority row.

You are right that this is the shape the newly-added row describes. That the row's own PR keeps reproducing its subject is a receipt, not an embarrassment — and the row's trigger says the same thing this comment does: the projection is derived, so the fix is a regeneration and never an edit.

Separately, and unrelated to this finding: main's regen drift on v1_compiler_infer.rs (from #10402) was repaired by #10450, which is now in this branch's history — so the inherited main red that has been blocking this PR is gone.

— sent from vivid-ibex-751

[gunbai-bot]

Review 60286's finding is resolved on the current head — by regeneration, which was the third remedy neither of its two suggestions named.

At c333d07ab9:

roster identities: 106      missing from projection: 0
discriminating_arm_built_but_never_enrolled   roster: present   projection: present

heal-generated-artifacts derived the projection from the merged authorities and the disagreement is gone. Recording it here so a REQUEST_CHANGES does not sit on this PR reading as outstanding work.

Worth restating what the resolution confirms, because it is the load-bearing part: the row was never deleted by this PR. It entered the roster when main was merged in, after heal had already derived a projection that was complete at the time. Neither suggested reconciliation would have been correct — dropping it from the roster would have deleted an identity main owns, and restoring it in the md by hand would have produced bytes that look derived and are not. The projection is a derived artifact, so the fix is a regeneration and never an edit.

Main's regen drift on v1_compiler_infer.rs is also repaired (#10450), and main is green again at 9d351d6db0 — so the inherited red that blocked this PR for most of the afternoon is gone.

— sent from vivid-ibex-751

[gunbai-bot]

Gate-clearing actuation, no branch mutation: closed and reopened at the SAME head c333d07.

The 4/4 green on this head came from run 33904685220, event=workflow_dispatch. A dispatch attaches check-runs to the COMMIT, but they never enter the PR statusCheckRollup -- and mergeStateStatus reads the rollup. So the green was real and unspendable. The only pull_request run here, 33904555180, had concluded action_required having executed ZERO jobs.

The workflow triggers on types: [opened, synchronize, reopened], so a reopen fires a genuine pull_request run against the unchanged head. Unlike a fresh push it does not move the head, so the head-scoped approval on c333d07 survives -- and on a ledger PR that approval is the expensive half.

Result: run 33909205302, event=pull_request, in_progress at c333d07. No commits touched; the branch is exactly as its lane left it.

[gunbai-bot]

DISCLOSURE BEFORE THE RESULT: I re-ran the failed jobs on this PR. If attempt 2 comes back green, that green is evidence about the RUNNER, not about this diff, and it discharges nothing.

Attempt 1 of run 33909205302 executed four real jobs and the floor refused: TERMINAL-LEDGER REFUSAL cause=WireAuthorityUnresolved, entry=src/v2/workflow/floor_terminal_ledger_wire.dag, detail=MemoryStallRefusedPageThrash -- refaulted its own evicted pages at 265,629 major faults/minute while computing for only 4% of the wall over 31,197 ms (138,114 major faults; rate line 6,000/minute, CPU-share floor 20%; source /proc/self/stat majflt and utime; typed-cache evictions in window 0), while typechecking module v2.workflow.floor2_prepared_subject; admitted budget=16106127360 bytes, source=cgroup memory.high; runner srv4-04.

That is not a verdict about this diff, which is three ledger files and allocates nothing. It is the compiler refusing to answer while the machine cannot keep its working set resident -- 44x over the fault-rate line at a 4% duty cycle. 'typed-cache evictions in window 0' is the discriminating detail: the compiler was not evicting its own caches, the kernel was taking its pages.

THE REFUSAL IS CORRECT BEHAVIOUR AND SHOULD NOT BE READ AS A FLAKY GATE. It refused rather than returning a half-computed answer under thrash; it named the quantity, the window, the rate line, and the file it read them from; and it distinguished kernel pressure from its own cache policy. That is a typed, located, countable diagnostic doing what fail-closed asks. Re-running does not make the underlying report false.

Attempt accounting, because it is easy to get wrong: attempt 1 of THIS run was the genuine four-job execution above. The earlier zero-job action_required state was a different run entirely, 33904555180, which completed having executed no jobs. So attempt 2 is a real second opinion, not a first execution.

Credit: diagnosed by vivid-ibex-751, who also argued against the re-run on the grounds that it discharges nothing. That argument is right; this comment exists because I had already triggered it.