Skip to content

test(kanban): pin partial loss x unverifiable survivor to the single upstream raise - #849

Closed
Kyzcreig wants to merge 2 commits into
mainfrom
daedalus-opus/t_ac0e595c-partial-loss-pin
Closed

Kyzcreig wants to merge 2 commits into
mainfrom
daedalus-opus/t_ac0e595c-partial-loss-pin

Conversation

@Kyzcreig

@Kyzcreig Kyzcreig commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

Residue of ARGUS round 2 on #837 (card t_ac0e595c). FleetReview's P1 consequence does NOT reproduce on today's tree — but the structural concern under it is real, and this is the pin for it.

The gap, measured

On the PARTIAL-LOSS path (one recorded repo published + on disk, one recorded-but-vanished) the branch performs no verification of its own. It builds recovered straight out of explicit and feeds it into the len(refs) == len(repos) + 1 arithmetic that decides completion. The only thing between an operator's CLOSED/fake --survivor-pr and a durably stored survivor is the single _verified_explicit() raise upstream at kanban_survivor.py:324.

Nothing pinned that dependency. The existing coverage misses exactly the intersection:

test covers
test_partial_loss_keeps_both_the_surviving_repo_and_the_operator_ref partial loss + VERIFIED survivor
test_stale_bases_with_an_unverifiable_survivor_pr_still_refuses ALL-GONE path + unverifiable survivor
(nothing) partial loss + unverifiable survivor

Reproduced first, not inferred. Monkeypatching _verified_explicit to the presence-only world FleetReview described completes a partial-loss card and durably stores a CLOSED PR as the lost repo's survivor:

MUTATION= presence-only _verified_explicit
RESULT= COMPLETED ok= True
STATUS= done
SURVIVOR= {"kind": "ref", "refs": [{"remote": "https://github.com/example/project.git",
           "branch": "refs/pull/68/head", "sha": "45915633360000...", "pr": "example/project#68",
           "state": "CLOSED", "external": true, "repository": "gone"}, ...]}
UNVERIFIABLE_REF_STORED= True

What this PR adds

Three refusal pins on the intersection — CLOSED-unmerged PR, nonexistent PR, unresolvable --survivor-ref (both flags) — plus a verified-ref completion as anti-vacuity teeth, proving the refusals are about verification and not about the flag being inert on this branch.

Assertions check shape, not just outcome: no survivor row persisted, no run-metadata survivor, and specifically no repository: "." entry from a fall-through to the external path.

The remote fixture gains missing (gh lookup fails the way a nonexistent PR does) and tips (what a bare git ls-remote <url> advertises, empty by default). Both are stubbed so no test reaches the network — the gate must not pass by relying on a network failure.

RED-PROOF — the load-bearing part

A pin that passes on today's tree for a reason unrelated to the partial-loss branch is vacuous. Mutating _verified_explicit to return an UNVERIFIED ref instead of raising:

== UNMUTATED (today's tree) ==
SUITE= 76 passed        PINS= all GREEN

== MUTATED: _verified_explicit returns an UNVERIFIED ref, never raises ==
MUTATION_APPLIED= 1
PINS= {closed_pr: RED, nonexistent_pr: RED, unresolvable_ref: RED}
PINS_KILLED_MUTANT= 3/3
VERDICT= PINS THE MECHANISM

RESTORED_BYTE_IDENTICAL= True
SHA256 d8d416beb97262bf before and after

3/3. The pins pass today only because the raise is there — which is exactly the property that was unpinned.

DECISION: test-only, no defense-in-depth duplication

The card asked for this to be stated, not left implicit.

_verified_explicit() is the correct single choke point: it is the one place both flags are converted from claim to authority, and it already guards both the workspace-MISSING and the partial-loss branch. Re-verifying inside the branch means a second remote round-trip against the same claim, with two copies of the "is this authority" rule free to drift — and drift would land in the safe direction only by luck.

The actual defect was never missing verification; it was that a load-bearing dependency was unpinned. A test that goes RED the moment the raise moves, narrows, inlines, or short-circuits fixes precisely that, at zero runtime cost and zero new verification surface. The branch keeps its correctness stake honestly: it now has a test that fails if its upstream guarantee is withdrawn.

Verification

  • 76 passed / 0 failed across all four survivor files, 5/5 consecutive runs (baseline 72, +4 new). Run under .venv/bin/python -P with PYTHONPATH — bare python3 resolves to a 3.11 without yaml and gives a false "no tests collected" red.
  • No implementation file touched. No existing guard weakened; every arm in the existing matrix stays green.
  • ruff 0.15.10 clean; git diff --check clean.

Pre-existing flake, attributed not hand-waved

An intermittent 1-test failure appeared with a different victim each run. Attributed by measurement rather than assumed:

  • base @ a26f2700bd, serial: 8/8 green; single victim test solo: 20/20 green
  • my tree with the 4 new tests deselected: 5/5 green
  • base @ a26f2700bd with 4 concurrent copies: 4/4 runs degraded (1 failed / 2 errors / 4 errors / 1 error), zero of my code present

Conclusion: inherited, load-induced git-subprocess contention under parallel pytest. Not introduced here. Serial runs on this branch are 5/5 clean at 76.

Stacking

Branched from a26f2700bd (#837's head) and targets that branch, since the partial-loss branch it pins only exists in #837. hermes_cli/kanban_survivor.py is a multi-way hotspot (#837, #796, #842, #839) — whoever lands last re-runs all four survivor files on the merged tree.

Upstream: kanban_survivor.py is absent from NousResearch/main (fork-only) — verified, so there is no upstream surface for this.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

…ishes

`preserve()` records `bases` at dispatch from the git checkout then in the
workspace. When that checkout is later gone but the workspace DIRECTORY
survives (evidence, logs, qa-output), the recorded-repository guard raised
unconditionally -- on the branch where `explicit` is never consulted. The
`--survivor-pr` / `--survivor-ref` remedy the error names is only read on the
workspace-MISSING branch, so that state had no reachable remedy at all: a
reviewer who edited nothing and had nothing to preserve could not complete,
even with a valid remote-verified PR.

Consult `explicit` before that raise. With no operator survivor the guard is
unchanged and still fails closed -- that is what protects unpushed work. The
relaxation is for OPERATOR flags only; a mined hint cannot reach it.

Two things the first cut got wrong, both found by mutation and fixed:

- `claimed = True` is required, or a verified survivor is accepted but never
  recorded.
- On a PARTIAL loss (some recorded repos survive, one vanished), the survivors
  of the rest satisfied the completion on their own and silently dropped the
  operator's ref for the lost repo. `recovered` seeds it into `refs`, and the
  ref-kind count accounts for it -- without that arithmetic the card falls
  through to `_external` and records a DUPLICATE ref under repository ".".

Verified:
- 7 new tests in tests/hermes_cli/test_kanban_survivor_stale_bases.py.
- 5/5 mutations killed (revert the fix; drop the guard; drop `claimed = True`;
  drop `recovered` seeding; revert the ref-count arithmetic). Source restored
  and sha256-compared byte-identical after the sweep.
- 959 passed / 0 failed / 3 skipped across all 81 tests/hermes_cli kanban files.
- ruff 0.15.20 clean; `git diff --check` clean.
- Live proof on an isolated HERMES_KANBAN_SANDBOX=1 board through the real CLI:
  refusal -> exit 1 + reason + remedy, card left `ready`; a real remote-verified
  `--survivor-pr #835` -> exit 0, "Completed <id>",
  survivor recorded at b4707a3; a nonexistent PR -> exit 1, still refused.

NOT DONE, and deliberately: the card's item 3 asks to fix `hermes kanban
complete` exiting 0 on this refusal. That does not reproduce on this tree. A
raised SurvivorUnavailable is a ValueError and kanban.py already catches it,
prints `kanban: <reason>` to stderr, and returns 1; main.py propagates a nonzero
handler return via sys.exit. Measured exit 1 on fork/main BEFORE this change,
through both `python -m hermes_cli.main` and the deployed `hermes` wrapper. What
was genuinely missing was the remedy text -- the message named no way out -- so
the HINT is now appended, and two CLI-level tests pin exit-nonzero-with-hint and
exit-zero-on-success. The card's item 4 (should a reviewer-lane completion
consult `bases` at all) is a contract question left open, not silently decided.
…upstream raise

On the PARTIAL-LOSS path (one recorded repo published, one recorded-but-
vanished) the branch performs NO verification of its own. It builds `recovered`
straight out of `explicit` and feeds it into the `len(refs) == len(repos) + 1`
arithmetic that decides completion. The ONLY thing between an operator's
CLOSED/fake `--survivor-pr` and a durably stored survivor is the single
`_verified_explicit()` raise upstream at kanban_survivor.py:324.

Nothing pinned that dependency. The existing coverage misses exactly this
intersection: test_partial_loss_keeps_both_the_surviving_repo_and_the_operator_ref
tests partial loss with a VERIFIED survivor;
test_stale_bases_with_an_unverifiable_survivor_pr_still_refuses tests an
unverifiable survivor on the ALL-GONE path.

Add three refusal pins (CLOSED-unmerged PR, nonexistent PR, unresolvable
--survivor-ref) plus a verified-ref completion as anti-vacuity teeth, and
extend the `remote` fixture to stub `gh` failure and `git ls-remote` tips so
no test reaches the network. The refusal assertions check SHAPE, not just
outcome: no survivor row persisted, no run metadata survivor, and in
particular no `repository: "."` entry from a fall-through to the external path.

DECISION -- test-only, no defense-in-depth duplication in the branch.
`_verified_explicit()` is the correct single choke point: it is the one place
both flags are converted from claim to authority, and it guards BOTH the
workspace-MISSING and the partial-loss branch. Re-verifying inside the branch
would mean a second remote round-trip against the same claim, with two copies
of the "is this authority" rule free to drift apart -- and the drift would be
silent in the safe direction only by luck. The real defect was that the
dependency was load-bearing and unpinned; a test that goes RED when the raise
moves fixes exactly that, at zero runtime cost and zero new verification
surface. The branch keeps its correctness stake honestly: it now has a test
that fails if its upstream guarantee is withdrawn.

Verified:
  * 76 passed / 0 failed across all four survivor files, 5/5 consecutive runs
    (baseline was 72; +4 new). Run under .venv/bin/python -P with PYTHONPATH.
  * RED-PROVED. Mutating `_verified_explicit` to the presence-only behaviour
    (return an UNVERIFIED ref instead of raising) turns all 3 refusal pins RED:
      PINS_KILLED_MUTANT= 3/3   VERDICT= PINS THE MECHANISM
    kanban_survivor.py restored byte-identical (sha256 d8d416beb97262bf before
    and after). The pins pass on today's tree only because the raise is there.
  * No implementation file touched; no existing guard weakened or relaxed.
  * ruff 0.15.10 clean; git diff --check clean.
@Kyzcreig

Copy link
Copy Markdown
Collaborator Author

FleetReview

PARTIAL — ensemble escalated: judge transient failure

This review did not reach a trusted verdict, so it is not a gate pass and the findings below may be incomplete. They are posted so they can be read rather than lost in a terminal record.

Reviewed with 2 of 3 model families — openai unavailable.

Confidence: 1/5

Findings

  • P1 hermes_cli/kanban_survivor.py:371 — Partial loss of 2+ recorded repos: only the first gets a survivor, yet the completion arithmetic declares the card fully survived
  • P2 hermes_cli/kanban_survivor.py:350 — A card rescued by --survivor-pr/--survivor-ref can never be reclaimed: the guard raises above the cleanup branch that is documented to reuse the recorded survivor
  • P2 tests/hermes_cli/test_kanban_survivor_stale_bases.py:233 — Three new refusal tests never reach the partial-loss branch they claim to cover
  • P0 tests/hermes_cli/test_kanban_survivor_stale_bases.py:43 — Fixture returns None

FleetReview provenance · models: C=claude-code-opus-5, D=grok-4.6, G=grok-4.6 · cost: $5.23 · duration: 13m 58s · rounds: 1 · files examined: 1

Base automatically changed from daedalus-opus/t_49c1dec1-survivor-escape to main September 22, 2026 01:50
@Kyzcreig

Copy link
Copy Markdown
Collaborator Author

Superseded by #852.

#849 was branched off #837's branch before that PR was squash-merged as e9b406c59b; its parent commit is now redundant history and GitHub reports it CONFLICTING. #852 re-ports the identical change onto live main — both land test-file blob 6d237689c1, verified equal — and carries the full red-proof. Closing this one so there is a single PR for the change.

@Kyzcreig Kyzcreig closed this Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant