fix(kanban): log git's returncode and stderr when survivor capture fails - #856
Merged
Merged
Conversation
`_git` raised a constant `survivor_unavailable: git inspection failed` and
discarded the returncode and stderr, so a real capture defect and a purely
environmental fault were indistinguishable at the call site. Establishing
which cost a full attribution pass per occurrence (t_ac0e595c), and
kanban_survivor.py has five open PRs whose reviewers each pay that tax.
Root cause of the occurrences that prompted this (card t_169d6e46): NOT a
survivor defect and not a RAM-disk race. fleet/ramscratch-env.sh exported a
FIXED `--basetemp=/Volumes/ramscratch/pytest` into every daedalus-family
worker shell; pytest rm_rf()s an explicit --basetemp at session start with no
numbered subdir and no lock, so concurrent workers deleted each other's live
tmp_path and git exited 128 "cannot change to '<path>': No such file or
directory". Measured with 2 concurrent sessions of the survivor suite:
10/12 sessions red with the fixed basetemp, 0/18 red without it. The env fix
lands separately in hermes-home; this commit is what makes the next
occurrence diagnosable in one log line instead of an attribution pass.
The raised message is unchanged (it is persisted to held_reason, the event
log and stderr, and open PRs key on it). Detail goes to the log, redacted
through kanban_external_survivor.redact, the same helper that guards an
echoed claim.
Verified:
- 2 new tests in test_kanban_survivor_authority.py, born-red 2/2 against
unmodified fork/main (assert 'rc=128' in '' / IndexError on no records)
- teeth: mutating the log to unredacted stderr kills ONLY the redaction
test (1 failed, 8 passed), so it is not a vacuous green
- 74/74 x3 across all four survivor test files
- ruff clean on both changed files
Collaborator
Author
FleetReviewReviewed with 2 of 3 model families — openai unavailable. Confidence: 4/5 Findings
FleetReview provenance · models: C=claude-code-opus-5, D=grok-4.6, G=grok-4.6 · cost: $5.71 · duration: 18m 37s · rounds: 1 · files examined: 2 |
Kyzcreig
added a commit
that referenced
this pull request
Sep 23, 2026
fork/main advanced 83 commits since 0dc715c, including the survivor stack (#837 #842 #848 #856 #872 #879 #886 #888) which rewrote kanban_survivor.py from 475 to 1516 lines. Four conflicts, all in kanban_survivor.py, resolved toward main's shapes: _git union: main's input= (needed by _present_commits/_rev_list) plus this PR's timeout= (needed by _content_advisory's 120s fetch). _capture kept main's extraction; folded this PR's canonical-tree fallback + mirror_hint advisory INTO it, so the _explain_broken_object_store classifier still wraps every object-reading step. ref arm kept main's 'not bundles and len(refs) == len(repos) + len(carried)' (the #842/#848 carried-survivor accounting) and this PR's canonical sidecar. The pre-#848 unbound-claim path is NOT reintroduced: _verified_explicit, _unbound_keys, _reusable and _bound are main's, untouched. The landed arm in preserve() and _verify_landed/_landed_contains_history merged without conflict. Verified: 11 survivor files (4 from this PR + 7 landed since): 255 passed, 0 failed the PR's own 4 files: 98 passed, 0 failed mutation, dirty-tree guard neutered: 16 passed, 2 FAILED mutation, history binding neutered: 15 passed, 3 FAILED ruff on kanban_survivor.py + kanban_db.py: clean git diff --check: clean
This was referenced Sep 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Card: t_169d6e46. Test-hermeticity investigation; this is the impl half (item 2 of the card).
What the "flake" actually was
NOT a survivor defect, and NOT a RAM-disk / racy-clean-index race. Root cause is outside this repo:
~/.hermes/fleet/ramscratch-env.shexported a fixed--basetemp=/Volumes/ramscratch/pytestinto every daedalus-family worker shell (
terminal.shell_init_files). pytestrm_rf()s an explicit--basetempat session start — no numbered subdir, no lock(
_pytest/tmpdir.py::TempPathFactory.getbasetemp, read in 9.1.1:if basetemp.exists(): rm_rf(basetemp)).Two concurrent worker pytest sessions therefore delete each other's live
tmp_pathtrees mid-run.Instrumented capture of the swallowed stderr (subprocess spy over
subprocess.run):Measured, one variable (basetemp fixed vs per-session), same host, same RAM disk:
--basetemp(status quo), N=2 concurrent--basetemp, N=3 concurrent--basetempper session, N=2--basetemp, TMPDIR on the RAM disk (the fix), N=3This also answers the card's review note that a green streak could not distinguish
"fixed" from "trigger absent": the red is now reproducible on demand
(
repro.sh fixed 2→ red first try), so the green arm is a real control.The env fix lands separately in hermes-home (
fleet/ramscratch-env.sh+ a detector,fleet/tests/test-ramscratch-env.sh, born-red against the pre-fix file).What this PR changes
Only the thing that is this repo's problem:
_gitraised a constantsurvivor_unavailable: git inspection failedand threw away the returncode and stderr, so anenvironmental fault and a real capture defect are indistinguishable at the call site. That is
why attributing it cost a full pass (t_ac0e595c), and
kanban_survivor.pyhas five open PRs whosereviewers each pay that tax.
held_reason, the event log and stderr,and open PRs key on that string.
kanban_external_survivor.redact, the samehelper that guards an echoed claim. Git stderr can carry a credential-bearing remote URL.
Verification
fork/main(
assert 'rc=128' in '',IndexErroron zero log records).(
1 failed, 8 passed) — the pair is not a vacuous green.ruff checkclean on both changed files.Hotspot / coordination
hermes_cli/kanban_survivor.pyis a multi-way hotspot (#848, #846, #842, #839, #796). Checked eachopen PR's diff: only #796 touches this helper, and only its signature line
(
check=True→check=True, timeout=30), which does not overlap the body edited here. Diff is+66/−1 across 2 files, one impl hunk.
Upstream
kanban_survivor.pyand its tests are 404 at NousResearch/hermes-agent — no upstream surface, sono upstream PR is owed for this half.
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.