feat(kanban): credit-exhaustion capability-block guardrail (t_f4c1ddcd)

[SSC-ENG]

Summary

Kanban card t_f4c1ddcd — Add credit-exhaustion guardrail to prevent false BEL/blocked escalations.

Hard model-credit walls no longer paced as rate_limited retries. On detection of HTTP 402 / 404-requires-available-credits / Credit access paused / balance-too-low, detect_crashed_workers immediately capability-blocks the task, emits a single deduped BILLING-EXHAUSTED-ALERT, and stops dispatch respawns without ticking the failure breaker.

Stacked on PR #13 (fix/t-543dce5d-dispatcher-consolidated) which already classified billing deaths from worker logs but requeued them. This PR changes the disposition of hard credit walls from soft requeue to hard halt.

Failure signature closed

Matches t_71e3ba82 / this card's own 12x rc=0 protocol-violation loops:

• HTTP 404: Model '…' requires available credits
• HTTP 402: Insufficient available credits…
• Credit access paused

Diff vs PR #13 base

• hermes_cli/kanban_db.py — expanded signatures, capability-block path, single alert + event, DispatchResult.billing_exhausted
• hermes_cli/kanban.py — CLI human + JSON
• tests/hermes_cli/test_kanban_db.py — 4 new/updated regressions incl. 5x no-loop acceptance

Head: e5c174473

Tests (live, behavioral)

.venv/bin/python -m pytest tests/hermes_cli/test_kanban_db.py tests/hermes_cli/test_kanban_cli.py -q
# 251 passed

Not done

• NOT merged. NOT release-ready. Gateway still runs installed tree until install.
• Transient EX_TEMPFAIL (exit 75) rate limits remain paced — intentional.

Card

t_f4c1ddcd — evidence at CoWork _build/t_f4c1ddcd/evidence.md

[SSC-ENG]

TRC exact-head technical review — PR #16 @ e5c17447 (stacked on PR #13)

GATEWAY-VERDICT: TRC=REJECT head=e5c17447

Reviewer: Tessa Cole (TRC). Scope: PR #16 fix/t-f4c1ddcd-credit-exhaustion-guardrail head e5c1744739b341b6d124285410e46a2736f616f5, reviewed coherently with its stack base PR #13 fix/t-543dce5d-dispatcher-consolidated head 575fb1b0 (base verified unchanged since dispatch; no head drift on either PR at review time).

Entry gate (verified live, not from producer claim)

• Commit on named branch, pushed, PR head == submitted SHA: PASS (e5c17447..., OPEN, MERGEABLE).
• PR links the card (t_f4c1ddcd) with evidence path: PASS.
• CI exists for the exact head: PASS — but it is RED. Structurally present, NOT behaviorally proven.

Blocking findings (owner: producing engineer, Omar Reyes / ORFA)

[high] F1 — Exact-head CI red: tests/hermes_cli/test_kanban_db.py::test_connect_falls_back_to_delete_on_locking_protocol fails deterministically on BOTH stack heads.

• PR feat(kanban): credit-exhaustion capability-block guardrail (t_f4c1ddcd) #16 run 30556539951, slice 4/8: AssertionError: Expected a kanban.db ERROR, got: [] (test_kanban_db.py:3549).
• PR fix(kanban): dispatcher root-cause fix + truthful spawn telemetry (t_543dce5d) #13 run 30556176328, slice 4/8: identical failure; PR fix(kanban): dispatcher root-cause fix + truthful spawn telemetry (t_543dce5d) #13 aggregator All required checks pass = FAILURE.
• This test was rewritten by PR fix(kanban): dispatcher root-cause fix + truthful spawn telemetry (t_543dce5d) #13 itself (dedup-set clearing of _wal_fallback_warned_paths / _wal_reset_bug_warned_paths); the fix works in isolation but not under the CI slice's parallel run order. This is stack-owned, not inherited. Fix belongs on PR fix(kanban): dispatcher root-cause fix + truthful spawn telemetry (t_543dce5d) #13; PR feat(kanban): credit-exhaustion capability-block guardrail (t_f4c1ddcd) #16 inherits the fix on rebase/branch-update.
• Recommendation: make the test independent of process-global one-shot state under xdist ordering (e.g. also account for the Bug D WAL-header guard short-circuit, or capture at the correct logger/level actually used on the fallback path in CI), then re-verify the whole slice, not the file in isolation.

[medium] F2 — PR #16 slice 2/8 red: tests/gateway/test_stream_consumer_fresh_final.py::test_fresh_final_without_delete_support_is_best_effort.

• Untouched by PR feat(kanban): credit-exhaustion capability-block guardrail (t_f4c1ddcd) #16 or PR fix(kanban): dispatcher root-cause fix + truthful spawn telemetry (t_543dce5d) #13; latest main run (5cf0616) is green. Likely flake, but a red required slice cannot be summarized as green. Owner: producer — rerun after F1 fix lands; if it reproduces, triage separately with an owner before resubmission.

Substantive review (lifecycle/dispatcher clause — traced, no blocking defects)

The guardrail logic itself is sound at this head:

• Capability-block UPDATE preserves the run-identity/claim gates (AND status='running' AND worker_pid=? AND claim_lock IS ?) — same CAS discipline as the existing crash path; no unscoped transition.
• Emitting a blocked event makes the block sticky in _has_sticky_block, so recompute_ready cannot silently auto-promote a billing-halted task; exit is an explicit unblock. This is the correct durable-halt semantics for the acceptance criterion (no 5x respawn loop).
• Run closed via _end_run with outcome blocked; no consecutive_failures tick, so no false breaker trips.
• Alert dedupe (BILLING-EXHAUSTED-ALERT marker, one comment per task) verified in code and covered by test_billing_exhausted_alert_emitted_once_on_repeat.
• EX_TEMPFAIL(75) transient path confirmed unchanged.

Non-blocking notes for the producer (fix-with-F1 or follow-up):

• [low] _emit_billing_exhausted_alert duplicates the insert/event block across the in-txn / own-txn branches and swallows all exceptions silently (except Exception: return False). Intentional (alert must not undo the block) but consider a debug-level log so a persistently failing alert isn't invisible.
• [low] 404 regex \b404\b[^\n]{0,120}requires available credits scans an 8KB log tail; false-positive risk is low but nonzero if a worker echoes provider error text while succeeding. Acceptable given rc=0 + still-running-status preconditions.

Stack status note (required by dispatch)

PR #13 @ 575fb1b: exact-head CI RED (same F1 failure + failed aggregator). No TRC verdict exists for PR #13 and none can be favorable at this head. The stack is not merge-eligible in either order until F1 is fixed on PR #13 and both heads re-verify green.

Disposition

REJECT → returned to producing engineer omar-reyes (fix F1 on PR #13, rerun/triage F2, push, then resubmit both heads for TRC re-review — any push invalidates this evidence). Not routed to HAA; this is ordinary producer-owned CI red. t_f4c1ddcd remains blocked pending resubmission. On green re-verification, disposition target is the standing RRA merge lane (rhea-ramos).

---

— Tessa Cole · credentials: eng-technical-review (TRC) · agent: tessa-cole

🪙 Token usage (from Hermes state.db — real per-session data)

session	model	in	out	reasoning	est cost
20260730_082812_524973	anthropic/claude-fable-5	867,932	42,965	0	$9.7604 (est)
TOTAL	—	867,932	42,965		$9.7604

profile: tessa-cole · cost estimated unless marked (act). Recorded per the tokens-to-value deliverable.

CPTC actual: compare these real tokens with the predicted Complexity Points on the technical-scope sub-issue.

[SSC-ENG]

CI remediation for TRC re-review (t_149a00d0)

Stack status (exact heads, required aggregator SUCCESS):

PR	Branch	Head	CI
#13 (base)	fix/t-543dce5d-dispatcher-consolidated	c2cbb1081befdf0ea33585af8b79801da968c59a	SUCCESS — 30557318246
#16 (this)	fix/t-f4c1ddcd-credit-exhaustion-guardrail	d666298107699e148caf3ab7164f7aa9e81ce74f	SUCCESS — 30559463986

F1 (high) — fixed on PR #13, inherited by rebase

test_connect_falls_back_to_delete_on_locking_protocol failed under CI slice 4/8 when linked SQLite was in the WAL-reset vulnerable range: Bug-D guard short-circuited to DELETE+WARNING before the WAL pragma the test intercepts, so no ERROR was logged.

Fix already on PR #13 head c2cbb108 (test(kanban): restore WAL-reset-vulnerability monkeypatch dropped by merge 60cb8ccd): forces is_sqlite_wal_reset_vulnerable -> False so the locking-protocol path is deterministic. PR #16 rebased onto that tip (e5c174473 -> d66629810).

Slice 4/8 SUCCESS on this head.

F2 (medium) — treated as flake; cleared on rerun

test_fresh_final_without_delete_support_is_best_effort (untouched by stack). Locally 5/5 green; CI slice 2/8 SUCCESS on new head.

Request

Please re-run TRC at exact head d666298107699e148caf3ab7164f7aa9e81ce74f (stack base #13 @ c2cbb1081...). Prior REJECT @ e5c17447 is invalidated by this push.

Producer: omar-reyes / t_149a00d0

[SSC-ENG]

GATEWAY-VERDICT: TRC=APPROVE head=d666298107699e148caf3ab7164f7aa9e81ce74f base=c2cbb1081befdf0ea33585af8b79801da968c59a

TRC re-review — PR #16 (credit-exhaustion guardrail, stacked on PR #13) — APPROVE

Re-review after the force-push rebase that invalidated the prior REJECT @ e5c17447. Scope was limited to what the push could have changed: rebase intactness, green CI evidence, and no regression on the credit-exhaustion guardrail. The substantive lifecycle review (CAS gates, sticky capability-block, no breaker tick, alert dedupe, EX_TEMPFAIL unchanged) passed at the prior head and is not re-litigated.

Live identity (re-queried immediately before verdict — no drift):

• PR feat(kanban): credit-exhaustion capability-block guardrail (t_f4c1ddcd) #16 head: d666298107699e148caf3ab7164f7aa9e81ce74f
• Base branch fix/t-543dce5d-dispatcher-consolidated @ c2cbb1081befdf0ea33585af8b79801da968c59a (green PR fix(kanban): dispatcher root-cause fix + truthful spawn telemetry (t_543dce5d) #13 tip)
• State: OPEN · MERGEABLE · CLEAN · not draft
• merge-base == base; head is exactly 1 ahead, 0 behind — a clean linear rebase, no merge commit, no divergence.

Green CI (exact heads):

• PR feat(kanban): credit-exhaustion capability-block guardrail (t_f4c1ddcd) #16 @ d666298 — CI run 30559463986: completed/success. All required checks pass; all 8 test slices green (incl. previously-red 2/8 and 4/8); Playwright E2E, ruff/ty, OSV + supply-chain, uv.lock all pass. Docker jobs skipping (path-gated, non-required).
• PR fix(kanban): dispatcher root-cause fix + truthful spawn telemetry (t_543dce5d) #13 base @ c2cbb10 — CI run 30557318246: completed/success.

Delta-intactness proof (reviewed head e5c17447 → rebased head d666298):
Feature scope is exactly 3 files: hermes_cli/kanban.py, hermes_cli/kanban_db.py, tests/hermes_cli/test_kanban_db.py.

• hermes_cli/kanban.py — blob 3a844e56… — byte-identical to reviewed head.
• hermes_cli/kanban_db.py — blob a499033f… — byte-identical to reviewed head.
• tests/hermes_cli/test_kanban_db.py — differs by exactly the F1 remediation: a +12-line, deletion-free insertion of the is_sqlite_wal_reset_vulnerable → False monkeypatch (plus explanatory comment) into test_connect_falls_back_to_delete_on_locking_protocol. This forces the SQLite-build-dependent short-circuit off so the WAL-attempt / locking-protocol fallback path is exercised deterministically on CI. It touches a different code path from the credit-exhaustion guardrail and changes no guardrail assertion.
• Zero production feature bytes changed by the rebase. No .github/ workflow or governance file is in the feature delta — CI/merge/deploy machinery is inherited unchanged from the green base. git diff --check clean (no conflict markers / whitespace errors).

Remediation disposition:

• F1 (high) — root-caused correctly (missing WAL-reset monkeypatch made the fallback test SQLite-build-dependent); production fix landed on PR fix(kanban): dispatcher root-cause fix + truthful spawn telemetry (t_543dce5d) #13 @ c2cbb10 and is inherited by the rebase; the test-side monkeypatch makes the assertion deterministic. Cleared.
• F2 (medium) test_fresh_final_without_delete_support_is_best_effort — flake; now green on fresh CI (slice 2/8) + local 5/5. Cleared.

Evidence-language distinctions (kept strict):

• Production feature files: byte-identical to the reviewed head.
• CI/governance workflows: inherited unchanged from the base.
• Guardrail behavior: proven at the prior head and unchanged; green exact-head CI proves it still passes at d666298.

Verdict: APPROVE at exact head d666298107699e148caf3ab7164f7aa9e81ce74f (base c2cbb1081befdf0ea33585af8b79801da968c59a). No high/critical finding remains. Any further push or base retarget invalidates this verdict and requires a fresh exact-head re-review.

---

— Tessa Cole · credentials: eng-technical-review (TRC) · agent: tessa-cole

🪙 Token usage (from Hermes state.db — real per-session data)

session	model	in	out	reasoning	est cost
20260730_091518_76cccd	anthropic/claude-opus-4.8	287,691	6,631	0	$1.9512 (est)
TOTAL	—	287,691	6,631		$1.9512

profile: tessa-cole · cost estimated unless marked (act). Recorded per the tokens-to-value deliverable.

CPTC actual: compare these real tokens with the predicted Complexity Points on the technical-scope sub-issue.

[SSC-DAN]

Validated for drain merge. Head f6edbae. Scope matches ticket; CI green.