v6.7 Tranche 1: kanban_complete verification gates (#28, #62, #64) - #43104
Closed
jarvis-stark-ops wants to merge 6 commits into
Closed
v6.7 Tranche 1: kanban_complete verification gates (#28, #62, #64)#43104jarvis-stark-ops wants to merge 6 commits into
jarvis-stark-ops wants to merge 6 commits into
Conversation
…callers
Two tool schemas advertised `required: []` (or just `["mode"]`) but
enforced additional requirements in the Python handler. Anthropic-tuned
models like Claude read the prose "REQUIRED PARAMETERS: …" hints and
emit the fields. xAI grok-4.3 reads the JSON Schema literally and omits
both/all of them, then loops on the Python validator's `tool_error`.
This blocked autonomous orchestration profiles (e.g. JARVIS in the
1Team-Engineering hermes-jarvis fleet) running on grok from completing
or patching anything.
Fix: express the real constraints in the schema using JSON Schema
2020-12's `anyOf` / `oneOf`. Both branches use only `required` so the
constraint is honored by every conformant validator.
- kanban_complete: anyOf summary or result (matches handler line 543)
- patch: oneOf mode=replace (path+old_string+new_string)
| mode=patch (patch)
(matches handler in _handle_patch)
Schema validation tested with jsonschema 4.25.1 (Draft 2020-12):
- kanban_complete: 5/5 cases match expected validity
- patch: 6/6 cases match expected validity
Handlers unchanged — they still tolerate both shapes for CLI legacy
callers that bypass the schema layer (e.g. `hermes kanban complete
--result "..."`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
schema: express conditional-required constraints for grok-style tool callers
…de (#8) Closes #7. Problem The kanban dispatcher loop in `_kanban_dispatcher_watcher` can silently stop cycling while the gateway process stays alive — observed twice in one day (2026-06-07). Cause is unclear (possibly hung `dispatch_once()`, sqlite lock, or event-loop livelock). Without instrumentation we can't tell a dead loop from an idle gateway. This unblocks investigation of #5 (HTTP 429 → delayed retry) and #6 (provider auth crashes → skip-not-stall), both of which were blocked on "is the dispatcher even running?" being answerable. Solution Write a heartbeat JSON to `$HERMES_HOME/state/dispatcher_health.json` at the end of every dispatcher cycle (success AND exception paths AND cancellation). Schema (v1, stable contract): schema_version: 1 last_cycle_ts: float # unix seconds, end of cycle last_cycle_iso: str # UTC ISO 8601 with Z suffix cycle_started_at: float cycle_duration_seconds: float interval_seconds: float # configured cadence (default 60) cycles_since_start: int # monotonic; first cycle = 1, 0 means "never wrote" any_spawned_this_cycle: bool spawned_total_this_cycle: int # count across all boards ready_pending: bool # ready queue non-empty consecutive_bad_ticks: int # mirrors the existing HEALTH_WINDOW counter gateway_pid: int cycle_error: str | null # exception text if cycle errored Detection rule for monitors: stall = time.time() - last_cycle_ts > 2 × interval_seconds dead = last_cycle_ts > 5 minutes ago AND gateway PID still alive Implementation - New method `GatewayRunner._write_dispatcher_heartbeat` (gateway/run.py). Uses existing `atomic_json_write` for crash-safe writes. - Called from `_kanban_dispatcher_watcher` at end of every iteration. - Heartbeat-write failures are wrapped in try/except so they can NEVER kill the dispatcher (heartbeat is a diagnostic, not a dependency). - Cancellation path also writes a final heartbeat with cycle_error="cancelled" before re-raising — so monitors can distinguish clean shutdown from crash. - Locals (`cycles_since_start`, `any_spawned`, etc.) initialized at top of the loop body BEFORE any try block so they're defined for the heartbeat call even if zombie-reap or main tick throws. Tests (5/5 passing) - Schema-v1 contract pinned (all 13 keys, types, ISO Z suffix) - Cycle-error path recorded correctly - Two writes overwrite (not append) - First-cycle-is-1 contract (monitors treat 0 as "never wrote") - Auto-creates `state/` dir if missing Follow-up (separate issues) - Extend `hermes gateway status` to read this file and show stall age - Schema v2: add `gateway_started_at` for uptime computation without ps - Optional Prometheus textfile exposition for node_exporter setups Co-authored-by: Jarvis <jarvis@Kaipos-Mac-mini.local> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ot just runtime (#9) Closes #6. Problem Worker startup calls `resolve_runtime_provider` to acquire credentials for the primary provider. If that raises AuthError (xAI OAuth token expired, Anthropic logged out, Codex revoked), the worker crashes before AIAgent's runtime fallback loop ever gets a chance — even though the user has explicitly configured a fallback chain for exactly this case. Observed in the v6.6 incident 2026-06-07: xAI OAuth token went missing mid-session and every subsequent worker crashed at startup despite having `fallback_providers: [openai-codex/gpt-5.5, xai-oauth/grok-4.3]` configured. Solution New helper `_resolve_runtime_with_fallback` wraps the primary-resolution call. On AuthError, iterates the configured fallback chain (read once from `get_fallback_chain(cfg)`) until one succeeds. If all fail, re-raises the LAST AuthError so cli.py's exit handling can surface it. Three safety bounds preserved (informed by code-review): 1. **Explicit CLI pin** — `hermes -z --model X --provider Y ...` should NOT silently downgrade. When `model` OR `provider` was a non-empty CLI arg, the helper re-raises primary AuthError verbatim, no fallback attempt. 2. **Rate-limit AuthError on primary** — falling through to other providers would burn their quota in milliseconds (the "quota amplification" footgun). Detected via existing `is_rate_limited_auth_error()` — re-raise immediately; existing rate-limit handling (cli.py exit 75) gets the task requeued. 3. **Remaining-chain handoff to AIAgent** — when fallback lands on chain entry [N], AIAgent's runtime fallback loop should only see entries AFTER N (not the dead primary, not the entry we just used). The helper now returns `(runtime, effective_model, landed_at_index, remaining_chain)` and the caller passes `remaining` to AIAgent's `fallback_model`. Implementation - `hermes_cli/oneshot.py:33-110` — new helper (testable at module level). - `hermes_cli/oneshot.py:439-460` — call site updated; reads chain once, detects explicit_pin from CLI args, passes remaining_chain to AIAgent. - AIAgent receives the correctly-sliced chain via `fallback_model=_fb`, preserving existing runtime-fallback semantics for mid-conversation failures. Tests (9/9 passing) — tests/cli/test_oneshot_runtime_fallback.py - primary succeeds → no fallback attempted, full chain preserved for AIAgent - primary fails, first fallback succeeds → effective_model advances, remaining_chain sliced correctly - two failures → third succeeds, slicing correct - all fail → LAST AuthError propagates (not primary's) - empty chain → primary error verbatim - fallback without model → effective_model preserved - explicit_pin=True → no fallback, primary error verbatim - rate-limit AuthError → no fallback, primary error verbatim - same provider in chain → no infinite loop, advances to next entry Code-review pre-merge: reviewer caught silent-downgrade regression, stale chain handoff, and quota-amplification footgun. All three addressed. Follow-up (separate issues, not blocking) - Consider applying the same pattern to `gateway/run.py:_resolve_runtime_agent_kwargs` and `cli.py:4881-4914` for a consistent worker-startup contract across surfaces. - Optional: emit a metric/heartbeat counter when fallback fires so we can detect "constantly failing primary" silently. Co-authored-by: Jarvis <jarvis@Kaipos-Mac-mini.local> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rs (#10) Closes #5. Problem The non-quiet `hermes -p <profile> chat -q "..."` path (the actual invocation pattern Marvel team kanban workers use, e.g. `chat -q "work kanban task X"`) never applied the kanban EX_TEMPFAIL exit-code mapping. A rate-limited worker exited 0 by virtue of `cli.chat()` returning cleanly. The dispatcher's reap classifier then treated rc=0 as a "protocol violation" (worker exited cleanly without calling `kanban_complete` or `kanban_block`) and auto-blocked the task. This was the v6.6 incident root cause. Tony+Tchalla re-reviews (t_77ac35a7, t_a30a88db) hit Codex 429, exited 0, dispatcher auto-blocked, chain stalled 2 hours until quota reset — even though `KANBAN_RATE_LIMIT_EXIT_CODE = 75` and the mapping at cli.py existed (just only on the QUIET single-query path). Solution 1. Extract the exit-code mapping into `_worker_exit_code_from_result(result)` at module level. Returns 0 / 1 / 75 per these rules: - None or non-dict or success → 0 - Failure outside a kanban worker → 1 - Failure inside a worker with failure_reason ∈ {rate_limit, billing} → 75 - Any other failure inside a worker → 1 2. Refactor the quiet path (~line 16172) to call the helper instead of inline. 3. Make `cli.chat()` stash its run_conversation result on `self._last_run_result` so the non-quiet caller can inspect failure metadata after chat() returns only the response string. Reset to None at __init__ AND at start of every turn — invariant doesn't depend on early-return order (caught by code review). 4. Wire the non-quiet path (~line 16197) to call the helper after chat() returns. 5. Exception path inside chat() synthesizes `{failed:True, error:..., completed:False}` so single-query callers still apply mapping (rate-limit branch doesn't fire because no failure_reason — correctly falls through to exit 1). Tests (9/9 passing) — tests/cli/test_worker_exit_code_from_result.py - None result → 0 - Non-dict result → 0 - Success result → 0 - Failure outside kanban (no HERMES_KANBAN_TASK env) → 1, regardless of reason - Rate-limit inside kanban → KANBAN_RATE_LIMIT_EXIT_CODE (75) - Billing inside kanban → 75 (same recovery story) - Other failures inside kanban → 1 - Missing failure_reason field inside kanban → 1 (defensive) - Rate-limit without HERMES_KANBAN_TASK env → 1 (human CLI gets generic exit) Combined with #6 (worker-startup fallback) and #7 (dispatcher heartbeat), both already merged, the worker-startup → dispatcher-detect → next-retry loop is now operationally robust: - #7 — silent stalls detectable via heartbeat JSON - #6 — primary provider auth crash falls through to fallback chain at startup - #5 (this) — rate-limit failures exit 75 so dispatcher requeues without burning the retry counter Code-review pre-merge: reviewer caught a stale-stash bug (previous turn's `_last_run_result` leaking into a downstream consumer if chat() takes an early return path). Fixed by initializing the stash to None in __init__ AND resetting at the top of each chat() turn — invariant pinned. Follow-up (separate issues, not blocking) - `_print_exit_summary()` shows "Resume this session with:" even on rate-limit failure in the non-quiet path. Pre-existing; not introduced by this PR. - Non-quiet branch doesn't check `HERMES_KANBAN_GOAL_MODE` env var (only quiet path runs `_run_kanban_goal_loop_q`). If a goal_mode worker ever spawns via the non-quiet path, the goal loop silently skips. Pre-existing. Co-authored-by: Jarvis <jarvis@Kaipos-Mac-mini.local> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds three pre-write-txn gates that fire before complete_task transitions a task to done. Mirrors the existing _verify_created_cards / HallucinatedCardsError pattern: any violation is recorded as an audit event and raised, so worker state is unchanged and the worker can retry after fixing the underlying issue. Closes hermes-jarvis#28 (repo hygiene gate) Closes hermes-jarvis#62 (workspace-diff verification) Closes hermes-jarvis#64 (per-role runtime floor) Context: hermes-jarvis#61 (bootstrap-paradox case study) ## The three gates 1. verify_runtime_floor — per-role floor on completed_at - started_at. build roles 5min, review roles 90s, orchestration roles 0. Catches Tony's 20-second "approve" verdicts and Friday's 59-second "implemented 7 dispatcher gates" claims. 2. verify_workspace_diff — when a non-review worker on a dir/worktree workspace claims to have produced code, git diff against the tracking base must show actual changes. Catches Friday's "Wave A gates implemented" with an empty diff on the branch. 3. verify_no_stray_artifacts — rejects untracked or tracked artifacts matching patterns the swarm has historically committed by accident: *evidence*, commit-hash*, triage/*, tmp-*, and tracked files with no extension and no shebang (the "all prior block evidence files" failure mode from agent-dashboard PR #1). ## Opt-outs Workers may bypass individual gates via per-call metadata keys: x_fast_justified → allow_below_floor x_no_code → allow_no_code x_stray_ok → allow_stray Opt-outs are recorded as part of the completed event for audit. ## Tests 28 new tests cover the exact 2026-06-09 failure modes (Tony 20s, Friday 59s + empty diff, PR-1 "all prior block evidence files") plus clean-path passes and opt-outs. 258 passed / 0 failed in the wider kanban+complete+task test suite — zero regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Author
|
Wrong target — re-opening against 1Team-Engineering/hermes-agent. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds three pre-write-txn gates in
complete_taskthat catch the failure modes the 2026-06-09 v6.7 build chain demonstrated. See #61 for the full case study — 9 tasks reported done in ~10 minutes with zero real deliverables.Pattern mirrors the existing
_verify_created_cards/HallucinatedCardsErrorflow athermes_cli/kanban_db.py:3607. State unchanged on rejection; worker can retry after fixing the issue.Gates
verify_runtime_floor(closes #64) — per-role floor oncompleted_at - started_at. build roles 5min, review roles 90s, orchestration 0. Catches Tony's 20s reviews and Friday's 59s "implementation".verify_workspace_diff(closes #62) — non-review workers ondir/worktreeworkspaces that claim implementation work must produce a non-emptygit diffagainst the tracking base. Catches Friday's empty branch + fabricated summary.verify_no_stray_artifacts(closes #28) — rejects*evidence*,commit-hash*,triage/*,tmp-*, and tracked files with no extension and no shebang. Catches the agent-dashboard PR Terminal tool #1 "all prior block evidence files" failure mode.Opt-outs
metadata.x_fast_justified→ bypasses runtime floormetadata.x_no_code→ bypasses workspace-diffmetadata.x_stray_ok→ bypasses hygiene gateOpt-outs recorded on the completed event for audit.
Test plan
tests/cli/test_kanban_completion_gates.py— each pin pins the exact 2026-06-09 failure modes plus clean-path passes and opt-outspytest tests/cli/ tests/tools/ -k 'kanban or complete or task'→ 258 passed / 0 failedkanban_completetimeWhy this PR alone (Tranche 1)
v6.7 has 11 total issues. This PR addresses the three that would have rejected the 2026-06-09 build chain on the first attempt. Tranche 2 covers #29/#30/#31 (reviewer-field validation + auto-spawn integrative review + reviewer SOULs in hermes-jarvis). Tranche 3: #32/#33/#34/#63/#65 (doc-drift, gh-auth propagation, respawn-guarded exemption, PR-existence verification, auth-claim verification).
🤖 Generated with Claude Code