feat(dashboard): emit bounded slash worker telemetry - #3
Conversation
* fix(kanban): route not-clean reviews to remediation * fix(kanban): defer incomplete review routing * fix(kanban): persist clean review handoffs * fix(kanban): persist deferred review routing gates * fix: bind re-review routing to remediation SHA * chore: register remediation commit author * fix: gate rereviews on completed remediation * fix(kanban): preserve deferred review gate on archive --------- Co-authored-by: Hermes Agent <hermes-agent@users.noreply.github.com> Co-authored-by: Codex <codex@openai.com>
#5) * fix(kanban): capture worker exit codes on Windows via retained Popen handles On Windows the dispatcher abandoned each worker's Popen handle at spawn and reap_worker_zombies() was a no-op, so worker exit codes were never observed. Every worker death surfaced as an opaque 'pid N not alive' crash, which made four protections dead code on this platform: - rate-limit sentinel exits (code 75) were counted as crashes instead of requeueing without a failure — quota walls breaker-blocked entire boards - clean-exit protocol violations were never classified - real exit codes never reached error text or events - PID-only liveness probes could confuse a dead worker with an unrelated process that recycled its PID (claim extensions on zombies) Fix, all behind os.name == 'nt' (POSIX behavior unchanged): - retain each spawned worker's Popen handle in a registry keyed by pid - reap_worker_zombies() polls retained handles and records returncodes into the existing _recent_worker_exits registry - _classify_worker_exit() reads returncodes directly on nt (no waitpid encoding) and falls back to the retained handle when no reap ran - new _worker_alive() prefers the retained handle over PID probes in detect_crashed_workers and release_stale_claims; an open process handle also prevents the kernel from recycling the pid at all Exited handles are retained for the exit-record TTL (so same/next-tick liveness checks stay handle-backed) and then released. Tests: 6 new Windows-path tests; _exited_status() test helper made platform-aware, which also un-breaks the two existing rate-limit tests on Windows hosts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: overnight-autonomous-2026-07-12 * fix(kanban): persist breaker override limits, add global quota-wall brake and crash backoff Three dispatcher-hardening fixes from the 2026-07-11 mass-block forensics: 1. Breaker override-limit persistence. The systemic-fingerprint and protocol-violation fast paths trip _record_task_failure at failure_limit=1, but recompute_ready resolves the limit independently (per-task max_retries else the CONFIG limit) and re-promoted every override-tripped card in the same tick (cf=1 < 2) into a second doomed spawn — inverting the protection. 18 of 28 cascade cards show the same-second gave_up->promoted->crash pair. Override trips now stamp their effective limit into the task's max_retries (persist_limit=True) so both readers agree; the docstring's false 'never disagree' claim is corrected. 2. Global quota-wall brake. When a worker dies on a provider usage wall, every new spawn is doomed until the window resets, but the dispatcher kept spawning fresh workers every tick (23 ticks, 64 crashed sessions on 07-11). A board-wide wall is now recorded in a new kanban_meta table — by the worker itself just before exiting with the EX_TEMPFAIL sentinel (works even where exit codes are invisible to the dispatcher), and by detect_crashed_workers when it observes a rate-limited exit — and dispatch_once spawns nothing until the wall expires (reclaim/promote bookkeeping still runs). resets_at is parsed from the provider error when present (epoch or ISO-8601), else defaults to the rate-limit cooldown; walls only extend and are capped at 6h. 3. Crash backoff. check_respawn_guard gains a 'crash_backoff' reason: after a failed run, respawn waits base * 2^(consecutive_failures-1) seconds (HERMES_KANBAN_CRASH_BACKOFF_BASE_SECONDS, default 120s, capped 30min, 0 disables) so an outage shorter than the dispatch interval can no longer burn a card's whole failure budget. Tests: 10 new; existing spawn-failure breaker tests explicitly disable the new backoff (they assert immediate-retry mechanics). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: overnight-autonomous-2026-07-12 * fix(kanban): validate card skills at authoring time; operator-channel auto-subscribe fallback Two authoring-time protections from the 2026-07-11 board forensics: 1. Skill-roster validation in create_task. Worker spawns force-load the card's skills via 'hermes --skills X'; cli.py hard-fails when EVERY requested skill is missing from the assignee profile's roster, so a decomposer that assigns skills its assignee doesn't have spawn-kills the worker at argparse (37 deterministic deaths, breaker-blocked cards). create_task now checks the requested skills against the assignee profile's installed roster (SKILL.md dirs under the profile's HERMES_HOME/skills) and strips unknown names, recording them as skills_stripped on the created event. Conservative: when the roster cannot be determined (unknown profile, no skills tree) nothing is stripped; HERMES_KANBAN_SKILL_VALIDATION=0 disables. 2. Operator-channel fallback in _maybe_auto_subscribe. Cards created with no session context (dispatcher, decomposer, cron) previously got NO notification subscription — 41 of 43 blocked cards on the 07-11 board were born silent. When kanban.operator_channel ('platform:chat_id[:thread_id]') is configured, such cards now subscribe that channel, delivered via kanban.operator_notifier_profile (default 'default' — must be a profile with a connected adapter). No-op when unconfigured. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: overnight-autonomous-2026-07-12 * fix(kanban): let explicit re-queues bypass the active_pr respawn guard The active_pr guard defers any card with a GitHub PR URL in a comment from the last 24 hours, with no bypass. PR-steward cards (refresh / remediate / un-draft lanes) cite their PR in nearly every comment, so any lane needing a second spawn - park->release, block->unblock, reclaim - froze for the full window. On 2026-07-12 this stalled merge-eligible PRs (CLEAN verdict, all checks green) at the un-draft hand-off for hours and starved the serial merge train. Mirror the recent_success guard's existing bypass: an explicit re-queue event (status, promoted, unblocked, reclaimed) recorded AFTER the newest PR-URL comment is a deliberate re-run request and allows the spawn. The anti-duplicate-PR intent is preserved: a card that just opened a PR and crashed still cannot silently respawn. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(kanban): restrict active_pr bypass to operator-driven requeues Review remediation for the exact-head NOT CLEAN verdict: - Only reclaim events with payload manual=true bypass (release_stale_claims auto-reclaims are the crash-after-opening-a-PR case the guard suppresses). - Strict > on created_at: integer-second clocks made >= treat a pre-comment same-second event as after the comment. - New tests: real release_stale_claims producer path does not bypass; manual reclaim_task does; same-second tie fails closed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(kanban): fail closed on non-literal manual reclaim payloads Review remediation round 2: the manual-reclaim discriminator accepted any truthy value and crashed on valid non-object JSON ([] escaped the narrow except as AttributeError). Require isinstance(dict) plus payload.get(manual) is True, catch broadly, fail closed. New test covers truthy-string, numeric, list, and null payloads. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Hermes Agent <hermes-agent@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # scripts/release.py # tui_gateway/server.py
|
VERDICT: NOT CLEAN Exact-head review against base
Validation at this exact head:
|
The profile-local MCP probe blocks HermesCLI's one-time tool snapshot on wait_for_mcp_discovery(), whose bound defaults to 1.5s. A cold CI runner needs longer than that to spawn a fresh interpreter, import mcp.server.fastmcp, and finish the stdio handshake, so the snapshot was taken before 'profileprobe' connected and /tools returned only the built-ins (slice 4/8 failure on PR #3). thread.join returns the instant discovery completes, so pinning mcp_discovery_timeout=20 in the test config costs ~0s locally while giving CI ample headroom. Reproduced the failure locally by forcing the bound to 0.01s.
|
Superseded — re-homed into hermes-desktop-overlay (patches/current-uncommitted.patch, commit 5cd32f0). The deployment builds from upstream NousResearch/main + the overlay, not this fork. This fork is being reset to upstream and archived; it now exists solely to hold upstream PR NousResearch#61868. Closing. |
2 participants
Draft Status
Opened as draft pending fresh exact-SHA independent review and repository checks for replacement head
babf754d34553f46b3a1a6626538bd52c0923d69.Summary
mainwith ordinary merge commit427f7d790071c1579c139a04a9b2dd0fb9803dcf, preserving the forkAUTHOR_MAPentry, current main mappings, profile-home propagation, and telemetry spawn-failure handlingworker_stoppedrequires confirmed exit, bootstrap status propagates, genuine recovery is counted while planned replacement is not, and incident/recovery fingerprints link durablytui_gateway.serverlogger path reaches dashboardgui.logwith adversarial privacy/non-duplication coverageKanban idempotency key:
t_44c56fb8Validation
uv run --frozen pytest -q -p no:cacheprovider tests/tui_gateway/test_slash_worker_telemetry.py— 24 passeduv run --frozen pytest -q -p no:cacheprovider tests/tui_gateway/test_slash_worker_mcp_discovery.py— 1 passeduv run --frozen pytest -q -p no:cacheprovider tests/tui_gateway/test_slash_worker_profile_home.py— 4 passeduv run --frozen pytest -q -p no:cacheprovider tests/tui_gateway/test_protocol.py— 75 passeduv run --frozen pytest -q -p no:cacheprovider tests/test_tui_gateway_server.py tests/test_tui_gateway_ws.py tests/test_tui_gateway_loop_noise.py tests/test_tui_gateway_queue_on_busy.py— 338 passeduv run --frozen --extra dev ruff check tests/tui_gateway/test_slash_worker_telemetry.py tests/tui_gateway/test_slash_worker_mcp_discovery.py tui_gateway/server.py tui_gateway/slash_telemetry.py tui_gateway/slash_worker.py— passeduv run --frozen python -m py_compile tui_gateway/server.py tui_gateway/slash_telemetry.py tui_gateway/slash_worker.py tests/tui_gateway/test_slash_worker_telemetry.py tests/tui_gateway/test_slash_worker_mcp_discovery.py— passedgit diff --checkand stagedgit diff --cached --check— passedtest_protocol.pyreplaces method registry entries; 664 passed); the affected protocol, profile-home, telemetry, MCP, and server suites above all pass in clean isolated processesPrivacy / Non-duplication Evidence
Deployment Impact
Code and tests only. No live rollout, runtime/config/channel/token/alert/task/service/queue/process/restart, production telemetry, or collector mutation was performed.
Rollback
Revert
babf754d34553f46b3a1a6626538bd52c0923d69(and the ordinary branch refresh merge if reverting the full remediation generation). Telemetry remains fail-open and command protocol execution does not depend on request-ID replay state.Agent Notes
s00rz-hermes/hermes-agentcodex/dashboard-slash-worker-true-northbabf754d34553f46b3a1a6626538bd52c0923d69427f7d790071c1579c139a04a9b2dd0fb9803dcfMerge Readiness