Skip to content

fix(kanban): hold external/interactive claims so the dispatcher stops re-dispatching live operator sessions - #51386

Open
vk0dev wants to merge 1 commit into
NousResearch:mainfrom
vk0dev:pr/dispatcher-external-claim-hold
Open

fix(kanban): hold external/interactive claims so the dispatcher stops re-dispatching live operator sessions#51386
vk0dev wants to merge 1 commit into
NousResearch:mainfrom
vk0dev:pr/dispatcher-external-claim-hold

Conversation

@vk0dev

@vk0dev vk0dev commented Jun 23, 2026

Copy link
Copy Markdown

Problem

A live interactive operator session (e.g. a Claude Code session in tmux) that runs hermes kanban claim <id> does not emit kanban heartbeats or other Hermes API traffic. Its claim_expires / last_heartbeat_at therefore go stale exactly like a wedged worker, and release_stale_claims() cannot tell the two apart. It reclaims the task, and the next dispatcher tick spawns a parallel worker on a task a human is actively working on — double work, merge conflicts, and wt/* worktree sprawl.

This is distinct from the cgroup-throttle duplication that RECLAIM_DEFER_GRACE_SECONDS addresses: here the claim holder is a healthy interactive process that legitimately never heartbeats.

Fix — explicit external/interactive claim hold

  • New nullable tasks.claim_kind column ('worker' | 'external'; NULL = legacy/worker). Additive migration via _migrate_add_optional_columns; existing boards keep current behaviour.
  • claim_task(external=True) marks the claim 'external' and holds it indefinitely (claim_expires = NULL). Such a row is invisible to all three reclaimers: release_stale_claims (filters claim_expires IS NOT NULL), enforce_max_runtime and detect_crashed_workers (filter worker_pid IS NOT NULL — an external hold has none). A claim_kind guard in release_stale_claims adds defense-in-depth.
  • CLI: hermes kanban claim <id> --external (opt-in; the default claim path is unchanged — still a worker claim with a TTL). An external hold is released only by an explicit complete / block / reclaim.
  • complete / block / reclaim clear claim_kind, so a reclaimed external task is cleanly worker-claimable again (operator recovery path for a dead interactive session).

Reclaim of genuinely dead/wedged worker claims is unchanged (regression-guarded).

Tests

Adds coverage in test_kanban_db.py and test_kanban_cli_dispatch_passthrough.py: external claim survives the stale sweep (indefinite + forced-expiry guard), worker claim still reclaimed, --external CLI contract, complete/block/reclaim clear the hold, and the additive migration. Full affected-file suites pass (239 tests).

🤖 Generated with Claude Code

…patch

A live interactive Claude Code session (tmux ccz/ccp) that runs
`hermes kanban claim <id>` does not emit kanban heartbeats or Hermes API
traffic, so its claim_expires/last_heartbeat_at go stale exactly like a
wedged worker. release_stale_claims() could not tell the two apart and
reclaimed the task, after which the dispatcher spawned a parallel coder
worker on it — the t_0781f106 parallel-work incident (double work, merge
conflicts, wt/* worktree sprawl).

Introduce an explicit external/interactive claim hold:

- New nullable `tasks.claim_kind` column ('worker' | 'external'; NULL =
  legacy/worker). Additive migration; existing boards keep current
  behaviour.
- claim_task(external=True) marks the claim 'external' and holds it
  indefinitely (claim_expires = NULL). Such a row is invisible to all
  three reclaimers: release_stale_claims (filters claim_expires IS NOT
  NULL), enforce_max_runtime and detect_crashed_workers (filter
  worker_pid IS NOT NULL — an external hold has none). A claim_kind guard
  in release_stale_claims adds defense-in-depth.
- CLI: `hermes kanban claim <id> --external` (opt-in; default claim path
  unchanged, stays a worker claim with a TTL). Released only by explicit
  complete / block / reclaim.
- complete/block/reclaim clear claim_kind so a reclaimed external task is
  cleanly worker-claimable again (operator recovery path for a dead
  interactive session).

Real reclaim of dead/wedged WORKER claims is unchanged (regression-
guarded). Tests cover: external claim survives stale sweep (indefinite +
forced-expiry guard), worker claim still reclaimed, CLI --external
contract, complete/block/reclaim clear the hold, and the additive
migration.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@alt-glitch alt-glitch added type/bug Something isn't working comp/cron Cron scheduler and job management P3 Low — cosmetic, nice to have labels Jun 23, 2026

@tonydwb tonydwb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Verdict: Approved

Well-designed fix for the kanban dispatcher re-dispatching live operator sessions. External (interactive) claims are now held indefinitely and never auto-reclaimed.

Looks Good

  • New claim_kind column (worker vs external) cleanly discriminates claim types
  • External claims set claim_expires = NULL and worker_pid = NULL, making them invisible to all three reclaimers
  • DB migration is additive (new column, backward-compatible with legacy rows)
  • CLI --external flag with clear help text explaining the behavior
  • Defense-in-depth: release_stale_claims additionally guards on claim_kind
  • Well-documented with extensive comments explaining the design rationale

Reviewed by Hermes Agent

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for isolating the interactive-operator claim case. The underlying issue remains reproducible on current main: hermes_cli/kanban.py:1816-1818 creates the normal finite-TTL claim, and hermes_cli/kanban_db.py:3634-3713 can reclaim it when no worker PID exists.

Problems

  • The proposed external hold would still be reclaimed by the newer detect_stale_running watchdog. It selects every running task at hermes_cli/kanban_db.py:6466-6472, treats a NULL heartbeat as stale at :6483-6486, and resets the task at :6508-6513. This path is not covered by the PR's claim_kind guard.
  • The new public CLI flag is absent from the documented claim syntax at website/docs/user-guide/features/kanban.md:642.

Suggested changes

  • During salvage, guard detect_stale_running for external holds and add a regression test for that watchdog.
  • Reconcile claim-kind cleanup with current typed-block branches (hermes_cli/kanban_db.py:4825-4946) and document --external.

This is an automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants