Skip to content

fix(kanban): add heartbeat grace window to prevent reclaiming slow workers (#23025) - #23108

Closed
KhanCold wants to merge 2 commits into
NousResearch:mainfrom
KhanCold:fix/23025-kanban-stale-claim-grace
Closed

fix(kanban): add heartbeat grace window to prevent reclaiming slow workers (#23025)#23108
KhanCold wants to merge 2 commits into
NousResearch:mainfrom
KhanCold:fix/23025-kanban-stale-claim-grace

Conversation

@KhanCold

Copy link
Copy Markdown

Bug Description

Kanban workers on the default board repeatedly entered a stall loop:

  1. Worker spawns (status: running)
  2. Worker produces no output
  3. Worker is reclaimed after ~15 min (status: reclaimed, error: stale_lock)
  4. Dispatcher immediately respawns a new worker
  5. New worker immediately hits the same pattern

This was particularly bad for slow models like kimi-k2.6.

Root Cause

release_stale_claims() reclaimed any running task whose claim_expires had passed, regardless of whether the worker had heartbeated recently. The default claim TTL is 15 minutes, but slow models can take longer between heartbeats.

Fix

Add a 5-minute heartbeat grace window to release_stale_claims():

  • If a worker heartbeated within the last 5 minutes, skip reclaim even if claim_expires has passed
  • Workers that are truly stuck (no heartbeat for >5 min) are still reclaimed
  • This gives slow-model workers extra time while still catching dead workers

Tests

Added tests/hermes_cli/test_kanban_stale_claim_grace_regression_23025.py with 3 tests:

  • Does NOT reclaim when worker heartbeated 2 minutes ago
  • DOES reclaim when worker last heartbeated 10 minutes ago
  • DOES reclaim when worker never heartbeated

All tests pass.

Fixes #23025

@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/cli CLI entry point, hermes_cli/, setup wizard labels May 10, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Duplicate of PR #23071 which addresses the same issue #23025 with a PID-liveness approach rather than heartbeat grace window.

@KhanCold KhanCold closed this May 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

recurring stall loop: kanban workers repeatedly reclaimed with stale_lock, zero output across respawns

2 participants