Skip to content

fix(kanban): a sleep-drifted start-time fingerprint must not release a live worker's claim - #118353

Open
liuhao1024 wants to merge 4 commits into
NousResearch:mainfrom
liuhao1024:liuhao/cron-bugfix-118326
Open

liuhao1024 wants to merge 4 commits into
NousResearch:mainfrom
liuhao1024:liuhao/cron-bugfix-118326

Conversation

@liuhao1024

@liuhao1024 liuhao1024 commented Sep 21, 2026 •

Copy link
Copy Markdown

What does this PR do?

On macOS, psutil.Process.create_time() readings of an unchanged live process drift forward across every sleep/wake by the accumulated sleep. The kanban stale-claim reaper compares that reading against the spawn-time fingerprint with exact equality, so after a wake it concludes the PID was recycled: it releases a live worker's claim without sending any signal, a duplicate worker spawns beside the original, both write the same card, and the failure breaker eventually blocks the card on pure infrastructure artifacts (#118326).

This PR makes claim liveness (_worker_alive) tolerate a forward start-time drift — same instantiation epoch and a start-time delta within _SAME_WORKER_START_DRIFT_CEILING (12 h of accumulated sleep, centiseconds) — so a drifted-but-live worker keeps its claim (extended, not reclaimed). The same rescue also unblocks the worker-side _claim_is_live fence, so a worker finishing right after a wake can close its own run again.

The signalling path stays exact: _pid_recycled (which gates every SIGTERM/SIGKILL decision) still requires an exact fingerprint match, so a genuinely recycled PID is never signalled. Follow-up to a field report on this PR: the drift rescue also had to stop the exact-vs-tolerant split from leaking into two actions —

  • _terminate_reclaimed_worker's recycle branch no longer marks a drifted-but-live worker terminated (which released the claim beside it and re-created the duplicate-spawn loop); it reports survival (signal_refused), so the existing _worker_survived_termination path defers the reclaim (reclaim_deferred).
  • the SIGKILL escalation in _terminate_reclaimed_worker and enforce_max_runtime now re-checks the exact _pid_recycled before escalating, so a tolerant-alive verdict on a recycled near-match never turns into signal permission.

A claim wrongly held against a recycled near-match is at worst held until its heartbeat goes stale, at which point the existing #29747 wedged-worker backstop releases the claim — the fail-closed direction. This mirrors the reasoning #111454 applies to the cron ledger, scoped to the kanban paths that PR does not cover.

Related Issue

Fixes #118326

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • hermes_cli/kanban_db_dispatch.py: added _SAME_WORKER_START_DRIFT_CEILING and _drifted_spawn_still_matches; _worker_alive now rescues a same-epoch fingerprint whose start half drifted forward within the ceiling. No change to _pid_recycled itself.
  • hermes_cli/kanban_db_dispatch.py (follow-up): _terminate_reclaimed_worker reports a drifted-but-live worker as surviving instead of terminated (claim defers, never released beside the live worker), and both its SIGKILL step and enforce_max_runtime's re-check the exact fingerprint before escalating — drift tolerance never becomes signal permission. Operator reclaim_task keeps its human-override release; the terminal reaper retries drifted-live workers on the next tick.
  • tests/hermes_cli/test_kanban_worker_pid_fingerprint.py: five regression tests (drifted worker keeps its claim; drifted fingerprint is never signalled nor released on the stale-heartbeat reclaim path — now asserts reclaim_deferred; a gross 13 h drift is still treated as a recycle; the post-SIGTERM recycled near-match never takes the SIGKILL on either the reclaim or the timeout path).

How to Test

  1. pytest tests/hermes_cli/test_kanban_worker_pid_fingerprint.py -q — all 9 tests pass (4 pre-existing + 5 new/updated). Observed result: 9 passed.
  2. Red check: with the kanban_db_dispatch.py change stashed, test_sleep_drifted_fingerprint_keeps_the_live_worker fails (the drifted worker is reclaimed); with the change applied it passes.
  3. Neighbouring regression sweep: pytest tests/hermes_cli/test_kanban_db.py tests/hermes_cli/test_kanban_terminal_worker_reaper.py tests/hermes_cli/test_kanban_reclaim_claim_lock_guard.py tests/hermes_cli/test_kanban_complete_live_claim_guard.py tests/hermes_cli/test_kanban_dispatch_lock.py tests/hermes_cli/test_kanban_dispatch_tick_hook.py tests/hermes_cli/test_kanban_dispatch_claim_allowlist.py tests/hermes_cli/test_kanban_gateway_restart_handoff.py tests/hermes_cli/test_kanban_worker_pid_fingerprint.py -q — 89 passed, 6 skipped (plus test_infrastructure_spawn_refusal_never_charges_the_card, which fails identically on the unmodified baseline of this branch and is unrelated to this change).
  4. ruff check hermes_cli/kanban_db_dispatch.py tests/hermes_cli/test_kanban_worker_pid_fingerprint.py — passes.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS 15 (arm64)

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

For New Skills

N/A — no skill added.

Screenshots / Logs

N/A

…a live worker's claim

macOS psutil create_time() readings of an unchanged process drift forward across
sleep/wake; the stale-claim reaper read the drift as PID recycling and released a
live worker's claim without a signal (NousResearch#118326). Claim liveness (_worker_alive) now
tolerates a forward start-time drift on the same instantiation epoch; the signal
path (_pid_recycled) stays exact so a recycled PID is still never signalled.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management sweeper:risk-automation Sweeper risk: may affect CI, automerge, label sync, or maintainer automation labels Sep 21, 2026
@kyssta-exe

Copy link
Copy Markdown

Summary

Fixes #118326: on macOS, psutil create_time() readings drift forward across sleep/wake, so the stale-claim reaper misread a live worker as a recycled PID, released its claim, and spawned a duplicate beside it. Claim liveness now tolerates same-epoch forward drift within a 12h ceiling while the signalling path stays exact.

What changed

  • hermes_cli/kanban_db_dispatch.py: new _SAME_WORKER_START_DRIFT_CEILING + _drifted_spawn_still_matches(); _worker_alive() rescues a same-epoch fingerprint whose start half drifted forward within the ceiling. _pid_recycled and all signalling paths untouched.
  • tests/hermes_cli/test_kanban_worker_pid_fingerprint.py: three tests — drifted worker keeps claim (extended, not reclaimed), drifted worker is never signalled on the stale-heartbeat path, 13h gross drift still counts as recycle.

Strengths

Findings

  • hermes_cli/kanban_db_dispatch.py (_worker_alive): liveness now costs up to two separate _process_fingerprint reads per call (_pid_recycled then the rescue re-read), which can straddle a concurrent change. Fine at reaper cadence, but threading one already-read fingerprint through both checks would remove the TOCTOU window. Non-blocking polish.

Verdict

Looks good to merge.

Reviewed using Hermes-Agent

@liuhao1024

Copy link
Copy Markdown
Author

Thanks for the careful review!

Agreed that in the worst case _worker_alive can read _process_fingerprint twice (once in _pid_recycled, once in the rescue). I kept the re-read deliberately so _pid_recycled and every signalling path stay byte-identical to before — the diff only widens liveness. At reaper cadence the window is benign: a torn read can only hold a recycled PID until the heartbeat backstop releases it, never signal it. Threading one already-read fingerprint through both checks is reasonable follow-up polish if maintainers want it.

@01startsales

Copy link
Copy Markdown

Independent field confirmation of #118326 on macOS 27.0 (arm64, psutil 7.2.2, Python 3.11.16, Hermes v0.21.3 → v0.21.4), plus one gap we hit when carrying a very similar fix locally.

What we observed. A Prometheus-profile kanban worker (run 56) was reclaimed as "crashed" while the process was still alive after a sleep/wake; a duplicate worker spawned beside it and both wrote to the same card. We carried a local patch on hermes_cli/kanban_db_dispatch.py (same shape as this PR: _worker_alive tolerates same-epoch start-time drift; _pid_recycled stays exact for signalling) and it has been running on our gateway since 2026-09-22 with a finite re-test (one card, 300 s cap, no retries) completing via the worker's own kanban_complete with no manual recovery.

Gap worth closing in this PR. With liveness tolerant but the signal path exact, two callers still treat "strict mismatch on a live pid" as terminated, so the claim is released beside a worker that is actually alive:

  1. _terminate_reclaimed_worker: the early branch if _kb._pid_alive(pid) and _pid_recycled(pid, started_at): info["terminated"] = True; info["pid_recycled"] = True fires for a drifted-but-live worker (the test test_sleep_drifted_fingerprint_is_never_signalled encodes this as the intended outcome: claim released, no signal). Not signalling is correct, but releasing the claim re-creates the duplicate-spawn the issue is about. We changed it to info["terminated"] = not _worker_alive(pid, started_at) and, when still alive, signal_refused = True so the reclaim defers (existing _worker_survived_termination path holds the claim and emits reclaim_deferred).
  2. The SIGKILL step after SIGTERM in both _terminate_reclaimed_worker and enforce_max_runtime: _worker_alive (tolerant) says "still alive" → SIGKILL is sent without re-checking the exact fingerprint. We added if _pid_recycled(pid, started_at): signal_refused / continue immediately before _sigkill, so drift tolerance never turns into a signal permission.

Isolated tests for both (32 cases: first-probe drift vs post-SIGTERM drift, ±1 / ±200 native units, epoch change, dead pid, None/junk/legacy/UNVERIFIED) plus an SQLite probe of release_stale_claims holding the claim (reclaimed=0, reclaim_deferred event, run ended_at NULL) are in a local branch; happy to push them as a follow-up commit here or as a separate PR on top of this one — whichever the maintainers prefer. Not opening a competing PR.

Regression on v0.21.4 (11 kanban test files): 191 passed / 2 skipped / 1 pre-existing failure (test_kanban_db.py::test_infrastructure_spawn_refusal_never_charges_the_card, fails identically on unpatched HEAD).

Field report on NousResearch#118326 (PR review): the drift rescue made claim liveness
tolerant, but two paths still acted on the exact fingerprint's word alone.

- _terminate_reclaimed_worker's recycle branch marked a drifted-but-live
  worker terminated, so the reclaim released the claim beside it and the
  duplicate-spawn loop re-formed. It now reports survival (signal_refused)
  and the caller defers via the existing reclaim_deferred path.
- The SIGKILL escalation in _terminate_reclaimed_worker and
  enforce_max_runtime trusted the tolerant _worker_alive verdict without
  re-checking the exact fingerprint, so a recycled near-match could take
  the SIGKILL. Both now refuse against _pid_recycled.
@liuhao1024

Copy link
Copy Markdown
Author

Thanks for the detailed field confirmation and for pinpointing both gaps — you're right on both counts, and they're now closed in d8664c5.

  1. Claim release beside a drifted-live worker. _terminate_reclaimed_worker's recycle branch now checks _drifted_spawn_still_matches before deciding: a drifted-but-live worker is reported as surviving (signal_refused + start_time_drifted), so the existing _worker_survived_termination path holds the claim and emits reclaim_deferred — exactly the defer semantics you described. The former test_sleep_drifted_fingerprint_is_never_signalled encoded the release behavior, so it is now ..._never_signalled_nor_released and asserts running + reclaim_deferred.

  2. SIGKILL escalation. Both _terminate_reclaimed_worker and enforce_max_runtime now re-check the exact _pid_recycled immediately before _sigkill; a tolerant-alive verdict that fails the exact fingerprint takes no SIGKILL and is reported as surviving (the reclaim path) / left to settle on a later tick (the timeout path). Drift tolerance no longer becomes signal permission.

Operator reclaim_task keeps its human-override release (unchanged), and the terminal reaper now simply retries drifted-live workers on the next tick instead of clearing their evidence.

Local regression on this branch: the fingerprint suite is 9/9 and the adjacent kanban suites pass (89 passed / 6 skipped across test_kanban_db.py, the reaper, dispatch and claim-guard files). The single failure, test_kanban_db.py::test_infrastructure_spawn_refusal_never_charges_the_card, reproduces identically on the unmodified baseline — same pre-existing failure you saw on v0.21.4.

Since the code paths are now closed here, I kept the fix minimal rather than importing your 32-case branch — but if you spot a probe shape the two new tests (test_recycled_near_match_after_sigterm_never_takes_the_sigkill, test_max_runtime_never_sigkills_a_recycled_near_match) don't cover, please do push them as a follow-up commit; happy to take it.

@alt-glitch alt-glitch removed the comp/cron Cron scheduler and job management label Sep 24, 2026
@youyoutu405-source

Copy link
Copy Markdown

Reproduced the direction gap on this branch (d8664c5f36) with a claim-level probe against a live process (same harness shape as test_kanban_worker_pid_fingerprint.py: record fingerprint, expire the claim, run release_stale_claims). The cross-process create_time disagreement reported in #118326 (comment) is bidirectional — one observed live pair |179022797843 vs |179022797743, exactly 100 cs apart — and _drifted_spawn_still_matches rescues only one direction (return 0 < delta <= _SAME_WORKER_START_DRIFT_CEILING, kanban_db_dispatch.py:427). A probe reading 100 cs below the recorded start time of the same live process is still concluded recycled:

So the "covers both cases" note above holds for forward drift only; the negative half of the observed ±1 s probe disagreement reproduces the failure without any sleep/wake.

Two shapes that would cover it (for author evaluation):

  1. Keep the forward ceiling for accumulated-sleep drift, and widen the near-match rescue with the direction-symmetric comparator the other liveness surfaces already use: same epoch AND (0 < delta <= _SAME_WORKER_START_DRIFT_CEILING OR start_time_fingerprints_match(recorded_start, current_start) — gateway/status.py, default tolerance 200 = 2 s, as in cron/executions.py, gateway/delivery_ledger.py, tools/async_delegation.py).
  2. Or abs(delta) in the ceiling check — simpler, though a 12 h negative window is wider than the probe disagreement needs, and a genuinely recycled PID should read newer than the dead worker's record, never older.

The signalling path stays exact either way.

…agreement

The cross-process create_time disagreement in NousResearch#118326 is bidirectional
(~100 cs either way), but the drift rescue only covered a forward delta,
so a reaper reading a live worker's start time below the spawn-time
record still concluded recycled and released the claim beside it. Keep
the forward sleep-drift ceiling and route the negative half through the
same start_time_fingerprints_match comparator the other liveness
surfaces share (epoch still exact, ±2 s); the signalling path stays
exact.
@liuhao1024

Copy link
Copy Markdown
Author

Thanks for the claim-level probe and the precise line reference — you're right that the rescue covered only the forward half, and the negative reading is exactly the release-beside-a-live-worker loop this PR exists to close.

Pushed 80d1f1b with your first shape: the forward ceiling is kept for accumulated-sleep drift, and the negative half now routes through the same start_time_fingerprints_match comparator the other liveness surfaces share (gateway/status.py, default tolerance 200 = 2 s), with the instantiation epoch still compared exactly. I skipped abs(delta) against the ceiling for the reason you gave: a recycled PID re-starts after the recorded worker died, so it reads newer, never older — only the same live worker can read below its spawn-time record, and that disagreement is ~1 s, not 12 h. The signalling path (_pid_recycled) stays exact either way.

Regression coverage mirrors the matrix from your probe and mlahatte's local checks: a −100 cs same-epoch reading keeps the worker alive and the claim extended (this test fails on the pre-fix code, verified locally), +100 cs stays covered, a −300 cs gap beyond the comparator tolerance is still recycled, and an identical start under a different epoch remains recycled.

@youyoutu405-source

Copy link
Copy Markdown

Local verification of this PR complete (campaign-style, independent reruns on both sides; full evidence archived). Verdict from the verification side: good to merge.

  • Bug-level repro (three scenarios against the PR head 80d1f1b3 vs its base fe429480, same script both sides):
  • Tests: the targeted fingerprint suite is 11 passed on the head vs 4 passed on the base (+7 new cases, all green). The full -k fingerprint/reclaim/stale acceptance run: 5 failed / 1113 passed on the head vs 5 failed / 1106 passed on the base — identical failure names on both sides (environmental), zero newly-introduced failures.
  • No over-correction: true-dead processes are still reclaimed (_pid_alive short-circuits before the rescue path) and the _pid_recycled signalling path is unaffected (checked in code and in the B run's zero-signal assertion).

One field note for the rollout: we carry a local tolerance patch (±2 s comparator only) on our box. It covers neither the 27-minute drift nor the signal-authorization case — and its stale-heartbeat path actually recorded SIGTERM+SIGKILL against a live PID in scenario B — so we'll retire it in favor of this PR once it lands, after a behavior-equivalence check. No production changes were made from our side.

@liuhao1024

Copy link
Copy Markdown
Author

Thank you for the campaign-style verification — especially the base-vs-head symmetry of the three bug-level scenarios and the polarity sweep at the boundaries; that's the strongest evidence this PR has gotten.

The acceptance-run parity (identical environmental failure names on both sides, +7 green targeted cases) and the no-over-correction checks (true-dead still reclaimed via the _pid_alive short-circuit, signalling path untouched) match what the suite was built to pin down, so I'm glad it held up under independent reruns.

Your field note about retiring the local ±2 s patch is a useful datapoint too: scenario B's stale-heartbeat SIGTERM+SIGKILL against a live PID is exactly the release-beside-a-live-worker failure mode this PR closes, so the behavior-equivalence check should be mostly a formality once it lands.

@ottotheoperator

Copy link
Copy Markdown

Confirming this from a Mac mini running the dispatcher: after sleep/wake, live workers were read as recycled, their claims were released, and duplicates were spawned on the same worktree.

One edge worth a test: on macOS current_instantiation_epoch() is "" (no /proc). So the "epoch half must still match" condition in _drifted_spawn_still_matches is "" == "", which is always true there. The drift rescue works as intended on macOS, but it rests on the start-time delta alone, with no reboot witness. After a reboot, a different process that reuses the PID with a create time up to 12 hours after the recorded one would be rescued as "our worker". It would not be signalled, since _pid_recycled stays exact, but its claim would be extended beside it.

A possible fix is for current_instantiation_epoch() to use sysctl kern.boottime on Darwin as the boot witness. That closes this edge and gives macOS the same reboot protection Linux has. I left related notes on #117931: that PR's None path does not trigger on macOS, because the fingerprint "|<start>" is readable there.

…Darwin

Review follow-up (ottotheoperator): on macOS current_instantiation_epoch()
was "" (no /proc), so the 'epoch half must still match' guard in
_drifted_spawn_still_matches degraded to "" == "" — after a reboot, a
different process reusing the PID with a create time inside the 12 h
sleep-drift ceiling was rescued as our worker and its claim extended
beside it. Read the boot time via psutil.boot_time() (pinned core
dependency, reads sysctl kern.boottime in-process) so a reboot
invalidates the epoch on Darwin as it already does on Linux; an
unreadable boot time keeps the epoch "" (check disabled, never
fail-closed).
@liuhao1024

Copy link
Copy Markdown
Author

Thank you for the Mac mini confirmation and for the precise hole: you're right that on macOS the epoch half was "" == "" (always true), so the drift rescue rested on the start-time delta alone and a post-reboot PID re-user inside the 12 h ceiling would keep its claim extended. Pushed 389709db to address it.

Implementation notes:

  • current_instantiation_epoch() now reads the boot time on Darwin via psutil.boot_time() instead of staying "". psutil is a pinned core dependency here and reads the sysctl process table in-process, so it is the same kern.boottime witness you suggested without a fork+exec per call (gateway/status.py already prefers it for the same reason). A reboot now changes the epoch on macOS exactly as it does on Linux, and _drifted_spawn_still_matches rejects the rescue on the epoch mismatch. An unreadable or non-positive boot time still yields "" — the check is disabled, never fail-closed, as before.
  • Tests: epoch-level (Darwin witness is read, boot-stable, and falls back to "" on junk/failure) and fingerprint-level (a PID re-user whose start lands inside the drift ceiling is rescued on the same boot but foreign — claim released, zero signals — once the epoch witness changes).

Your related note on #117931 also checks out from the code: that PR's None path can't trigger on macOS because _process_fingerprint() there returns a readable "|<start>" rather than None — with this change it becomes "<boottime>|<start>", still readable, so the None-vs-readable distinction is unaffected.

@alt-glitch alt-glitch added comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery and removed comp/cli CLI entry point, hermes_cli/, setup wizard labels Sep 26, 2026
@ottotheoperator

Copy link
Copy Markdown

Thank you, that was quick! psutil.boot_time() is a better choice than shelling out to sysctl, and the fingerprint-level test (the same PID re-user, rescued on the same boot but foreign once the epoch changes) is exactly the case I was worried about.

Once this lands we'll drop our local stopgap from #117931 on the Mac mini, since it would otherwise switch off your drift rescue. Much appreciated.

@liuhao1024

Copy link
Copy Markdown
Author

Glad the shape works for you. On psutil.boot_time(): it keeps the epoch half on the same process-introspection library the fingerprint already uses (create_time), with no extra subprocess in the reclaim path — and since the epoch is only compared when both sides report one, the Linux /proc-backed value and the macOS boot-time value stay symmetric.

Sounds like the right sequencing on your side: once this lands, dropping the #117931 stopgap should be safe precisely because the drift rescue then owns the sleep/wake case end to end. Thanks again for the Mac mini confirmation and the edge report that surfaced the macOS ""-epoch hole.

@alt-glitch alt-glitch added P3 Low — cosmetic, nice to have and removed P2 Medium — degraded but workaround exists sweeper:risk-automation Sweeper risk: may affect CI, automerge, label sync, or maintainer automation labels Sep 26, 2026
@alt-glitch alt-glitch added P2 Medium — degraded but workaround exists comp/cli CLI entry point, hermes_cli/, setup wizard sweeper:risk-automation Sweeper risk: may affect CI, automerge, label sync, or maintainer automation and removed comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have labels Sep 26, 2026
@henrynguyeninfo1

Copy link
Copy Markdown

Adding a live field report in case it helps get this reviewed. We hit this on macOS 26 (arm64), on main @ d9d122a003.

  • 18:40:26: gateway.log shows kanban dispatcher [default]: spawned=1 reclaimed=0 crashed=1. Run 33 (a sly-demon research worker in the middle of a long media download) was classified crashed, and run 34 was spawned into the same worktree.
  • 19:05: run 33 was demonstrably still alive. It posted a card comment, and later found it couldn't make terminal transitions because the run-ownership check (correctly) rejected its stale run id.
  • 19:06:30: kanban dispatcher: reaped 1 zombie worker(s), pids=[72551].
  • Result: two workers wrote to one worktree for about 25 minutes.

We traced it to the exact-equality "<boot>|<start>" branch of _pid_recycled in hermes_cli/kanban_db_dispatch.py. We didn't confirm whether a sleep/wake happened in that window. Locally, comparing the start-time component with gateway.status.start_time_fingerprints_match, while keeping the boot epoch exact and unreadable-means-foreign, fixed it for us. That is the same direction as this PR, so +1 here rather than opening a duplicate.

@liuhao1024

Copy link
Copy Markdown
Author

Thanks for the live field report — and for tracing it to the exact branch.

One clarification that I think squares your trace with the PR: main @ d9d122a003 does not include this fix, so what you hit is the unfixed behavior of #118326 exactly as reported — and the crash classification path you traced is the one this PR widens. The dispatcher's crashed=1 booking comes from _reclaim_dead_workers, which consults _worker_alive; on main, _worker_alive's epoch+start comparison is the exact-equality "<boot>|<start>" check you identified, so any read disagreement recycles the live worker, releases its claim, and spawns a duplicate into the same worktree — your run 33/34 sequence.

On this PR's head that same call is tolerant before it is exact: an exact match still decides immediately, but a mismatched reading on the same boot is rescued either by the forward-drift ceiling (accumulated sleep/wake, up to 12 h) or — the half your report likely exercised — by the shared gateway.status.start_time_fingerprints_match comparator (±2 s read disagreement, either direction). A sleep/wake is not required to trigger it: #118326's second report observed the same live process read ±100 cs apart across processes, which is why the negative half routes through that comparator. The boot epoch half stays exact, and since 389709db Darwin has a real witness for it (psutil.boot_time()), so a post-reboot PID re-user inside the ceiling is still foreign.

So your local fix and this PR are the same shape — same comparator for the start half, exact boot epoch — and your +1 is much appreciated. Once this lands, the run-33 scenario should be held by the rescue instead of being booked crashed.

@facit

facit commented Oct 9, 2026

Copy link
Copy Markdown

Field report from macOS, supporting this PR: with #117931 applied, the dispatcher still reclaimed a live worker because of start-time drift.

Setup: Mac mini (Apple Silicon, macOS 26.6), Hermes main as of 2026‑10‑09 plus #117931, gateway-embedded dispatcher, max_in_progress: 1, one local llama.cpp model at a time. A kanban worker (Qwen3.5‑9B) had been running a single turn for 39 minutes. The gateway had been restarted once in between; the worker survived it, since it runs in its own session.

What happened: the reclaim pass declared the worker dead and spawned a second worker for the same card. A diagnostic log line we added just before _classify_dead_worker printed:

kanban: reclaiming task t_523f884a although pid 3843 still exists (stored fingerprint '|179158652125', now '|179158652225')

The epoch half is empty on macOS, and the start-time reading drifted by exactly +100 centiseconds (1.00 s). _pid_recycled compares the "<epoch>|<start>" string exactly, so a 1 s drift counts as a recycled PID. #117931 doesn't help here, because the fingerprint was readable; it was just 1 s off. The run lost 39 minutes of the agent's context.

Same symptom with another model: earlier the same day, on the same machine with a different model (Bonsai‑2 27B), the dispatcher recorded three false pid … not alive crashes on live workers over about 6 hours. One of them left two workers thrashing a single llama.cpp slot. Those happened before we applied #117931 or added the diagnostic, and the run rows no longer hold the fingerprints. So we can't say whether they were this drift or the unreadable-fingerprint case #117931 handles, but the pattern was the same: a long-running live worker reclaimed under memory pressure.

Local stopgap until this lands: same epoch plus start_time_fingerprints_match(rec_start, cur_start) in the "|" branch of _pid_recycled, with the exact compare as the junk fallback. That's essentially the near-match rule here, minus the forward/backward and reaper-signalling distinctions this PR adds. We'll switch to this PR once it's merged. It would be great to see it land alongside #117931; on macOS the two cover different failures (unreadable vs drifted fingerprint).

@alt-glitch alt-glitch added the comp/gateway Gateway runner, session dispatch, delivery label Oct 10, 2026
@liuhao1024

Copy link
Copy Markdown
Author

Thanks for the field report — the '|179158652125' vs '|179158652225' reading (epoch halves matching, start time drifting forward by exactly 100 cs) is precisely the case this PR's rescue layer covers:

  • _worker_alive still starts from the exact _pid_recycled compare, but a same-epoch live near-match falls through to _drifted_spawn_still_matches, where a forward delta within _SAME_WORKER_START_DRIFT_CEILING (12 h, the accumulated sleep/wake half) keeps the claim alive. +100 cs is comfortably inside it.
  • On the reclaim path the same rescue reports start_time_drifted + signal_refused, so the caller defers (reclaim_deferred) instead of releasing the claim or spawning the second worker — your 39-minute context loss is the exact failure mode that branch exists to prevent. A drifted reading is also never signalled.
  • Your stopgap is essentially this PR's rule; the deltas here are that the near-match lives in the rescue layer rather than inside _pid_recycled (so the signalling path keeps the exact compare and can never target a drifted stranger), and that the two directions get different bounds: forward sleep-drift up to the 12 h ceiling, either-direction read disagreement up to the shared 200 cs tolerance every other liveness surface uses.

One note on the empty epoch half: the latest commit on this branch (389709db87) gives the epoch half a kern.boottime witness on Darwin via psutil.boot_time(), so new fingerprints won't have an empty epoch — and legacy rows recorded with an empty epoch still rescue fine on the "" == "" match.

Agreed on #117931 being complementary rather than overlapping: unreadable fingerprint vs. readable-but-drifted. Together they cover both halves of the macOS failure mode you hit.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:risk-automation Sweeper risk: may affect CI, automerge, label sync, or maintainer automation type/bug Something isn't working

Projects

None yet

8 participants