Repository navigation
Conversation
…ng them doomed Kanban workers dispatched from a dispatcher running under a Type=oneshot systemd unit were spawned as direct children of the unit's cgroup and SIGKILLed at service exit with zero log output (NousResearch#113612). The helper's early-return treated every non-gateway topology as safe in-process, silently ignoring require_restart_safe_scope=True. Now a non-gateway dispatcher with INVOCATION_ID set falls through to the same systemd-run --scope wrap (and fails loud when no scope can be created); terminal dispatchers keep the in-process path. authored with AI assistance (Muse, Meta's Muse Spark) under the contributor's direction; reviewed and approved by the contributor before submission.
alt-glitch
added
type/bug
|
Independent verification on the PR head (b669d32): tests/cron/test_restart_safe_worker.py passes locally, 27/27 on Linux (canonical runner). The condition split is exact: no INVOCATION_ID keeps the in-process path, systemd-managed non-gateway without the strict requirement keeps it too, and require_restart_safe_scope under a oneshot unit now scope-wraps instead of spawning children into a dying cgroup. Fails loud when no scope can be created. No findings. |
SummaryDispatchers running under a non-gateway systemd unit ( What changed
Strengths
Findings
VerdictLooks good to merge Reviewed using Hermes-Agent |
teknium1
added a commit
that referenced
this pull request
Sep 18, 2026
…spatch scope-wraps workers Two defects at the same spawn boundary (hermes_cli/kanban_db_dispatch.py:: _restart_safe_worker_argv -> tools/process_registry.py::restart_safe_gateway_child_argv). #114720 — a managed-gateway dispatcher whose user D-Bus session had gone away raised the restart-safe-scope RuntimeError at spawn; the ordinary spawn except-clause fed it to _record_task_failure, which advanced consecutive_failures and at failure_limit parked the card as a bare `blocked` (block_kind NULL) with a `gave_up` run. Nothing about the card had run. - restart_safe_gateway_child_argv raises RestartSafeScopeUnavailable (RuntimeError subclass) so the dispatcher can tell a host refusal from a card failure. - _record_task_failure(infrastructure=True): run + event carry `infrastructure: true`, consecutive_failures is left alone, the breaker never trips; the card stays `ready` with the linger remedy in last_failure_error and a warning in the dispatcher log. - check_respawn_guard returns `infrastructure_cooldown` while the latest run is such a refusal inside the rate-limit cooldown window, so a dead bus is retried spaced instead of every tick. - require_restart_safe_scope=True keeps its hard-fail semantics under the managed gateway. #113612 — `hermes kanban dispatch` from an operator's Type=oneshot systemd unit spawned unmanaged workers: the gate `_is_supervised_gateway_process()` is False for a CLI dispatcher, so the helper returned the bare argv and the workers died at the unit's cgroup teardown with empty logs, while the `running` rows waited for reclaim. - New outlives_parent=True (kanban only; cron blocks its caller and keeps the old gate): under any systemd unit (INVOCATION_ID) a fire-and-forget worker is scope-wrapped when the user bus is reachable. Without a bus it degrades with a loud once-per-process warning naming the consequence and both remedies (enable-linger, KillMode=process) rather than refusing — the unit's KillMode/lifetime is unknowable and a long-lived sequencer without linger must keep working. Docs: kanban.md "Workers and systemd cgroups" + guard reasons; cron.md note. Based on analysis from PR #113624. Co-authored-by: Kevin Rajan <7121943+kvnloo@users.noreply.github.com>
Collaborator
|
Thanks @kvnloo. Your change was salvaged into #114886 with your authorship preserved (co-authored); #114886 — fix(kanban): host spawn refusals no longer park cards as blocked; oneshot-unit dispatch workers survive unit exit (#114720, #113612, salvage #113624) — is now merged on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Kanban workers dispatched from a dispatcher running under a
Type=oneshotsystemd unit were spawned as direct children of the unit's cgroup and SIGKILLed at service exit, with zero log output (Python's stdout is block-buffered to a file, so the SIGKILL before the ~4–8 KiB flush leaves the log empty). The board then loops spawn → kill → reclaim forever.restart_safe_gateway_child_argvearly-returnedin_processfor every non-gateway topology, silently ignoringrequire_restart_safe_scope=True. Now a non-gateway dispatcher withINVOCATION_IDset falls through to the samesystemd-run --user --scopewrap the gateway path uses (and fails loud via the existing_degraderaise when no user scope can be created). A plain terminal dispatcher (noINVOCATION_ID) keeps the in-process path, andrequire_restart_safe_scope=Falsecallers (cron's default) are unchanged.Related Issue
Fixes #113612
Type of Change
Changes Made
tools/process_registry.py:restart_safe_gateway_child_argvnow routes non-gateway systemd-unit dispatchers through the restart-safe scope wrap whenrequire_restart_safe_scope=True; terminal dispatchers keepin_process.tests/cron/test_restart_safe_worker.py: 4 new invariant tests (oneshot scope wrap, loud failure without a bus, terminal dispatcher unchanged, require=False unchanged).How to Test
INVOCATION_IDset and_is_supervised_gateway_process()False:restart_safe_gateway_child_argv(..., require_restart_safe_scope=True)returnsin_process(silent passthrough — the bug).systemd-run --scope-wrappedscopeddispatch (user bus reachable) or raisesRuntimeError: cannot create restart-safe systemd scope…with the linger remedy.scripts/run_tests.sh tests/cron/test_restart_safe_worker.py— 26 passed, 1 skipped (pre-existing live-host guard); the 2 new behavior tests fail on base (red-on-base proven).test_kanban_cli_dispatch_passthrough,test_kanban_dispatch_claim_allowlist,test_kanban_dispatch_lock,test_kanban_dispatch_tick_hook).Checklist
Code
fix(scope):,feat(scope):, etc.)scripts/run_tests.shand all tests passDocumentation & Housekeeping
docs/, docstrings) — or N/A (docstring updated in the helper)_IS_LINUX)authored with AI assistance (Muse, Meta's Muse Spark) under the contributor's direction; reviewed and approved by the contributor before submission.