Repository navigation
Cron jobs run on systemd gateways without a user D-Bus session (salvage #102431) - #109311
Merged
kshitijk4poor merged 3 commits intoSep 12, 2026
Conversation
kshitijk4poor
added
comp/cron
A systemd-supervised gateway (INVOCATION_ID set) with no user D-Bus session (containers, minimal LXCs, supervisors without linger) fails EVERY scheduled job at dispatch: restart_safe_gateway_child_argv() raises, run_one_job() records a failure, and the only symptom is silently skipped executions (a missed nightly backup, dead watchdogs, no alert). Cron now degrades to a direct external subprocess with a once-per-process warning instead of raising, unless cron.require_restart_safe_scope=true (config.yaml, default false) restores fail-closed. Degraded jobs keep process separation and the full NousResearch#101940 ownership handoff - only cgroup isolation is lost, so a mid-job gateway restart kills the worker and the execution ledger records exactly that. The dispatch is a GatewayChildDispatch NamedTuple (in_process / scoped / degraded) so the degraded case can never collapse into the "not managed, stay in-process" sentinel - the failure mode that would recreate the restart-interruption edge NousResearch#101940 closed. Kanban stays fail-closed (require_restart_safe_scope=True at its call sites): its workers are long-lived agentic runs, so the degrade policy is limited to bounded cron jobs in this PR. Addresses the NousResearch#102431 review: the env-var flag became a config key per AGENTS.md (no new HERMES_* non-secret vars), Kanban keeps fail-closed instead of updating its tests to a degraded contract, main's enable-linger remedy message is preserved, and the degrade warning fires once per process.
Follow-up to the cherry-picked NousResearch#102431 fix, addressing the review findings: - The two real-helper scheduler tests ran the Linux-only helper unmarked and failed on macOS/Windows; the surviving one is now `linux_only`. - `_warn_scope_degraded_once` used a bare `os.getuid()` that tripped scripts/check-windows-footguns.py --all (lint lane red). The remedy text is now built once in the helper and passed into the warning, so the "scope binary vanished" case no longer warns about a missing D-Bus. - Tests trimmed to the invariant bar: degraded != in_process and warns once (helper level); default config still Popens externally and `require_restart_safe_scope: true` raises (scheduler level, real helper). Dropped the stubbed duplicate, the standalone config-raise test and the in_process half already covered by the existing passthrough test. - `GatewayChildDispatch.reason` had no reader outside a test; removed. - Both degrade branches share one local `_degrade(detail)`. - The per-fire config read uses `load_config_readonly()` (no deepcopy) with the same `except Exception` guard as the sibling `failure_nudge_threshold` read, so a config error no longer escapes the launcher. - Kanban's no-run-id guard fails closed for any non-`in_process` mode instead of matching one enum value. - Rationale restated in six places collapsed to the helper docstring; `cron.require_restart_safe_scope` documented in the cron user guide.
…notice The warning fires once per process but was prefixed with the first job's unit suffix, reading as a per-job notice for a host-level condition. Drop the suffix; the message now says what applies to every later dispatch.
kshitijk4poor
force-pushed
the
fix/cron-scope-degrade-salvage-102431
branch
from
September 12, 2026 17:38
b9691a8 to
78100e2
Compare
kshitijk4poor
enabled auto-merge (rebase)
September 12, 2026 17:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cron jobs on a systemd-managed gateway with no user D-Bus session (containers, minimal LXCs, a service user without linger) run again instead of failing at dispatch on every fire.
Salvage of #102431 by @kiwipaulrob — the fix commit is cherry-picked with authorship preserved; the second commit folds the review findings on top. Closes #102431's symptom; residual population after #105346 is hosts with genuinely no user manager.
Behaviour
tools/process_registry.py::restart_safe_gateway_child_argvreturns aGatewayChildDispatch(mode, argv)withmode ∈ {in_process, scoped, degraded}, so a degraded dispatch can never collapse into the "not managed, stay in-process" sentinel (the v1 blocker).cron/scheduler.py::_launch_external_cron_worker):scopedanddegradedboth launch an external worker with the existing fix(cron): preserve active turns across gateway restarts (salvage #101877) #101940 ownership handoff;degradedruns the direct command (process separation kept, cgroup isolation lost) and warns once per gateway process. New config keycron.require_restart_safe_scope(defaultfalse) restores the fail-closed raise with theenable-lingerremedy.hermes_cli/kanban_db_dispatch.py) passesrequire_restart_safe_scope=True— long-lived workers stay fail-closed; its no-run-id guard fails closed for any non-in_processmode.website/docs/user-guide/features/cron.md("Restart-safe workers under systemd").Follow-up commit (review findings)
linux_only.os.getuid()in the warning trippedcheck-windows-footguns.py --all(lint lane); the remedy text is built once and passed into the warning, which also fixes the "scope binary vanished" case warning about a missing D-Bus.GatewayChildDispatch.reasonremoved (no reader). Both degrade branches share one local_degrade(detail).load_config_readonly()with the same guard as the siblingfailure_nudge_thresholdread.Validation
run_tests.shontests/cron/test_restart_safe_worker.py,tests/hermes_cli/test_kanban_gateway_restart_handoff.py,tests/tools/test_process_registry.py)RuntimeErroron every dispatch, 0 Popen. Branch: default → 1 external Popen, nosystemd-run, one warning across 2 dispatches;require_restart_safe_scope: true→ raises, 0 Popen; unmanaged →in_processruff,check-windows-footguns.py --all,check_compat_pointers.py,git diff --checkOpen policy question (unchanged from the original PR)
Default is degrade (
cron.require_restart_safe_scope: false), which flips the fail-closed default #101940 introduced. Theneeds-decisionlabel on #102431 stands; flipping the default is a one-line change inhermes_cli/config_defaults.pyif maintainers prefer fail-closed. Kanban is unaffected either way.