Repository navigation
fix(nix): NixOS module lingers the service user so restart-safe cron workers get a user scope (#110628, salvage #110629) - #111489
Merged
Merged
Conversation
The NixOS module produces the one topology the restart-safe cron worker cannot run in: a system service (`User=hermes`, `isSystemUser = true`) whose uid has no systemd user manager. `run_one_job` hands every fire to `_launch_external_cron_worker` → `restart_safe_gateway_child_argv`, which requires a transient `systemd-run --user --scope` and fails closed when it cannot get one: Restart-safe cron worker dispatch failed: cannot create restart-safe systemd scope for gateway child: systemd-run --user --scope is unavailable (usually no reachable user D-Bus session at /run/user/995/bus). On a system-level service install, run `sudo loginctl enable-linger <gateway-user>` and restart the gateway. So on a stock `services.hermes-agent.enable = true` host, no cron job runs at all — every fire is recorded as a failed execution before an agent starts, and the occurrence is consumed, so a weekly job does not retry for another week. The error names the remedy, but nothing in the module applies it, and the terminal-worker path with the same cgroup-isolation goal degrades gracefully instead (`process_registry.py`: "worker shares the gateway cgroup"), so nothing else on the host looks wrong. Three parts, because linger alone leaves a startup race: - `linger = lib.mkDefault true` on the created user: the declarative `loginctl enable-linger`, which gives the uid a user manager and with it /run/user/<uid>/bus. mkDefault, so an operator can still refuse it. - after/wants linger-users.service, the unit that runs enable-linger. - a bounded preStart wait for the socket. logind starts user@<uid>.service asynchronously, and `run_gateway` resolves XDG_RUNTIME_DIR / DBUS_SESSION_BUS_ADDRESS exactly once at startup (802f0f9), so a bus that appears after ExecStart is one the process never sees for its lifetime. Non-fatal after 10s: a gateway without cron beats no gateway. All three are gated on `lingerEnabled`, read back off `config` rather than assumed: with `createUser = false` the operator owns the user, nothing here knows whether they lingered it, and the unit must not block on a bus that may never arrive. Test: nix/checks.nix `cron-worker-user-scope` relates the unit's `User=` to that user's `linger`, requires the ordering and the wait when it lingers, and requires neither when the module does not own the user. Red on the parent commit for the first three arms. Needs nixpkgs >= 25.05 for `users.manageLingering`.
…ent the cron/linger dependency The scope-dispatching cron worker runs inside the container there, so a host user manager would start for nothing (as PR #110641 by @liuhao1024 also gated it). nix-setup.md gains the note operators need when they declare the user themselves.
૮ >ﻌ< ა ci reviewran on 73d27d7 — chore: map contributor email for @haizaar
|
alt-glitch
added
type/bug
13 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On NixOS,
services.hermes-agent.enable = truenow gives the service uid a systemd user manager (linger), so gateway-dispatched cron jobs can create their restart-safesystemd-run --user --scopeinstead of running unscoped or, undercron.require_restart_safe_scope, failing every fire.Fixes #110628
nix/nixosModules.nix:users.users.<user>.linger = lib.mkDefault (!cfg.container.enable)undercreateUser; the nativehermes-agentunit is orderedafter/wantslinger-users.serviceand gets a bounded, non-fatalpreStartwait (≤10 s) for/run/user/<uid>/bus—loginctl enable-lingerreturns beforeuser@<uid>.serviceis up, andrun_gatewayadopts the bus env exactly once at startup. All gated onlingerEnabledread back offconfig, socreateUser = falsenever blocks on a bus that may never arrive.nix/checks.nix:cron-worker-user-scopeevaluates the module and asserts the four properties (linger set, ordering, preStart wait present, no wait for an unmanaged uid).website/docs/getting-started/nix-setup.mdgets the linger/cron note including thecreateUser = falseremedy.Root cause: a system unit (
User=hermes,isSystemUser = true) is neither a login session nor a lingering user, so logind never startsuser@995.serviceand there is no bus forsystemd-run --user. The module never applied the remedy the runtime error names.Premise vs current main: on
origin/mainthe default path no longer fails closed —restart_safe_gateway_child_argvdegrades to a direct external subprocess (GatewayChildDispatch("degraded"), landed in560b6d2e8/ v2026.9.14) and onlycron.require_restart_safe_scope: trueraises. The module half is still needed for restart-safe workers to actually exist on a stock NixOS install. The/bin/trueprobe payload half (haizaar's follow-up comment) landed in #105365.Live verification: no
nixon this host — static check only.git diffof the salvaged commit re-read line by line against the module (createUserblock, MODE A native unit); the runtime default was probed in Python:restart_safe_gateway_child_argv([...], require_restart_safe_scope=False)with the probe forced False →mode == "degraded";True→ the descriptiveRuntimeError. Thenix flake checkjob in CI (.github/workflows/nix.yml) runs the newcron-worker-user-scopecheck.Related: #110803 (same failure class, a stale True probe on a non-Nix host) is fixed separately.
Credit
Salvages #110629 by @haizaar (cherry-picked with authorship). #110641 by @liuhao1024 proposed the same
lingerline gated on!container.enable; that gate is carried here in the follow-up commit.Infographic