fix(cron): desktop ticker defers to a live gateway instead of racing it

[Sora-bluesky]

What

When hermes gateway run and hermes serve (Hermes Desktop) share a HERMES_HOME, the desktop cron ticker's standalone delivery path has no live platform adapters. So when serve wins the tick race, Feishu interactive cards are silently downgraded to plain text, with no error and no warning (#66629).

This gates the desktop ticker on a read-only, record-based owner probe of the gateway runtime lock, through the can_dispatch seam that cron.scheduler.tick() already re-checks under .tick.lock. Desktop dispatches only when no live gateway owns this HERMES_HOME, and resumes on its own once the gateway stops.

Why read-only (observer must not kill owner)

The probe never opens the OS lock. It classifies ownership from the JSON lock record and a live-PID check, so a frequent desktop-side probe cannot serialize against acquire_gateway_runtime_lock() and cannot make gateway startup lose its own lock. This is a property of the construction: the probe never touches the lock, so there is no timing window in which a collision could occur. An earlier attempt in this branch used an OS-lock probe with an acquire retry to paper over the collision, but a bounded retry cannot survive a probe descheduled by Defender, a debugger, or OS pressure. This design removes the mechanism instead.

How

gateway/status.py: probe_gateway_runtime_lock() -> "held" | "free" | "unknown" establishes ownership only from positive evidence, in order:

1. record present and parseable, else unknown (crashed mid-write, or the sub-millisecond window between _try_acquire_file_lock and the record write)
2. _pid_exists(pid), else free (crash: OS released the lock, record stale)
3. start_time present on BOTH the record and the live process, else unknown
4. recorded start_time == live start_time, else free (PID reuse)
5. record["hermes_home"] is a usable string and matches the probed lock's parent, compared with the host's path + case semantics (_same_hermes_home), else unknown/free
6. the live command line belongs to this profile by exact argv token, else free

hermes_cli/web_server.py: _no_live_gateway() dispatches on free/unknown and defers on held. An unknown result logs a warning so a hidden probe failure stays visible. It is wired via an inspect.signature guard so providers without can_dispatch still work.

Exact-profile matching (the load-bearing correctness fix)

Step 6 previously used _command_line_belongs_to_profile(), which matched the profile by substring (--profile work ⊂ --profile worker). Combined with a stale record whose honest hermes_home still named work after PID reuse plus a start-time collision, the probe could report work running when only a worker gateway was live, which stalled work's cron. This rewrites the matcher to compare an exact argv token (--profile/-p, space and = forms, or a normalized HERMES_HOME= token), with the explicit --profile/-p selector authoritative over a conflicting HERMES_HOME=. This is the root fix and also corrects every other caller of the matcher (get_runtime_status_running_pid); it incidentally fixes a cross-platform bug where a forward-slash HERMES_HOME= never matched on a backslash host.

Scope and known residuals

• The fix is desktop-only: gateway startup is not serialized on any lock. Serializing it would let a desktop probe stall or fail the gateway, which is worse than the original silent degradation.
• advance_next_run under .tick.lock guarantees at-most-once, so no double delivery is possible on an unknown fail-open.
• One residual predates this change. For the default/root profile a bare gateway carries no profile flag and its HERMES_HOME comes from the environment (invisible on argv), so a PID-reuse plus start-time collision can only be disambiguated by the self-asserted recorded_home. That case is collision-only and already present in get_running_pid.
• A record written before hermes_home was persisted reads as unknown until that gateway restarts (fail-open: dispatch + warning).
• is_gateway_runtime_lock_active() is intentionally left as a separate follow-up: it opens with "a+" and unlinks on PermissionError, so it is unsuitable for a hot-path probe, but it is authoritative for lifecycle commands and is not on the desktop cron path. Redesigning that observer/owner split is a distinct change.

Test

NO_COLOR=1 uv run --extra dev python -m pytest tests/cron/test_desktop_cron_gateway_gate.py tests/gateway/test_status.py -q

• Headline RED→GREEN through the public _start_desktop_cron_ticker: the ungated ticker fires while a live gateway is recorded; the gated one stays quiet and resumes when the record clears.
• Unit tests pin every branch of the 3-state probe: never opens the OS lock, dead PID, PID reuse, null start_time (either side), TOCTOU permission flip, RecursionError on a deeply-nested corrupt record, cross-profile hermes_home, platform-aware home comparison, and the exact-token matcher (work must not match --profile worker; --profile authoritative over HERMES_HOME=; the sibling-profile + start-time-collision repro returns free).
• Every regression test verified RED against the prior implementation via git stash, GREEN after. Full local run: green on Windows (pre-existing unix-permission / POSIX-path baseline failures excluded).

[alt-glitch]

This was generated by AI during triage.
Related to open #52259, #44049, and #46207: this uses the built-in scheduler provider's can_dispatch hook with a cached PID probe, while the others use direct or scheduler-level ownership checks. Maintainer choice is needed.

[Sora-bluesky]

Good flag — comparing the four approaches for the maintainers, factually:

• fix(cron): defer desktop ticker to gateway scheduler owner #44049 (Jun 11): introduces a scheduler-level ownership concept (cron/scheduler.py + provider changes + chronos plugin changes; 6 files). Deterministic owner, largest surface.
• fix(desktop): skip dashboard cron tick while the profile gateway is alive #46207 (Jun 14): startup-time guard in web_server.py — skips ticks while the profile gateway is alive, with its own probe helper (2 files).
• fix(desktop): don't start a cron scheduler when a live gateway owns cron (#52202) #52259 (Jun 25): doesn't start the scheduler at all when a live gateway owns cron (cron/scheduler.py + web_server.py; 5 files).
• fix(cron): desktop ticker defers to a live gateway instead of racing it #67418 (this PR): per-tick gate through the built-in provider's existing can_dispatch hook + the existing get_running_pid_cached() probe — no scheduler/provider changes, 1 source file, and the desktop ticker auto-resumes when the gateway stops (the startup-time approaches leave cron dead for desktop-only users if the gateway exits later in the session).

All four fix the same race; the choice is genuinely the maintainers' (ownership model vs. liveness gate). If one of the earlier PRs is preferred, happy to close this one — I'd only suggest carrying over the per-tick auto-resume property, whichever lands. Also noting for triage: #66629 is effectively the same underlying race as #52202 / #43965.

[teknium1]

Thanks for tracing the desktop/gateway delivery split. The underlying race is present on current main: the desktop starts a provider without adapters at hermes_cli/web_server.py:141-159, while the gateway starts one with live adapters at gateway/run.py:21659-21682.

Problems

• The new gate at hermes_cli/web_server.py:191 is a liveness observation, not an ownership claim. The built-in provider checks it before tick() (cron/scheduler_provider.py:195), and tick() checks it again only after taking .tick.lock (cron/scheduler.py:3911). A gateway can become live after that final check and before desktop execution claims the due work, so the degraded desktop delivery path remains possible during the handoff.

Suggested changes

• Establish a shared atomic scheduler-owner/claim protocol for the desktop and gateway paths, and add a deterministic gateway-start-during-dispatch test.

Automated hermes-sweeper review.

[teknium1]

This supplies only a non-atomic liveness observation. The built-in provider checks it before tick(), then tick() checks it again after .tick.lock is acquired; a gateway can become live after that final read and before desktop execution claims a due job. Please use a shared ownership/claim protocol and cover that transition deterministically.

[Sora-bluesky]

You're right that the gate is a liveness observation, not an ownership claim — a gateway starting inside the check→dispatch window can still take delivery quality for that one handoff tick. That residual TOCTOU is inherent to any liveness-gate approach; closing it fully requires the atomic owner/claim protocol, which is exactly #44049's design, and I'd rather not duplicate that surface in a fourth PR. As laid out in the comparison above: this PR is the minimal harm-reduction option (shrinks the degraded-delivery window from "every tick" to "at most one handoff tick per gateway start") with per-tick auto-resume; #44049 is the complete-fix option. If the maintainers pick the ownership model, happy to close this in its favor — ideally carrying the auto-resume property over.

[Sora-bluesky]

Thanks for the detailed trace. The gate is evaluated inside cron_tick under .tick.lock, right after the lock and before get_due_jobs() (cron/scheduler.py:3930), so the double-dispatch race is already closed: .tick.lock serializes the two tickers, recurring occurrences are claimed by advance_next_run() under the lock, and one-shots go through claim_dispatch()'s CAS.

The residual you're pointing at is a gateway going live during get_due_jobs() itself. I tried re-checking right before the claim, but get_due_jobs() already mutates state (it persists run_claim for due one-shots and fast-forwards stale recurring next_run_at for a catch-up), so deferring after it would strand that work rather than hand it cleanly to the gateway. Closing the window safely means making scan/gate/claim atomic (or a shared owner-lease with fencing), which is a real change; the residual it removes is a single tick delivered as plain text during a gateway-startup race, with no lost or double delivery. Happy to build the atomic-owner version if you want it for this PR, or leave the current gate as the pragmatic close.

[Sora-bluesky]

Redesigned around your ownership-vs-liveness point.

The gate now probes gateway.lock directly, read-only, instead of the cached PID: held -> defer, free -> dispatch, unknown -> a read-only PID identity fallback that requires a matching start_time, so a recycled PID belonging to a different HERMES_HOME's gateway can't false-positive. It opens the lock "r+" (no O_CREAT), so unlike the shared is_gateway_runtime_lock_active it never creates or unlinks the file.

Dropped the startup drain barrier from the earlier attempt: .tick.lock + advance_next_run already dedupe a valid occurrence under the lock, so it was redundant, and its "provable ordering" claim was false on a drain timeout anyway.

Decision-table and probe-classifier tests included; the two residuals (a hand-off micro-race, and the blind fail-open biased to cron liveness) are written up in the commit body.

[GottZ]

This was generated by AI during triage.

Summary

Nine PRs are associated with this issue complex: six change Desktop-versus-Gateway cron arbitration, while #67135, #67198, and #67204 add scheduler-provider tests without changing ownership or delivery behavior. The fix diffs range from obsolete direct-ticker guards and lifecycle suppression to current-provider deferral; the recorded cross-issue best-fix verdict selects #44050 for #43965, #52202, and #66629.

Related pull requests

• fix(cron): defer desktop ticker to gateway scheduler owner #44049 partial — (+211/-13) — n/a: Adds scheduler-level runtime-lock deferral, threads it through the provider contract, and opts the Desktop ticker in; this addresses the reported provenance and adapterless-delivery cause but substantially duplicates fix(cron): yield desktop ticker to running gateway #44050 and also changes the Chronos interface.
• fix(cron): yield desktop ticker to running gateway #44050 best fix — (+219/-14) — n/a: The recorded best fix for [Bug]: Desktop dashboard cron ticker can execute jobs instead of launchd gateway, breaking macOS TCC/FDA provenance #43965, Desktop dashboard cron scheduler races the gateway; when it wins a tick it has no live adapter and delivery hangs until script_timeout (600s) #52202, and Desktop serve cron ticker silently degrades Feishu interactive cards to plain text #66629 carries narrow owner deferral through the active built-in provider path while preserving the Gateway default and external-provider contracts. Its updated diff explicitly addresses the keep_open review by adding provider-boundary, owner/no-owner, Gateway-default, and external-provider coverage.
• fix(desktop): skip dashboard cron tick while the profile gateway is alive #46207 partial — (+83/-2) — n/a: Checks Gateway liveness before each Desktop tick and fails open, but patches the removed direct cron_tick() loop instead of the active provider path. Despite the keep_open review on fix(desktop): skip dashboard cron tick while the profile gateway is alive #46207, its requested provider-aware port is already implemented in fix(cron): yield desktop ticker to running gateway #44050, so this variant is superseded.
• fix(desktop): yield cron ownership to running gateway #47358 partial — (+73/-3) — n/a: Uses get_running_pid() to stand down the old Desktop ticker and directly targets macOS TCC requester provenance. Despite the automated keep_open verdict on fix(desktop): yield cron ownership to running gateway #47358, its production hunk and tests bypass the current resolve_cron_scheduler().start() path introduced by abbd864, while fix(cron): yield desktop ticker to running gateway #44050 covers that path.
• fix(desktop): don't start a cron scheduler when a live gateway owns cron (#52202) #52259 duplicate — (+456/-16) — n/a: Suppresses Desktop scheduling at startup, stops an active built-in ticker when a Gateway later appears, and separately bounds standalone sends to 30 seconds. Its keep_open verdict supports salvage, but the ownership path does not automatically resume after a later Gateway exit; the distinct timeout work should be split from the ownership duplicate.
• test(cron): add coverage for pre-set stop event #67135 related — (+26/-0) — n/a: Adds only pre-set stop_event lifecycle coverage and changes no scheduler-ownership, live-adapter, or Feishu-delivery behavior. Its automated keep_open verdict supports retaining it independently for hermes_cli/cron.py background ticker lacks dedicated unit tests #67102.
• test(cron): cover ticker interval, adapter/loop forwarding, and recovery ordering #67198 related — (+155/-0) — n/a: Adds interval fidelity, adapter/loop forwarding, recovery ordering, and compatibility-shim tests without introducing a Desktop-versus-Gateway ownership guard. Its automated keep_open verdict supports independent scheduler-provider coverage.
• test(cron): cover ticker inter-tick interval pacing (#67102) #67204 related — (+46/-0) — n/a: Adds a real-time inter-tick pacing test and makes no production change to scheduler arbitration or delivery. Its keep_open review supports the test but requests correction of the comment claiming a jitter margin that the assertion does not implement.
• fix(cron): desktop ticker defers to a live gateway instead of racing it #67418 partial — (+849/-16) — n/a: Adds a provider-aware per-tick gate backed by a read-only Gateway lock-record classifier, automatic Desktop resumption, and extensive PID/profile/status hardening. Despite the keep_open review and subsequent redesign, the current author discussion still documents a handoff micro-race before work is claimed; the ownership result overlaps fix(cron): yield desktop ticker to running gateway #44050, while independently useful status hardening can be split.

Duplicates

#44049 and #44050 implement substantially the same runtime-lock-backed scheduler deferral. #46207 and #47358 are obsolete direct-ticker variants; the ownership portions of #52259 and #67418 address the same race through broader lifecycle, timeout, or status-probe changes. #67135, #67198, and #67204 are independent test PRs, not ownership-fix duplicates.

Suggested consolidation

Author action: rebase #44050 onto current main, retaining its built-in-only provider integration and regression coverage; this preserves the recorded best-fix verdict without making a merge recommendation. Then close #44049, #46207, and #47358 as duplicates of #44050; for #52259, split out the standalone-send timeout before closing its ownership portion as duplicate, and for #67418, split any independently justified status/profile hardening before closing its ownership gate as duplicate. These closures explicitly differ from the keep_open reviews on #46207, #47358, #52259, and #67418 because the first two patch the obsolete direct-ticker path, while the latter two combine the same ownership fix with broader salvageable work. Keep #67135 and #67198 in their separate scheduler-test triage, and request the documented comment correction on #67204.

Complex graph

flowchart TD
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I43965(["issue #43965 (open)"])
    I52202(["issue #52202 (open)"])
    I66629(["issue #66629 (open)"])
    subgraph Dup44049 ["PRs duplicating each other"]
        P44049["PR #44049 (open)"]
        P44050["PR #44050 (open)"]
        P46207["PR #46207 (open)"]
        P47358["PR #47358 (open)"]
        P52259["PR #52259 (open)"]
        P67418["PR #67418 (open)"]
    end
    P67418 -.->|partial| I43965
    P67418 -.->|partial| I52202
    P67418 -.->|partial| I66629
    class I43965 open
    class I52202 open
    class I66629 open
    class P44049 open
    class P44050 open
    class P46207 open
    class P47358 open
    class P52259 open
    class P67418 open
    class P44050 best
    class P44050 best
    class P44050 best
    class P67418 target
    click I43965 "https://github.com/NousResearch/hermes-agent/issues/43965"
    click I52202 "https://github.com/NousResearch/hermes-agent/issues/52202"
    click I66629 "https://github.com/NousResearch/hermes-agent/issues/66629"
    click P44049 "https://github.com/NousResearch/hermes-agent/pull/44049"
    click P44050 "https://github.com/NousResearch/hermes-agent/pull/44050"
    click P46207 "https://github.com/NousResearch/hermes-agent/pull/46207"
    click P47358 "https://github.com/NousResearch/hermes-agent/pull/47358"
    click P52259 "https://github.com/NousResearch/hermes-agent/pull/52259"
    click P67418 "https://github.com/NousResearch/hermes-agent/pull/67418"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 9 pull requests and 3 issues in this complex. Each diff was read against this issue; Assessment working set: 114 kB of PR diffs, 42 kB of issue/PR text, 15 kB of discussion (22 comments), 25 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

[boblin]

Additional repro on Linux + SSH mode: the race causes jobs to fail, not just duplicate

We hit this on Linux with Hermes Desktop in SSH mode, where the symptom is worse than duplicate delivery. Worth capturing here since the existing reports (#57191, #66629) focus on Windows duplicates and Feishu cards.

Setup: hermes gateway run as a systemd service on the host; Hermes Desktop on a workstation connects to that same host over SSH, which starts hermes serve --isolated there. Both tick.

Symptom: cron jobs fail non-deterministically with FileNotFoundError: [Errno 2] No such file or directory: 'yt-dlp', depending on which ticker wins — i.e. on whether Desktop happens to be open.

Why it fails rather than duplicates: the two tickers do not share an environment. A probe run from inside an actual cron job returned:

{"ppid": 2266888,
 "parent_cmdline": "... hermes serve --isolated ... --ssh-session-token-file ...",
 "path": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin",
 "hermes_home": null,
 "which": {"yt-dlp": null, "node": null, "simplex-chat": null}}

The gateway's own environment has ~/.local/bin on PATH and HERMES_HOME set (both come from its systemd unit). The SSH-spawned backend inherits a non-interactive SSH environment instead: no ~/.local/bin, and HERMES_HOME unset. Any job calling a user-installed tool by bare name breaks, and anything relying on HERMES_HOME gets a different home.

Diagnostic trap worth noting: /proc/<gateway-pid>/environ shows the correct PATH, so the environment looks fine — the job simply wasn't run by the gateway. Check the job's actual ppid first.

On the code comment at hermes_cli/web_server.py:236 (v0.20.0, 3aeff239b):

# Desktop-spawned backends (HERMES_DESKTOP=1) fire cron jobs themselves,
# since the app has no gateway running the scheduler.
if os.getenv("HERMES_DESKTOP") == "1":

That assumption does not hold in SSH mode — the Desktop connects to a host where the gateway is running, and the ticker starts unconditionally anyway. This PR's "defer to a live gateway" approach fixes our case too.

Workaround in the meantime: put tools that jobs invoke by bare name on the system PATH (/usr/local/bin) so both environments resolve them, and never rely on HERMES_HOME being set — fall back to ~/.hermes.

[Sora-bluesky]

@boblin Thanks for this — the SSH-mode repro is the first report showing the race producing failures rather than duplicates, and the ppid diagnostic trap you describe (gateway's /proc/<pid>/environ looking healthy while the job ran elsewhere) is worth having on record for anyone else chasing a phantom FileNotFoundError.

One clarification so the thread stays accurate for the maintainers: your case is covered by the per-tick deferral approach generally, not by this PR uniquely. I checked #44050 (the fix the 2026-08-03 triage above records as "best fix") against your scenario:

• Its gate is evaluated per tick, inside tick() itself, not once at startup, so a gateway that starts or stops after the SSH backend is already running gets picked up on the next tick.
• The SSH-spawned backend does carry HERMES_DESKTOP=1: the Desktop app builds the remote command as env HERMES_DESKTOP=1 hermes serve --isolated ... (apps/desktop/electron/remote-lifecycle.ts:454), so fix(cron): yield desktop ticker to running gateway #44050's opt-in condition is satisfied on your setup too. (Caveats: this holds for the default in-process cron provider, and remote hosts are Linux/macOS only per SUPPORTED_REMOTE_OS.)

So your report confirms the deferral approach holds up under SSH mode, but it doesn't by itself pick between the open candidates. I should also correct something I wrote on this thread back in July: I described some of the other PRs as startup-time guards. That was wrong. #46207 rechecks gateway liveness on every ticker iteration, and #52259's head adds a per-tick can_dispatch recheck, so they are not startup-only either. Sorry for the mischaracterisation.

Two factual differences that do separate them, for whoever decides:

1. How liveness is probed. fix(cron): yield desktop ticker to running gateway #44050's check goes through is_gateway_runtime_lock_active(), which opens the gateway lock file, attempts a real OS lock (flock/msvcrt), and unlinks the file on PermissionError (gateway/status.py:925-943 on its branch). This PR's probe_gateway_runtime_lock() never creates, acquires, or unlinks the lock (pinned by test_probe_never_touches_the_os_lock). A liveness check that can take or delete the lock it is checking seemed worth avoiding, which is why this PR added the read-only classifier.
2. Current mergeability. As of today fix(cron): yield desktop ticker to running gateway #44050 conflicts with main (InProcessCronScheduler.start: profile_homes vs defer_to_gateway_owner), while this PR is clean. That's a rebase away from changing, so I mention it only as present state, not as an argument.

The triage's suggestion — land one ownership fix, split anything independently useful — still seems right to me, and which gate lands is the maintainers' call. If #44050 is preferred, the read-only probe here is the piece I'd suggest carrying over.