fix(desktop): don't start a cron scheduler when a live gateway owns cron (#52202)

[JoaoMarcos44]

Fixes #52202

Summary

When a launchd/systemd hermes gateway run service and the desktop app share one HERMES_HOME, two in-process cron schedulers race. The desktop dashboard backend (HERMES_DESKTOP=1) starts its own scheduler in hermes_cli/web_server.py::_start_desktop_cron_ticker, alongside the gateway's. They coordinate via cron/.tick.lock (winner-takes-tick), which prevents double-fire but not capability: the desktop backend is spawned by the GUI and lacks the gateway's live adapters and inference env. When it wins a tick for a job delivering to a live platform (e.g. Telegram), the job runs without an adapter and stalls until a timeout, delivering the generic provider timeout. Fallback chain was exhausted or unavailable. It is intermittent — a coin flip on which process grabs the lock each tick.

Failure flow

                 cron/.tick.lock  (winner-takes-tick — mutual exclusion only,
                        |          NOT a guarantee of execution capability)
        +---------------+----------------+
        v                                v
  [gateway run]                     [desktop backend]  HERMES_DESKTOP=1
  adapters = LIVE, loop = LIVE      adapters = None, loop = None
  inference env OK                  inference env absent (spawned by GUI)
        |                                |
        v                                v
   run_one_job -> run_job           run_one_job -> run_job
        |                                |
   delivers via LIVE adapter -> OK  no live adapter -> cold/standalone path
                                         |  stalls until the timeout fires
                                         v
                                 failure summarized + delivered:
                                 "provider timeout. Fallback chain was
                                  exhausted or unavailable."

Root cause

1. web_server.py::_lifespan started the ticker whenever HERMES_DESKTOP=1, without checking for a live gateway — even though that check already exists and is used elsewhere (hermes_cli/cron.py::find_gateway_pids, gateway/status.py::is_gateway_running).
2. cron/.tick.lock (cron/scheduler.py::tick) coordinates tick exclusion only, not capability. The winner may lack adapters/env — violating the invariant exactly one scheduler executes jobs per HERMES_HOME.
3. Latent secondary bug: in the standalone delivery path (cron/scheduler.py::_deliver_result), the primary asyncio.run(coro) send was unbounded — a wedged platform HTTP call could hang the cron worker indefinitely.

Note on the reported "600s"

The issue reports Script timed out after 600s for a trivial no_agent job. For transparency: in the code, _DEFAULT_SCRIPT_TIMEOUT = 120 (not 600), and the script timeout applies to the script, not delivery. The number 600 matches HERMES_CRON_TIMEOUT (the agent inactivity limit, default 600), not the script timeout — so the report conflates two distinct timeouts. This does not weaken the fix: whichever timeout fires, it only fires because the desktop is executing a job it shouldn't. Removing that execution covers every branch (script / delivery / agent inactivity).

Fix

1. Detect a live gateway before starting the ticker (primary)

hermes_cli/web_server.py, inside _lifespan. Under HERMES_DESKTOP=1, call is_gateway_running() first:

• Live gateway detected → the desktop stays a passive observer and never starts its own scheduler. The gateway becomes the sole cron executor for that HERMES_HOME.
• No gateway (desktop-only install) → the ticker starts as before.
• Detection error → assume no gateway and start the ticker (safe failure direction: a desktop-only install never loses cron).

if os.getenv("HERMES_DESKTOP") == "1":
    gateway_alive = False
    try:
        from gateway.status import is_gateway_running
        gateway_alive = is_gateway_running()  # PID file + runtime lock, with stale cleanup
    except Exception:
        gateway_alive = False
    if gateway_alive:
        _log.info(
            "Desktop backend: live gateway detected — not starting an own cron "
            "scheduler (the gateway is the sole cron executor for this HERMES_HOME)."
        )
    else:
        cron_stop = threading.Event()
        cron_thread = threading.Thread(
            target=_start_desktop_cron_ticker, args=(cron_stop,),
            daemon=True, name="desktop-cron-ticker",
        )
        cron_thread.start()

This is the first option suggested in the issue ("detect a live gateway.pid / gateway lock and skip").

2. Bound the standalone delivery send (defense in depth)

cron/scheduler.py, in _deliver_result. The primary send is now bounded by asyncio.wait_for(..., timeout=30), mirroring the budget the threadpool fallback already had.

def _bounded_send():
    return asyncio.wait_for(
        _send_to_platform(platform, pconfig, chat_id, cleaned_delivery_content,
                          thread_id=thread_id, media_files=media_files),
        timeout=30,
    )
coro = _bounded_send()
try:
    result = asyncio.run(coro)
except RuntimeError:
    coro.close()  # avoid "coroutine was never awaited"
    with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool:
        future = pool.submit(asyncio.run, _bounded_send())
        result = future.result(timeout=35)

Behavior matrix

Scenario	Before	After
Gateway + desktop, gateway wins tick	OK	OK (unchanged)
Gateway + desktop, desktop wins tick	stalls → timeout → failure delivered	desktop never executes; gateway is sole executor ✅
Desktop only (no gateway)	runs cron	runs cron (unchanged) ✅
Gateway only	OK	OK (unchanged)
Server hermes dashboard (no HERMES_DESKTOP)	n/a	branch not reached ✅

Cross-platform

The liveness check reuses the same primitives the gateway itself uses, each with explicit per-OS handling:

• _pid_exists: psutil → ctypes OpenProcess/WaitForSingleObject (Windows) / os.kill(pid, 0) (POSIX).
• Runtime lock: msvcrt.locking (Windows) / fcntl.flock (POSIX).
• get_running_pid() does no subprocess/network calls (file I/O only) → no startup delay on any OS.

Verified on Windows 11; valid for macOS (launchd) and Linux (systemd), on x86_64 and ARM64.

Tests

• New test_ticker_skipped_when_gateway_alive: with HERMES_DESKTOP=1 and a live gateway, the desktop ticker must not run.
• Existing TestDesktopCronTicker, tests/cron/test_scheduler.py, and tests/cron/test_scheduler_provider.py pass (79 fix-related tests green on the rebased base).

python -m pytest tests/hermes_cli/test_web_server.py::TestDesktopCronTicker -q
python -m pytest tests/cron/test_scheduler.py tests/cron/test_scheduler_provider.py -q

Alternatives considered

Alternative	Why not
Hand off delivery to the gateway via IPC	Much more surface/complexity; a new channel to maintain
Validate adapter capability in the lock winner	Treats the symptom, not the duplicate schedulers; inference env is hard to auto-detect reliably
Desktop never executes jobs	Breaks the desktop-only install, where the ticker is the only executor

Known caveats (not regressions)

• Divergent HERMES_HOME/profile between gateway and desktop: if they run in different homes, the desktop may not see the gateway and start the ticker; .tick.lock is anchored on the shared default root, so the race could recur in that edge case. Pre-existing. Optional mitigation: re-check liveness per tick.
• Startup race: if the desktop comes up before the gateway, the check sees "no gateway" and starts the ticker. Optional mitigation: periodic re-check.

A standalone write-up is also added in docs/fixes/issue-52202-desktop-cron-gateway-race.md.

[alt-glitch]

This was generated by AI during triage.

Related: fix PR for #52202. Detects a live gateway before the desktop backend starts its own cron ticker (passive-observer when one exists, fail-open otherwise), plus a 30s bound on the standalone delivery send. Restores the one-scheduler-per-HERMES_HOME invariant.

[teknium1]

Thanks for tracing the desktop/gateway ownership collision. The premise remains valid on current main: hermes_cli/web_server.py:136-154 documents that the desktop ticker has no live adapters, while _lifespan starts it unconditionally at hermes_cli/web_server.py:197-205; the gateway independently starts its adapter-backed provider at gateway/run.py:20831-20846.

Problems

• The proposed liveness check runs only during desktop startup. If desktop starts first and the gateway starts later, both schedulers remain active. That leaves the PR's stated one-scheduler invariant unmet; the PR body also identifies this startup race.
• The cron/scheduler.py delivery hunk predates current-main hardening in the same block. cron/scheduler.py:1889-1955 now contains shutdown and per-target fallback handling from 242c9639a8 and 8aab8be50c; preserve those paths when carrying over the timeout bound.

Suggested changes

• Re-check gateway ownership during desktop ticking and stop the desktop ticker when the gateway becomes live; cover desktop-first/gateway-second startup.
• Add a direct standalone-send timeout test while retaining the current delivery fallback behavior.

Automated hermes-sweeper review.

[JoaoMarcos44]

@teknium1 done