Skip to content

fix(cron): wait for hermes cron run instead of detaching - #86739

Closed
Adolanium wants to merge 1 commit into
NousResearch:mainfrom
Adolanium:fix/cron-oneshot-run-waits
Closed

fix(cron): wait for hermes cron run instead of detaching#86739
Adolanium wants to merge 1 commit into
NousResearch:mainfrom
Adolanium:fix/cron-oneshot-run-waits

Conversation

@Adolanium

Copy link
Copy Markdown
Contributor

What does this PR do?

hermes cron run is a one shot command. The process ends as soon as the command returns.

If the shell still has a leftover Desktop or gateway session key, the run was sent to a background thread in that same process. The process then exited. The thread died. The job stayed claimed. The next hermes cron run failed.

This PR makes hermes cron run wait in the current process until the job finishes. A leftover session key cannot detach it.

Gateway and live agent cronjob(action="run") still use the background path. Those processes stay alive.

Before the new run, the CLI also marks leftover execution rows as unknown when their owner process is already dead.

Related Issue

Fixes #86721

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Security fix
  • Documentation update
  • Tests (adding or improving test coverage)
  • Refactor (no behavior change)
  • New skill (bundled or hub)

Changes Made

  • tools/cronjob_tools.py: add allow_background (default true). hermes cron run sets it false so the job runs inline.
  • hermes_cli/cron.py: one shot run always passes allow_background=False. It also calls recover_interrupted_executions() first.
  • tests/tools/test_cronjob_run_background.py: with a session key bound, allow_background=False still runs inline and never dispatches.
  • tests/hermes_cli/test_cron.py: the CLI run command forces inline and reaps dead claims.

How to Test

  1. On current main, if HERMES_SESSION_KEY is set, hermes cron run <id> can return at once, leave executions.status=claimed, and leave a dead async_delegations row. The next run can fail.
  2. On this branch, hermes cron run <id> waits until the job finishes and prints Ran now: succeeded. or Ran now: failed.
  3. python -m pytest tests/tools/test_cronjob_run_background.py tests/hermes_cli/test_cron.py -q (28 passed on Windows 11).
  4. A live gateway or Desktop cronjob(action="run") still returns a background handle.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run the focused cron run suites and they pass
  • I've added tests for my changes
  • I've tested on my platform: Windows 11

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior or N/A

One-shot hermes cron run must finish the job in this process.
A leftover HERMES_SESSION_KEY was enough to claim the job, start
a background thread, then exit and kill the runner.

allow_background=False on the CLI run path keeps the inline fire.
Dead execution rows are marked unknown before the new claim.
Gateway cronjob(action=run) still detaches.

Fixes NousResearch#86721
@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management comp/tools Tool registry, model_tools, toolsets sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state labels Aug 15, 2026

@trevorgordon981 trevorgordon981 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct direction and cleanly scoped. Forcing allow_background=False for one-shot hermes cron run is the right fix for #86721 — a one-shot process that detaches a runner thread and then exits kills the job mid-flight and leaves the fire claim hanging. The test test_allow_background_false_stays_inline_even_with_session_key is exactly the regression that matters (leftover Desktop/gateway HERMES_SESSION_KEY no longer triggers a doomed background dispatch). Two smaller notes.

1. recover_interrupted_executions() failure is silent

The call before the forced-inline run is wrapped in except Exception: pass. If recovery itself fails (e.g. the state.db it reads is locked or malformed), the user gets no signal and cron run proceeds — and could re-queue or collide with a claim the recovery was supposed to reap. This matches the surrounding defensive style, so it's not out of character, but for the specific purpose here — "reap dead claims so the inline run can proceed cleanly" — a silent recovery failure undermines the very thing the call exists for. Consider logging a warning (at minimum) rather than a bare pass, or only proceeding when recovery reports success.

2. allow_background defaults True — correct for the tool, but confirm no other one-shot path needs the same guard

cronjob(..., allow_background: bool = True) preserves the existing in-session tool behavior, and only the CLI cron run path sets it False. That's the right split. Worth confirming there's no other one-shot entry point (e.g. a --run flag on cron list, a gateway-side manual trigger, or a tool call from a headless script that also exits immediately) that should get the same treatment — otherwise #86721 can resurface through a sibling path.

Tests

Good — test_oneshot_run_forces_inline_and_reaps_dead_claims asserts allow_background=False is passed and recover_interrupted_executions is invoked, and the test_allow_background_false_stays_inline_even_with_session_key case covers the leftover-env regression with dispatch_async_delegation asserted-not-called and inline run_one_job called once. Missing: a test for the recovery-failure path (finding #1).

@teknium1

Copy link
Copy Markdown
Contributor

Thanks @Adolanium — you diagnosed this first (3 hours before the fix that landed), and your analysis of the one-shot-process/detached-runner failure mode was exactly right.

The fix merged today in #86853 resolves #86721 through a slightly different mechanism: hermes cron run now declares its delivery channel stateless (scoped ContextVar) before invoking the cron API, so async_delivery_supported() gates off the background dispatch and the run executes synchronously to completion — same outcome as your allow_background=False parameter, without widening the tool signature. The stale-claim reap you also wired in is covered by the periodic dead-owner reclaim added to the scheduler tick in the same PR.

Closing as resolved on main. Appreciate the fast, well-tested report-to-fix turnaround — the regression tests in your PR shaped the coverage that landed.

@teknium1 teknium1 closed this Aug 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management comp/tools Tool registry, model_tools, toolsets P1 High — major feature broken, no workaround sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cron run from one-shot CLI orphans the job: async delegation dies with the calling process, execution stuck 'claimed' forever

4 participants