Skip to content

fix(cron): run manual jobs without blocking the caller - #53395

Closed
izumi0uu wants to merge 1 commit into
NousResearch:mainfrom
izumi0uu:fix/cron-run-nonblocking-52705
Closed

fix(cron): run manual jobs without blocking the caller#53395
izumi0uu wants to merge 1 commit into
NousResearch:mainfrom
izumi0uu:fix/cron-run-nonblocking-52705

Conversation

@izumi0uu

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes the synchronous cronjob(action='run') path so a manual cron run starts immediately without blocking the caller until the job finishes.

Before this change, the model tool claimed the job and then called run_one_job() inline. That kept the current agent/tool turn open for the full runtime of the cron task. This patch reuses the scheduler's persistent pools for one-off manual dispatch, so manual runs still execute right away but now return after the job is queued.

Related Issue

Fixes #52705

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✅ Tests (adding or improving test coverage)

Changes Made

  • Added cron.scheduler.dispatch_job() as the single-job, non-blocking counterpart to tick(sync=False), reusing the existing running-job guard and workdir-aware pool selection.
  • Switched tools.cronjob_tools._execute_job_now() to claim the job and dispatch it through the scheduler pool instead of calling run_one_job() inline.
  • Updated CLI /cron run messaging to describe immediate background execution instead of “next scheduler tick”.
  • Added regression coverage for the non-blocking manual-run path and the updated CLI output.

How to Test

  1. Run /Users/idah/.hermes/hermes-agent-2/.venv/bin/pytest -q tests/tools/test_cronjob_run_immediate.py
  2. Run /Users/idah/.hermes/hermes-agent-2/.venv/bin/pytest -q tests/hermes_cli/test_cron.py
  3. Run /Users/idah/.hermes/hermes-agent-2/.venv/bin/pytest -q tests/cron/test_parallel_pool.py

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS 15.7.7 (Darwin 24.6.0 arm64)

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cron Cron scheduler and job management comp/cli CLI entry point, hermes_cli/, setup wizard labels Jun 27, 2026

@tonydwb tonydwb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Verdict: Approved

Good fix for cron manual jobs blocking the caller. The new _submit_job_with_running_guard function correctly checks for duplicate running jobs and releases the running set in a finally block. The contextvars.copy_context() pattern is correct for thread pool isolation.

Looks Good

  • Clean separation of guard logic into its own function
  • Running-set membership always released via finally block
  • Context isolation with copy_context is appropriate
  • No security concerns

Reviewed by Hermes Agent

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing the caller-blocking manual-run path; current main still invokes run_one_job() inline from tools/cronjob_tools.py:622-641, so the underlying issue remains valid.

Problems

  • The proposed extracted submission helper does not carry forward current main's shutdown protections in cron/scheduler.py:3663-3702: preflight detection of interpreter finalization plus cleanup when pool.submit() races shutdown. Those protections were added in 8aab8be50.
  • The proposed claim-loss branch is older than the current behavior at tools/cronjob_tools.py:625-637, which distinguishes paused/disabled/missing jobs from a held claim (7ecc822e1). Preserve that distinction during salvage.

Suggested changes

  • Extract the current guarded submission logic, including both shutdown paths, into the reusable helper.
  • Preserve the current claim-loss reason selection and add regression coverage for it after changing the dispatch model.

This is an automated hermes-sweeper review.

Comment thread cron/scheduler.py
@teknium1 teknium1 added sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 15, 2026
Extract current main's guarded scheduler submission path so manual runs reuse workdir-aware pools, running-job dedupe, execution-ledger ownership, ContextVar isolation, and both interpreter-shutdown protections.

Preserve paused, disabled, missing, and held-claim result reasons while returning immediately after successful background dispatch.

Tested: scripts/run_tests.sh -j 4 tests/cron tests/tools/test_cronjob_run_immediate.py tests/hermes_cli/test_cron.py (756 passed)

Refs: NousResearch#52705
@izumi0uu
izumi0uu force-pushed the fix/cron-run-nonblocking-52705 branch from 5b09dfa to dbed6e1 Compare July 21, 2026 14:39
@izumi0uu

Copy link
Copy Markdown
Contributor Author

Addressed in dbed6e1. Extracted current main’s complete guarded scheduler submission path into a reusable module-level helper and routed both ticker and manual dispatch through it, preserving the interpreter-shutdown preflight, submit-race cleanup, running-job dedupe, execution-ledger ownership, ContextVar isolation, and workdir-aware pool selection. Manual cronjob(action='run') now claims and queues the job immediately, returning execution_pending=true without waiting for run_one_job() to finish. I also preserved the distinct paused/disabled, missing, and genuinely held-claim responses, and added regressions covering both shutdown paths, claim-loss reason propagation, and the real non-blocking execution flow.

teknium1 added a commit that referenced this pull request Aug 7, 2026
Salvaged from PR #53395 by @izumi0uu: the fire claim's 300s TTL is
routinely outlived by real cron jobs, so claim_job_for_fire alone
cannot stop a manual cronjob(action='run') from double-firing a job
the ticker (or another manual run) is still executing.

Extract the ticker's _submit_with_guard running-set check into shared
module-level helpers (try_register_running_job / release_running_job)
and register manual runs through the same set — one dedupe owner, no
drift. Manual runs also become visible to get_running_job_ids (the
gateway shutdown drain, #60432) and mark_running_jobs_interrupted,
which previously could not see them.

The background dispatch path pre-checks the running set so a mid-run
job reports 'already running' in the tool response immediately
instead of as a delayed error completion event; the authoritative
atomic check remains in _run_claimed_job on the worker.

Co-authored-by: izumi0uu <izumi0uu@gmail.com>
teknium1 added a commit that referenced this pull request Aug 7, 2026
Salvaged from PR #53395 by @izumi0uu: the fire claim's 300s TTL is
routinely outlived by real cron jobs, so claim_job_for_fire alone
cannot stop a manual cronjob(action='run') from double-firing a job
the ticker (or another manual run) is still executing.

Extract the ticker's _submit_with_guard running-set check into shared
module-level helpers (try_register_running_job / release_running_job)
and register manual runs through the same set — one dedupe owner, no
drift. Manual runs also become visible to get_running_job_ids (the
gateway shutdown drain, #60432) and mark_running_jobs_interrupted,
which previously could not see them.

The background dispatch path pre-checks the running set so a mid-run
job reports 'already running' in the tool response immediately
instead of as a delayed error completion event; the authoritative
atomic check remains in _run_claimed_job on the worker.

Co-authored-by: izumi0uu <izumi0uu@gmail.com>
@teknium1

teknium1 commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Thanks @izumi0uu — the strongest idea in this PR outlived the PR itself: your observation that manual runs and the ticker need ONE shared in-flight dedupe owner (the fire claim's 300s TTL can't prevent double-fires on long jobs) was salvaged into #80807, now on main. The ticker's _submit_with_guard running-set check is extracted into shared try_register_running_job()/release_running_job() helpers and manual runs register through the same set — which also made them visible to the gateway shutdown drain (#60432). You're credited as co-author on the commit (3671c9f).

The non-blocking half went a different way than the fire-and-forget submit here: #80807 routes manual runs through the async-delegation rail so the outcome re-enters the conversation as a completion event rather than requiring status polling. Closing with thanks — the dedupe guard is your contribution in the tree.

@teknium1 teknium1 closed this Aug 7, 2026
POWERFULMOVES pushed a commit to POWERFULMOVES/PMOVES-hermes-agent that referenced this pull request Aug 10, 2026
Salvaged from PR NousResearch#53395 by @izumi0uu: the fire claim's 300s TTL is
routinely outlived by real cron jobs, so claim_job_for_fire alone
cannot stop a manual cronjob(action='run') from double-firing a job
the ticker (or another manual run) is still executing.

Extract the ticker's _submit_with_guard running-set check into shared
module-level helpers (try_register_running_job / release_running_job)
and register manual runs through the same set — one dedupe owner, no
drift. Manual runs also become visible to get_running_job_ids (the
gateway shutdown drain, NousResearch#60432) and mark_running_jobs_interrupted,
which previously could not see them.

The background dispatch path pre-checks the running set so a mid-run
job reports 'already running' in the tool response immediately
instead of as a delayed error completion event; the authoritative
atomic check remains in _run_claimed_job on the worker.

Co-authored-by: izumi0uu <izumi0uu@gmail.com>
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
Salvaged from PR NousResearch#53395 by @izumi0uu: the fire claim's 300s TTL is
routinely outlived by real cron jobs, so claim_job_for_fire alone
cannot stop a manual cronjob(action='run') from double-firing a job
the ticker (or another manual run) is still executing.

Extract the ticker's _submit_with_guard running-set check into shared
module-level helpers (try_register_running_job / release_running_job)
and register manual runs through the same set — one dedupe owner, no
drift. Manual runs also become visible to get_running_job_ids (the
gateway shutdown drain, NousResearch#60432) and mark_running_jobs_interrupted,
which previously could not see them.

The background dispatch path pre-checks the running set so a mid-run
job reports 'already running' in the tool response immediately
instead of as a delayed error completion event; the authoritative
atomic check remains in _run_claimed_job on the worker.

Co-authored-by: izumi0uu <izumi0uu@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cronjob(action='run') blocks the calling agent until the cron session finishes

4 participants