Skip to content

fix(cron): parallelize script jobs with workdirs - #61774

Open
soria-clawd-bot wants to merge 1 commit into
NousResearch:mainfrom
soria-clawd-bot:fix/noagent-workdir-concurrency
Open

fix(cron): parallelize script jobs with workdirs#61774
soria-clawd-bot wants to merge 1 commit into
NousResearch:mainfrom
soria-clawd-bot:fix/noagent-workdir-concurrency

Conversation

@soria-clawd-bot

Copy link
Copy Markdown

Summary

  • pass no-agent cron workdirs to subprocess cwd instead of process-global os.chdir
  • keep agent workdir jobs serialized while allowing script-only watchdogs to run in parallel
  • record scheduled_at, dispatch_started_at, and dispatch_lag_seconds in saved run artifacts

Proof

  • HERMES_HOME=$(mktemp -d) EMAIL_HOME_ADDRESS= uv run pytest -q tests/cron — 673 passed
  • uv run ruff check on touched files — passed
  • git diff --check — passed

@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/cron Cron scheduler and job management labels Jul 10, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related: this lands in a saturated competing cluster on cron/scheduler.py _run_job_script workdir handling — #21397 (earliest-open, general workdir→subprocess-cwd), #42274 / #56005 (no_agent-specific), #60475 (general). All open. This PR uses the same general mechanism (thread workdir into subprocess cwd, drop the inert process-global os.chdir) plus a parallel-pool serialization change and dispatch-lag artifacts. Not marking a duplicate — flagging the cluster so a maintainer can pick one approach.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for isolating the no-agent cwd to the subprocess; the premise is present on current main: cron/scheduler.py:2535-2549 mutates the process cwd, and tick() serializes every workdir job at cron/scheduler.py:3640-3647.

Problems

  • cron/scheduler.py:3349 in this PR derives scheduled_at from next_run_at. That is not valid for recurring external-scheduler fires: cron/scheduler_provider.py:100-105 claims before reloading the job, while cron/jobs.py:1708-1710 and 1743-1747 advance next_run_at during that claim. The proposed artifact can therefore record the next future occurrence and a zero lag instead of the occurrence that fired.

Suggested changes

  • Keep the subprocess-cwd/pool change focused, and split the dispatch metadata; alternatively carry the pre-claim scheduled time explicitly into run_one_job().
  • Cover the CronScheduler.fire_due() recurring-job path if retaining the metadata.

Automated hermes-sweeper review.

Comment thread cron/scheduler.py
@@ -3346,6 +3347,37 @@ def _teardown_cron_agent(agent, job_id: str) -> None:
logger.debug("Job '%s': failed to reap stale auxiliary clients: %s", job_id, e)


Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

next_run_at is not reliably the occurrence being dispatched. CronScheduler.fire_due() claims before it reloads the job, and claim_job_for_fire() advances recurring next_run_at; this will save a future scheduled time and zero lag for external recurring fires. Preserve the pre-claim fire time explicitly, or split this telemetry from the cwd fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants