Skip to content

fix(cron): clean up HERMES_CRON_SESSION env var after each job run (#56771) - #57124

Open
AlexFucuson9 wants to merge 1 commit into
NousResearch:mainfrom
AlexFucuson9:fix-cron-env-leak-56771
Open

fix(cron): clean up HERMES_CRON_SESSION env var after each job run (#56771)#57124
AlexFucuson9 wants to merge 1 commit into
NousResearch:mainfrom
AlexFucuson9:fix-cron-env-leak-56771

Conversation

@AlexFucuson9

Copy link
Copy Markdown
Contributor

Problem

When the cron scheduler runs in-process via InProcessCronScheduler (same Python process as the gateway), run_job() sets os.environ["HERMES_CRON_SESSION"] = "1" but never removes it in the finally block. Since os.environ is process-global and shared across all threads, this sentinel leaks into interactive gateway sessions.

Impact: After any cron job runs, execute_code and terminal commands in Telegram/gateway interactive sessions are blocked with the cron-deny approval error until the gateway process is restarted.

Root Cause

  • cron/scheduler.py:2444 — sets os.environ["HERMES_CRON_SESSION"] = "1"
  • cron/scheduler.py:3025+finally block cleans up TERMINAL_CWD, ContextVars, session DB, agent resources — but not HERMES_CRON_SESSION
  • tools/approval.py:2655check_code_execution_approval() reads HERMES_CRON_SESSION from os.getenv() and blocks execution

The codebase already uses ContextVars for per-job session state to avoid exactly this class of cross-session pollution, but HERMES_CRON_SESSION was left as a raw os.environ write.

Fix

Add os.environ.pop("HERMES_CRON_SESSION", None) to the finally block in run_job(), alongside the existing TERMINAL_CWD cleanup. This ensures the env var is only present while a cron job is actively running.

Fixes #56771

HERMES_CRON_SESSION was set via os.environ (process-global) in run_job()
but never removed in the finally block. When the cron scheduler runs
in-process via InProcessCronScheduler (same process as the gateway),
this sentinel leaked into interactive gateway sessions, causing
execute_code and terminal commands to be blocked with the cron-deny
approval error.

Add os.environ.pop('HERMES_CRON_SESSION', None) to the finally block
alongside the existing TERMINAL_CWD cleanup.

Fixes NousResearch#56771
@alt-glitch alt-glitch added type/bug Something isn't working comp/cron Cron scheduler and job management sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state P2 Medium — degraded but workaround exists labels Jul 2, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Competing fix cluster for #56771 (all open, all different sites/mechanisms): this PR (#57124) cleans up the leaked HERMES_CRON_SESSION at the source (os.environ.pop in run_job's finally); #56796 replaces the process-global env var with a per-job ContextVar (gateway/session_context.py); #56784 reorders the approval-context check so gateway session indicators take precedence over the cron flag (consumer side). This one is the smallest surgical source-side fix. Flagging for a maintainer to pick the canonical approach; not marking any as a duplicate.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for isolating the source of the leak. The current main implementation still has the process-global marker at cron/scheduler.py:2812, so the underlying report is valid.

Problems

  • The added cleanup is unsafe with concurrent jobs. cron/scheduler.py:3741-3788 dispatches workdir-less jobs in parallel; when one job reaches this pop, another active cron job can lose its cron classification. The approval gates at tools/approval.py:2173, 2700, and 3121 would then no longer apply approvals.cron_mode to that remaining job.
  • Cleanup only after run_job() returns still leaves an interactive gateway request misclassified while any cron job is active: the process-global flag is set at cron/scheduler.py:2812, and _is_gateway_approval_context() prioritizes it at tools/approval.py:241-245.
  • This diff adds no regression test for the cross-thread gateway/cron case.

Suggested changes

  • Scope the cron marker per job (for example with the existing ContextVar session-state approach) and have the approval gates read that scoped value.
  • Add a concurrent-thread regression test showing gateway approval remains interactive while real cron work still honors approvals.cron_mode.

Automated hermes-sweeper review.

Comment thread cron/scheduler.py
os.environ.pop("TERMINAL_CWD", None)
else:
os.environ["TERMINAL_CWD"] = _prior_terminal_cwd
# Remove the process-wide cron sentinel so it does not leak into

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This process-global cleanup is unsafe when due jobs run concurrently: a completed job can clear the marker while another cron job is still active, causing its approval checks to stop enforcing approvals.cron_mode. The cron marker needs per-job scoped state rather than a shared env-var lifetime.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: execute_code blocked in interactive Telegram gateway session because HERMES_CRON_SESSION env var leaks from cron scheduler into user's shell

3 participants