Skip to content

fix(cron): pop HERMES_CRON_SESSION env var after last cron job finishes - #69740

Closed
im47cn wants to merge 1 commit into
NousResearch:mainfrom
im47cn:fix/cron-session-env-cleanup-73195
Closed

fix(cron): pop HERMES_CRON_SESSION env var after last cron job finishes#69740
im47cn wants to merge 1 commit into
NousResearch:mainfrom
im47cn:fix/cron-session-env-cleanup-73195

Conversation

@im47cn

@im47cn im47cn commented Jul 23, 2026

Copy link
Copy Markdown

Problem

When the cron scheduler runs inside a gateway process (InProcessCronScheduler, the default), os.environ["HERMES_CRON_SESSION"] = "1" is set in run_job() and never cleaned up. Since os.environ is process-global, the flag leaks into user-interactive sessions handled by the same gateway process.

Result: execute_code, web_request, and other tools gated on check_execute_code_guard or check_all_command_guards are blocked with:

BLOCKED: execute_code runs arbitrary local Python (...). 
Cron jobs run without a user present to approve it.

Users replying to cron-delivered messages (e.g. Feishu/Telegram threads) get their session incorrectly classified as a cron session.

Root Cause

cron/scheduler.py:2920:

os.environ["HERMES_CRON_SESSION"] = "1"

Comment on line 2918: "This env var is process-wide and persists for the lifetime of the scheduler process — every job this process runs is a cron job."

The intent was for a dedicated scheduler process — but the gateway co-locates the scheduler. Once any cron job runs, every subsequent user session in the same gateway process inherits HERMES_CRON_SESSION=1.

Fix

In run_job()'s finally block, use _running_job_ids as a reference counter — pop HERMES_CRON_SESSION only when no other cron jobs are still executing:

with _running_lock:
    remaining = _running_job_ids - {job_id}
    if not remaining:
        os.environ.pop("HERMES_CRON_SESSION", None)

This is safe under _running_lock which already serialises concurrent job tracking.

Tests

Added tests/cron/test_cron_session_env_cleanup.py (4 tests):

  • Single job finishes → flag removed
  • Concurrent jobs → flag preserved until last one finishes
  • Thread-safe with lock → no race on pop
  • Approval guard no longer sees cron after cleanup

All 788 existing cron + approval tests pass without modification.

Evidence

Confirmed in production gateway process (2026-07-23 BJT):

$ echo $HERMES_CRON_SESSION
1
$ echo $HERMES_GATEWAY_SESSION  
1

User session (Feishu gateway, 08:45 BJT) blocked 3 consecutive execute_code calls with the cron-deny error.

The cron scheduler sets os.environ['HERMES_CRON_SESSION'] = '1' in
run_job() to gate approval checks on cron_mode. This is process-global
and never cleaned up — when the scheduler runs inside a gateway process
(InProcessCronScheduler), the flag leaks into user-interactive sessions.

Affected tools blocked by the stale flag: execute_code, web_request, and
any tool gated on check_execute_code_guard or check_all_command_guards
when cron_mode defaults to 'deny'.

Fix: in run_job's finally block, use _running_job_ids as a reference
counter — pop the env var only when no other cron jobs are still
executing. This is safe under _running_lock which already serializes
concurrent job tracking.

Closes NousResearch#73195.
@im47cn

im47cn commented Jul 23, 2026

Copy link
Copy Markdown
Author

Closing in favor of a root-cause fix. The reference-counting approach is a band-aid: the check-then-pop window still races under concurrent job execution. The root cause is that HERMES_CRON_SESSION uses process-global os.environ while the rest of the session state (HERMES_SESSION_PLATFORM etc.) already migrated to task-local ContextVars. Will submit a proper ContextVar migration.

@im47cn im47cn closed this Jul 23, 2026
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cron Cron scheduler and job management sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state labels Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management P2 Medium — degraded but workaround exists sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants