Skip to content

fix(cron): reset cron-delivery ContextVars to _UNSET after a job - #82362

Open
aldoeliacim wants to merge 1 commit into
NousResearch:mainfrom
aldoeliacim:fix/cron-auto-deliver-contextvar-leak-r2
Open

fix(cron): reset cron-delivery ContextVars to _UNSET after a job#82362
aldoeliacim wants to merge 1 commit into
NousResearch:mainfrom
aldoeliacim:fix/cron-auto-deliver-contextvar-leak-r2

Conversation

@aldoeliacim

Copy link
Copy Markdown
Contributor

What

run_job() sets the HERMES_CRON_AUTO_DELIVER_* ContextVars while a job runs so the job's own send_message can detect a duplicate auto-delivery target. On the way out it "cleared" them by setting "" — but an empty string is the explicitly cleared state that suppresses the os.environ fallback in get_session_env. Correct WHILE the job runs; wrong after it: the empty value leaks into every later caller in the same thread/task (cross-run / cross-test pollution), so a subsequent consumer that should have fallen back to os.environ sees "explicitly no target" instead.

Fix: reset_cron_delivery_vars() in gateway/session_context.py returns the vars to the _UNSET sentinel (never-set state), and run_job() calls it in the job-teardown path instead of the "" loop.

Why

  • The scheduler thread is long-lived: one job's teardown must not change delivery-target resolution for the next job or for non-cron callers on the same thread.
  • The distinction between "explicitly cleared" ("", suppress fallback) and "never set" (_UNSET, allow fallback) is the documented contract of get_session_env — teardown was landing on the wrong side of it.

How to test

scripts/run_tests.sh tests/gateway/test_session_env.py -- -q

11 tests pass, including the new one: after reset_cron_delivery_vars(), get_session_env falls back to os.environ again (fails against the previous "" teardown).

Tested on Linux (aarch64).

Supersedes auto-closed #47089 (head fork deleted in a remote swap; GitHub can't reopen a PR whose head repo is gone). Rebased onto current main — upstream's run_job teardown gained cwd-lock and non-dispatcher token handling since; this change composes with both.

run_job() sets the HERMES_CRON_AUTO_DELIVER_* ContextVars while a job runs so
the job's own send_message can detect a duplicate auto-delivery target. The
cleanup in the finally block set them back to "" — but "" is the
"explicitly cleared" state that suppresses the os.environ fallback in
get_session_env, NOT the "never set" (_UNSET) sentinel. So after any cron run,
every later caller in the same thread/task reads an empty cron-delivery value
instead of falling back to os.environ.

In a long-lived scheduler process this lets a finished cron job shadow a later
same-thread caller's auto-delivery env. It also surfaces as cross-test
pollution: after tests/cron/test_scheduler.py runs, the duplicate-target
detection in tests/tools/test_send_message_tool.py reads the leaked empty
ContextVar instead of the env the test set.

Fix: add session_context.reset_cron_delivery_vars(), which restores the three
cron-delivery vars to _UNSET, and call it in run_job's finally cleanup instead
of set(""). The per-job set("") at job START is unchanged (correct: each job
starts from a clean explicitly-empty state before its target is resolved).

Adds regression tests in tests/gateway/test_session_env.py asserting the vars
return to _UNSET and that get_session_env's os.environ fallback works again
after a reset.

Related: NousResearch#43370 (fixes the sibling HERMES_CRON_SESSION leak via the token
mechanism; this covers the auto-delivery vars it does not touch), NousResearch#8866.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state labels Aug 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants