Skip to content

fix(send_message): HERMES_CRON_SESSION ContextVar isolation (stop in-process cron flag latch leaking interactive sends to home) - #88

Merged
Kyzcreig merged 3 commits into
mainfrom
fix/cron-session-contextvar-latch
Jun 22, 2026
Merged

Kyzcreig merged 3 commits into
mainfrom
fix/cron-session-contextvar-latch

Conversation

@Kyzcreig

Copy link
Copy Markdown
Collaborator

Closes the dominant, proven cause of the recurring send_message origin-leak (interactive turns landing in the home channel). v1 (#71) and v2 (#83) were real but secondary.

Root cause (proven live)

The gateway ticks the cron scheduler in-process (gateway/run.py:17161). The first cron job set os.environ["HERMES_CRON_SESSION"]="1" process-globally (cron/scheduler.py:1586) and never cleared it (only TERMINAL_CWD + ContextVars are reset in the finally). So after the first cron tick, every interactive turn read cron=True -> send_message's _has_messaging_origin() cron-guard returned False first -> NOT_IN_TURN -> home channel.

Caught live on a main turn (gateway PID 95240, already running v2): a terminal subprocess showed HERMES_CRON_SESSION=1 in the gateway process, and a bare send_message returned "Sent to discord home channel" while the user was in #fix-issues. The cron-session marker was the latch; this also explains the "main-turn seam" v2 could not prove.

Fix

  • gateway/session_context.py — HERMES_CRON_SESSION is now a task/thread-isolated ContextVar (registered in _VAR_MAP), with token-based set_cron_session/clear_cron_session (nestable) and a context-aware is_cron_session() reader (ContextVar -> os.environ fallback for CLI/standalone-scheduler/test back-compat).
  • cron/scheduler.py — run_job sets the marker via the ContextVar (before the per-job copy_context so the main agent inherits it) and resets it in finally. No more process-global os.environ write.
  • tools/approval.py + tools/send_message_tool.py — the 6 readers use is_cron_session(). Approval keeps cron deny-gating; the send resolver + tripwire are no longer poisoned by a latched flag.
  • tools/delegate_tool.py (B1 — caught by Opus review) — cron jobs spawn subagents in a bare ThreadPoolExecutor that does not inherit contextvars. Without a rebind, the contextvar-only change would have flipped cron subagents from deny -> auto-approve dangerous commands / execute_code (a security regression introduced by the fix). Fixed by capturing cron-ness at spawn (_is_cron_child) and re-binding it in the child-run wrapper (_bind_child_cron_session), mirroring v2's send-origin rebind.

Tests (RED-proven where load-bearing)

  • tests/gateway/test_cron_session_contextvar.py — thread isolation, copy_context inheritance, env fallback, interactive-not-poisoned after a cron cycle, cron-guard-first preserved.
  • tests/tools/test_cron_subagent_session.py — cron child dangerous-cmd + execute_code STILL BLOCKED; a RED control proves the rebind is load-bearing (without it -> auto-approve); interactive parent's child not over-marked; marker reset after the child run.
  • 64 tests green incl. all back-compat suites (test_cron_approval_mode, test_send_message_origin, test_subagent_send_origin). Grep gates clean: no prod os.environ["HERMES_CRON_SESSION"]= writer, no prod raw env_var_enabled("HERMES_CRON_SESSION") reader.
  • Pre-existing unrelated reds (fail on clean fork/main): test_hardline_blocklist (reboot detection) + TestDelegateHeartbeat (wall-clock timing).

Spec: docs/specs/PRD-send-message-origin-leak-v3-cron-session-latch.md (Opus Pass-1 folded — B1 blocker -> Phase 5 rebind, B2 fake-green -> real subagent deny test, registration gate, ordering constraint).

@github-actions

github-actions Bot commented Jun 22, 2026 •

Copy link
Copy Markdown

🔎 Lint report: fix/cron-session-contextvar-latch vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 11350 on HEAD, 11348 on base (🆕 +2)

🆕 New issues (4):

Rule Count
unresolved-import 2
invalid-assignment 1
unresolved-attribute 1
First entries
tests/run_agent/test_credits_notices_toggle.py:76: [invalid-assignment] invalid-assignment: Object of type `None` is not assignable to attribute `_credits_session_start_micros` of type `int`
tests/tools/test_cron_subagent_session.py:24: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
tests/gateway/test_cron_session_contextvar.py:19: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
tools/delegate_tool.py:1352: [unresolved-attribute] unresolved-attribute: Unresolved attribute `_is_cron_child` on type `AIAgent`

✅ Fixed issues (2):

Rule Count
unresolved-attribute 2
First entries
tests/run_agent/test_credits_notices_toggle.py:76: [unresolved-attribute] unresolved-attribute: Unresolved attribute `_credits_session_start_micros` on type `AIAgent`
run_agent.py:2940: [unresolved-attribute] unresolved-attribute: Object of type `Self@get_credits_spent_micros` has no attribute `_credits_session_start_micros`

Unchanged: 5936 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

@greptile-apps

greptile-apps Bot commented Jun 22, 2026 •

Copy link
Copy Markdown

Greptile Summary

This PR replaces the process-global os.environ["HERMES_CRON_SESSION"] write with a task/thread-isolated ContextVar, fixing a proven bug where the first in-process cron tick permanently latched the flag and caused every subsequent interactive turn's send_message to misroute to the home channel. It also closes the security regression (B1) that would have dropped approval deny-gating for cron-spawned subagents, which run in a bare ThreadPoolExecutor that does not inherit context variables.

  • gateway/session_context.py: Adds _CRON_SESSION ContextVar registered in _VAR_MAP, with token-based set_cron_session/clear_cron_session and a context-aware is_cron_session() reader (ContextVar → os.environ fallback for back-compat with tests and CLI callers).
  • cron/scheduler.py: Replaces the process-global env write with set_cron_session() / clear_cron_session() in run_job's finally, placed before the per-job copy_context() so the main agent inherits the marker.
  • tools/delegate_tool.py: Captures cron-ness at subagent spawn (_is_cron_child) in the parent thread and re-binds it in _run_with_thread_capture via _bind_child_cron_session / _clear_child_cron_session, preserving deny-gating across the executor boundary.
  • tools/approval.py + tools/send_message_tool.py: Migrates all 6 env_var_enabled("HERMES_CRON_SESSION") calls to is_cron_session(), with a lazy-import wrapper in approval.py to avoid circular imports.

Confidence Score: 5/5

Safe to merge — the fix is narrowly scoped to replacing a process-global env write with an identical-semantics ContextVar, following patterns already established in the same file for the cron delivery vars.

The root cause is well-understood and the fix is minimal-diff: a direct swap of the process-global flag for a task-isolated ContextVar, token-based clear, and an explicit rebind at the one boundary (ThreadPoolExecutor) that does not auto-propagate context. clear_session_vars hardcodes its var list and does not touch _CRON_SESSION; set_session_vars has no _CRON_SESSION parameter, so registration in _VAR_MAP only affects get_session_env reads. The delegate boundary capture-and-rebind mirrors the pre-existing _bind_child_send_origin pattern exactly. Tests cover thread isolation, copy_context inheritance, the bad-token exception fallback, nestability, a RED control proving the rebind is load-bearing, and the end-to-end send-target resolution after a cron cycle.

No files require special attention — all changed files follow established codebase patterns and have targeted test coverage for their load-bearing paths.

Important Files Changed

Filename Overview
gateway/session_context.py Adds _CRON_SESSION ContextVar with correct _UNSET default, registered in _VAR_MAP, and exposes set/clear/is_cron_session() helpers; exception path in clear_cron_session correctly restores _UNSET (not "") to preserve os.environ fallback.
cron/scheduler.py Replaces process-global os.environ write with set_cron_session() before copy_context() and clear_cron_session() in finally; follows the identical pre-existing pattern for _ctx_tokens.
tools/delegate_tool.py Captures _is_cron_child at spawn with inner try/except, adds _bind_child_cron_session/_clear_child_cron_session helpers, and wires them into _run_with_thread_capture alongside the existing send-origin rebind.
tools/approval.py Adds _is_cron_session() lazy-import wrapper with an error-level log on the fallback path; migrates all 4 env_var_enabled("HERMES_CRON_SESSION") call sites.
tools/send_message_tool.py Migrates both env_var_enabled("HERMES_CRON_SESSION") calls in _has_messaging_origin() and _warn_unexpected_home_fallback() to the context-aware is_cron_session().
tests/gateway/test_cron_session_contextvar.py Covers thread isolation, copy_context inheritance, env fallback, bad-token exception path, nestability, _VAR_MAP registration gate, and the end-to-end send-target resolution after a cron cycle.
tests/tools/test_cron_subagent_session.py RED-control test proves rebind is load-bearing; positive tests assert dangerous-cmd and execute_code stay BLOCKED inside a cron child; negative test confirms interactive-parent children are not over-marked.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant GT as Gateway Process
    participant CT as Cron Worker Thread
    participant IT as Interactive Turn
    participant DT as Delegate Child Thread

    Note over GT: Before fix: os.environ["HERMES_CRON_SESSION"]="1" latched forever

    rect rgb(200, 240, 200)
        Note over CT: After fix — cron job lifecycle
        CT->>CT: set_cron_session() → token (ContextVar only)
        CT->>CT: copy_context() — main agent inherits marker
        CT->>CT: "agent runs, tool calls — is_cron_session()=True"
        CT->>DT: delegate_task() — bare ThreadPoolExecutor (no ContextVar inheritance)
        Note over DT: _build_child_agent captures _is_cron_child=True
        DT->>DT: _bind_child_cron_session(child) → set_cron_session()
        DT->>DT: "check_dangerous_command → is_cron_session()=True → BLOCKED"
        DT->>DT: finally: _clear_child_cron_session(token)
        CT->>CT: finally: clear_cron_session(token) → ContextVar reset to _UNSET
    end

    rect rgb(200, 220, 255)
        Note over IT: Concurrent interactive turn (same process, different context)
        IT->>IT: "is_cron_session() → ContextVar=_UNSET → os.environ unset → False"
        IT->>IT: "_has_messaging_origin() → cron=False → check session platform"
        IT->>IT: _resolve_send_target() → ORIGIN (current channel)
    end

    Note over GT: is_cron_session() reads ContextVar first,<br/>falls back to os.environ for CLI/test back-compat
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant GT as Gateway Process
    participant CT as Cron Worker Thread
    participant IT as Interactive Turn
    participant DT as Delegate Child Thread

    Note over GT: Before fix: os.environ["HERMES_CRON_SESSION"]="1" latched forever

    rect rgb(200, 240, 200)
        Note over CT: After fix — cron job lifecycle
        CT->>CT: set_cron_session() → token (ContextVar only)
        CT->>CT: copy_context() — main agent inherits marker
        CT->>CT: "agent runs, tool calls — is_cron_session()=True"
        CT->>DT: delegate_task() — bare ThreadPoolExecutor (no ContextVar inheritance)
        Note over DT: _build_child_agent captures _is_cron_child=True
        DT->>DT: _bind_child_cron_session(child) → set_cron_session()
        DT->>DT: "check_dangerous_command → is_cron_session()=True → BLOCKED"
        DT->>DT: finally: _clear_child_cron_session(token)
        CT->>CT: finally: clear_cron_session(token) → ContextVar reset to _UNSET
    end

    rect rgb(200, 220, 255)
        Note over IT: Concurrent interactive turn (same process, different context)
        IT->>IT: "is_cron_session() → ContextVar=_UNSET → os.environ unset → False"
        IT->>IT: "_has_messaging_origin() → cron=False → check session platform"
        IT->>IT: _resolve_send_target() → ORIGIN (current channel)
    end

    Note over GT: is_cron_session() reads ContextVar first,<br/>falls back to os.environ for CLI/test back-compat
Loading

Reviews (2): Last reviewed commit: "fix(send_message): address Greptile P2s ..." | Re-trigger Greptile

Comment thread tools/approval.py
Comment thread gateway/session_context.py
Apollo added 2 commits June 22, 2026 10:58
…e in-process cron flag latch leaking interactive sends to home

Root cause (PRD v3): the gateway ticks the cron scheduler IN-PROCESS. The first
cron job set os.environ["HERMES_CRON_SESSION"]="1" process-globally and never
cleared it, so every later interactive turn read cron=True -> send_message origin
resolver took the cron-guard-first branch -> NOT_IN_TURN -> home channel. Caught
live on a main turn (gateway PID 95240, v2-deployed); the v2 subagent fix was real
but secondary.

Fix:
- gateway/session_context.py: HERMES_CRON_SESSION is now a task/thread-isolated
  ContextVar (registered in _VAR_MAP) with set_cron_session/clear_cron_session
  (token-based, nestable) and a context-aware is_cron_session() reader
  (ContextVar -> os.environ fallback for CLI/standalone/test back-compat).
- cron/scheduler.py: run_job sets the marker via the ContextVar (BEFORE the
  per-job copy_context so the main agent inherits it) and resets it in finally.
  No more process-global os.environ write.
- tools/approval.py + tools/send_message_tool.py: the 6 readers now use the
  context-aware is_cron_session() (approval keeps cron deny-gating; send resolver
  + tripwire no longer poisoned by a latched flag).
- tools/delegate_tool.py (B1, Opus Pass-1 blocker): cron jobs spawn subagents in
  a bare ThreadPoolExecutor that does NOT inherit contextvars. Capture cron-ness
  at spawn (_is_cron_child) and re-bind it in the child-run wrapper
  (_bind_child_cron_session) so cron subagents KEEP their approval deny-gating —
  without this, contextvar-only would have flipped cron subagents to auto-approve
  dangerous commands/execute_code (a security regression introduced by the fix).

Tests (RED-proven where load-bearing): tests/gateway/test_cron_session_contextvar.py
(thread isolation, copy_context inheritance, env fallback, interactive-not-poisoned
after a cron cycle, cron-guard-first preserved) + tests/tools/test_cron_subagent_session.py
(cron child dangerous-cmd + execute_code STILL BLOCKED; RED control proves the
rebind is load-bearing; interactive parent child not over-marked; reset after run).
64 tests green incl. all back-compat suites; grep gates clean (no prod os.environ
writer, no prod raw reader). Pre-existing unrelated reds: test_hardline_blocklist
(reboot detection) + TestDelegateHeartbeat (timing) both fail on clean fork/main.

Spec: docs/specs/PRD-send-message-origin-leak-v3-cron-session-latch.md (Opus Pass-1 folded).
… is the dominant leak cause; Opus Pass-1 folded (B1 cron-subagent approval regression → Phase 5 rebind)

(cherry picked from commit 1e1903e36f00599f899cc88a9f86a8935c330947)
@Kyzcreig
Kyzcreig force-pushed the fix/cron-session-contextvar-latch branch from 345d2e0 to d7730e1 Compare June 22, 2026 17:58
…est cron-backstop docstring

- session_context.clear_cron_session: on a cross-context reset() failure, restore
  _UNSET (not "") so get_session_env still falls through to os.environ — setting ""
  pinned a non-_UNSET value and silently defeated the I5 back-compat fallback.
  + test_clear_with_bad_token_restores_env_fallback.
- approval._is_cron_session: correct the docstring — the os.environ except-fallback
  is NOT a reliable backstop for in-process gateway cron now that run_job no longer
  writes os.environ; the import failure is effectively impossible, so log loudly
  rather than claim cron gating is "never silently lost".
@Kyzcreig
Kyzcreig merged commit d12ac48 into main Jun 22, 2026
37 checks passed
@Kyzcreig
Kyzcreig deleted the fix/cron-session-contextvar-latch branch June 22, 2026 18:12
@Kyzcreig
Kyzcreig restored the fix/cron-session-contextvar-latch branch September 21, 2026 10:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant