Skip to content

fix(cron): multiplex delivery uses owning profile's bot token and adapter (#83182) - #83557

Closed
dongjiang1989 wants to merge 1 commit into
NousResearch:mainfrom
dongjiang1989:fix-cron
Closed

dongjiang1989 wants to merge 1 commit into
NousResearch:mainfrom
dongjiang1989:fix-cron

Conversation

@dongjiang1989

Copy link
Copy Markdown

Summary

Closes: #83182

Under a multiplex gateway (multiple per-profile Telegram bots etc.), cron jobs owned by a secondary profile deliver to the wrong bot/chat. Telegram logs show Thread 9539 not found ... retrying without message_thread_id — the thread is dropped because the bot token used at delivery time does not own it.

The root cause has two parts:

  1. cron/scheduler.py::run_one_job installs the job-owning profile's secret scope, then resets it in the finally block between run_job and _deliver_result. By the time delivery runs, current_secret_scope() is Noneload_gateway_config()_getenv() falls back to os.environ (empty in the multiplex unit) — TELEGRAM_BOT_TOKEN resolves to the wrong bot (or nothing).

  2. The gateway starts the cron ticker with the shared runner.adapters dict (the default profile's live adapters). _deliver_resultresolve_delivery_transport(...) only ever consults that shared dict. It never reaches Gateway._profile_adapters[<job profile>], so even when the token happens to be right, cron delivery cannot reach the secondary profile's live adapter.

This is distinct from existing issues #51853 and #54675 — their fix (PR #59315, commit 0f154e7) covers only adapter startup, not cron delivery-time scope reset / shared-adapter usage.

What does this PR do?

Two fixes, both required to close the bug:

  • Part 1 — Scope lifetime: Move reset_secret_scope from the inner finally (after run_job) to an outer finally (after delivery). The profile's secret scope now stays installed through both execution and delivery, so load_gateway_config() picks up the right TELEGRAM_BOT_TOKEN from the owning profile's .env.

  • Part 2 — Per-profile adapters: gateway/run.py builds profile_adapters_by_home — a mapping from resolved profile home path to that profile's live adapter map (sourced from runner._profile_adapters for secondary profiles, runner.adapters for the default). This map is threaded through the cron chain: scheduler_provider.start_start_multiplexcron_tickrun_one_job_deliver_result. run_one_job resolves the owning profile's adapter map via the current hermes home (already set by _start_multiplex's per-profile override), so delivery picks the right bot token.

When profile_adapters_by_home is not set (single-profile mode, direct CLI ticks, external providers), the code falls back to the shared adapters dict — fully backward compatible.

Related Issue

Fixes #83182

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor
  • 🎯 New skill

Changes Made

cron/scheduler.py

  • Part 1: Restructure the try/finally in run_one_job. The inner finally: reset_secret_scope(_scope_token) is replaced with an outer finally that wraps both run_job and the delivery block. Agent teardown stays in the inner finally (its existing contract, unchanged).
  • Part 2: Add profile_adapters_by_home parameter to tick and run_one_job; add profile_adapters parameter to _deliver_result. run_one_job resolves the owning profile's adapters from the map keyed by the current hermes home path. _deliver_result prefers the profile-specific adapters over the shared dict when both are present.
  • DeliveryRouter and resolve_delivery_transport call sites updated to use the resolved delivery_adapters.

cron/scheduler_provider.py

  • InProcessCronScheduler.start and _start_multiplex accept and propagate profile_adapters_by_home down to cron_tick.

gateway/run.py

  • At cron start, when multiplex_profiles is on, build profile_adapters_by_home from runner._profile_adapters (secondary profiles) plus runner.adapters (default profile), keyed by resolved hermes home path. Pass to the cron scheduler via cron_start_kwargs.

tests/cron/test_cron_multiplex_delivery_83182.py (new)

  • TestSecretScopeThroughDelivery — verifies the scope-reset ordering (deliver → reset, not reset → deliver).
  • TestProfileAdapterSelection — verifies profile-specific adapters are preferred over shared, with fallback.
  • TestRunOneJobProfileAdaptersResolution — verifies run_one_job correctly looks up the right adapter map from profile_adapters_by_home.
  • 6 new regression tests, all passing.

tests/cron/test_run_one_job.py / test_preflight_config.py

  • Adapt fake_deliver signatures to accept the new profile_adapters=None kwarg. No behavioral changes.

How to Test

  1. scripts/run_tests.sh tests/cron/test_cron_multiplex_delivery_83182.py — 6 tests pass.
  2. scripts/run_tests.sh tests/cron/test_run_one_job.py tests/cron/test_preflight_config.py — 13 tests pass, no regression.
  3. End-to-end under a multiplex gateway with two Telegram profiles: create a cron job owned by the secondary profile, wait for it to fire. Pre-fix: delivers via the default bot, logs Thread ... not found. Post-fix: delivers via the secondary profile's bot to the correct chat.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix
  • I've run the test suite and the affected tests pass
  • I've added tests for my changes
  • I've tested on my platform: macOS

Documentation & Housekeeping

  • I've updated relevant documentation (docstrings) — added inline comments explaining both fixes
  • I've updated cli-config.yaml.example if I added/changed config keys — N/A (no config changes)
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture — N/A
  • I've considered cross-platform impact — N/A (no platform-specific code)
  • I've updated tool descriptions/schemas if I changed tool behavior — N/A

@alt-glitch alt-glitch added type/bug Something isn't working comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery platform/telegram Telegram bot adapter area/auth Authentication, OAuth, credential pools area/config Config system, migrations, profiles P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades needs-decision Awaiting maintainer decision before any implementation labels Aug 11, 2026
@dongjiang1989

Copy link
Copy Markdown
Author

cc @alt-glitch PTAL, thanks

…pter (NousResearch#83182)

Two root causes under multiplex gateway:

1. Secret scope reset before delivery:
   run_one_job's inner finally block reset the profile's secret scope
   after run_job returned but BEFORE _deliver_result ran. load_gateway_config
   → _getenv fell back to os.environ (empty in the multiplex unit) —
   TELEGRAM_BOT_TOKEN resolved to the wrong bot and delivery routed to
   the wrong chat.
   Fix: move reset_secret_scope to an outer finally so the scope stays
   installed through BOTH execution and delivery.

2. Shared adapters dict for delivery:
   The cron ticker received only runner.adapters (default profile's live
   adapters). Secondary-profile jobs therefore delivered via the default
   profile's adapter — even though Gateway._profile_adapters[profile] had
   the right per-profile adapter live and ready.
   Fix: gateway/run.py builds profile_adapters_by_home (resolved hermes
   home path → per-profile adapter map) and passes it through the cron
   chain: scheduler_provider → tick → run_one_job → _deliver_result.
   run_one_job resolves the owning profile's adapter map via the current
   hermes home (set by _start_multiplex's per-profile override), so
   delivery picks the right bot token.

Files changed:
  cron/scheduler.py           - scope restructure + profile_adapters param chain
  cron/scheduler_provider.py  - propagate profile_adapters_by_home
  gateway/run.py              - build profile_adapters_by_home from runner
  tests/cron/test_cron_multiplex_delivery_83182.py - 6 new regression tests
  tests/cron/test_run_one_job.py                   - fake_deliver signature
  tests/cron/test_preflight_config.py              - fake_deliver signature

Fixes NousResearch#83182

Signed-off-by: dongjiang <dongjiang1989@126.com>
ayushnangia added a commit to ayushnangia/hermes-agent that referenced this pull request Aug 14, 2026
…ousResearch#80921)

Deterministic, LLM-free conformance cells against the real SessionDB with
real SIGKILL mid-write, per the tracking issue's spot-probe method:

- cell 1: acknowledged-append durability + recovery determinism (adapted
  from the issue's 29.5K probe, scaled kill window, identical assertions)
- cell 2: consume-once under 8-process concurrent claim_handoff
- cell 3 (new): compression-rotation atomicity — never a compression-ended
  parent without a continuation (NousResearch#80337 contract; NousResearch#80487 recovery context)
- cells 4-5: documented stubs interlocked with NousResearch#82956-NousResearch#82959 and
  NousResearch#83197/NousResearch#83557

Journal-mode matrix (resolver default / DELETE / WAL-with-skip-gate) per
cell; every wait deadline-bounded; writers asserted alive at kill time.
@Enough1122

Copy link
Copy Markdown
Contributor

AI code review — automated review for reference, author can ignore or act on any point.

fix(cron): multiplex delivery uses owning profile's bot token and adapter (#83182)

  1. gateway/run.py — the new block iterates for _pname, _phome in profile_homes: assuming every element is a 2-tuple, but the log line a few lines above defensively handles string elements (p[0] if isinstance(p, tuple) else p). If profile_homes can contain plain strings, this unpacking raises ValueError; the surrounding try/except Exception swallows it, so per-profile adapters silently never get wired (multiplex cron keeps the old wrong-bot behavior with only a warning). Please confirm the element shape and, if strings are possible, normalize before unpacking.
  2. gateway/run.py — the per-profile adapter maps are shallow-copied once at gateway startup (dict(_amap)). If a secondary profile's adapters are later replaced wholesale (reconnect/restart rebuilding the adapter dict), cron delivery keeps using the stale snapshot. Resolving the map lazily at delivery time (or refreshing it per tick) would better match the "live adapter map" intent in the comments.
  3. Minor: _deliver_result's profile_adapters docstring calls it the "live adapter map", but the value is a startup snapshot; a note saying so would prevent future maintainers from assuming object identity with runner.adapters.

@dongjiang1989

Copy link
Copy Markdown
Author

Thanks for the review. Analyzed all 3 points:

1. Tuple unpacking — not an issue. profiles_to_serve() has return type List[Tuple[str, Path]] and always yields 2-tuples. The defensive isinstance(p, tuple) check in the log line is legacy paranoia, not a real code path. No normalization needed.

2. Stale adapter snapshot — not an issue in practice. Adapter instances in _profile_adapters[profile] are never replaced after registration — see gateway/run.py:13866: if platform not in profile_map: profile_map[platform] = adapter. Reconnects mutate the existing adapter in place; they do not swap the object. The shallow dict(_amap) copy shares the same adapter instances with the gateway, so cron sees the same live state.

3. Docstring — fair nit but the behavior is correct. "Live adapter map" refers to the fact that the dict holds live adapter references (shared with gateway), not that the dict itself is refreshed. Adding "(shared references to live adapter instances)" would clarify without changing semantics; will do if desired.

RecursiveIntell pushed a commit to RecursiveIntell/Ares that referenced this pull request Aug 25, 2026
One xfail per open member of the class mapped on NousResearch#82936: profile-scoped
state resolved from ambient process state at use time instead of bound
to the owning profile/session at creation. Test-only; fixes nothing;
flips to XPASS as per-site fixes land.

Members: NousResearch#82936 (multiplex terminal env), NousResearch#81952 (corrupt config silent
fallback), NousResearch#83346 (ambient session-key profile), NousResearch#80318 (profile scope
hides root MoA presets), NousResearch#83197/NousResearch#83557 (cron delivery scope reset before
delivery).

(cherry picked from commit ee79a0f7b6396e41f64df7ed76e9a759edbf8719)
kshitijk4poor pushed a commit that referenced this pull request Aug 27, 2026
…80921)

Deterministic, LLM-free conformance cells against the real SessionDB with
real SIGKILL mid-write, per the tracking issue's spot-probe method:

- cell 1: acknowledged-append durability + recovery determinism (adapted
  from the issue's 29.5K probe, scaled kill window, identical assertions)
- cell 2: consume-once under 8-process concurrent claim_handoff
- cell 3 (new): compression-rotation atomicity — never a compression-ended
  parent without a continuation (#80337 contract; #80487 recovery context)
- cells 4-5: documented stubs interlocked with #82956-#82959 and
  #83197/#83557

Journal-mode matrix (resolver default / DELETE / WAL-with-skip-gate) per
cell; every wait deadline-bounded; writers asserted alive at kill time.
and7777 pushed a commit to and7777/hermes-agent that referenced this pull request Aug 27, 2026
…ousResearch#80921)

Deterministic, LLM-free conformance cells against the real SessionDB with
real SIGKILL mid-write, per the tracking issue's spot-probe method:

- cell 1: acknowledged-append durability + recovery determinism (adapted
  from the issue's 29.5K probe, scaled kill window, identical assertions)
- cell 2: consume-once under 8-process concurrent claim_handoff
- cell 3 (new): compression-rotation atomicity — never a compression-ended
  parent without a continuation (NousResearch#80337 contract; NousResearch#80487 recovery context)
- cells 4-5: documented stubs interlocked with NousResearch#82956-NousResearch#82959 and
  NousResearch#83197/NousResearch#83557

Journal-mode matrix (resolver default / DELETE / WAL-with-skip-gate) per
cell; every wait deadline-bounded; writers asserted alive at kill time.
ayushnangia added a commit to ayushnangia/hermes-agent that referenced this pull request Aug 29, 2026
One xfail per open member of the class mapped on NousResearch#82936: profile-scoped
state resolved from ambient process state at use time instead of bound
to the owning profile/session at creation. Test-only; fixes nothing;
flips to XPASS as per-site fixes land.

Members: NousResearch#82936 (multiplex terminal env), NousResearch#81952 (corrupt config silent
fallback), NousResearch#83346 (ambient session-key profile), NousResearch#80318 (profile scope
hides root MoA presets), NousResearch#83197/NousResearch#83557 (cron delivery scope reset before
delivery).
@teknium1

teknium1 commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Thanks @dongjiang1989 — you proposed hoisting the secret-scope reset out of the run-block finally 12 days before the equivalent fix landed, and your follow-up on the review points (tuple shape, adapter identity stability) was correct. Reviewed against current origin/main: both halves of this PR are now in via #99375

Closing as redundant with credit; #83182 is closed as fixed with your name on it.

@teknium1 teknium1 closed this Sep 2, 2026
RecursiveIntell pushed a commit to RecursiveIntell/Ares that referenced this pull request Sep 3, 2026
One xfail per open member of the class mapped on NousResearch#82936: profile-scoped
state resolved from ambient process state at use time instead of bound
to the owning profile/session at creation. Test-only; fixes nothing;
flips to XPASS as per-site fixes land.

Members: NousResearch#82936 (multiplex terminal env), NousResearch#81952 (corrupt config silent
fallback), NousResearch#83346 (ambient session-key profile), NousResearch#80318 (profile scope
hides root MoA presets), NousResearch#83197/NousResearch#83557 (cron delivery scope reset before
delivery).

(cherry picked from commit ee79a0f7b6396e41f64df7ed76e9a759edbf8719)
melon-xf added a commit to melon-xf/hermes-agent that referenced this pull request Sep 3, 2026
…ousResearch#80921)

Deterministic, LLM-free conformance cells against the real SessionDB with
real SIGKILL mid-write, per the tracking issue's spot-probe method:

- cell 1: acknowledged-append durability + recovery determinism (adapted
  from the issue's 29.5K probe, scaled kill window, identical assertions)
- cell 2: consume-once under 8-process concurrent claim_handoff
- cell 3 (new): compression-rotation atomicity — never a compression-ended
  parent without a continuation (NousResearch#80337 contract; NousResearch#80487 recovery context)
- cells 4-5: documented stubs interlocked with NousResearch#82956-NousResearch#82959 and
  NousResearch#83197/NousResearch#83557

Journal-mode matrix (resolver default / DELETE / WAL-with-skip-gate) per
cell; every wait deadline-bounded; writers asserted alive at kill time.
zapabob pushed a commit to zapabob/hermes-agent-windows that referenced this pull request Sep 5, 2026
…ousResearch#80921)

Deterministic, LLM-free conformance cells against the real SessionDB with
real SIGKILL mid-write, per the tracking issue's spot-probe method:

- cell 1: acknowledged-append durability + recovery determinism (adapted
  from the issue's 29.5K probe, scaled kill window, identical assertions)
- cell 2: consume-once under 8-process concurrent claim_handoff
- cell 3 (new): compression-rotation atomicity — never a compression-ended
  parent without a continuation (NousResearch#80337 contract; NousResearch#80487 recovery context)
- cells 4-5: documented stubs interlocked with NousResearch#82956-NousResearch#82959 and
  NousResearch#83197/NousResearch#83557

Journal-mode matrix (resolver default / DELETE / WAL-with-skip-gate) per
cell; every wait deadline-bounded; writers asserted alive at kill time.
RecursiveIntell pushed a commit to RecursiveIntell/Ares that referenced this pull request Sep 5, 2026
One xfail per open member of the class mapped on NousResearch#82936: profile-scoped
state resolved from ambient process state at use time instead of bound
to the owning profile/session at creation. Test-only; fixes nothing;
flips to XPASS as per-site fixes land.

Members: NousResearch#82936 (multiplex terminal env), NousResearch#81952 (corrupt config silent
fallback), NousResearch#83346 (ambient session-key profile), NousResearch#80318 (profile scope
hides root MoA presets), NousResearch#83197/NousResearch#83557 (cron delivery scope reset before
delivery).

(cherry picked from commit ee79a0f7b6396e41f64df7ed76e9a759edbf8719)
RecursiveIntell pushed a commit to RecursiveIntell/Ares that referenced this pull request Sep 10, 2026
One xfail per open member of the class mapped on NousResearch#82936: profile-scoped
state resolved from ambient process state at use time instead of bound
to the owning profile/session at creation. Test-only; fixes nothing;
flips to XPASS as per-site fixes land.

Members: NousResearch#82936 (multiplex terminal env), NousResearch#81952 (corrupt config silent
fallback), NousResearch#83346 (ambient session-key profile), NousResearch#80318 (profile scope
hides root MoA presets), NousResearch#83197/NousResearch#83557 (cron delivery scope reset before
delivery).

(cherry picked from commit ee79a0f7b6396e41f64df7ed76e9a759edbf8719)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/auth Authentication, OAuth, credential pools area/config Config system, migrations, profiles comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery needs-decision Awaiting maintainer decision before any implementation P2 Medium — degraded but workaround exists platform/telegram Telegram bot adapter sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cron delivery uses wrong bot/chat under multiplex (secret-scope reset before delivery + shared-adapter dict)

4 participants