Skip to content

fix(cron): bind the profile secret scope for the dispatch-failure notice - #130431

Closed
skyeyesec333 wants to merge 1 commit into
NousResearch:mainfrom
skyeyesec333:fix/cron-dispatch-failure-secret-scope
Closed

skyeyesec333 wants to merge 1 commit into
NousResearch:mainfrom
skyeyesec333:fix/cron-dispatch-failure-secret-scope

Conversation

@skyeyesec333

@skyeyesec333 skyeyesec333 commented Oct 1, 2026 •

Copy link
Copy Markdown

Fixes #131585

Resubmission of #127310. It closed unmerged on 2026-09-30 when I deleted the fork it came from by mistake, and GitHub can't reopen it. Rebased onto main at bd32c71. Since #126982 merged, the fix uses its _install_fire_secret_scope() helper instead of building the scope by hand. Earlier review and test notes are on #127310.

What does this PR do?

The failure notice for a restart-safe cron worker dispatch failure (added in #124013 for #123401) can't be delivered under a multiplexed gateway. It fails with the same plain-language error the notice is meant to replace:

ERROR cron.scheduler: Job '<id>': Restart-safe cron worker dispatch failed: cron external worker exited before ownership acknowledgement (exit 1); ...
ERROR cron.scheduler: Delivery failed for job <id>: Hermes could not read this profile's TELEGRAM_HOME_CHANNEL (an internal profile-scoping bug on the multiplexed gateway, not your configuration). Run `hermes gateway restart`; if it keeps happening, report it with `hermes debug share`.

Impact: this only affects gateways serving more than one profile (the default once a second profile exists), but there it undoes #124013 in the case it was added for. #123401 was a 9.5-hour worker outage with 106 failed runs and no notices; on a multiplexed gateway the same outage today still sends none for jobs that deliver to a home channel. The incident row and last_error are still recorded, so nothing is lost; it just isn't pushed to anyone.

run_one_job calls _deliver_crash_failure from the dispatch-failure branch, which returns before _run_one_job_body ever runs. _run_one_job_body is where the owning profile's secret scope gets installed for delivery. So the notice runs with no scope, and _home_env_lookup / _env_home_target_chat_id read the home channel through get_secret, which raises UnscopedSecretError when multiplex is on and no scope is set. The incident row is still written, but the notice itself never goes out, so the outage #123401 wanted to surface is still silent for any job that delivers to a home channel.

I hit this on a multiplexed gateway during the worker ModuleNotFoundError: ruamel outage (#125820 and friends): three dispatch failures in a row on a 5-minute job, each followed by the delivery error above.

The fix wraps the _deliver_crash_failure call in _install_fire_secret_scope() / _reset_fire_secret_scope(), the pair _run_one_job_body and record_unknown_worker_outcome (the post-handoff notice) already use. The post-handoff branch is unchanged.

Related Issue

Follow-up to #123401 / #124013. #126982 (merged 09-30) added the routed-fire helpers this version now uses; it didn't touch this branch. On main, record_unknown_worker_outcome already wraps the post-handoff notice the same way.

One narrower case this PR leaves alone: _run_one_job_body's own except path also calls _deliver_crash_failure, and an exception raised before that function installs its scope (claim_dispatch, mark_execution_running) would hit the same unscoped read. I haven't seen that in practice, so I kept this PR to the branch that fails in production.

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • cron/scheduler.py: in run_one_job's pre-handoff dispatch-failure branch, wrap _deliver_crash_failure in _install_fire_secret_scope() / _reset_fire_secret_scope().
  • tests/cron/test_restart_safe_worker.py: test_dispatch_failure_notice_resolves_the_owning_profiles_home_channel runs a failed handoff with multiplex active and the home channel only in the profile's .env, and checks the notice resolves it, the incident is recorded, the outcome is delivered, and no scope is left installed afterwards.

How to Test

  1. scripts/run_tests.sh tests/cron/test_restart_safe_worker.py
  2. On current main the new test fails and the log shows the same Hermes could not read this profile's TELEGRAM_HOME_CHANNEL line as production; with this change the file passes (35 passed).
  3. Negative controls: dropping the scope install fails the new test (the notice resolves nothing), and dropping the reset fails its no-leak check (current_secret_scope() is still set after the fire). Each control fails only the new test.

I also ran all of tests/cron plus the secret-scope tests: 163 files, 1645 passed, 3 skipped. I did not run the full pytest tests/ suite.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass (targeted suites only, see above)
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: Linux (kernel 7.0), Python 3.14

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings), or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys, or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows, or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide, or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior, or N/A

run_one_job's pre-handoff dispatch-failure branch (NousResearch#124013) calls
_deliver_crash_failure before _run_one_job_body, which is where the
firing profile's secret scope is installed for delivery. Under a
multiplexed gateway the notice then resolves the home channel through
get_secret with no scope, raises UnscopedSecretError, and never leaves.

Wrap the notice in _install_fire_secret_scope/_reset_fire_secret_scope,
the pair _run_one_job_body and record_unknown_worker_outcome already use.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cron Cron scheduler and job management area/profiles Multi-profile isolation, HERMES_HOME scoping sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Oct 1, 2026
kshitijk4poor added a commit that referenced this pull request Oct 3, 2026
…h-notice exit

_run_one_job_body's except branch sends _deliver_crash_failure when anything in
the body raises, but the firing profile's secret scope was only installed after
claim_dispatch and mark_execution_running. A raise from either sent the crash
notice with no scope: under multiplex get_secret fails closed
(UnscopedSecretError, "Hermes could not read this profile's
TELEGRAM_HOME_CHANNEL") and the notice never left. Same class as the
dispatch-failure branch fixed for #131585 (and called out in #130431's body).

Install the scope first thing in the try so every exit through the except is
scoped; the outer finally already resets it. The handoff path stays outside it
(#107692), and the external-worker payload path and the post-handoff notice
already run scoped.

Extends the #130431 test with a claim_dispatch-raising case.

Refs #131585
Co-authored-by: skyeyesec333 <skyeyesec333@users.noreply.github.com>
@kshitijk4poor

Copy link
Copy Markdown

Thanks @skyeyesec333 — this landed in #132446 (merged, 8d15ca4) with your commit kept as authored (c8815a5). We also moved the body's scope install ahead of claim_dispatch, the second case your description mentions, and wrapped the install/reset in a small context manager. Closing as salvaged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/profiles Multi-profile isolation, HERMES_HOME scoping comp/cron Cron scheduler and job management P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Cron dispatch-failure notice still isn't delivered on multi-profile gateways (#123401 follow-up)

3 participants