Skip to content

fix(cron): add spend-neutral provider reset backoff - #102073

Open
jayleaton wants to merge 1 commit into
NousResearch:mainfrom
jayleaton:wt/t_a545d3a5
Open

jayleaton wants to merge 1 commit into
NousResearch:mainfrom
jayleaton:wt/t_a545d3a5

Conversation

@jayleaton

@jayleaton jayleaton commented Sep 3, 2026 •

Copy link
Copy Markdown

Summary

  • parse provider reset hints and persist deterministic bounded backoff after HTTP 429 failures
  • suppress unattended fires until backoff expiry while preserving durable failure state and automatic resume
  • keep explicit manual runs safe with a dedicated atomic backoff override, including dashboard, background, legacy-provider, and concurrent-pause paths
  • make Chronos arm and reconcile one-shots at the effective backoff expiry

Test plan

  • HERMES_TEST_FILE_RETRIES=0 scripts/run_tests.sh tests/cron/test_provider_rate_limit_backoff.py tests/cron/test_scheduler_provider.py tests/plugins/test_chronos_cron.py tests/tools/test_cronjob_run_immediate.py tests/tools/test_cronjob_run_background.py tests/cron/test_ticker_stall_60703.py tests/hermes_cli/test_web_server_cron_profiles.py -q (142 passed)
  • python -m compileall -q cron/jobs.py cron/rate_limit_backoff.py cron/scheduler.py cron/scheduler_provider.py hermes_cli/web_server.py plugins/cron_providers/chronos/__init__.py tools/cronjob_tools.py
  • git diff --cached --check

Operational notes

  • Live job dabcf70cf427 (eng-completion) remains active on its unchanged every 10m schedule.
  • Durable history confirms recovery after the task's original terminal 429 (2336eaa607544e3ba33fae88e5b76def failed, followed by 29282100a7c549b7b7464c73ad794f81 completed).
  • Its latest provider reset is still in the future; the candidate parses the live error to a bounded reset time, while persistence/restart/automatic-resume behavior is covered deterministically in tests.
  • No model, provider, reasoning, or X-job configuration was changed.

Do not merge from this task; review only.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cron Cron scheduler and job management comp/cli CLI entry point, hermes_cli/, setup wizard comp/plugins Plugin system and bundled plugins sweeper:risk-automation Sweeper risk: may affect CI, automerge, label sync, or maintainer automation labels Sep 3, 2026
@jayleaton

Copy link
Copy Markdown
Author

@Teknium Ready for review. Exact candidate f40e397 is deployed to the live default gateway; the active eng-completion job suppressed its otherwise-due unattended fire during the persisted provider reset window without changing cadence/model/provider, and all 142 focused tests pass. Please do not merge until review is complete.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management comp/plugins Plugin system and bundled plugins P2 Medium — degraded but workaround exists sweeper:risk-automation Sweeper risk: may affect CI, automerge, label sync, or maintainer automation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants