Skip to content

fix(cron): don't attribute no_agent script failures to a provider - #77648

Closed
Yanir-R wants to merge 1 commit into
NousResearch:mainfrom
Yanir-R:fix/cron-no-agent-failure-attribution
Closed

fix(cron): don't attribute no_agent script failures to a provider#77648
Yanir-R wants to merge 1 commit into
NousResearch:mainfrom
Yanir-R:fix/cron-no-agent-failure-attribution

Conversation

@Yanir-R

@Yanir-R Yanir-R commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Problem

_summarize_cron_failure_for_delivery in cron/scheduler.py classifies a failed cron job by substring-matching the error prose"timed out", "429", authenticat|authoriz — and maps any hit onto a provider-shaped explanation. It never consults the job's execution mode, which is present in the same job dict it is passed.

A no_agent job is its script: run_job short-circuits it before any model is reached ("no LLM involvement"). Provider timeouts, rate limits, auth errors and fallback chains are therefore structurally impossible for such a job — yet those branches are tested first.

Reproduction

_run_job_script reports a timeout as f"Script timed out after {script_timeout}s: {path}". That string contains "timed out", so a shell script exceeding its timeout is delivered to chat as:

⚠️ Cron 'x' failed: provider timeout. Fallback chain was exhausted or unavailable.

for a job that never opened a socket. The operator is sent to inspect model routing and fallback configuration while the actual fault is a shell script. "429" or "authentication" appearing anywhere in a script's output misfires the same way.

from cron.scheduler import _summarize_cron_failure_for_delivery as f
job = {"name": "nightly-job", "no_agent": True, "script": "nightly.sh"}
f(job, "Script timed out after 900s: /home/u/.hermes/scripts/nightly.sh")
# -> "⚠️ Cron 'nightly-job' failed: provider timeout. Fallback chain was
#     exhausted or unavailable. Full details saved in cron output."

Fix

Gate the three provider branches on not job.get("no_agent") and let script jobs fall through to the existing generic cleaner, which already reports the real error and names the script. No new message text — the correct output already existed on the fall-through path:

⚠️ Cron 'nightly-job' failed: Script timed out after 900s: /home/u/.hermes/scripts/nightly.sh

Agent-mode jobs are unaffected.

The auth branch already carries a word-boundary guard so "oauth" and "4015" do not trip it, which addresses one substring false-positive; gating on mode removes the remaining class for script jobs.

Tests

The summarizer had no direct coverage — the only test referencing it (tests/cron/test_shutdown_interrupt.py) patches it out and asserts on its call arguments, so the classification logic itself was never exercised.

Adds parametrized cases to tests/cron/test_cron_no_agent.py pinning both directions:

  • script jobs are never blamed on a provider, including when their own output contains "429", "authentication" or "ReadTimeout"
  • agent-mode jobs keep the existing provider timeout / provider rate limit / provider authentication error summaries unchanged

Notes

No change to what data reaches the delivery channel beyond routing these errors to the existing generic path: script stdout/stderr already passes through redact_sensitive_text, and the timeout error is a fixed string plus the script path.

`_summarize_cron_failure_for_delivery` classifies a failed job by
substring-matching the error prose — "timed out", "429",
`authenticat|authoriz` — and maps any hit onto a provider-shaped
explanation, without consulting the job's execution mode.

A `no_agent` job IS its script: `run_job` short-circuits it before any
model is reached. Provider timeouts, rate limits, auth errors and
fallback chains are therefore structurally impossible for it, yet those
branches are tested first.

`_run_job_script` reports a timeout as "Script timed out after {n}s:
{path}". That contains "timed out", so a shell script exceeding its
timeout is delivered to chat as:

  ⚠️ Cron 'x' failed: provider timeout. Fallback chain was exhausted
    or unavailable.

for a job that never opened a socket, sending the reader to inspect
model routing while the actual fault is a shell script. "429" or
"authentication" appearing anywhere in a script's output misfires the
same way.

Gate the three provider branches on `not job.get("no_agent")` and let
script jobs fall through to the existing generic cleaner, which already
reports the real error and names the script. No new message text.

The auth branch carries a word-boundary guard so "oauth" and "4015" do
not trip it, which addresses one substring false-positive; gating on
mode removes the remaining class for script jobs.

Tests: the summarizer had no direct coverage — the only test referencing
it patches it out and asserts on its arguments. Adds parametrized cases
pinning both directions: script jobs are never blamed on a provider
(including when their output contains "429" or "authentication"), and
agent-mode jobs keep the existing provider summaries unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cron Cron scheduler and job management duplicate This issue or pull request already exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Aug 3, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Duplicate of #70977 — same fix at the same site: gate the provider-classification branches in _summarize_cron_failure_for_delivery on not job.get('no_agent'). Related: #60593 (alternate no_agent summary branch), #59379 (broader script-timeout family). A maintainer should pick the canonical PR.

@Ruanjq98

Ruanjq98 commented Aug 3, 2026

Copy link
Copy Markdown

Code Review: #77648

Verdict: Approve

File symlink normalization security: adds path resolution checks. Good hardening.

LGTM - Reviewed diff. Changes are sound.

teknium1 added a commit that referenced this pull request Aug 13, 2026
…composed no_agent gate

The cherry-picked tests predate #85508's honest fallback-chain phrasing
and each other: assertions pinned the old 'exhausted or unavailable'
literal and #83188's no_agent fallback-note behavior, which #77648's
mode gate supersedes (no provider classification at all for no_agent
jobs). Assert the composed contract instead.
@teknium1

Copy link
Copy Markdown
Contributor

Merged via PR #85536 (rebase-merge) — your commit was cherry-picked onto current main with your authorship preserved in git log.

Yours was the earliest of three independent fixes for this bug (#79451, #81629 followed), and the mode-gate approach with the strongest test coverage — the parametrized "script prose must never pick the blamed subsystem" cases landed as you wrote them. Thanks!

bobaba76 pushed a commit to bobaba76/hermes-agent that referenced this pull request Aug 27, 2026
…composed no_agent gate

The cherry-picked tests predate NousResearch#85508's honest fallback-chain phrasing
and each other: assertions pinned the old 'exhausted or unavailable'
literal and NousResearch#83188's no_agent fallback-note behavior, which NousResearch#77648's
mode gate supersedes (no provider classification at all for no_agent
jobs). Assert the composed contract instead.
melon-xf added a commit to melon-xf/hermes-agent that referenced this pull request Sep 3, 2026
…composed no_agent gate

The cherry-picked tests predate NousResearch#85508's honest fallback-chain phrasing
and each other: assertions pinned the old 'exhausted or unavailable'
literal and NousResearch#83188's no_agent fallback-note behavior, which NousResearch#77648's
mode gate supersedes (no provider classification at all for no_agent
jobs). Assert the composed contract instead.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management duplicate This issue or pull request already exists P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants