fix(cron): fail jobs when pre-run script fails instead of greenwashing as ok - #20326
Closed
konsisumer wants to merge 1 commit into
Closed
fix(cron): fail jobs when pre-run script fails instead of greenwashing as ok#20326konsisumer wants to merge 1 commit into
konsisumer wants to merge 1 commit into
Conversation
…g as ok When a cron job's pre-run script returned non-zero (or timed out), run_job captured the error, injected it into the agent prompt, and let the LLM generate a response — which then marked the job last_status=ok. Operators saw "ok" while the data-collection step had crashed and the agent was fabricating output from a polluted prompt. Short-circuit run_job at the same point the no_agent path already does: on script failure, return success=False with the script output as the error and a "pre-run script failed" status in the job doc. mark_job_run already records this as last_status=error, and _process_job already delivers the error to the user. Fixes NousResearch#20301
alt-glitch
added
type/bug
Collaborator
Contributor
Author
|
Closing — deferring to #20323 by @liuhao1024 which addresses the same. Reopen if that PR stalls. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stop marking cron jobs as
last_status=okwhen their pre-runscriptfailed.What changed and why
cron/scheduler.py::run_jobnow short-circuits on pre-run script failure — when_run_job_scriptreturnssuccess=False, the job exits withsuccess=False, the script output as the error, and a "pre-run script failed" status doc, instead of injecting the error into an agent prompt and burning an LLM call.mark_job_runalready recordslast_status=errorforsuccess=False, and_process_jobalready delivers the failure message to the configured target.no_agentpath's failed-script handling (cron/scheduler.py:912-929), so both paths now behave consistently: a broken pre-run script always surfaces aserror, never asok.TestRunJobWakeGate.test_script_failure_does_not_trigger_gate(which previously asserted the old greenwashing behavior — "agent DID wake despite the gate-like text") to assert the new contract: agent is not invoked,success is False, and the script error is captured. The gate-bypass safety property the original test cared about (don't honorwakeAgent: falsefrom a failed script's stderr) is still upheld — we just fail-fast instead of waking the agent.How to test
pytest tests/cron/ -q— all 318 cron tests pass (one pre-existing failure ontest_script_empty_output_notedreproduces on main, unrelated to this fix).pytest tests/run_agent/test_exit_cleanup_interrupt.py -q— passes (these tests patch_build_job_promptand exerciserun_jobcleanup paths).scriptpointing to a script thatexit 1s, run a tick, and checkcron list—last_statusis nowerrorwith the script's stderr inlast_error.What platforms tested on
Fixes #20301