feat(cron): per-job max_turns override + record budget stops as PARTIAL

[colingreig]

What does this PR do?

Two related changes to run_job in cron/scheduler.py, plus the plumbing to make the new field settable through the normal API.

1. Per-job max_turns override (feature)
max_iterations now resolves the turn budget in priority order: job.max_turns (a positive int field on the job record) → agent.max_turns config → max_turns config → 90. Long-running worker jobs need a larger turn budget than the default without raising the global limit for every other cron job. The field is settable through the normal API — create_job() gains a max_turns parameter and the cronjob tool exposes it on create/update. Because bool is an int subclass, it's excluded explicitly so max_turns: true falls back to the default rather than silently capping a run at one turn.

2. Budget stops recorded as partial, not hard failure (bug fix)
Previously, any result with completed is False was wrapped in RuntimeError, so a job that hit its iteration/turn budget but still produced a substantive multi-KB handoff report was stored as last_status=failed with the entire report buried in last_error — indistinguishable from a turn-1 crash. This is the inverse over-correction of the mis-reporting fixed in #17855.

Now, when turn_exit_reason starts with max_iterations_reached or budget_exhausted, failed is not True, and the final response is ≥ 300 chars (to reject the short boilerplate finalize_turn injects when the post-budget summary call itself fails), the job is recorded as a success with a ⚠️ PARTIAL — iteration budget hit banner prepended to the report, which then flows through the normal success path. Genuine failures and empty/short results still raise as before.

Note: this records the partial run through the existing success path, so last_status becomes ok (the banner in the delivered/saved report is the signal). A distinct partial status value (for separate TUI rendering) would be a reasonable follow-up but is intentionally out of scope here to keep the change focused.

Related Issue

Related: #17855 (follow-on). That fix stopped API failures being mis-reported as last_status=ok; this handles the inverse, where legitimate partial work was mis-reported as total failure. #17855 is closed, so this is not a Fixes. No open issue currently tracks this.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)

Changes Made

• cron/scheduler.py — run_job: resolve max_iterations from a positive non-bool job.max_turns before config/default; route max_iterations_reached/budget_exhausted stops with a ≥ 300-char report to partial success instead of RuntimeError.
• cron/jobs.py — create_job(): optional max_turns parameter, stored on the record only when a positive non-bool int (omitted otherwise, so existing records are unchanged).
• tools/cronjob_tools.py — cronjob tool: max_turns parameter threaded to create/update + a max_turns property in the tool schema.
• tests/cron/test_run_job_partial_budget.py — new: partial-success path, the ≥ 300-char floor, genuine-failure passthrough, and the max_turns override (incl. bool/zero/negative/string guards).
• tests/cron/test_jobs.py — create_job max_turns persistence/validation.
• tests/tools/test_cronjob_tools.py — tool create/update max_turns passthrough.

How to Test

1. New coverage: pytest tests/cron/test_run_job_partial_budget.py tests/cron/test_jobs.py tests/tools/test_cronjob_tools.py -q
2. Cron regression: pytest tests/cron/ -q
3. Manual: create a job with a low max_turns (e.g. 3) and a task needing more turns. The run is stored as last_status=ok with the report readable (prefixed ⚠️ PARTIAL — iteration budget hit) instead of last_status=failed with the report buried in last_error.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass — ran the relevant suites instead: tests/cron/ + tests/tools/test_cronjob_tools.py = 527 passed. Did not run the full tests/ locally (unrelated collection errors from optional deps not installed in my env, e.g. mcp); CI covers the full suite.
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: macOS 15 (Darwin 25.3.0), Python 3.11

Documentation & Housekeeping

• I've updated relevant documentation (docstrings on create_job + tool schema description) — or N/A
• I've updated cli-config.yaml.example if I added/changed config keys — N/A (max_turns is a per-job record field, not a top-level config key)
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
• I've considered cross-platform impact — N/A (string-prefix and integer comparisons only; no platform-specific code)

[tonydwb]

Code Review Summary

Verdict: Approved

Clean, well-scoped fix/feature with comprehensive tests. No issues found.

• Logic is correct and focused
• Tests cover the new behavior
• No security concerns
• Good error handling

---

Reviewed by Hermes Agent

[liuhao1024]

Verified the per-job max_turns override and partial-success budget recording.

1. bool exclusion: isinstance(max_turns, int) and not isinstance(max_turns, bool) is the correct guard — True == 1 and False == 0 would silently set 1-turn or 0-turn budgets without it. All four code paths (create_job, update, run_job resolution, tool schema) use the same validation.

2. Precedence chain: _job_max_turns → _cfg.agent.max_turns → _cfg.max_turns → 90 — job-level override correctly wins over all global config sources. The or chain with None from invalid values falls through as expected.

3. Partial-success logic: failed is not True AND exit_reason starts with "max_iterations_reached"/"budget_exhausted" AND final_response >= 300 chars — the three-way guard correctly separates genuine failures from budget-exhausted partial work. The 300-char floor rejects finalize_turn's short completion boilerplate while preserving multi-KB handoff briefs.

4. Backward compatibility: Jobs without max_turns field are unaffected — job.get("max_turns") returns None, validation returns False, falls through to config/default. No migration needed.

5. Test coverage: test_run_job_partial_budget.py covers all four cases (partial success, short-report failure, genuine failure with failed=True, per-job override precedence) with controlled agent result dicts. Thorough.

Clean implementation.

[andrewjocom]

Great fix for the inverse of #17855. We hit this exact mis-reporting in production and went one step further than this PR — sharing in case it informs a follow-up.

In our deployment, a 23:30 daily-summary LLM cron (max_iterations=45) repeatedly starved its final required step (an mcp_obsidian_mem_remember_text KG-persistence call sequenced last). When earlier steps burned the budget, the run hit max_iterations_reached(45/45), produced a fallback summary, and — under the current code path this PR also routes through the success path — was stored as last_status=ok. The summary delivered fine (good UX), but the green ok lied: the KG write never happened and the operator had no signal. We only caught it by grepping the agent log for the missing tool call.

That's why we'd argue for the "reasonable follow-up" the PR notes as out of scope: a distinct partial status value, not ok + a banner. Reasons from the field:

• last_status is consumed by monitoring/ops dashboards and the cronjob run tool's own success flag. A banner buried in the delivered report is invisible to anything that branches on last_status. Routing through the success path means every consumer now treats a starved-but-summarised run as fully healthy.
• The length gate (≥300 chars) rejects short boilerplate, but it can't distinguish "complete run" from "ran out of budget mid-pipeline with required steps skipped." Both look like success.

Our local implementation (4 files, applied + verified live):

• run_job() threads a partial bool through a 5-tuple return (success, output, final_response, error, partial).
• mark_job_run(..., partial=False) sets last_status="partial" (and a truthful last_error) when partial, while keeping success=True so delivery + recurring cadence are unaffected.
• cron list renders partial in amber; the manual-run tool treats partial as success (output was delivered).

This keeps your #17855 guarantee (genuine failures still error) and adds a third, honest state instead of overloading ok.

Happy to open a follow-up PR or extract our scheduler.py/jobs.py diff if you want to fold a distinct partial status into this work.

[teknium1]

Thanks for the focused per-job budget work. The per-job max_turns capability is still absent on current main (cron/scheduler.py:2899), so that portion remains useful.

Problems

• The new PARTIAL branch changes only response text. run_one_job() will still call mark_job_run(..., success=True) (cron/scheduler.py:3452, :3531-3533), and mark_job_run() persists every success as last_status="ok" (cron/jobs.py:1485). The immediate-run tool also treats only "ok" as successful (tools/cronjob_tools.py:643). A partial run would therefore remain falsely healthy to status consumers, as the production report in this discussion describes.
• Current main already delivers non-empty max_iterations_reached(...) fallback reports (cron/scheduler.py:3226-3246; regression test tests/cron/test_scheduler.py:1489-1534, added by ae7e85742). Please reconcile with that implementation rather than replacing the older branch wholesale.

Suggested changes

• Preserve the per-job override, but thread an explicit partial outcome through run_job → run_one_job → mark_job_run, then update status consumers and add a real-path temporary-HERMES_HOME regression test.

Automated hermes-sweeper review.

[teknium1]

This only annotates response text. run_job() still returns success=True, so mark_job_run() persists last_status="ok" and the immediate-run tool reports the run healthy. Please thread an explicit partial outcome to storage and consumers instead of using the banner as the sole signal.