feat(cron): retry-on-failure for agent jobs (configurable, opt-in)

[girnarholdings]

What

Adds an optional per-job retry config so agent jobs (script: null, run inside run_conversation) retry on transient failures (LLM timeout, rate-limit, network blip) instead of waiting hours for the next cron tick.

"retry": {"max_attempts": 2, "delay_seconds": 300}

On failure, the scheduler re-arms next_run_at to now + delay_seconds (up to max_attempts times) before reverting to the cron schedule. The normal ticker picks up the retried job via its existing next_run_at <= now due-check.

Why

Agent briefing jobs (e.g. BoltNews pre/post/mid-day, earnings daily pipeline) fail transiently on DeepSeek API timeouts. Currently a single transient failure = no briefing that cycle, and mark_job_run re-arms next_run_at to the next cron tick (hours away). The bash retry wrapper shipped for no_agent summarizer jobs cannot wrap an LLM session — this closes that gap at the scheduler layer.

Design (minimal, rides existing mechanisms)

cron/jobs.py — schema + reset:

• create_job accepts retry: {max_attempts, delay_seconds} (validated; defaults None = no retry).
• Persists retry + retry_count: 0 in the job dict.
• mark_job_run resets retry_count = 0 on success.

cron/scheduler.py — _record_job_outcome helper:

• Wraps both mark_job_run call sites in run_one_job (normal + exception paths).
• Always records the honest outcome first, then — if retry configured, run failed, attempts remain — post-overrides next_run_at = now + delay and bumps retry_count. The override runs after mark_job_run because its own next_run_at = compute_next_run(...) would otherwise clobber the retry time.
• The _consume_interrupted_flag guard stays outside the helper so an interrupted run is never retried.

No changes to: get_due_jobs, compute_next_run, the ticker loop, claim_dispatch. The retry rides the existing due-check.

Opt-in / non-regression

Jobs without a retry field hit the exact same code path as today (_record_job_outcome calls mark_job_run and returns — the retry branch is gated on job.get("retry")). The test_no_retry_config_failure_preserves_behavior test pins this.

Full tests/cron/ suite: 668 passed, 9 pre-existing env failures (croniter-missing in the base Python + gateway/fire-claim env tests) — identical to clean main. CI has croniter==6.0.0 installed (core dep in pyproject.toml), so most of those 9 pass there.

Tests (7 new in TestJobRetry)

• test_create_job_with_retry_config — config persisted, retry_count=0
• test_create_job_without_retry_defaults_none — back-compat
• test_invalid_retry_max_attempts_rejected / test_invalid_retry_delay_rejected — validation
• test_retry_re_arms_next_run_at_on_failure — next_run_at = now+delay, retry_count=1
• test_retry_resets_on_success — recovery clears the counter
• test_no_retry_config_failure_preserves_behavior — non-regression guard

Test plan

• CI ci.yml orchestrator + python lint/test sub-workflows pass
• all-checks-pass gate green