fix: restore ready-for-work label when a dispatched job fails after exhausting retries - #478
Conversation
…ries (#463) Queue::fail now reports whether a failure was terminal (retries exhausted) vs a requeue. WorkerPool gains a terminal_failure_hook fired from run_in_process on that terminal case; the daemon wires it to dashboard::orphan_reconcile::handle_terminal_job_failure, which swaps the item off dispatched onto needs-manual-dispatch (or ready-for-work as a fallback) -- the clean-failure counterpart to reconcile_orphaned_jobs, which only ever covered a job whose process crashed mid-flight. Previously an item whose dispatched job failed for an ordinary reason (no billing balance, transient network error) stayed labeled dispatched forever, invisible to every future discovery tick. Agentflare-Agent: claude-code Agentflare-Branch: task/463-failed-jobs-retries-exhausted-leave-item Agentflare-Item: 463
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 53 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (7)
Comment |
getappz
changed the title
dispatched, never restored to ready-for-work — silently undispatchable forever
…d judge replies (#516) The SDD-loop judge parser found the first '{' and last '}' in the whole reply and parsed the slice between them. Reproduced live (item #478, 2026-08-15): trailing commentary containing its own unrelated braces, or a JSON object missing a required field, either corrupted the extracted span or hard-failed the step. Prefer an explicit fenced code block (```json ... ```) when present, and otherwise scan forward from the first '{' for its own matching '}' (tracking nesting depth and skipping brace-like bytes inside string literals) instead of blindly spanning to the last '}' anywhere in the reply. A genuine parse failure -- including syntactically valid JSON missing a field -- still returns Err, retried by the step's own RetryPolicy as before. Agentflare-Agent: claude-code_2-1-233_agent Agentflare-Branch: fix/harden-judge-decision-parser
…wnstream (#546) real_agent_send_hook fed every role's raw --output-format stream-json transcript (one JSON object per line: system init, tool_use, ..., final result) straight into role_reply, since #498 deleted the old coder step's transcript parsing without giving sdd_loop's shared hook an equivalent. For the judge specifically, parse_judge_decision then parsed the transcript's first line -- a valid-but-action-less system/init event -- instead of the judge's actual decision on the last line, hard-failing every judge turn with "missing field `action`" (items #478/#502/#503). Restore transcript parsing (parse_claude_reply, agent_launch.rs) and apply it to every Claude Code role's reply via clean_agent_reply, not just the judge's, since role_reply also gets embedded verbatim into build_judge_prompt. Agentflare-Agent: claude-code Agentflare-Branch: task/489-sdd-loop-judge-reply-deterministically-m Agentflare-Item: 489
1 participant
Committed as
635d1c8on the current branchtask/463-failed-jobs-retries-exhausted-leave-item.Summary
Fixed item #463: items whose dispatched job cleanly failed after exhausting
max_retriesstayed labeleddispatchedforever, invisible to every future discovery tick.Root cause:
agentflare_jobs::Queue::fail()decides retry-vs-terminal but never told any caller which happened.orphan_reconcile.rs's label-restore logic only fired for crashed/orphaned jobs (state = 'running'at daemon startup), never for a job that ran to completion and cleanly failed.Fix:
Queue::fail()now returnsResult<bool, Error>—truewhen the failure was terminal.WorkerPoolgainswith_terminal_failure_hook(...);run_in_processinvokes it exactly once, only on a genuine terminal failure (verified by a new test asserting it fires once after 2 retries, not on either retry).dashboard/server.rs) wires that hook to a newdashboard::orphan_reconcile::handle_terminal_job_failure, which removes thedispatchedlabel and addsneeds-manual-dispatch(falling back toready-for-workif that label was never created for the project) — avoiding a bare requeue that would just retry-loop against the same broken agent (e.g. the reproduced billing-balance failure).NEEDS_MANUAL_LABELpub(crate)insupervisor.rsas the shared source of truth.Added tests covering: the label swap to
needs-manual-dispatch, the fallback toready-for-work, a no-op for non-in_processjobs, and theWorkerPoolhook firing correctly through the real retry/terminal-failure flow. All new and pre-existing tests pass;cargo fmtclean.