AI runs: the cron's claim pass honours the attempt ceiling — a queued run with no attempts left is abandoned, not re-claimed for ever (Q40, second half; migration 0479)

[NewWorldVenture]

Stacked on #901 (→ #900). Migration number 0479 reserved by the coordinator on #699 (comment 5970781386); file and scope announced before writing (5970918142).

The defect (finalaudit Q40, "the two claim paths disagree")

family_automation_runs.attempt is the run's abandonment budget. Three readers agree about it: the recovery arm of claim_ai_runs dead-letters an executing run whose lease ran out once attempt >= max_attempts ("Run abandoned after the maximum number of attempts."); claimRun refuses to lease a run at that ceiling; and since #901 the executor hands the budget back only when a slice made progress, so the counter is the number of slices in a row that completed nothing.

The claiming half of claim_ai_runs never read it. Its candidates were every due ready / scheduled_followup run, and each claim added one. A run parked back in the queue with no attempts left was therefore claimed by every tick for ever (attempt 6, 7, 8 …) while the resume button, the kick after an approval answer and a step re-run all refused it, silently. The dead-letter arm never reached it, because it ran only for an expired lease on an executing row, and a parked run holds no lease. Unreachable by people, unkillable by the worker.

The fix

supabase/migrations/0479_a_run_with_no_attempts_left_is_abandoned_not_reclaimed.sql — one create or replace function public.claim_ai_runs(integer, integer), in 0263's shape:

• the recovery statement gains a second arm: a ready / scheduled_followup run with no live lease, no cancel in flight and attempt >= max_attempts takes the existing failed arm (state, 0022's legacy status, error, completed_at) and 0263's reconcile (steps, request ledger, run_failed event), so the family is told instead of being shown "Scheduled" for ever. A run whose cancel is in flight is left to end cancelled; a follow-up not yet due is abandoned now rather than on the day, since it would never have run;
• the candidates query gains and attempt < max_attempts, the same ceiling claimRun applies to one row;
• (second commit) the reconcile gains a fourth bounded write, for both dead-letter arms: the dead runs' approvals still pending become cancelled (as cancelRun closes them; not expired, time did not run out) with decided_at set and no decider, found by run_id and by the step whose approval_id names them (0251 lets run_id be null). 0263 left them open, so a run that died with an approval in flight kept a card in "Needs your decision" until expires_at, and a parent who decided it approved work that would never start: foldIntoRun never re-opens a terminal run.

Unchanged: the expired-lease arms, the lease token's type, the increment, the ordering, for update skip locked, the grants (0263's revoke/grant pair restated byte for byte). Depends on 0250, 0263, 0253, 0292 (and 0093/0251 for approval_requests); nothing in 0475–0478 is referenced. tests/migration-version-safety.test.ts pin → 0480.

app/api/cron/ai-runs/route.ts, lib/ai/runs/store.ts — with the ceiling enforced, an unexecuted claim must not spend the budget. The tick claims a wave of four and gives each a slice only while 12 s remain; a run it never reached was handed back ready but kept the attempt the claim had charged. Five such hand-backs under a backlog would have abandoned a run that was never tried. claimRuns now reports the post-claim attempt, and the hand-back returns it (Math.max(0, attempt - 1)), which also makes the "exactly as it was found" comment true. The write's row-count gate (C1-S9-63) is untouched.

Tests and probes

• docs/audit/a-run-with-no-attempts-left-is-abandoned-check.sql (new, globbed by run-probes.sh): five runs and three pending approvals, one call, one family, rolled back. A due ready run at 5/5 is not claimed and is abandoned with its step failed, its request failed, exactly one run_failed event and both of its approvals (one by run_id, one reachable only through the step) cancelled with decided_at and no decided_by; a scheduled_followup at 5/5 due in three days is abandoned too; a run at 5/5 with a cancel in flight and one holding a live lease are untouched; the control at 4/5 is claimed (executing, attempt 5, live lease) with its step, request, timeline and approval untouched. Setup is checked before the call.
• tests/helpers/claim-ai-runs.ts (new): the predicate-for-predicate in-memory stand-in for the function, moved out of a-run-that-threw-is-not-stranded.test.ts and given the ceiling, plus claimAiRunsMigration() which finds the current definition by name so a renumbering at merge time cannot break the pins.
• tests/a-run-with-no-attempts-left-is-abandoned.test.ts (new, 17 cases) drives the real cron route and the real claimRun: five stalling slices (three per tick, as the route's own loop allows) then the next claim abandons instead of claiming; a progressing slice in the middle resets the count so abandonment comes after the tenth slice, not the fifth; what the cron abandons, claimRun had already refused, and the refusal now carries a reason; the not-yet-due follow-up; cancel in flight / live lease left alone; one below the ceiling gets its slice (finishes if it finishes; abandoned only after a fifth stall); a run claimed in a wave but handed back unexecuted is not charged. Source pins on the migration (both arms, the ceiling, the reconcile bounded to v_dead four times, the approvals write's shape, the grants) and on the probe's control.
• tests/a-successful-slice-must-not-spend-a-retry.test.ts: the "two claim paths still disagree" pin now pins that they agree, keeping 0250's original as the recorded premise.
• Probe headers for 0263 (dead-letter-reconcile-check.sql) and the RPC grants name 0479 as the function's current writer.

Evidence (local PostgreSQL 16.13 harness, disposable)

• The new probe fails against 0263's body (a ready run with no attempts left was claimed again) and passes against 0479's, in a rolled-back transaction; applied twice (reapply) it still passes; the two existing claim_ai_runs probes (0263 dead-letter, stranded run) and the RPC-grants probe pass against the new body.
• Second commit: the extended probe passes with the approvals write and, against the first commit's body, fails at the abandoned run's approval (by run_id) was left pending.
• Concurrency: with the function committed on the harness, 30 committed rows (20 due at 0/5, 10 due at 5/5) and three concurrent callers: 20 claimed with 20 distinct lease owners, 10 abandoned, 0 left ready, max attempt 5, exactly one run_failed event per abandoned run, 10 requests failed / 20 still executing. No double claim, no double dead-letter.
• 46 related test files / 678 tests pass (incl. the version pin); tsc --noEmit and ESLint clean. Second commit: 12 suites / 167 cases, eslint clean.
• Mutation controls (each reverted): hand-back keeps the charge → 1 case fails; stand-in without the ceiling and the queue arm → 5 fail; migration without the queue arm and the candidates ceiling → 3 fail (source pins).

Not done here

• Nothing applied anywhere but the local disposable database; production's ledger still stops at 0177 (PROD-DB-0177), and no apply, ledger stamping, credential or deployment action is taken or authorised by this PR.
• Steps left awaiting_approval under an abandoned run keep that state, exactly as 0263 leaves them for a dead worker; widening that list is a separate decision. Their approval is now closed, so nothing asks a parent to decide for them.
• Retarget to claude/ai-runs-release-recoverable-20261003 when AI runs: a slice that made progress no longer spends the run's abandonment budget (Q40, first half) #901 merges, and to main when AI runs: a run whose slice threw is handed to the recovery pass, not stranded (LIBRARY-3E7235F73734, API-9BCA8F281320) #900 does.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KP9rt5rVQ9jDMpZp2xBy3K

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
bubaly	Ignored		Oct 4, 2026 7:32am UTC