AI runs: a run a person asks for again gets its attempt budget back (Q40 follow-up)

[NewWorldVenture]

Stacked on #904 (base claude/ai-runs-claim-ceiling-20261003) for the claim-pass stand-in in tests/helpers/claim-ai-runs.ts. Retarget as the stack lands. No migration.

What was wrong

attempt is the run's abandonment budget. At max_attempts the recovery arm of claim_ai_runs dead-letters the run (#904) and claimRun refuses to lease it (already on main). Four writes put a run back to ready because a member of the family asked for it:

• resumeRun, rerunStep, editStepInput in lib/ai/runs/controls.ts
• the decision that returns a run to the queue in lib/services/approvals/index.ts

Each one tells the person Bubaly will pick the run up. None touched attempt. For a run that had spent its budget that promise failed twice: the kick after the control went through claimRun, which answered claimed: false without a word, and the next cron tick abandoned the run again. "Try that step again" on an abandoned run produced a run abandoned again, with nothing a person could do about it. An approval granted to a run parked at the ceiling was consumed by a run that then died.

Change

• freshBudget() in lib/ai/runs/store.ts returns { attempt: 0 }. The four writes above spread it. A human asking again is a new budget, the way a slice that made progress is (budgetReset in the executor).
• The Q40 pin in tests/a-successful-slice-must-not-spend-a-retry.test.ts that said the executor's park was the only reset now names both helpers and still forbids a bare attempt: 0 in the intake, the kick, the cron route, the controls and the approvals service.

Second commit — review fix (a66ac1269, review 5973301004)

The four re-entry writes went through the plain updateRun, which matches on id and family alone, so two of them holding the same snapshot both landed: two final approval folds on one run — the first puts it in the queue, a worker claims it (executing, attempt charged, a live lease), and the second, still holding the snapshot that said awaiting_approval, restored ready, zeroed the attempt and cleared the worker's live lease. resumeRun and rerunStep wrote from their snapshots the same way. The lease fence #901 gave the worker does not reach a write a person's control makes.

• store.updateRunAsObserved(scope, runId, observed, patch) — compare-and-set on state, attempt and lease_owner as the caller observed them (.is('lease_owner', null) when the snapshot had none). Not the state alone: a run can be executing before and after under two different workers.
• foldIntoRun, resumeRun, rerunStep, editStepInput write through it. Zero rows is "someone else moved this run first": the fold has nothing left to do (its released steps are in the graph the running slice reloads every pass) and sends no kick; resume reports the run is not paused; re-run and edit say the run changed and nothing was overwritten (the re-queued step is the claim's to pick up; the saved plan version stays unreferenced). The source case in the attempts-back test counts the four observed writes and pins that no bare ready write remains.
• tests/a-stale-fold-cannot-re-arm-a-claimed-run.test.ts (6) — the in-memory client wrapped so the first run read hands the re-entry its snapshot and, before it can act, runs the other party (the real decide and claimRun over the same tables): the two-fold/claim race (B's step released, the worker's claim/attempt/lease untouched, one kick in total); a fold whose snapshot holds lands with the fresh budget; re-run a step against a claim; re-run a step on an executing run whose slice parked and was re-claimed meanwhile (same state, different attempt and lease); resume against another tab's resume plus a claim; a control whose snapshot holds lands.

Tests

• New tests/a-run-a-person-asks-for-again-gets-its-attempts-back.test.ts (5 cases). Real controls, real claimRun and the claim-pass stand-in against the in-memory client: a step re-run on an abandoned run is claimed by the kick and taken by the claim pass instead of being dead-lettered; a resume from paused at the ceiling is claimable; the control row at the ceiling with no person asking is refused by the kick and abandoned by the tick, which is what a person's run met before; a source pin that every ready write in both files carries the fresh budget (and, since the second commit, is the observed compare-and-set).
• tests/run-controls.test.ts: the three payload assertions gain attempt: 0.
• tests/approval-execution.test.ts: the approve case now parks the run at the ceiling and asserts the budget is back.

Evidence

• First commit: mutation controls — dropping freshBudget from rerunStep fails 4 cases; dropping it from the decision fails 2. Typecheck and eslint clean. The 15 suites that scan these source files or the tests directory pass (316 cases).
• Second commit: mutation controls (files restored and checksummed) — the fold back on updateRun 1 failure; the controls back on updateRun 2; the compare on state alone 1. 78 run-spine suites (1159 tests) green, the time-sensitive files also under TZ=America/Los_Angeles; tsc and ESLint clean. In-memory harness and source only; no PostgreSQL race test is claimed.

Not done

• A run resumed at the ceiling that stalls another five times is abandoned again. That is intended: the loop needs a person each time.
• pauseRun and cancelRun keep the plain write: neither hands back a budget or clears a lease that is not already the person's to clear.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KP9rt5rVQ9jDMpZp2xBy3K

---

Generated by Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
bubaly	Ignored		Oct 4, 2026 7:32am UTC

[NewWorldVenture]

Independent review of exact head 41e8978d3c5e73305844e5e4399c1b767dd23aa3 found that resetting the attempt budget still has an unguarded stale-control race. In foldIntoRun (lib/services/approvals/index.ts, around lines 420–476), a resolved approval can write the run back to ready, set attempt: 0, and clear its lease through the generic updateRun path. That update filters by run id and family only (lib/ai/runs/store.ts, around lines 534–550). If two final approval folds observe completion together, one can make the run ready and the worker can claim it; the other stale fold can then restore ready/attempt 0 and clear the worker's live lease while it is executing. resumeRun/rerunStep also write from previously loaded snapshots without a state/attempt compare.

The new tests prove a previously abandoned run can be resumed with a fresh budget, but the source-shape assertion that every ready patch contains freshBudget() does not exercise this interleaving. Please guard the transition against the observed state/attempt/lease (or serialize it) and add a deterministic two-fold/claim race regression. Current evidence is source-level plus the in-memory harness; no PostgreSQL race test or production execution is claimed. I left the Claude-owned branch unchanged.

[NewWorldVenture]

Confirmed, and closed in a66ac1269 (pushed with this comment, on top of the stack's merge-down of #901's lease fence and #904).

The guard. store.updateRunAsObserved(scope, runId, observed, patch) is the compare-and-set on the run as the caller observed it — state, attempt and lease_owner together (.is('lease_owner', null) when the snapshot had none). Not the state alone: a run can be executing before and after under two different workers, which a state compare would not tell apart. foldIntoRun, resumeRun, rerunStep and editStepInput all write through it. Zero rows is "someone else moved this run first" and each caller leaves the run where that someone put it: the fold has nothing left to do (the steps it released are in the graph the running slice reloads every pass) and sends no kick; resume reports the run is not paused; re-run and edit say the run changed and nothing was overwritten (the re-queued step is the claim's to pick up; the saved plan version stays unreferenced). The source case in a-run-a-person-asks-for-again… now counts the four observed writes and pins that no bare ready write remains in either file.

The deterministic race (tests/a-stale-fold-cannot-re-arm-a-claimed-run.test.ts, 6 cases). The in-memory client is wrapped so that the first family_automation_runs read hands the re-entry its snapshot and, before it can act, runs the other party — the real decide for the other approval and the real claimRun over the same tables — then the stale re-entry proceeds:

• two final folds and a claim: B's fold (snapshot awaiting_approval, attempt 2, no lease) runs after A's fold put the run in the queue and the kick's claim took it; B's decision is recorded and its step released, the run stays executing, attempt 1, the worker's lease and expiry untouched, and exactly one kick was sent (A's);
• a fold whose snapshot still holds lands with the fresh budget and kicks;
• re-run a step against a claim between read and write: the claim stands, the person is told, the step is queued for the worker's reload;
• re-run a step on an executing run whose slice parked and was re-claimed meanwhile — same state, different attempt and lease — the write does not land (this is the case that needs more than a state compare);
• resume against another tab's resume followed by a claim: "That run is not paused.", claim untouched;
• a control whose snapshot holds lands.

Mutation controls, files restored and checksummed after each: the fold back on updateRun → 1 failure; the three controls back on updateRun → 2; the compare reduced to state only → 1 (the re-leased executing case). 78 run-spine suites (1159 tests) green, the time-sensitive files also under TZ=America/Los_Angeles; tsc and ESLint clean.

Evidence is the in-memory harness and source, as before; no PostgreSQL race test or production execution is claimed. pauseRun and cancelRun are unchanged (neither hands back a budget or clears a lease that is not already the person's to clear).

---

Generated by Claude Code