[brentkelly]

Distributed board 5/5 (T3O-66, part of T3O-60). A worker that goes offline currently leaves its cards to time out and recover, and spawn duplicate threads. Removing a worker also has no defined handling for the cards still placed on it.

• Offline: signals from a worker that isn't reconciled are held. Timeouts skip steps on a paused worker and credit the time it was offline. Usage-limit probes skip those cards. Slots stay held, and cards show a Worker offline notice with Force-move….
• Reconnect: a scoped reconcileWorker runs before held signals are replayed and before schedule(). It never respawns a live thread.
• Removal: board.workers.removePreview groups the affected cards. Cards with no worktree fall back to the default worker. The rest become placementOrphaned: running steps are stopped if the worker is reachable, stalled with worker-removed, and their slots released. Re-adding the worker reattaches them.
• Force-move: board.cardWorkerMove accepts force and restartFromBase. The move continues from the remote branch, or restarts from base after a second confirmation.
• Web: notice ranking, the force-move dialog, removal-dialog grouping and the header attention pill. User docs are updated.

🤖 Generated with Claude Code

[brentkelly]

High-level summary

This PR finishes the distributed board's failure handling. Signals from an unreconciled worker are held, and a scoped reconcileWorker runs on reconnect before those signals are replayed and before schedule(). Timeouts and usage-limit probes skip paused workers, and timeout reference times are shifted by the time a worker was offline. Worker removal and its preview are decided atomically in the decider, as fallback, orphaned or reattached outcomes. Force-move gains abandon and restart-from-base phases, and the web gets the notice ranking, the force-move dialog and the header attention pill. The structure is sound: removal outcomes are decided with the removal event, reconcile reuses the boot path, and local-only boards are left alone.

This round has 1 critical, 3 improvement and 1 nitpick finding. The critical one is the PR's headline force-move path. A step stranded on an offline or removed worker, once requeued on its new worker, reads its old thread's state before checking which worker that thread is on. Against the real RemoteThreadGateway, that read answers unknown and liveThread fails. The failure leaks the slot that was just acquired and aborts the whole schedule() pass. The test for this path passes only because the fake gateway answers for offline workers. The improvements are:

• a reconnect that falls between two syncs is missed, because the sync samples current status rather than the transition
• "no remote branch" is inferred from any failure to fetch the remote
• restart-from-base moves cards to Build whatever their stage

None of these are cosmetic.

[brentkelly]

Caution

Critical (1/1): a requeued stranded step reads its old, unreachable thread first, which fails, leaks a slot and aborts schedule()

After a force-move (or a re-pin of an orphaned card), the step stalled with worker-removed is requeued, and admitBuildCandidate / admitPlanStep call resumeExistingStepThread. That function calls threadGone(state.threadId) before the T3O-65 worker-mismatch check below it. The thread still lives on the old worker, which by definition is offline or removed. ThreadGatewayRouter routes the read to RemoteThreadGateway.threadState, which returns UNKNOWN_THREAD_STATE when there is no link, and liveThread turns that into a ThreadReadError.

In admitBuildCandidate, slots.acquire has already succeeded at that point, and nothing catches the failure:

• the slot on the new worker leaks
• the error escapes the for loop in schedule(), so every candidate ordered after this card is not admitted on that pass
• the same thing happens on every later pass until the old worker comes back, leaking one slot each time

The new test a force-move strands the step on the offline worker, then runs it where it lands passes only because threadGateway.testkit's threadState answers for any thread regardless of whether its worker is online.

Suggested fix: Do the link/worker comparison first, and return "no-thread" without touching the gateway when the thread's worker differs from where the card runs now. In the harness, make the fake gateway answer UNKNOWN_THREAD_STATE for threads linked to an offline or removed worker, so this test exercises the real behaviour.

[brentkelly]

Fixed in f8491c3. resumeExistingStepThread now checks whether the link's worker matches where the card runs before it reads the thread, so a thread on an offline or removed worker is never read through the gateway. The force-move test a force-move strands the step on the offline worker, then runs it where it lands (workerOffline.test.ts) now wraps the gateway so the W1 thread reads UNKNOWN_THREAD_STATE, the way RemoteThreadGateway answers for an unreachable worker. It fails before the fix (no spawn: liveThread throws after the slot is acquired) and passes after it.

[brentkelly]

Adjudication (round 1): fix-upheld. resumeExistingStepThread now returns no-thread on a worker mismatch before threadGone reads the old thread. The force-move test's gateway now answers UNKNOWN_THREAD_STATE for the W1 thread. I put the old check order back and the test failed, so it really catches this bug. With the fix in place it passes.

[brentkelly]

Warning

Improvement (1/3): a worker that drops and returns between two syncs is never treated as having been offline

syncWorkerReach samples the current workersNow status and ignores the transitions. placement.changes is a Stream<void> tick, and the reactor queue is serial. If a worker goes offline and comes back before the queued workers item runs, a busy queue or a short network blip is enough. In that case the item sees online, the worker is still in reconciledWorkers, and nothing happens:

• no reconcile runs before the resumed signal stream is processed, which the brief requires on every reconnect
• unreachableSince is never set, so the offline time is not credited, and sweepTimeouts can time out and recover a step whose thread was simply out of reach

gateThreadSignal has the same gap, because it trusts reconciledWorkers membership.

Suggested fix: Key reconciliation on a per-connection generation or epoch from WorkerConnections (or record each transition as it is observed) rather than sampling status. A worker whose connection generation changed since its last reconcile is then unreconciled even if it reads online now, and its offline interval comes from the connection's own timestamps.

[brentkelly]

Fixed in 0087641. BoardPlacement.changes now emits the summary each change produced, not a void tick. The reactor's subscriber records every worker that isn't online in droppedWorkers as each summary arrives, along with when it first saw it. syncWorkerReach consumes that set first: it un-reconciles the worker and starts its time out of reach from when the drop was seen, so a worker that is already back online still gets the scoped reconcile and the offline credit. gateThreadSignal and workerPausedNow treat a worker with a pending drop as unreconciled, so its signals are held right away. New test: a drop and reconnect both landing before the supervisor looks is still reconciled (workerOffline.test.ts). It fails before the fix (0 recoveries) and passes after it. I chose a timestamped drop set over a per-connection epoch because the summary carries no epoch, and adding one would mean changing the WorkerConnections and contracts types for the same result.

[brentkelly]

Adjudication (round 1): fix-upheld. changes now carries each summary. The subscriber records every non-online worker in droppedWorkers as the summary arrives. syncWorkerReach un-reconciles those workers and credits their offline time, and workerReconciled gates signals while a drop is pending. The new drop-and-reconnect test passes.

Residual, non-blocking: in production, BoardWorkers.states is Stream.merge(...).pipe(Stream.mapEffect(() => state)). It reads the current state when each tick is pulled, not the state that caused the tick. The test's fake placement emits the real value, so it does not cover this. The remaining window is now the scheduling latency of an idle consumer fiber, not the serial queue's processing time. A real reconnect needs a network handshake, so this is negligible. Mapping each connections.states value into the summary would close it completely.

[brentkelly]

Warning

Improvement (2/3): any failure to read the remote is reported as "no remote branch", which offers Restart from base

LocalBoardWorkspace.remoteBranchTip returns null both when the remote lacks the branch and when it cannot be reached ("null when the remote does not have it (or cannot be reached)"). On top of that, .pipe(Effect.orElseSucceed(() => null)) folds every other failure into null: a target worker whose RPC dropped, a missing checkout, or an auth failure on fetch.

On the force path, null becomes outcome: "no-remote-branch", and the dialog then asks "Restart {card} from {base}?". Confirming forgets the card's branch (worktree: null) and cuts Build again from base. So a transient fetch failure on the target can lead a human to throw away a card's pushed work, with copy telling them it was never pushed. The non-force path treats a null tip as a hard refusal, which is safe. The force path turns it into an offer to discard.

Suggested fix: Make the workspace distinguish "branch absent" from "couldn't ask". For example, return a tagged result, or fail on fetch and transport errors and keep null for a confirmed-missing ref. Offer no-remote-branch only for the former, and refuse with "Couldn't reach the remote from {target}" otherwise.

[brentkelly]

Fixed in 1cd4a01. LocalBoardWorkspace.remoteBranchTip now asks git ls-remote --heads first. An empty answer means the branch is absent and returns null. A failed ls-remote or a failed fetch fails with BoardWorkerGitError ('The remote … couldn't be reached.'). No remote configured at all still returns null. In BoardCardMove, checkForced refuses when the read fails, and the non-force path does the same when there is no source to move from, instead of folding the failure into no-remote-branch. Tests: a remote that can't be asked refuses the force-move rather than offer a restart (BoardCardMove.test.ts) fails before the fix and passes after it. BoardWorkspace.contract.test.ts now points origin at a missing path and asserts remoteTipUnreachable === 'BoardWorkerGitError' on both the local and the remote (RPC) workspace.

[brentkelly]

Adjudication (round 1): fix-upheld. remoteBranchTip runs ls-remote --heads first. A non-zero exit or a thrown error fails with BoardWorkerGitError. Empty output returns null (branch absent). A failed fetch after a positive ls-remote also fails instead of returning null. In both checkForced and the move, the mover turns a failed read into refused(...couldn't reach the remote...), so only a confirmed-absent branch gets no-remote-branch. The new mover test and the contract test pass.

[brentkelly]

Warning

Improvement (3/3): restart-from-base moves the card to Build whatever stage it is in, and ignores a refused move

The docblock says "A card past Build goes back to it first", but the guard is card.stage !== build.stageId, so any other stage triggers it. Two cases go wrong:

• A card in a stage before Build that still holds a worktree, for example one sent back to Planning, is jumped forward into Build with override: true, skipping the planning the human just asked for.
• A Done card with a kept worktree (T3O-52) that a human force-moves off an offline worker is dragged back out of Done.

Also, dispatchStamped's boolean is discarded. If board.card.move is refused, restarted is still recorded. The card is then left in its later stage with worktree: null and its branch forgotten, and nothing tells the human.

Suggested fix: Compare boardStageIndex(board, card.stage) with Build's index, and move only when the card is past Build and not Done. Refuse the restart, or leave the stage alone, for earlier stages and for Done. If the stage move is refused, return refused(...) before recording restarted. Add tests for a pre-Build card and a Done card.

[brentkelly]

Fixed in 1cd4a01. checkForced refuses no-remote-branch for a Done card, so restart can't be reached for it. restartFromBase moves the card back to Build only when isBoardStageAtOrAfterBuild holds. A card not yet at Build keeps its stage, its branch is forgotten, and Build cuts a fresh branch from base when the card gets there. I chose that over refusing, because a refusal would leave a pre-Build card stuck on a dead worker with no way out. The board.card.move result is now checked: if the move is refused, the mover records a failed move phase and returns refused, rather than recording restarted. Tests that fail before the fix and pass after it: with no remote branch, a Done card is not restarted and with no remote branch, a card not yet at Build restarts in its own stage. The refused-stage-move branch has no direct test, because the mover's fixture decider has no way to refuse an override move.

[brentkelly]

Adjudication (round 1): fix-upheld. restartFromBase only moves a card back to Build when isBoardStageAtOrAfterBuild. Done cards are refused earlier, in checkForced. A refused stage move now records failed and returns false, and the caller (BoardCardMove.ts:549) turns that into refused. The Done-card and pre-Build tests pass.

[brentkelly]

Note

Nitpick (1/1): requeueResolvedPlacementStalls's docblock sits on provisionPlacedBuildStep

This docblock (which predates this PR, and which the PR extends with the T3O-66 paragraph) is followed directly by provisionPlacedBuildStep's own docblock. Editors and hover documentation attach neither block to requeueResolvedPlacementStalls, which sits about 20 lines below.

Suggested fix: Move this comment block down so it sits directly above const requeueResolvedPlacementStalls.

[brentkelly]

Fixed in f8491c3: the docblock now sits directly above requeueResolvedPlacementStalls.

[brentkelly]

Adjudication (round 1): fix-upheld. The docblock now sits directly above requeueResolvedPlacementStalls.

[brentkelly]

High-level summary — round 2 (re-review at 1cd4a01)

This PR completes the distributed board's failure handling. A worker the board can't reach pauses its steps: they aren't timed or recovered and keep their slots. When it comes back, a reconcile scoped to that worker runs before its held signals are released, and its offline time is credited to the step timeout. Removal decides each card's fallback or hold atomically in the decider, and stops and stalls running steps. Force-move continues from the remote branch, or restarts from base after a second confirmation. The web side adds the Worker removed / Worker offline notices, the force-move and removal dialogs, and the header indicator.

All five round-1 fixes hold up on re-reading. The worker-mismatch check now runs before the thread read. Drops are recorded as each summary arrives. ls-remote tells an absent branch from an unreachable remote. Restart only sends cards at or after Build back to it, and refuses Done cards. Reading the code fresh turned up two new improvement findings and nothing critical. First, a reconcile pass that fails partway forgets which threads it already decided, so held signals from before it can be replayed against steps it already recovered or parked. Second, a step parked on a removed worker (awaiting input, paused, or stalled) is neither stalled nor requeued by removal or force-move, so it stays bound to an unreachable thread. That second gap can also leave a provider usage limit uncleared. Both are edge paths with no test coverage. The core offline, reconnect and removal flows look sound.

Status of round-1 findings

#	prior finding	status
r1-c1	🔴 A requeued stranded step reads its old, unreachable thread first, leaking a slot and aborting schedule()	✅ closed
r1-i1	🟡 A worker that drops and returns between two syncs is never treated as having been offline	✅ closed. Residual tick-pull latency is negligible.
r1-i2	🟡 Any failure to read the remote is reported as no-remote-branch, which offers Restart from base	✅ closed
r1-i3	🟡 Restart-from-base moves the card to Build from any stage and ignores a refused move	✅ closed
r1-n1	🟢 requeueResolvedPlacementStalls docblock sits on provisionPlacedBuildStep	✅ closed

[brentkelly]

Warning

Improvement (1/2): a reconcile that only partly finishes forgets which threads it already decided, so the held signals it keeps are replayed against those steps later

reconcileWorker returns null if any one step's liveThread read fails. That happens when the worker drops again partway through the pass. When it does, syncWorkerReach hits continue here. It doesn't clear heldSignals for that worker, and it throws away the decidedThreads set built so far. But the steps that pass already reconciled stay reconciled: settleStep, recoverStep and parkStepForInput have dispatched, and a recovered step may already be running a nudge turn.

On the next attempt (the next timeout-sweep tick, or the next reconnect), the steps that pass 1 recovered or parked usually come back as resume-watch or reschedule. Their thread ids are no longer in decidedThreads. So the signals held from before pass 1 are replayed through processThreadSignal. For example, a held turn-ended reaches a step that has already been recovered and is running its nudge. A held turn-requested reaches a step that pass 1 parked, and handleTurnStartRequested resumes a stalled step. The builder's notes say this filter exists to stop exactly this, a turn-end held from before the reconcile causing a second recovery. That protection only holds when the first pass finishes. There is no test for a partial pass followed by a retry.

Suggested fix: Keep the decided threads per worker across attempts, for example decidedSinceDrop: Map<BoardWorkerId, Set<string>>. Merge each pass's set into it, and clear it only when the worker is marked reconciled and its held signals have been released. Alternatively, drop every held signal for a thread that any pass on that worker decided. Add a test where one step's read fails on the first pass, the retry succeeds, and a held turn-end for the first step does not trigger a second recovery.

[brentkelly]

Fixed in 2c97278. Decided threads now live in a per-worker decidedThreads map that reconcileWorker adds to on every attempt, including partial ones. The map is cleared only when the held signals are released, or when the worker is removed or forgotten. Test: a reconcile that fails partway still drops the held signals it already decided in workerOffline.test.ts. One step's read fails on the first pass while the other step is recovered onto its backoff rung. On the retry, the held board turn-start and turn-end are not replayed. Before the fix the step was woken off its rung and recovered a second time (expected 2 to equal 1).

[brentkelly]

Adjudication (round 2): fix-upheld. Checked at a0bae48. In 2c97278, decidedThreads is now a per-worker map that lasts across reconcile attempts. A pass that fails partway still adds the threads it advanced, recovered or parked. The map is cleared only after a whole pass releases the held signals, or when the worker is removed or unregistered. The new test "a reconcile that fails partway still drops the held signals it already decided" fails when 2c97278 is reverted and passes now (15/15 in workerOffline.test.ts).

[brentkelly]

Warning

Improvement (2/2): removal and force-move only stall running and completing steps, so a parked step stays bound to a thread on the removed worker that nobody can reach

stallStepsOnRemovedWorker skips every step that isn't running or completing. stallStrandedStep and boardCardWorkerRemovalOutcome apply the same filter. A step parked on the removed worker is skipped too: awaiting-input (a question or a stop), paused, or stalled for usage-limit or needs-a-human. Its thread link still names that worker. What happens next depends on whether the card has a worktree.

• No worktree (for example, a planning step waiting on a question): the card gets fallback and moves to the default worker with no notice. The step still shows Input needed. The human's answer would have to go to a thread on a worker that is gone, and gateThreadSignal drops that worker's signals anyway. Nothing ever requeues it, because requeueResolvedPlacementStalls only picks up worker-removed and provider-not-on-worker stalls.
• Has a worktree: the card is orphaned and the human force-moves it. stallStrandedStep skips the parked step, and when the move lands the requeue skips it too. The card arrives on its new worker still parked on the old, unreachable thread.
• Usage-limit stall: slotKeyFor resolves to the removed worker, so workerPausedNow is true for it forever. The provider-limit probe filters the step out, and if it was the instance's only parked card, the limit never clears.

None of the removal, fallback or force-move tests cover a parked step.

Suggested fix: On removal, and on a force-move or pin off an unreachable worker, also stall the card's parked step with worker-removed when its thread link names that worker. requeueResolvedPlacementStalls already restarts such a step fresh on the card's new worker through the link-worker check in resumeExistingStepThread. Add a decider or supervisor test with an awaiting-input step on the removed worker, for both the fallback case and the orphaned-then-force-moved case.

[brentkelly]

Fixed in a0bae48. A new stepBoundToThread predicate covers running, completing, awaiting-input, and stalled steps other than worker-removed that have a thread. stallStepsOnRemovedWorker and stallStrandedStep both use it, so parked steps on the removed or abandoned worker are stalled worker-removed too. Removal then calls requeueResolvedPlacementStalls(), so a fallback card restarts on its new worker straight away. A held card still waits for its move. A usage-limit park on the removed worker no longer counts toward its instance's cooldown, so the limit can clear. A human paused step is deliberately left alone. The decider refuses recover-step on paused, and the human's resume goes through resumeExistingStepThread, whose worker check starts afresh on the card's current worker. boardCardWorkerRemovalOutcome is unchanged: a parked step doesn't hold the card, and it now restarts on the default instead. Tests: removing the worker restarts a fallback card's parked step on the default (awaiting-input) and a force-move off an offline worker takes a parked step with it (usage-limit). Both fail before the change and pass after. The existing removal test now sets placementOrphaned as the decider does.

[brentkelly]

Adjudication (round 2): fix-upheld. Checked at a0bae48. stepBoundToThread now covers running, completing, awaiting-input, and any stall except worker-removed, and both stallStepsOnRemovedWorker and stallStrandedStep use it. The removal path then calls requeueResolvedPlacementStalls, which still skips orphaned and moving cards, so only fallback cards restart. slotKeyFor takes the worker from the thread link, so a fallback card whose pin was already cleared still resolves to the removed worker. Leaving a human Pause alone is reasoned and documented. Both new tests (fallback parked step, and a force-move of a usage-limit-parked step) fail with the fix reverted and pass now.

[brentkelly]

High-level summary — round 3 (re-review at a0bae48)

This PR adds the failure half of the distributed board. Cards on an offline worker pause: their steps are not timed and not recovered, their slots stay held, and they get an amber Worker offline notice. On reconnect, a reconcile scoped to that worker runs before any held signal is released, and the offline interval is credited to the step's timeout. Removing a worker previews the affected cards, falls back the ones with no worktree to the default worker, and holds the rest as placementOrphaned; running steps there are stopped and stalled. Force-move continues a card from its pushed branch, or restarts it from base after a second confirmation. Re-adding the worker reattaches held cards.

Both round-2 fixes hold. decidedThreads now persists per worker across reconcile attempts and is cleared only when the held signals are released or the worker is forgotten. stepBoundToThread widens removal and force-move to parked steps, and removal now requeues the fallback cards. I also re-read the removal decider, BoardWorkers.remove/removePreview, the force-move mover and dialog, the timeout-credit sweep, signal gating, the reattach path, and the web notice/pill code. The four directly relevant test files pass (66 tests). Nothing blocks. This round has 3 nitpicks and no critical or improvement findings.

Status of round-2 findings

#	prior finding	status
r2-i1	🟡 A partial reconcile forgets which threads it already decided, so held signals are replayed against those steps	✅ closed
r2-i2	🟡 Removal and force-move only stall running/completing steps, so a parked step stays bound to a thread on the removed worker	✅ closed

[brentkelly]

Note

Nitpick (1/3): a revoked or incompatible worker is described as offline and expected to reconnect

boardWorkerOutOfReach counts unauthorized and incompatible as trouble, so those cards get kind: "offline". The panel title is then {name} is offline and the body says the card "carries on from where it stopped when {name} reconnects". The header popover (BoardWorkersAttentionPill.tsx:51) says the same: "They carry on when it reconnects." A revoked worker never reconnects by itself; it needs re-pairing. An incompatible one needs an upgrade. Only the tooltip's why line tells these cases apart, so a human reading the panel may wait for a reconnect that will never happen.

Suggested fix: Pick the title and the closing sentence by worker.status.state. For example, "{name}'s access was revoked — re-pair it in Settings → Workers, or force-move the card", and the same for a T3o upgrade. Do the same in the popover row.

[brentkelly]

Fixed in 7e03784. boardCardWorkerTrouble now picks the panel title by status.state: "{name}'s access was revoked", "{name} needs a newer T3o", or "{name} is offline". The body ends "once {name} is paired again", "once {name} is upgraded", or "when {name} reconnects". The header popover row has a new resumesWhen field built by the same helper, so its "They carry on …" line no longer says reconnect for a revoked worker. The short label stays "Worker offline", the notice name the brief fixes. Test: cardWorker.test.ts › "tells a revoked or incompatible worker apart from one that will reconnect".

[brentkelly]

Note

Nitpick (2/3): "The running step is stopped" depends on the preview's display text

stopsAStep is byHand.some((card) => card.detail.endsWith("step running")). It parses the human-readable detail that BoardWorkers.removePreview builds at BoardWorkers.ts:521. Rewording that string on the server would silently drop the sentence from the dialog.

Suggested fix: Add a running: boolean (or a reason enum) to BoardWorkerRemovePreviewCard and branch on that.

[brentkelly]

Fixed in 1cbf1a9. BoardWorkerRemovePreviewCard has a structured running: boolean, set by BoardWorkers.removePreview from the same running/completing check that builds the detail text. stopsAStep now reads card.running. BoardWorkers.test.ts asserts running is true on the busy by-hand row and false on the fallback row.

[brentkelly]

Note

Nitpick (3/3): the restart confirmation says the card goes back to Build even when it won't

After the round-1 fix, restartFromBase moves a card back to Build only when it is at or after Build. A pre-Build card keeps its stage and only loses its branch. The confirmation still tells every card that it "goes back to the start of Build".

Suggested fix: Word it as "the card's branch is cut fresh from {base} when it reaches Build", or have the check answer say whether the stage changes and use that.

[brentkelly]

Fixed in 8b8ed12, with the wording driven by the check answer as suggested. The no-remote-branch answer has a new backToStage: the Build stage's label for a card at or past Build, using the same isBoardStageAtOrAfterBuild rule restartFromBase applies, or null for a card not yet at Build. The confirmation says "goes back to the start of {stage}" only when backToStage is set. Otherwise it says the card stays in its stage, its branch is forgotten, and Build later starts from base. Tests in BoardCardMove.test.ts: the restart test expects backToStage: "Building", and "a card not yet at Build restarts in its own stage" asserts backToStage: null from check.