Reassignments run in the source node's lane and batch by AgentStartBatchSize (GH-3749, GH-3748)

[jeremydmiller]

Closes #3749. Mitigates #3748 (root ack-decoupling stays open — see below). Part of the #3753 campaign.

What changed

GH-3749 defect 1 — wrong lane. ReassignAgent ran in the destination node's dispatcher lane while doing its blocking StopAgent round trip against the source. Lanes are strictly serial, so a source that was merely busy (every stop slow behind the projection-version-bump side-effect gate) froze every batched AssignAgents chunk queued for a perfectly healthy destination — the field measurement was 22 reassignments starving 1,290 placements on an idle node. ReassignAgent.DestinationNodeId is now OriginalNode.NodeId, so a slow or dead source wedges nothing but its own lane, restoring the invariant AgentCommandDispatcher's doc comment claims. The start half already cascades through Enqueue into the destination's own lane, so it still orders behind that node's queued work — which also covers the GH-3698 pending re-target case, so the narrow StopInSourceLane flag is gone.

GH-3749 defect 2 — never batched. Reassignment was the one command type batchCommands never chunked; a rebalance emitted thousands of individual stop round trips and AgentStartBatchSize had no effect on any of them (reporter verified at 50, 100, 400, 1500). New ReassignAgents batches all agents moving between the same two nodes, chunked by AgentStartBatchSize: one StopAgents round trip, then one AssignAgents cascade for exactly the confirmed-stopped subset — an unconfirmed stop is never followed by a start elsewhere, preserving the GH-3698 single-copy invariant. Unconfirmed stragglers are re-decided by a later evaluation.

GH-3748 mitigation — per Jeremy: every batched reply window, 30s/agent past 20 agents. Shared AgentBatchTimeouts.ReplyWindowFor(n) (30s + n×1s up to 20 agents, 30s + n×30s beyond) now backs AssignAgents, StopRemoteAgents (previously a flat 30s default at any batch size), and ReassignAgents. The old 1s-per-agent assumption is off by 1–2 orders of magnitude for a Marten shard behind a version bump (27s p50 / 215s max measured), so no batch size could ever satisfy the old window. Not the root fix: acknowledgements still wait for the work itself, and the single-agent 60s ack window (WolverineRuntime.Agents.cs) is untouched because the >20-agents condition can't apply to it — so #3748 should stay open for ack/work decoupling.

Evidence — new SlowTests harness (agent_reassignment_at_scale)

Miniature of the reported deploy: one node owning 240 old-generation agents with 2s stops (busy source), then a version-bump deploy — universe grows to 480 and two fresh nodes join, so reassignments and first-time placements are in flight together. Same knobs as the reported cluster (batch 50, parallelism 10).

metric	main @ 4d05fbd (unfixed)	this branch
both fresh nodes carrying agents	163.1s — one node at 0 for ~2.5 min while peers sat full	2.7s
full convergence (480 agents / 3 nodes)	246.3s	102.9s
worst backwards march of placed count	76 agents (323→321 visible in samples)	3
peak concurrent starts, cluster-wide	10 — pinned at ONE node's MaxAgentStartParallelism	20

The unfixed run reproduces every headline symptom of #3753: a node handed hundreds of placements holding zero, the total marching backwards, and cluster-wide start concurrency capped at a single node. The test fails on main and passes here (verified both ways), and the pre-existing GH-3698 agent_assignment_at_scale harness still passes.

⚠️ Note: SlowTests is compiled by wolverine.slnx but not run by any CI workflow or Nuke target — both scale harnesses are locally-run. The 261 CoreTests agent tests (11 new: lane keying, confirmed-subset cascade, value equality, chunking through EvaluateAssignmentsAsync, reply-window arithmetic) do run in CI.

🤖 Generated with Claude Code

https://claude.ai/code/session_013eR4GL278688VhyhrGcttJ