Agent commands wait on observed progress, not the clock (GH-3748, GH-3750)

[jeremydmiller]

Closes #3748. Closes #3750. Part of the #3753 campaign, building on #3757's reply-window mitigation.

The defect pair

Both issues are one protocol flaw seen from two sides: the batched agent command reply window did double duty as a work budget and a failure detector, and no number can be both.

• Agent command acknowledgements wait for the agent work to finish, so a slow start or stop always times out (6.24.2) #3748: acknowledgements only arrive when the agent work finishes, so a start or stop slower than the window (a Marten shard behind a projection version bump: 27s p50 / 215s max, measured) always times out — 43 lane timeouts and zero successful stops in seven minutes on the reporter's leader.
• A partially-fulfilled AssignAgents chunk is treated as complete, so agents still starting get re-placed onto other nodes #3750: the timeout is then treated as whole-chunk failure. The dispatcher releases its in-flight hold, the pending-assignment ledger's TTL (seconds) is long expired, and the next evaluation re-places agents that are still mid-start onto other nodes — 369 of 902 distinct agents started more than once in twelve minutes, every repeat on a different node, while ~5,600 never-placed agents waited.

What changed

1. A progress probe: QueryAgentPresence → AgentPresenceReport. The leader can ask a node which of the named agents are running right now, answered instantly from the in-memory agent registry. Presence confirms a start; absence confirms a stop (an agent the node has never seen is exactly as stopped as one it just stopped).

2. Remote StartAgents/StopAgents execute off the control lane (DeferredAgentCommandRunner). The database control endpoint is deliberately serial (MaxDegreeOfParallelism = 1), so a slow inline batch used to monopolize the node's entire control plane — stops queued behind it, and nothing could answer the progress probe. Batches now run on a background FIFO (strictly one at a time, preserving inter-batch ordering and the parallelism cap) and the control lane is free in microseconds. The typed reply still goes out only at true completion, correlated to the original conversation, so a pre-upgrade leader blocked in InvokeAsync<AgentsStarted> completes exactly as before. A per-agent stop-revocation sequence keeps the stop-after-start ordering the serial lane used to guarantee: a stop that lands while its target's start is queued or in flight either suppresses that start or reverts it immediately.

3. The leader races the reply against presence polling (AgentWorkConfirmation). The reply wins whenever it arrives — including from pre-upgrade nodes whose polls simply go unanswered and for whom the old reply window remains the bound (no rolling-upgrade regression in either direction). The polls add the missing piece: observed progress. One more agent confirmed within Durability.AgentProgressStallTimeout (default 5 min, the daemon side-effect gate's ceiling) keeps the wait alive, so a slow-but-converging node gets unbounded time while a wedged or mute one costs a bounded wait. Every agent stays held — dispatcher in-flight + pending ledger — for as long as the command is waiting, which is what stops the re-placement churn. On resolution, confirmed agents re-arm the ledger (ConfirmDispatched) so the one-snapshot gap before their assignment rows become visible can't re-decide an agent that just started. The reassignment single-copy invariant is unchanged: a start is only cascaded for agents confirmed gone from the source, now by acknowledgement or observation.

New knobs: Durability.AgentProgressPollInterval (10s) and Durability.AgentProgressStallTimeout (5 min).

Evidence

• New SlowTests harness slow_starts_outrun_reply_windows: 3 nodes, 36 agents at heterogeneous 6/12/18s per start (36s/72s/108s of work per chunk against 42s reply windows). Converges with every agent started exactly once and asserts the reporter's own churn metric (no agent completes a start on more than one node). Getting this green caught three real bugs in the initial implementation — it is a sensitive guard.
• 11 new unit tests over the confirmation race (progress resets the stall clock, mute-poll fallback to the legacy window, gone-mode, partial replies, cancellation) plus presence-query serialization/round-trip coverage.
• Full CoreTests (2192), both existing agent scale harnesses (agent_assignment_at_scale, agent_reassignment_at_scale), and the persistence agent/control-endpoint suites are green locally; full wolverine.slnx Release build is clean.

Honest note on the baseline: this miniature also passes on current main — the GH-3698/#3757 machinery already makes a 3-node/36-agent cluster eventually-convergent, and the field failure needs production scale (thousands of agents, 512 tenant databases, 5-minute gates, rolling pods) to turn "eventually" into "hours". The discriminating evidence for the root fix is the reporters' field data; what this PR removes is the mechanism — the window-as-failure-detector — rather than tuning its numbers.

🤖 Generated with Claude Code