Stop Balanced-mode host shutdown paying a full agent reply window (GH-3781)

[jeremydmiller]

Closes #3781.

IHost.StopAsync() on a Balanced-mode node holding a large agent load did not return. A dump of the
wedged process (dotnet-dump collect + dumpasync --coalesce — dotnet-stack cannot see this,
because the wedge is an await on the GC heap, not on a thread stack) put it here:

Task<AgentPresenceReport>                          <-- pending
 MessageRoute.sendAndAwaitReplyAsync<AgentPresenceReport>
  WolverineRuntime.InvokeAsync<AgentPresenceReport>
   AgentWorkConfirmation.AwaitCoreAsync             (the QueryAgentPresence poll)
    AssignAgents.ExecuteAsync
     AgentCommandDispatcher.runLaneAsync
      AgentCommandDispatcher.DisposeAsync           <-- awaiting the lane worker
       WolverineRuntime.teardownAgentsAsync
        WolverineRuntime.StopAsync -> Host.StopAsync

The leader was stuck in AgentCommandDispatcher.DisposeAsync(), waiting on a lane worker executing
an AssignAgents chunk aimed at a peer that had already shut down.

It was never a deadlock

I let the wedged repro run out instead of killing it:

	duration
agent_reassignment_at_scale, unfixed	17m 17s, passed
same, with the issue's .WaitAsync(20.Seconds()) teardown control	2m 18s
same, on this branch	1m 57s

~15 minutes inside StopAsync, which is exactly AgentBatchTimeouts.ReplyWindowFor(30) =
30s + 30 x 30s = 930s. So it is one full agent-batch reply window per queued command, per lane
— 25.5 minutes at the shipped AgentStartBatchSize = 50 — not "hangs forever". Nobody had
waited it out. That distinction is the fix: there is no lock to break, there is a timeout being paid
in full during shutdown because the cancellation that should short-circuit it was dropped.

Three compounding causes

1. ReplyListener silently dropped an already-cancelled token

cancellationToken.Register(onCancellation);   // <-- _completion is still null here
_completion = new TaskCompletionSource<T>(...);

CancellationTokenRegistration invokes the callback synchronously when the token is already
cancelled, so onCancellation's _completion?.TrySetException(...) fired against null and did
nothing. The listener then sat out its entire reply window.

This one is not agent-specific. Every shutdown path passes an already-cancelled token, so any
InvokeAsync<T> in Wolverine handed a cancelled token waited its whole window rather than failing
fast. Agent shutdown is just where that window is 25 minutes long. ReplyTracker additionally no
longer registers a listener that completed inside its own constructor — nothing would ever have
taken that entry back out.

2. The dispatcher held the wrong cancellation token

It was built with the runtime-wide Cancellation; NodeAgentController and
DeferredAgentCommandRunner both take _agentCancellation.Token. And StopAsync cancels
_agentCancellation at the top but only calls DurabilitySettings.Cancel() after
teardownAgentsAsync has returned — so the token that would end the lane loop was cancelled after
the thing awaiting it had already finished awaiting.

3. DisposeAsync drained its lanes instead of abandoning them

Writer.TryComplete() does not discard what is already buffered — ReadAsync keeps handing it out
— so shutdown executed every command still queued for a cluster the node was in the middle of
leaving, one reply window each. It now latches _disposing, empties the lane queues (releasing
their in-flight claims so a later leader can re-issue the work), and waits each lane against
LaneShutdownTimeout, logging the lane it gives up on. That last part is a backstop: the node's own
deregistration sits behind this in teardownAgentsAsync, and no future non-cancellable await inside
a command should be able to hold IHost.StopAsync() again.

Tests

Five new tests, all mutation-tested — each was confirmed to fail with only its own fix reverted,
not merely observed to pass:

• response_handling.a_token_that_is_already_cancelled_fails_the_listener_immediately (+ the
  after-registration twin as the control)
• per_destination_lane_dispatch.disposal_abandons_the_commands_still_queued
• per_destination_lane_dispatch.disposal_gives_up_on_a_lane_that_ignores_cancellation

Both dispatcher tests are deliberately written so the unfixed code fails them on an assertion
rather than by hanging — the first version deadlocked the test runner under mutation, which for a
regression test whose subject is a shutdown that never returns rather defeats the point.

Verified: CoreTests 2230/2230 green; all three SlowTests.Agents scale classes green (5m10s);
dotnet build wolverine.slnx -c Release clean, 0 warnings.

Deliberately not done

WolverineRuntime.StopAsync(CancellationToken) accepts the host's shutdown token and never uses it,
so HostOptions.ShutdownTimeout is inert for every slow teardown path. Threading it through looks
tempting but would cut off stopAllAgentsAsync, which GH-3604 deliberately built to use the whole
SIGTERM grace window for thousands of subscription shards — a 30s default would start SIGKILLing
mid-drain, which is the failure that change existed to fix. The bounded lane wait is the safer
backstop. Worth its own issue if we want ShutdownTimeout to mean something here.

Follow-on for #3779

With this in, agent_reassignment_at_scale.DisposeAsync needs no timeout guard and SlowTests can
go into a CI job. Bounding the teardown in the test alone would have made CI green while hiding a
production shutdown wedge.

🤖 Generated with Claude Code

https://claude.ai/code/session_0116vfBcKwcjWn8msM4ZjkuA