ProjectionCoordinatorBase.StartAsync disposes the old CancellationTokenSource without cancelling it, orphaning a live leadership loop

[jeremydmiller]

ProjectionCoordinatorBase.StartAsync replaces the cancellation source without cancelling the old one:

_cancellation?.SafeDispose();          // disposes, never cancels
_cancellation = new CancellationTokenSource();
_runner = Task.Run(() => executeAsync(_cancellation.Token), _cancellation.Token);

Disposing a CancellationTokenSource does not cancel it, and _runner is overwritten, so a second StartAsync leaves the previous executeAsync loop running with a token that will never be cancelled and a task no drainRunner() will ever await. ResumeAsync() is just StartAsync(default), so any ResumeAsync that is not preceded by PauseAsync orphans a live leadership loop for the rest of the process.

Consequences observed:

1. The orphan keeps competing for leadership after the coordinator is stopped. StopAsync drains only the newest runner and then calls distributor.ReleaseAllLocks(). The orphan's next poll sees HasLock == false and re-attains the leadership advisory lock — a stopped coordinator silently holding leadership again.
2. It strands the advisory-lock handle. Because the orphan is still polling while ReleaseAllLocks → AdvisoryLock.DisposeAsync drains, an acquire that completes after the drain lands in a dictionary nothing will dispose (AdvisoryLock strands a handle acquired after DisposeAsync — permanent 'idle in transaction' advisory-xact-lock session weasel#396). With Marten's default transaction-scoped advisory locks that leaves a backend idle in transaction on pg_try_advisory_xact_lock until process exit, which then permanently defeats Marten's high-water gap-liveness check — see Stopped host can leak the HotCold coordinator's advisory-xact-lock session ("idle in transaction"), which 9.16.1 gap detection treats as an eternal in-flight append — stalling every later daemon in the same process marten#5090.

Verified against Marten 9.21.0, 6/6 iterations: start a HotCold host, call IProjectionCoordinator.ResumeAsync() with no intervening PauseAsync, await host.StopAsync() — and the advisory-lock session is still open and idle in transaction after StopAsync returns every time. Without the extra ResumeAsync, 15/15 boot/stop/dispose cycles are clean, because PauseAsync cancels and drains before the locks are released.

Suggested fix

Make StartAsync idempotent-safe: cancel and drain any existing runner before starting a new one, rather than dropping it on the floor.

if (_cancellation != null)
{
    await _cancellation.CancelAsync();
    await drainRunner();
    _cancellation.SafeDispose();
}

That makes a double StartAsync / bare ResumeAsync equivalent to PauseAsync + StartAsync, which is what every caller already assumes. Worth asserting in a test that two StartAsync calls leave exactly one live loop.