Move the side-effect gate's warm-up out of the agent start path (the architectural half deferred from #594)

[erdtsieck]

Follow-up to #594. PR #596 shipped what it called the patch half and explicitly left this out:

Not in this PR — The architectural half of #594 — moving the warm-up out of the start path entirely
(start the agent side-effect-suppressed and let normal shard execution carry it to the prior mark), plus a
warm-up concurrency control independent of agent-start batching. That is the change that removes the
condition; this one makes the condition survivable and is small enough to ship now.

There is no open issue tracking that half, so this is it. I have now run the patch half at full scale and can
report both what it fixed and exactly what the remaining shape costs.

The patch half works — that part is settled

Same cluster as #594 (one Marten store over 512 tenant databases, 993 tenant shards of the bumped projection,
~6,485 agents, 5 nodes), same restored production snapshot, one projection bumped ProvidedCares 21 → 23.
Before: JasperFx 2.36.3. After: 2.37.2 with Wolverine 6.24.4.

	before (2.36.3)	after (2.37.2)
gate failures leaving a shard stopped	50 in one hour on one pod	0 across all pods
shards failing their start repeatedly	3 tenants at attempt 3	0
a timed-out warm-up that had reached the mark	retried from scratch	correctly treated as warmed up

Re-reading persisted progression instead of trusting the clock removed the whole retry storm. Thank you —
that was the right call and it is visibly effective.

What the remaining shape costs

Because the warm-up still runs inside the start path, an agent counts as assigned only once its replay has
finished. The practical effect is that the cluster's assignment table is a progress bar for the catch-up
rather than for the distribution, and it advances in bursts:

• while warm-ups run, agent starts across the whole five-node cluster drop to 3 in five minutes;
• then a chunk of ~400 lands at once;
• repeat.

Measured during one such pause: 65 warm-ups in flight simultaneously (median age 52s, oldest 159s, none
near the 300s ceiling). Nothing is wrong — every node is healthy, there are no timeouts, no churn, zero stops
— the cluster is simply busy replaying, and no placement can be confirmed until it stops being busy.

Overall: 2,916 of 6,485 agents after 25 minutes, on track for roughly an hour. The pacing item is a single
tenant with 899,979 events, replaying at 27,316 events/min — about 33 minutes on its own, during which the
agents queued behind it cannot start.

Why this matters beyond the clock

Three things fall out of coupling the two, and none of them is fixed by a longer ceiling:

1. You cannot tell a healthy deploy from a stuck one. During the hour, the assignment table looks exactly
   like the failure mode we reported in Node assignment is very slow, or never completes, when a release contains a projection version bump wolverine#3753 — nodes at zero, totals crawling. The only way
   to distinguish them is to go read pod logs for warm-up lines. An operator watching the obvious table gets
   the wrong answer.
2. Distribution decisions are made against a stale picture. The leader is placing agents whose starts will
   not confirm for minutes; every rebalance during that window is reasoning about a fleet that does not exist
   yet. Wolverine has now hardened against the consequences (Agent command acknowledgements wait for the agent work to finish, so a slow start or stop always times out (6.24.2) wolverine#3748, #3749, #3750, all
   fixed and confirmed working here), but it is treating a symptom.
3. Warm-up concurrency is emergent, not chosen. The 65 above is not a setting — it is however many
   gate-needing shards happened to fall in the in-flight chunk, which depends on AgentStartBatchSize and on
   what fraction of agents belong to the bumped projection (993 of 5,972 here). There is no way to say "use
   twelve replay workers", which is the knob an operator actually wants.

What we would like

Exactly what #596 described as deferred:

1. Start the agent side-effect-suppressed and let normal shard execution carry it to the prior mark. The
   start returns immediately, the assignment lands, and the catch-up becomes ordinary async projection work
   that the daemon already knows how to schedule, report on and pace.
2. A warm-up concurrency control independent of agent-start batching, so the replay load is a deliberate
   number rather than a side effect of how the distribution layer chunks agents.

Together those turn a version-bump deploy from "an hour in which the cluster is indistinguishable from broken"
into "assignment completes in seconds, projections catch up in the background where you can watch them".

Environment

JasperFx 2.37.2 / Marten 9.22.2 / Wolverine 6.24.4, .NET 10, 5 pods, DurabilityMode.Balanced with
UseWolverineManagedEventSubscriptionDistribution, one Marten store over 512 tenant databases with
Events.UseTenantPartitionedEvents. Canary on a restored production copy, so we can reproduce at full scale
and iterate quickly — happy to run whatever would be useful, including a build that moves the warm-up out of
the start path.

Related: #594 and #596 (the patch half), JasperFx/wolverine#3753 (the outcome-level report, still open until a
version-bump deploy converges quickly on its own).