Move the blue/green side-effect gate warm-up out of the agent start path (design half of #594)

[jeremydmiller]

Follow-up to #594, which was closed with only its patch half shipped in 2.37.0. This issue tracks the
half that actually removes the cost, and it is the one the reporting customer says gates them: "this is
the one that would let us deploy a projection version bump at all."

What 2.37.0 already fixed

• DaemonSettings.SideEffectGateTimeout replaces the hardcoded 5-minute ceiling (non-positive or
  Timeout.InfiniteTimeSpan opts out), mirroring StopAndDrainTimeout.
• A gate timeout is no longer automatically a failure. tryApplySideEffectVersionGateAsync re-reads
  persisted progression and treats reaching the prior mark as warmed up. This directly addressed the
  field report of a shard failing its start three times while its progression sat exactly on its
  high-water mark.
• A genuine timeout publishes a Paused shard state rather than leaving the shard silently stopped.

That converts a hard failure into a slow success. It does not make it faster.

What remains

The #480 blue/green gate still runs its side-effect-suppressed warm-up replay synchronously inside the
agent start path (startContinuousShardAsync → tryApplySideEffectVersionGateAsync → rebuildAgent,
before tryStartAgentAsync). A shard's first start therefore costs a full replay from the current
progression to the prior version's mark before the agent is considered started at all.

Measured on a restored-production canary — 512 tenant databases, one projection version-bumped across 993
tenant shards:

warm-up p50	27 s
warm-up p95	82 s
warm-up tail	215 s (an earlier sample measured 288.5 s against the then-fixed 300 s ceiling)
throughput	4.7 warm-ups / minute
peak observed warm-up concurrency	60
time before every shard has started once	~200 minutes

A shard start that normally costs milliseconds costs tens of seconds to minutes, for every shard, on every
version-bumped deploy.

Why this matters beyond the latency itself

Slow agent starts are the stated precondition for three open Wolverine issues:
JasperFx/wolverine#3748 (acknowledgement windows), JasperFx/wolverine#3749 (reassignment lane blocking)
and JasperFx/wolverine#3750 (partially-confirmed assignment chunks). Fixing those makes the precondition
survivable; fixing this removes it.

One caveat worth recording so the causal claim is not over-read: #3750 and #3748 do genuinely require slow
starts. #3749's phantom stops for a source absent from the node table look structural rather than
warm-up-induced — slow starts would widen the window and amplify it, not create it. So this issue should
not be treated as a reason to drop #3749.

Proposed direction

1. Move the warm-up out of the start path. Start the agent in Continuous mode but
   side-effect-suppressed, and let normal shard execution carry it to the prior version's mark,
   flipping side effects on when it crosses. The shard is then observably running and assignable from the
   first moment, and the replay is ordinary catch-up work rather than a blocking precondition. Note this
   changes what DetermineStartingPositionAsync and the gate currently rely on — today the gate's value
   comes precisely from the persisted progress being at the prior mark before Continuous starts.
2. Give warm-ups their own concurrency control, independent of agent-start batching. The observed peak
   of 60 concurrent warm-ups is an emergent property of how the distribution layer chunks agents; it is not
   something an operator can size today, and it is the knob that decides whether 993 warm-ups saturate the
   databases or trickle.

Open questions

• With side effects suppressed on a running Continuous agent, what observes the crossing of the prior
  mark, and how is that made crash-safe? The current design gets crash-safety for free because the gate
  triggers on current < prior and resumes from the persisted floor.
• Should the suppressed window be visible in ShardState / the console, so an operator can tell "running
  but not yet emitting side effects" from "running normally"? Today the distinction is invisible once the
  gate returns.
• Does an opted-in projection need a way to declare that its side effects are safe to replay, letting the
  gate be skipped entirely rather than optimized?

Verification

Nothing in this chain reproduces at dev scale. The reporting customer has a restored-production canary at
full scale (512 databases, ~6,500 agents) and has offered to run diagnostics against a pinned prerelease —
agreeing that measurement protocol is likely a precondition for validating any fix here.

[jeremydmiller]

Filed JasperFx/wolverine#3779 for the verification half of this: a test seam that makes FakeAgent's start cost configurable (with a long tail, since the field data is p50 27 s / p95 82 s / tail 215 s) plus an agent count on FakeAgentFamily, so #3753's convergence failures reproduce at dev scale instead of only on the reporting customer's restored-production canary.

That does not replace the canary — the pathologies here are emergent from real database contention and real chunk sizes, so sign-off on the design change proposed in this issue will still need feedback from the actual users. But it lets us reject bad fixes in minutes rather than round-tripping every candidate through someone else's production restore, and it gives the three already-closed defects (JasperFx/wolverine#3748, #3749, #3750) regression coverage they don't have today.

Holding this issue open pending that, and pending user feedback on the open questions above — particularly what observes the crossing of the prior mark on a running side-effect-suppressed agent, and how that is made crash-safe.