Make the blue/green side-effect gate survivable at scale (#594) - #596
Merged
Conversation
The gate from #480 runs its bounded, side-effect-suppressed warm-up replay inside the agent start path against a HARDCODED 5-minute ceiling. Reported from a 512-tenant-database deployment where warm-ups measure 27.3s p50, 81.8s p95 and 288.5s max -- twelve seconds inside the ceiling, so start failures were the next sample rather than a tail risk. This is the patch half of #594; moving the warm-up out of the start path entirely is tracked separately. - The ceiling is now DaemonSettings.SideEffectGateTimeout (default unchanged at 5 minutes). A non-positive value or Timeout.InfiniteTimeSpan opts out of the separate bound, matching StopAndDrainTimeout's semantics from #564. - A timeout is no longer automatically a failure. The gate re-reads PERSISTED progression and treats a shard that reached the prior version's mark as warmed up regardless of the clock. The reporter saw a shard fail its start three times on this timeout while its progression sat at exactly the prior mark, each retry re-running a warm-up that had nothing left to do. warmup.LastCommitted is not usable here because ReplayAsync disposes the agent on its way out, so the read goes to the database. - A genuine timeout now publishes a Paused shard state instead of leaving the shard silently stopped, so it is visible to a supervisor and resumes from its persisted floor. The disposed warm-up agent is still deliberately not registered -- it would report Running. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jeremydmiller
added a commit
that referenced
this pull request
Aug 3, 2026
…ath (#598, #610) (#615) * fix: never let one property getter lose a whole OptionsDescription (#590) An OptionsDescription is a diagnostic view built by calling arbitrary property getters on somebody else's configuration object, so any one of them can throw -- and losing the entire description over one bad property is a terrible trade. Reported from the field as JasperFx/wolverine#3740, where AzureServiceBusTransport.HostName threw a NullReferenceException for credential-based connections and a monitored service could consequently never build its ServiceCapabilities snapshot at all. - Skip set-only properties and indexers, neither of which can be read via PropertyInfo.GetValue(subject) (ArgumentException / TargetParameterCountException). Wolverine.Pulsar's PulsarTransport has a this[Uri] indexer, which was enough to make it undescribable. - Catch whatever a getter throws -- including from the [ChildDescription], [DescribeAsStringArray] and [DescribeAsConfigurationState] branches -- and record OptionsValue.Unreadable(...) in its place. - Report the exception TYPE only, never the message: descriptions get shipped to monitoring consoles and work hard to keep secrets out, and exception messages habitually quote the offending configuration value. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RainvDHdde5JUAiGEPEgWw * feat: move the blue/green side-effect gate warm-up out of the agent start path (#598, #610) The #480 gate ran its side-effect-suppressed warm-up as a bounded replay INSIDE startContinuousShardAsync, so a shard was not started -- and therefore, to a distributing host, not assigned -- until the replay finished. #596 made that survivable; this removes it. The agent now starts immediately and carries the warm-up itself: it runs Continuous from its own persisted progress with side effects suppressed, clamps its loading ceiling to the prior version's mark so no page straddles it, and enables side effects the moment COMMITTED progression reaches that mark. Crash safety is unchanged and comes from the same place -- the trigger is "persisted progress < mark", so an interrupted warm-up resumes suppressed. Suppression is a first-class flag on the agent rather than Rebuild mode. Rebuild mode also switches on the #525 deferred-write accumulator, whose flush waits for a range reaching the agent's high-water -- which a clamped warm-up never reaches -- so a deferred window would never flush, progression would stall below the mark and the gate would never lift. The executions keep the real mode for their own bookkeeping and pass Rebuild down only for the per-slice apply, which is where side effects are raised. Also: - MaxConcurrentSideEffectGateWarmupsPerDatabase, the warm-up concurrency control both issues asked for. Waiting for a slot never blocks the agent -- it is started, assigned and heartbeating throughout. - ShardState.SideEffectsSuppressed / SideEffectGateMark, so an operator can tell "running but not emitting side effects" from "running normally" without pod logs. - The optimized-replay shortcut is now guarded on suppression. A gated fresh deploy starts Continuous at LastCommitted 0, which is exactly that shortcut's trigger, and a store replay executor replays to its own high-water rather than the gate mark. - DaemonSettings.SideEffectGateTimeout is obsolete and ignored: there is no start-blocking wait left to bound. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(test): wait for the Started state instead of snapshotting it CI reproduced this on net9 as "Sequence contains no matching element" from the Last() call: ShardStateTracker publishes through a Block<ShardState>, so a subscribed observer runs on the block's consumer thread and is not guaranteed to have run by the time StartAllAsync returns. The test read the collected states straight after the call, which is the same race main's #609 fixed in the suite this branch deletes. The change was written while resolving that conflict but never staged -- `git diff HEAD~1` compares against the WORKING TREE, so the check that was supposed to confirm it landed in the merge commit was reading the uncommitted edit back to itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #594 (the patch half — see below).
The gate added in #480 runs its bounded, side-effect-suppressed warm-up replay inside the agent start path, against a hardcoded 5-minute ceiling. @erdtsieck has now run it at scale — one Marten store over 512 tenant databases, ~6,500 agents — and measured the warm-ups:
288.5s against a 300s ceiling is twelve seconds of headroom, so start failures were the next sample rather than a tail risk. And because a start that takes minutes blows Wolverine's agent-command acknowledgement windows, a rolling deploy carrying a projection version bump could not converge at all (JasperFx/wolverine#3748, #3749, #3750).
What changed
The ceiling is configurable —
DaemonSettings.SideEffectGateTimeout, default unchanged at 5 minutes. A non-positive value orTimeout.InfiniteTimeSpanopts out of the separate bound entirely, the same semanticsStopAndDrainTimeoutgot in Make the daemon per-shard StopAndDrain timeout configurable (currently hardcoded 5s) #564, which this deliberately mirrors rather than inventing a new shape.A timeout is no longer automatically a failure. The gate re-reads persisted progression and treats a shard that reached the prior version's mark as warmed up regardless of the clock. This is the concrete field symptom:
provided-cares:V23for one tenant failed its start three times onSideEffectGateTimeoutwhile its progression sat at exactly its high-water mark — the replay had finished, only the wait expired, and each retry re-ran a warm-up with nothing left to do. The check has to go to the database:warmup.LastCommittedis not usable becauseReplayAsyncdisposes the agent in itsfinallyon the way out.A genuine timeout leaves the shard observably Paused rather than silently stopped, so a supervisor can see it and it resumes from its persisted floor on the next start. The disposed warm-up agent is still deliberately not registered — the existing comment is right that it would misreport as
Running— so the Paused state is published through the tracker instead.Not in this PR
The architectural half of #594 — moving the warm-up out of the start path entirely (start the agent side-effect-suppressed and let normal shard execution carry it to the prior mark), plus a warm-up concurrency control independent of agent-start batching. That is the change that removes the condition; this one makes the condition survivable and is small enough to ship now.
Tests
EventTests/Daemon/SideEffectGateTimeoutTests.csdrives the realJasperFxAsyncDaemonover a substituted store/database with an execution that never completes its rebuild, so the gate's timeout fires deterministically: the default is still 5 minutes; a timeout that reached the prior mark starts continuous execution; one that did not publishes Paused and starts nothing; and zero/negative/infinite all opt out of the bound. FullEventTests(657) green.🤖 Generated with Claude Code