Simulate slow agent starts in the distribution harness so #3753's convergence failures reproduce at dev scale

[jeremydmiller]

Follow-up to #3753 and to JasperFx/jasperfx#598 (move the blue/green side-effect gate warm-up out of the agent start path).

Why

Every issue in the #3753 chain — #3748, #3749, #3750, JasperFx/jasperfx#594 and now JasperFx/jasperfx#598 — takes a slow agent start as its precondition. #3748 and #3750 genuinely require one; #3749's phantom stops look structural but are amplified by one.

Right now nothing in that chain reproduces at dev scale. The only verification site is the reporting customer's restored-production canary (512 tenant databases, ~6,500 agents, 5 nodes), which means:

• every candidate fix has to be shipped as a pinned prerelease and measured by someone else, on their schedule;
• we cannot write a regression test for any of the three defects already closed, so nothing stops them coming back;
• the emergent behaviour that actually bites — the observed peak of 60 concurrent warm-ups, the total assigned-agent count going down over a four-minute window — is invisible to dotnet test.

FakeAgent.StartAsync currently returns Task.CompletedTask. That is precisely the assumption the whole chain violates.

What to build

A test seam that makes agent start cost configurable, then a distribution test at a scale that matters:

• FakeAgent: a settable start/stop delay (and ideally a per-agent one, so a tail can be simulated rather than a uniform cost — the field data is p50 27 s / p95 82 s / tail 215 s, and it is the tail that breaks things).
• FakeAgentFamily: an agent count parameter. Names is a hardcoded 12; Node assignment is very slow, or never completes, when a release contains a projection version bump #3753 is about ~6,500.
• Optionally a virtual clock so a 200-minute convergence can be asserted without a 200-minute test.

src/Testing/CoreTests/Runtime/Agents/scale_safe_batch_starts.cs is the closest existing precedent and probably where this lands.

Assertions worth having

• With N agents whose starts cost a distribution with a long tail, every agent is assigned and running within a bounded number of distribution rounds.
• The total count of assigned agents is monotonic — the four-minute window where it went 2,401 → 1,885 is the clearest signal something is wrong, and it is cheap to assert.
• No node sits at zero while another holds everything (the 799 / 0 / 0 / 0 / 1 shape).
• A partially-fulfilled AssignAgents chunk is not treated as complete (regression guard for A partially-fulfilled AssignAgents chunk is treated as complete, so agents still starting get re-placed onto other nodes #3750).

Caveat, stated up front

A simulation is not a substitute for the canary. The pathologies in #3753 are emergent from real database contention, real node-table timing and real chunk sizes, and a fake-agent harness will reproduce the shape of the problem, not its magnitude. Sign-off on "a deploy carrying a projection version bump converges quickly on its own" almost certainly still needs the reporting customer's cluster, and they have offered to run diagnostics against a pinned prerelease.

The value here is the other direction: a dev-scale reproduction lets us iterate and reject bad fixes in minutes instead of round-tripping every candidate through someone else's production restore, and it gives the three closed defects regression coverage they do not currently have.

🤖 Generated with Claude Code

[jeremydmiller]

PR #3780 addresses this, but the issue's premise needs a correction first, and the follow-up is now blocked on a separate defect.

The harness already exists

src/Testing/SlowTests/Agents/ was built during the #3748/#3749/#3750 work and already does what this issue asks for, against real hosts with real delays:

• SlowStartingAgents.cs — AgentStartTelemetry with a per-agent DelayOverride (the tail, not a uniform cost), a mutable StopDelay, GrowUniverseTo for the version-bump deploy shape, and a cluster-wide PeakInFlight high-water mark
• agent_assignment_at_scale — the whole universe distributed across every node
• agent_reassignment_at_scale — already asserts the monotonicity this issue asks for, as worstBackslide against the field's 2,401 → 1,885
• slow_starts_outrun_reply_windows — chunks slower than the reply window converge without duplicate starts

So "nothing in that chain reproduces at dev scale" is out of date. The real gap is narrower and worse:

SlowTests runs in no CI workflow. Zero jobs in tests.yml, no Nuke target. Three well-built regression tests for this chain run nowhere, which is why they were easy to miss.

What #3780 adds

The fast tier underneath them: slow_agent_start_convergence in CoreTests, which drives the real NodeAgentController.EvaluateAssignmentsAsync round after round against a simulated 5-node / 603-agent cluster with a long-tailed start distribution. 0.5s, seeded and deterministic, in a job that already runs on every commit — where the SlowTests versions are minutes apiece and wall-clock-sensitive.

It answers this issue's "assertions worth having" list directly: every agent converges; the running count never goes backwards; no node starved while another holds everything; a partially-fulfilled AssignAgents chunk is not treated as complete. Plus the FakeAgent / FakeAgentFamily seams asked for (per-agent start/stop delay, configurable agent count, StartDelayPolicy for the tail).

Every assertion was verified against two mutations of the product code rather than merely observed to pass — details in the PR.

Why the CI wiring is not in that PR

I went to add a CIAgents job and could not, because agent_reassignment_at_scale never finishes. Its assertions all pass; the host shutdown in DisposeAsync then hangs forever, so dotnet test never reports. Reproduced 4/4. Bounding the teardown stop at 20s makes the suite pass in 2m18s.

Filed as #3781 with the stack evidence. That has to land before SlowTests can be a CI job — bounding the teardown inside the test would make CI green while hiding a real hang.

Still outstanding here

• SlowTests wired into a CI job (blocked on Balanced-mode host shutdown hangs indefinitely when the node holds a large number of agents #3781)
• the virtual clock, which only pays for itself once those tests are in CI and their wall clock matters

🤖 Generated with Claude Code

https://claude.ai/code/session_0116vfBcKwcjWn8msM4ZjkuA

[jeremydmiller]

Closing this — everything it asked for is in main, and the harness has already earned its keep.

The asks, mapped

Ask	Where it landed
FakeAgent settable start/stop delay	FakeAgent.StartDelay / StopDelay, plus StartCount to tell a start that is merely slow from one the leader has re-driven
per-agent delay, so a tail can be simulated rather than a uniform cost	FakeAgentFamily.StartDelayPolicy, applied in BuildAgentAsync
FakeAgentFamily agent-count parameter	FakeAgentFamily(string scheme, int agentCount) — plus an explicit-names overload so grouped, versioned agents can be modelled the way EventSubscriptionAgentFamily URIs are
"optionally a virtual clock"	Deliberately not built. The sim measures time in evaluation rounds instead. The field distribution is p50 27s / p95 82s / a 215s tail against a ~5s health check, so wall-clock would take hours and trade a deterministic assertion for a timing race. Rounds are seeded and reproduce exactly.

The assertions

All four, in CoreTests/Runtime/Agents/slow_agent_start_convergence.cs:

• bounded convergence — every_agent_converges_despite_a_long_tailed_start_distribution
• no starved node (the 799/0/0/0/1 shape) — no_node_is_starved_while_another_holds_everything
• partial AssignAgents chunk is not treated as complete (A partially-fulfilled AssignAgents chunk is treated as complete, so agents still starting get re-placed onto other nodes #3750) — covered here and in partially_confirmed_assignments.cs
• monotonic assigned count — folded into a_first_placement_wave_emits_no_stops_and_no_reassignments rather than left standing on its own. Against a stable node set the count can only fall if agents that had come up were stopped, so it is strictly implied by the zero-churn assertion; under mutation it failed for neither mutant, which is the definition of a test that cannot fail. It is documented there as the shape an operator actually sees.

Plus an_agent_is_never_started_on_two_nodes_at_once for GH-3698's failure mode, which the issue did not ask for and is the worst outcome in the chain.

The gap this issue actually surfaced

Writing it turned up something the issue did not know: SlowTests ran in no CI workflow at all — no job in tests.yml, no Nuke target. The real-host reproductions of this whole chain were dead code as far as CI was concerned.

That is now .github/workflows/slow-tests.yml — workflow_dispatch, 60-minute cap, deliberately out of the PR path (it is wall-clock bound by design; the first run of the whole project was still going when the standard 20-minute cap killed it). It is run before a release and after anything touching agent assignment.

Why the caveat in this issue held up

A simulation is not a substitute for the canary.

Right, and it is written into the sim's own docs. But the other direction paid off faster than expected: fixing the re-decision churn in #3852 exposed that StopAgents was still serial while StartAgents got bounded parallelism back in GH-3604. Nothing in the fast tier caught it, but SlowTests/Agents did, and the resulting numbers were 38.0s → 14.0s to fresh nodes and 176.3s → 31.1s to convergence. Without a harness that can express a slow start, that regression ships.

[jeremydmiller]

Closing per the comment above — every ask is in main, and the SlowTests CI gap it surfaced is closed by the manual slow tests workflow. Reopen if the harness turns out to be missing a shape you need.