CIPolecat red on main since 1dfdd42a1: bisected to a version-string bump, mechanism unknown

[jeremydmiller]

CIPolecat has been red on main since 1dfdd42a1. Bisected to a commit whose entire content is a version-string bump, which cannot be the mechanism — so something reads $(Version) at a distance and I could not find what.

Status: cause localised with high confidence, mechanism unknown. Nothing fixed.

The failure

18 tests fail, all with the same exception, clustered in two namespaces:

[FAILED] PolecatTests.Requirements.using_data_requirements.document_does_not_exist_attribute_happy_path
  — Wolverine.Runtime.Routing.IndeterminateRoutesException : Could not determine any valid
    subscribers or local handlers for message type PolecatTests.Requirements.EnsureNoDuplicatePcThing

failing namespace	count
PolecatTests.Requirements.using_data_requirements	28 log lines (~18 unique)
PolecatTests.strong_typed_identifiers	4
PolecatTests.handler_actions_with_returned_StartStream	2
PolecatTests.handler_actions_with_implied_polecat_operations	2

Exception breakdown across the run: 54 IndeterminateRoutesException, 24 RequiredDataMissingException, 24 ShouldAssertException, 1 InvalidOperationException. The supervisor reports
43 passed, 25 passed on retry, 18 failed (25 retries, 27 worker processes) — identically on every red run.

This is a handler-discovery failure: the message types have no chain, so InvokeAsync cannot route them.

The bisect

Six runs, each a workflow_dispatch on a throwaway branch with the tests.yml matrix trimmed to CIPolecat only (one job per run instead of 28). All within the same ~40 minute window, so runner-image and upstream-package drift are held constant.

commit	content	result	runs
7e8f8ba86	GH-3774 — completion continuations honor the metrics-silent tracker	pass	30726582635
57464ba91	GH-3773 — RavenDb/CosmosDb dead-letter readers	pass ×3	30727034697, 30727226425, 30727230786
1dfdd42a1	Bump to 6.24.4	fail ×3	30726791466 + organic 30724675101 (main) and 30725560867 (PR #3775)

git diff between the passing and failing probe branches is one line:

-        <Version>6.24.3</Version>
+        <Version>6.24.4</Version>

Under a coin-flip null, 3/3 vs 3/3 is p ≈ 1.6%; counting the two organic failures it is 5/5 red at the bumped version. Bounded on the other side by 11 consecutive green CIPolecat runs on main through 3ad7986b6.

Ruled out (tested, not assumed)

• GH-907 follow-up: MessageSucceededContinuation bypasses the metrics-silent tracker, so system traffic still records succeeded/effective-time #3774 — green at its own sha. Its diff adds an optional IMessageTracker parameter and a MessageContext.Tracker property; it never touches routing or discovery. This was my initial suspicion purely from filename adjacency (NoHandlerExecutor, NoHandlerContinuation) and it is wrong.
• RavenDb and CosmosDb dead-letter readers still abort the whole queue on one unreadable body #3773 — green ×3, and it touches only DeadLetterMessage.cs in the RavenDb and CosmosDb projects, which PolecatTests does not reference.
• NuGet resolution. Restore graphs at the two commits are identical — 112 libraries each, differing only in the version labels on the in-solution project references. There is no WolverineFx PackageReference anywhere; PolecatTests reaches Wolverine purely through ProjectReference. The Polecat package depends on JasperFx/JasperFx.Events only, never on WolverineFx.
• RememberedApplicationAssembly (process-wide static) silently pins handler discovery for every later implicit host — order-dependent test failures with no loud signal #3521 application-assembly pin. Zero occurrences of the "adopted application assembly … but this host was registered from …" warning in the failing logs.
• Worker crash / infrastructure. No crashed or indeterminate workers, no connection failures, no OOM. The 43 passed / 43 failed-on-attempt-1 split is a whole-worker wipeout, but the worker did not die — every one of its tests threw the same routing exception.
• A newly published Polecat. 5.9.1 was published 2026-07-31T22:39, before both the last green and the first red run.
• A missing package asymmetry. Both WolverineFx 6.24.3 and 6.24.4 are published on nuget.org.

Does not reproduce locally

Green in every local configuration tried, on macOS/arm64 against the docker-compose SQL Server:

• the failing class alone — 14/14
• the full CIPolecat shard filter in one process — 86/86
• the real supervisor via ./build.sh CIPolecat --disable-test-retry — 86/86
• all of the above at 1dfdd42a1, the known-red sha
• and again with Polecat pinned to 5.9.1

Leads worth taking first

1. Something reads $(Version) at runtime. That is the whole puzzle. Both failing areas are source-generator territory (JasperFx.Events.SourceGenerator, StronglyTypedId — note strong_typed_identifiers is among the failures), and Wolverine's runtime codegen is the obvious place an assembly version could leak into type resolution. Start by diffing the generated code between the two versions rather than the source.
2. Why only in CI. The local/CI gap is real and is worth closing on its own (see below) — whatever differs there is likely the same thing that makes the version visible.

Adjacent finding worth fixing regardless

Polecat is the only floating dependency in Directory.Packages.props:

<PackageVersion Include="Polecat" Version="[5.7.0,6.0.0)" />

A warm local cache resolves 5.7.0; CI restores fresh and gets 5.9.1. That is a silent two-minor-version gap between what a developer tests and what CI tests. It is not the cause here — pinning 5.9.1 locally still passes — but it will eventually produce a "works on my machine" that costs someone a day.

Things that will bite you

• CIPolecat is one of three shards. PolecatTests is split by namespace across CIPolecat / CIPolecatWorkflow / CIPolecatSagas (CI: CIAWS and CIPolecat jobs chronically hit the 20-minute execution timeout #3350); the failing namespaces live in the CIPolecat catch-all. Running the whole project locally is not the same test.
• The identical 43/25/18 on every red run is not by itself evidence of determinism. With 2 worker processes and a deterministic class partition, any whole-worker failure yields the same split. It took a controlled pass/fail pair at adjacent commits to establish determinism.
• Trimming the tests.yml matrix to a single target turns a bisect probe from 28 jobs into 1. Cheap enough to resample properly, which is what separated signal from luck here.

Reproduction harness

Three throwaway branches carry the trimmed-matrix probes, one per bisect point. The repo ruleset blocks branch deletion, so they are still there and are directly re-runnable with gh workflow run tests.yml --ref <branch>:

• tmp/bisect-gh3774 — at 7e8f8ba86 (green)
• tmp/bisect-mid — at 57464ba91 (green)
• tmp/bisect-tip — at 1dfdd42a1 (red)

They should be deleted once this is closed.

🤖 Generated with Claude Code

https://claude.ai/code/session_013eR4GL278688VhyhrGcttJ

[jeremydmiller]

Found it. Handler discovery was scanning xunit.v3.core instead of PolecatTests. Fix in #3777, validated in CI at the red sha.

The mechanism

The application assembly is resolved by walking the stack for the first frame outside the framework that isn't System*/Microsoft* — JasperFxOptions.DetermineCallingAssembly (JasperFxOptions.cs:243) and Wolverine's own determineCallingAssembly. Under an async test fixture the frames between the framework and the test class belong to the runner, so xunit.v3.core satisfied the filter and got adopted. ReSharperTestRunner is already on that skip list; xUnit was never added.

With the runner adopted, discovery scans an assembly containing none of the suite's handlers, HandlerGraph.CanHandle is false for every test message type, and InvokeAsync throws IndeterminateRoutesException.

Instrumented runs, all at 1dfdd42a1:

branch	adopted assembly	result
tmp/bisect-probe (6.24.4)	xunit.v3.core, 79 of 82 hosts, 0–15 chains	43 failed
tmp/bisect-probe-mid (6.24.3)	PolecatTests, 82 of 82, 141 chains	86 passed
tmp/bisect-probe-fix (6.24.4 + fix)	PolecatTests, 82 of 82, 141 chains	86 passed, 0 retries

Note the shape: 6.24.3 is uniformly correct, 6.24.4 is variable. That is a stack walk producing an unstable answer, not a configuration difference.

Corrections to the issue above

"18 tests fail" is wrong — 43 do. MaxRetriesPerRun = 25 (build/SupervisedTests.cs:142) capped the run at 25 retries, and all 25 passed. The other 18 were never retried; they are budget exhaustion, not a harder failure. Every one of the 25 fresh retry processes adopted PolecatTests — that is why they passed, and it was the strongest clue in the original logs.

The GH-3521 warning evidence was void. The warning fires on 81 of 82 hosts on a healthy local run, because RegistrationCallingAssembly resolves to Wolverine.SqlServer. Worse, the RememberedApplicationAssembly branch (WolverineOptions.Assemblies.cs:118-121) never calls the divergence check at all, so that path cannot warn. Its absence from the CI logs proved nothing.

"Diff the generated code, a hash may embed the version" — ruled out. ToSuffixedTypeName hashes type.FullNameInCode(); no version anywhere in it. Also ruled out: xUnit discovery order is byte-identical at the two versions (--list-tests diff is empty), and Bobcat returns discovery order verbatim at laneCount <= 1 (WorkPlan.cs), so ordering is not the channel either.

The bisect held up. Re-running both probe branches a day later reproduced tip-red / mid-green, so the version correlation is real, not luck.

What the version actually does — and what I could not explain

The failing set maps exactly onto the defect: every class in the shard that builds a host, invokes a message, and relies on conventional discovery failed; every class that pins ApplicationAssembly, calls Discovery.DisableConventionalDiscovery(), or never invokes a message passed. That is a static property, which is why the 43/25/18 split was identical on every red run.

What I could not determine is why a version string changes the stack-walk outcome. It clearly does — 82/82 correct at 6.24.3 vs 79/82 wrong at 6.24.4, same runner image, minutes apart. I tested and rejected the obvious candidate: the HasReferenceToJasperFxTool short-circuit at JasperFxOptions.cs:212 returns False at both versions locally with no reference-load failures, so it is not flipping. The remaining plausible channel is JIT inlining changing which frames survive on an async continuation, which would also explain why this never reproduced on macOS. I did not prove that, and the fix removes the dependency, so I stopped there.

Follow-ups

Filed upstream:

• DetermineCallingAssembly adopts the test-runner assembly, so type discovery scans a runner instead of the test assembly jasperfx#600 — the durable fix. DetermineCallingAssembly adopts the test runner; 2.37.0 has no test-runner exclusion at all, so Marten and Polecat hosts carry the same exposure. Never adopt a test-runner assembly as the application assembly (GH-3776) #3777 defends Wolverine at the boundary only.
• DetermineCallingAssembly returns intermediate Critter Stack assemblies, making the GH-3521 divergence warning fire on healthy hosts jasperfx#601 — the same walk returns intermediate Critter Stack assemblies (Wolverine.SqlServer), which is why the RememberedApplicationAssembly (process-wide static) silently pins handler discovery for every later implicit host — order-dependent test failures with no loud signal #3521 divergence warning fires on healthy hosts and why its absence here proved nothing.

Still open in this repo:

• The Wolverine half of the RememberedApplicationAssembly (process-wide static) silently pins handler discovery for every later implicit host — order-dependent test failures with no loud signal #3521 warning: the RememberedApplicationAssembly branch never calls CheckForDivergentApplicationAssembly, so that path cannot warn at all. Unfiled.
• Polecat's [5.7.0,6.0.0) range in Directory.Packages.props. The "adjacent finding" in the issue above is wrong and I should correct it: NuGet resolves a version range to the lowest applicable version, deterministically and regardless of cache state. 5.7.0 is published and listed, so local and CI both resolve 5.7.0 — I confirmed "Polecat/5.7.0" in project.assets.json, and there is no lock file for CI to diverge from. There is no dev/CI gap. The genuine (and opposite) point is that the range pins Wolverine to the floor forever and silently never picks up 5.8.x/5.9.x, so it is still worth replacing with an explicit version — just not for the reason given.

Probe branches tmp/bisect-probe, tmp/bisect-probe-mid and tmp/bisect-probe-fix are re-runnable alongside the three original ones; all six should go once this closes.

🤖 Generated with Claude Code

https://claude.ai/code/session_013eR4GL278688VhyhrGcttJ

[jeremydmiller]

Closed automatically by #3777 (46dd558c2). Confirming it actually took on main, not just on a probe branch — all three Polecat shards are green at the merge commit:

shard	result
CIPolecat	success
CIPolecatSagas	success
CIPolecatWorkflow	success

Root cause: handler discovery was scanning xunit.v3.core instead of PolecatTests. The application-assembly stack walk skips System*/Microsoft*/ReSharperTestRunner*, and under an async test fixture the frames between the framework and the test class belong to the runner — so the runner assembly satisfied the filter and was adopted. With no handler chains, every InvokeAsync threw IndeterminateRoutesException.

The version bump was a trigger, not a cause: it perturbed a stack walk that was already unstable. Why a version string changes stack shape is still unexplained — I ruled out the HasReferenceToJasperFxTool short-circuit, the codegen type-name hash and xUnit discovery order by direct test, and the remaining candidate (JIT inlining over async continuations) I did not prove. The fix removes the dependency on that walk, so it stopped mattering.

Carried forward:

• DetermineCallingAssembly adopts the test-runner assembly, so type discovery scans a runner instead of the test assembly jasperfx#600 — the durable fix. JasperFx 2.37.0 has no test-runner exclusion, so Marten and Polecat hosts have the same exposure; Never adopt a test-runner assembly as the application assembly (GH-3776) #3777 only defends Wolverine at its boundary.
• DetermineCallingAssembly returns intermediate Critter Stack assemblies, making the GH-3521 divergence warning fire on healthy hosts jasperfx#601 and The GH-3521 application-assembly warning has a hole on one path and fires on healthy hosts on the other #3778 — the RememberedApplicationAssembly (process-wide static) silently pins handler discovery for every later implicit host — order-dependent test failures with no loud signal #3521 warning, which has a hole on one path and fires on healthy hosts on the other. That combination is what made me discard the correct hypothesis early here.

The Polecat version-range finding originally filed in this issue was wrong, and is corrected in the comment above: NuGet resolves a range to the lowest applicable version deterministically, so there is no dev/CI gap. The range does pin Wolverine to the 5.7.0 floor indefinitely, which is a separate and much smaller problem.

Six tmp/bisect-* branches remain as the reproduction harness and can now go — the ruleset blocks branch deletion, so that needs a maintainer.

🤖 Generated with Claude Code

https://claude.ai/code/session_013eR4GL278688VhyhrGcttJ