Complete a tracked session only when ALL conditions are satisfied (GH-3824) - #3828
Merged
Merged
Conversation
…-3824) TrackedSession.IsCompleted() short-circuited on the first satisfied condition rather than requiring all of them: if (_conditions.Any(x => x.IsCompleted())) return true; ... return !_conditions.Any() || _conditions.All(x => x.IsCompleted()); which made the All(...) on the last line unreachable whenever any single condition was satisfied. With one condition ANY and ALL are identical, which is why this survived -- nearly every session registers just one. Wolverine.Tracking is public test-support API, so this is not only our problem: a user chaining two WaitForMessageToBeReceivedAt calls gets a session that returns when EITHER host handles the message and then asserts against handlers that may still be running. There was already a workaround implying someone hit this and routed around it -- WaitForExecutionOf<T> deliberately merges repeat calls into one condition object, and its doc comment says so in as many words. Found via Wolverine.RabbitMQ.Tests.end_to_end, tagged Category=Flaky and therefore excluded from CI. Its two persistent failures each chain three WaitForMessageToBeReceivedAt calls for a fan-out exchange, so the session returned as soon as the FIRST of the three receivers handled the message. The recorded triage had the symptoms right (both pass alone, fail in-class, fail in ~500ms on a null ColorHistory nowhere near the 30s timeout) but inferred the cause: that WaitForMessageToBeReceivedAt is satisfied by MessageFailed, so a message that arrived and then failed ended the session. Dumping the session -- the next step that triage itself prescribed -- refutes that: status=Completed, ZERO exceptions, message marked successful, at exactly one of three receivers. Measured, not assumed: full Wolverine.RabbitMQ.Tests on main 4 failed / 491 full Wolverine.RabbitMQ.Tests on branch 2 failed / 491 The two that remain are the known tracked-debt flakes (ConventionalRouting.send_from_one_node_to_another_all_with_conventional_routing and multi_tenancy_through_virtual_hosts.send_message_to_a_specific_tenant), failing identically on both sides. CoreTests: 2246 passed, 0 failed. Of 163 WaitForMessageToBeReceivedAt call sites, exactly one file chains more than one. The Category=Flaky tag comes off end_to_end, returning 20 tests to CIRabbitMQ. Also points the three Azure Service Bus "needs its own issue" notes at the issues now filed for them: GH-3825, GH-3826, GH-3827. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WHAuhdWS3XeAk16swV9G8m
This was referenced Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3824.
TrackedSession.IsCompleted()short-circuited on the first satisfied condition instead of requiring all of them:With a single condition
AnyandAllare identical, which is why this survived: nearly every session registers just one.Wolverine.Trackingis public test-support API, so this is not only our problem — a user chaining twoWaitForMessageToBeReceivedAtcalls gets a session that returns when either host handles the message, then asserts against handlers still running. A test that passes locally and fails under load, pointing nowhere near its cause.There was already a workaround implying someone hit this and routed around it rather than fixing it:
WaitForExecutionOf<T>deliberately merges repeat calls into a single condition object, and its doc comment says so — "Multiple calls combine into a single condition that requires every registered count to be reached." That is only necessary because multiple conditions don't AND together.How it surfaced, and a refuted hypothesis
Wolverine.RabbitMQ.Tests.end_to_endcarried[Trait("Category", "Flaky")], excluding 20 tests from CI. Its two persistent failures each chain threeWaitForMessageToBeReceivedAtcalls for a fan-out exchange.The in-file triage had every symptom right — both pass alone, fail in-class, fail in ~500ms on a null
ColorHistorynowhere near the 30s timeout — and then inferred the cause: that the wait is satisfied byMessageFailed, so a message that arrived and then failed ended the session early. It also recorded that the next step was to dump the session rather than infer.Doing that dump refutes the inference outright:
Nothing failed. The message succeeded — at exactly one of the three receivers, whose condition alone satisfied
Any. Alone on an idle machine all three finish inside the same millisecond; in-class under load the other two lag and theirColorHistoryis still null at assertion time.Verification — baselined, not assumed
Because this changes shared test infrastructure, the full RabbitMQ project was run on both sides:
mainend_to_end.use_fan_out_exchangeend_to_end.use_direct_exchange_with_binding_keyConventionalRouting…send_from_one_node_to_another…multi_tenancy_through_virtual_hosts.send_message_to_a_specific_tenantThe fix removes exactly its two targets and adds nothing. The two survivors are the known tracked-debt pair, failing identically on both sides.
CoreTests: 2246 passed, 0 failedend_to_endclass alone: 20/20 on three consecutive runsWaitForMessageToBeReceivedAtcall sites repo-wide; exactly one file chains more than oneWaitForMessageToBeReceivedAtwithWaitForExecutionOf; its comments describe wanting both, so the fix makes it match its own documentationwolverine.slnxRelease build clean, 0 warningsReviewers: the failure mode to watch for on CI is a timeout, not an assertion — a session that now correctly waits for something that never arrives.
Also in here
The
Category=Flakytag comes offend_to_end(20 tests back intoCIRabbitMQ), and the three Azure Service Bus "Needs its own issue" notes now point at the issues finally filed for them — #3825, #3826, #3827. Those were diagnosed product defects sitting behindFlakytags with nothing tracking them.🤖 Generated with Claude Code
https://claude.ai/code/session_01WHAuhdWS3XeAk16swV9G8m