GH-4135: report Received exactly once per receipt - #4141
Merged
Conversation
A listener with PartitionProcessingByGroupId recorded TWO Received events for one envelope while executing it exactly once. Two branches of HandlerPipeline report Received, and they are meant to be mutually exclusive: TryDeserializeEnvelope reports it from a finally block, and executeAsync reports it directly when the envelope already carries a Message. Partitioned processing goes through BOTH. ShardedExecutionBlock.DeserializeFirst deserializes up front -- it has to, because receiver-side grouping rules resolve the group id off the deserialized message -- and the envelope then reaches executeAsync already carrying a Message, taking the second branch. The reported symptom is that Received.SingleMessage<T>() fails against a system behaving correctly, with a message that reads as duplicate delivery and sends you looking for a routing bug that is not there. But IMessageTracker.Received is not only tracking: the same call increments the received counter, posts RecordReceived to the CritterWatch accumulator, and logs the receipt. So every partitioned listener has been double-counting received messages in production metrics and logging each receipt twice. Fixed with a WasTrackedAsReceived guard on the envelope rather than by removing either call site: both are correct for the path they serve, and the deserialization one must stay in its finally so a malformed payload is still reported as received before it is moved to the error queue. GlobalPartitionedInterceptor takes the same pre-deserialize route and is fixed by the same guard. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HxbBpN6PmRB5CMSSTdGNr3
Dump every tracked record for the probe message -- event type, sequence, service, node and envelope id -- into the assertion message. One full-suite run produced a Received count of 2 that did not reproduce in six isolated runs or in two further full-suite runs, and with no record dump there was nothing to tell a genuine broker redelivery apart from the defect this test exists to catch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HxbBpN6PmRB5CMSSTdGNr3
This was referenced Aug 27, 2026
Open
This was referenced Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4135.
The bug
A listener with
PartitionProcessingByGroupIdrecorded twoReceivedevents for one envelope while executing it exactly once. Reproduced verbatim from the issue's repro.Two branches of
HandlerPipelinereportReceived, and they are meant to be mutually exclusive:TryDeserializeEnvelopereports it from afinallyblockexecuteAsyncreports it directly when the envelope already carries aMessagePartitioned processing goes through both.
ShardedExecutionBlock.DeserializeFirstdeserializes up front — it has to, because receiver-side grouping rules (ByMessage,ByPropertyNamed, inferred grouping) resolve the group id off the deserialized message — and the envelope then reachesexecuteAsyncalready carrying aMessage, taking the second branch. That matches the reporter's guess exactly.It is bigger than the reported symptom
The issue reports that
Received.SingleMessage<T>()fails against a system behaving correctly, with a message that reads as duplicate delivery and sends you hunting a routing bug that is not there — it cost the reporter a full bisect.But
IMessageTracker.Receivedis not only tracking. The same call:_receivedCounterRecordReceivedto the CritterWatch accumulatorSo every partitioned listener has been double-counting received messages in production metrics and logging each receipt twice. The red test was the visible half.
The fix
A
WasTrackedAsReceivedguard on the envelope, rather than removing either call site — both are correct for the path they serve, and the deserialization one has to stay in itsfinallyso a malformed payload is still reported as received before it is moved to the error queue.GlobalPartitionedInterceptortakes the same pre-deserialize route and is fixed by the same guard.Verification
duplicate_received_with_partitioning_4135reproduces the issue's repro against a real broker. Confirmed red before, green after.drain_wait_for_prefetch) passes 2/2 in isolation on this branch and 2/2 onorigin/main, and every full-suite run tonight produced exactly one, always different, failure under local load. Re-running the full suite to confirm non-reproduction.dotnet build wolverine.slnx -c Release -f net9.0clean.🤖 Generated with Claude Code
https://claude.ai/code/session_01HxbBpN6PmRB5CMSSTdGNr3