GH-4188: see past the pass-through receiver wrappers before branching on the receiver - #4190
Merged
Merged
Conversation
… on the receiver ListeningAgent.EnqueueDirectlyAsync type-switched on the raw _receiver field, but that field is routinely a wrapper. ReceiverWithRules -- installed for any incoming envelope rule, which includes an endpoint-level MessageType or TenantId -- is unconditionally an ILocalQueue, so a wrapped NativeAck or Inline receiver matched the ILocalQueue branch ahead of its own and threw from inside ReceiverWithRules.EnqueueAsync, re-creating the exact GH-4011 failure on the durability agent's re-entry path (DLQ replay per GH-1942, scheduled-message firing). GlobalPartitionedInterceptor is not an ILocalQueue at all, so everything behind it fell through to the throwing else. A wrapped BufferedReceiver was the quieter case: it matched the generic ILocalQueue branch instead of its own, so the replay was enqueued rather than dispatched through a RetryOnInlineChannelCallback -- and that callback is what marks the inbox row handled on completion. The message ran; the row was left behind. Both wrappers now implement IReceiverWrapper, and Unwrap() peels them off (they nest) so branch SELECTION sees the receiver that actually executes messages. Dispatch stays on the outer receiver everywhere it can, so the wrappers keep applying incoming rules and re-routing globally partitioned messages; only the ILocalQueue branch enqueues into the unwrapped receiver, which is what ReceiverWithRules.EnqueueAsync delegated to anyway. Branch order is unchanged, and a null receiver still reaches the same throw. LatchReceiver hand-rolled a one-level ReceiverWithRules unwrap for the same reason (GH-3709) and so never latched anything behind the interceptor; it uses the shared helper now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 30, 2026
Merged
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4188.
The defect
ListeningAgent.EnqueueDirectlyAsynctype-switches over receiver implementations to decide how a replayed envelope re-enters the pipeline. It reads the raw_receiverfield — but that field is routinely a pass-through wrapper, not the receiver the switch is looking for.Endpoint.MaybeWrapReceiverinstallsReceiverWithRuleswheneverRulesForIncoming()yields anything: anIncomingRulesentry, an endpoint-levelMessageType, or an endpoint-levelTenantId.ReceiverWithRulesis unconditionally anILocalQueue, so a wrappedNativeAckReceiverorInlineReceivermatched theILocalQueuebranch ahead of its own and then threw from insideReceiverWithRules.EnqueueAsync, whoseInneris not a local queue:Same exception, same call sites — DLQ replay per GH-1942 and scheduled-message firing — as the bug #4011 was supposed to have closed. #4011 added the missing branch; it did not fix the branch never being reached.
GlobalPartitionedInterceptoris the second wrapper on the same path. It is not anILocalQueueat all, so everything behind it fell through to the throwingelse.A wrapped
BufferedReceiveris the quieter third case: it matched the genericILocalQueuebranch instead of its own, so the replay was enqueued rather than dispatched through aRetryOnInlineChannelCallback— and that callback is the only thing that marks the inbox row handled on completion (GH-1942). The message ran; the row was left behind.Noticed while working #4187 (GH-4186) and deliberately left out of scope there.
The fix
Both wrappers now implement a small internal
IReceiverWrapper { IReceiver Inner { get; } }, andUnwrap()peels them off — they nest,GlobalPartitionedInterceptor(ReceiverWithRules(inner))— so branch selection sees the receiver that actually executes messages.Dispatch stays on the outer receiver everywhere it can, so the wrappers keep doing their jobs: incoming envelope rules still get applied, globally partitioned messages still get re-routed. Only the
ILocalQueuebranch enqueues into the unwrapped receiver, which is exactly whatReceiverWithRules.EnqueueAsyncdelegated to anyway.Branch order is unchanged —
BufferedReceiveris still tested ahead of the genericILocalQueuebranch, so #4011 does not regress — and a null_receiverstill reaches the same throw it always did.One behavior change beyond the throw
The wrapped-
BufferedReceivercase now dispatches throughReceivedAsyncinstead ofEnqueueAsync, so incoming envelope rules are applied to a replayed envelope where before they were skipped. That is a deliberate consequence, not a side effect of fixing the inbox row: a replay should look like a delivery, and every other path into that endpoint already stamps the endpoint'sTenantId/MessageType. For the two rulesRulesForIncoming()synthesizes on its own —TenantIdRuleandMessageTypeRule, both unconditional assignments — re-running them on a DLQ replay is a no-op, because the envelope was stamped by the same rules when it first arrived. A user-supplied rule inIncomingRulescarries no such guarantee, but re-running one on a replay is exactly what a broker redelivery already does. For a scheduled message the result is what a live delivery to that endpoint would have produced. The buffered test asserts it (envelope.TenantId).It is limited to that one case. Wrapped NativeAck and Inline threw before, so they have no prior behavior to change, and an unwrapped
BufferedReceiverhas no rules to apply.LatchReceiverhand-rolled a one-levelReceiverWithRulesunwrap for the same reason (GH-3709) and so never latched anything behind the interceptor. It uses the shared helper now, which matters: an unlatched receiver'sDrainAsyncreturns immediately instead of waiting for in-flight handlers.Tests
CoreTests/Runtime/WorkerQueues/wrapped_receiver_enqueue_directly_4188.cs— five tests, all red before the change and each verified to fail for the right reason:InvalidOperationExceptionfromReceiverWithRules.EnqueueAsyncGlobalPartitionedInterceptorelseenvelope.Listenerwas null — noRetryOnInlineChannelCallbackLatchReceiverthrough the interceptorEach asserts the receiver's real shape first, so none of them can pass vacuously against an unwrapped receiver. The rule-carrying tests also assert the rule was applied (
envelope.TenantId), which would fail if dispatch bypassed the wrapper.Verification
CoreTestsgreen — 2679 passed, 2 skipped — including the existing ListeningAgent.EnqueueDirectlyAsync has no branch for a native-ack receiver #4011, NativeAck listeners always report QueueCount 0 and a LastQueueActivityAt frozen at construction #4186 andnative_ack_receiversuites.Bug_1942_replay_dlq_to_buffered_or_inlinegreen against Postgres.dotnet build wolverine.slnx -c Release -f net9.0clean, 0 warnings.Follow-up, not in this PR
Filed as #4191:
ReceiverHasFaultedand the receiver-rebuild path inStartAsynctype-testIFaultTrackingReceiveragainst the raw_receivertoo, so for any endpoint with an incoming rule a terminally faulted receiver (#506, CritterWatch#942) is never rebuilt. Same family, sameUnwrap()helper — which is why #4191 is blocked on this PR.🤖 Generated with Claude Code