Fix deadlock in DrainAsync causing Redis scheduling test failure#2294
Merged
jeremydmiller merged 1 commit intomainfrom Mar 12, 2026
Merged
Fix deadlock in DrainAsync causing Redis scheduling test failure#2294jeremydmiller merged 1 commit intomainfrom
jeremydmiller merged 1 commit intomainfrom
Conversation
When a rate-limited message triggers PauseListenerContinuation, the pause calls StopAndDrainAsync → DrainAsync from within the receiver block's execute function. The WaitForCompletionAsync added in #2288 waits for in-flight items to finish, but the current message IS an in-flight item — creating a deadlock that times out after DrainTimeout (30s), causing the Redis rate limiting test to fail. Fix: only wait for completion when Latch() was previously called (indicating shutdown via OnApplicationStopping), not when DrainAsync is the first to set _latched (indicating a pipeline-triggered pause). Closes #2291 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This was referenced Mar 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DurableReceiver.DrainAsync()andBufferedReceiver.DrainAsync()introduced by Latch receivers on ApplicationStopping for immediate graceful shutdown #2288WaitForCompletionAsynccall now only executes during actual shutdown (whenLatch()was called separately viaOnApplicationStopping), not during pipeline-triggered pausesRoot Cause
PR #2288 added
WaitForCompletionAsynctoDrainAsync()to wait for in-flight messages during graceful shutdown. However,DrainAsync()is also called from within the handler pipeline when a rate-limited message triggersPauseListenerContinuation:This circular dependency causes a 30-second deadlock (bounded by
DrainTimeout), after which the rate-limited message's retry window has long passed, and the test's 20-second polling timeout expires first.Fix
Use the
_latchedflag to distinguish the two call paths:OnApplicationStoppingcallsLatch()first →_latchedis alreadytruewhenDrainAsync()runs → safe to waitLatch()call →_latchedisfalsewhenDrainAsync()runs → skip the wait to avoid deadlockTest plan
rate_limited_messages_are_delayed_with_native_schedulingpasses consistently (was failing every run)Closes #2291
🤖 Generated with Claude Code