Skip to content

Stop dereferencing a null channel when a RabbitMQ listener is disposed mid-startup (GH-3842) - #3843

Merged
jeremydmiller merged 1 commit into
mainfrom
gh-3842/rabbit-listener-null-channel
Aug 5, 2026
Merged

Stop dereferencing a null channel when a RabbitMQ listener is disposed mid-startup (GH-3842)#3843
jeremydmiller merged 1 commit into
mainfrom
gh-3842/rabbit-listener-null-channel

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Closes #3842.

The defect

RabbitMqChannelAgent.EnsureInitiated() is best-effort and can return without a channel two different ways — it bails early when the agent is disposed, and it logs-and-swallows a failure to open one:

internal async Task EnsureInitiated()
{
    if (_disposed) return;                        // <-- no channel
    ...
    catch (Exception e)
    {
        Logger.LogError(e, "Error trying to start a new Rabbit MQ channel for {Endpoint}", this);
    }                                             // <-- also no channel

RabbitMqListener.CreateAsync() called Queue.DeclareAsync(Channel!, Logger) immediately afterwards. The ! stopped the compiler from objecting, and DeclareAsync takes a non-nullable IChannel, so the null travelled six frames before anything dereferenced it:

System.NullReferenceException : Object reference not set to an instance of an object.
   at Wolverine.RabbitMQ.Internal.RabbitMqQueue.DeclareAsync(IChannel channel, ILogger logger) in RabbitMqQueue.cs:line 341
   at Wolverine.RabbitMQ.Internal.RabbitMqListener.CreateAsync() in RabbitMqListener.cs:line 230
   at Wolverine.RabbitMQ.Internal.RabbitMqTransport.buildListener(...)
   at Wolverine.Transports.ListeningAgent.StartAsync()
   at Wolverine.Configuration.EndpointCollection.StartListenerAsync(...)

RabbitMqQueue.cs:341 is await channel.QueueDeclareAsync(...) — the first use of the parameter. Nothing there is wrong; it is just where the null finally lands.

In the field this is any host that disposes a listener while it is still starting: a fast restart, a failed startup, a cancelled host. You get an unexplained NRE in queue declaration instead of "this listener was disposed during startup."

How it was found

Chasing the intermittent failure of Bug_189_fails_if_there_are_many_messages_in_queue_on_startup. It passes 3/3 in isolation and fails on attempt 1 under the CIRabbitMQ target's worker processes — the race needs concurrent host lifecycle, which is exactly why single-test reruns kept looking green.

The change

CreateAsync captures the channel once and checks it. That also closes a smaller hole: reading the property five times in a row races a concurrent rebuild replacing it mid-method.

  • Disposed → return quietly and log at Debug. A host stopping while its listeners come up hits this routinely, and there is nothing left to build against.
  • Live agent, no channelInvalidOperationException naming the endpoint and queue, because that one is a real failure that EnsureInitiated merely logged.
  • EnsureInitiated now documents that it does not guarantee a channel, so the next caller does not have to rediscover this.

Worth noting the settle path in this same file already learned this: CanSettle() captures Channel into a local and null-checks it, and its doc comment cites Channel! throwing a NullReferenceException mid-reconnect. The create path never got the same treatment.

Verification

Both tests were run against the unfixed listener first and fail with the reported NullReferenceException; they pass after:

test unfixed fixed
disposed_during_startup_abandons_creation_quietly NullReferenceException pass
a_live_agent_that_cannot_open_a_channel_throws_something_diagnosable NullReferenceException pass

The second test disposes the transport's ListeningConnection so startNewChannel() genuinely fails. Nulling Channel by hand is not enough — CreateAsync calls EnsureInitiated() first, which would simply open a fresh channel, and the branch under test would never run while the test still passed.

For the red baseline I reverted only RabbitMqListener.cs, keeping the new IsDisposed accessor on the agent — that is test-visible plumbing rather than the fix, and reverting it too would have broken compilation and proved nothing.

Full CIRabbitMQ target after the fix: 492 passed, Bug_189 clean. The one remaining flake, multi_tenancy_through_virtual_hosts, is pre-existing tracked debt.

One honest caveat: Bug_189 is a race, so a clean run is encouraging rather than conclusive. The mechanism is confirmed and the tests pin it directly; whether it was the only mechanism behind that flake is a separate question — #3842 also records a second failure mode seen on CI (OperationInterruptedException / channel-1 KeyNotFound inside the client's receive loop) that this PR does not claim to address.

…rtup (GH-3842)

RabbitMqChannelAgent.EnsureInitiated() is best-effort and can return without a channel two
different ways: it bails early when the agent is disposed, and it logs-and-swallows a failure
to open one. RabbitMqListener.CreateAsync() called `Queue.DeclareAsync(Channel!, Logger)`
straight afterwards, so both outcomes surfaced as a bare NullReferenceException thrown from
RabbitMqQueue.DeclareAsync -- six frames from the actual cause, at a line where nothing is
wrong. The null-forgiving operator is what kept the compiler from saying so.

Reproduced as the intermittent failure of Bug_189_fails_if_there_are_many_messages_in_queue_
on_startup: it fails on attempt 1 and passes on retry under the CIRabbitMQ target's worker
processes, while passing 3/3 in isolation, because the race needs concurrent host lifecycle.

CreateAsync now captures the channel once and checks it, which also closes a smaller hole --
reading the property five times in a row races a concurrent rebuild replacing it mid-method.
Disposal during startup returns quietly, since a host that stops while its listeners are still
coming up hits that routinely and there is nothing left to build against. A live agent with no
channel throws InvalidOperationException naming the endpoint and queue, because that one is a
real failure that EnsureInitiated only logged.

The settle path in this same file already learned this lesson -- CanSettle() captures Channel
into a local and null-checks it, and its comment cites `Channel!` throwing a
NullReferenceException mid-reconnect. The create path never got the same treatment.

Tests, red on the unfixed listener with the reported NullReferenceException, green after:

- disposed_during_startup_abandons_creation_quietly
- a_live_agent_that_cannot_open_a_channel_throws_something_diagnosable, which disposes the
  transport's ListeningConnection so startNewChannel() genuinely fails. Nulling Channel by
  hand is not enough -- EnsureInitiated() would just open a fresh one and the branch under
  test would never run.

Full CIRabbitMQ target: 492 passed, Bug_189 clean (the one remaining flake,
multi_tenancy_through_virtual_hosts, is pre-existing tracked debt).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WHAuhdWS3XeAk16swV9G8m
@jeremydmiller
jeremydmiller merged commit 1526483 into main Aug 5, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RabbitMqListener.CreateAsync dereferences a null Channel when the agent is disposed during startup

1 participant