Skip to content

RabbitMqListener.CreateAsync dereferences a null Channel when the agent is disposed during startup #3842

Description

@jeremydmiller

RabbitMqListener.CreateAsync() dereferences a channel that EnsureInitiated() is documented to leave null, using the null-forgiving operator to suppress the compiler telling it so. Under concurrent host startup/shutdown this produces a bare NullReferenceException from inside RabbitMqQueue.DeclareAsync, ~6 frames from the actual cause.

The path

RabbitMqChannelAgent.EnsureInitiated() returns without establishing a channel when the agent is disposed — twice, once before taking the lock and once after:

internal async Task EnsureInitiated()
{
    if (_disposed)
        return;                       // <-- returns with Channel still null
    ...
    await Locker.WaitAsync();
    try
    {
        if (_disposed)
            return;                   // <-- and again

CreateAsync() then proceeds as though a channel is guaranteed (RabbitMqListener.cs:224-236):

public async Task CreateAsync()
{
    await EnsureInitiated();

    if (Queue.AutoDelete || _transport.AutoProvision)
    {
        await Queue.DeclareAsync(Channel!, Logger);     // <-- Channel is null here

The ! means the compiler cannot warn, and DeclareAsync takes a non-nullable IChannel, so the failure surfaces at the first dereference inside the callee:

System.NullReferenceException : Object reference not set to an instance of an object.
   at Wolverine.RabbitMQ.Internal.RabbitMqQueue.DeclareAsync(IChannel channel, ILogger logger) in RabbitMqQueue.cs:line 341
   at Wolverine.RabbitMQ.Internal.RabbitMqListener.CreateAsync() in RabbitMqListener.cs:line 230
   at Wolverine.RabbitMQ.Internal.RabbitMqTransport.buildListener(...) in RabbitMqTransport.cs:line 479
   at Wolverine.RabbitMQ.Internal.RabbitMqTransport.BuildListenerAsync(...) in RabbitMqTransport.cs:line 450
   at Wolverine.RabbitMQ.Internal.RabbitMqQueue.BuildListenerAsync(...) in RabbitMqQueue.cs:line 396
   at Wolverine.Transports.ListeningAgent.StartAsync() in ListeningAgent.cs:line 386
   at Wolverine.Configuration.EndpointCollection.StartListenerAsync(...)

RabbitMqQueue.cs:341 is await channel.QueueDeclareAsync(...) — the first use of the parameter. Nothing at that line is wrong; it is just where a null handed in six frames earlier finally gets dereferenced.

How it was found

Bug_189_fails_if_there_are_many_messages_in_queue_on_startup is intermittently failing. Reproduced locally by running the full CIRabbitMQ Nuke target (4 worker processes): failed attempt 1 with the NRE above, passed on attempt 2. It passes 3/3 when run in isolation — the race needs concurrent host lifecycle, which is why single-test reruns look green.

Note there appears to be a second, distinct failure mode for the same test. On CI (runs 30968222969 and 30969685499) it failed with a different exception:

RabbitMQ.Client.Exceptions.OperationInterruptedException : AMQP close-reason, initiated by Library, code=541
  ---- System.Collections.Generic.KeyNotFoundException : The given key '1' was not present in the dictionary
     at RabbitMQ.Client.Impl.SessionManager.Lookup(Int32 number)
     at RabbitMQ.Client.Framing.Connection.ReceiveLoopAsync(...)

That one is inside the client library's receive loop — a frame arriving for a channel whose session is gone. Both are lifecycle races around the same channel, and they may share a root cause, but I have only confirmed the mechanism for the NRE. Filing them together so the second is not lost; happy to split if they diverge.

Why it matters beyond the flaky test

The null-forgiving operator here is actively costing diagnosis time. EnsureInitiated()'s disposed-check is deliberate and correct — a disposed agent should not open a channel. The defect is that its caller has no idea that happened. Any real deployment that disposes a listener while it is starting (a fast restart, a failed startup, a cancelled host) gets an unexplained NRE in queue declaration rather than "this listener was disposed during startup."

Suggested fix

Make the postcondition explicit rather than assumed:

  1. Have CreateAsync() check for a disposed agent / null Channel after EnsureInitiated() and return early — disposal during startup is a legitimate outcome, not an error.
  2. If reaching that point with no channel is genuinely unexpected, throw with context (InvalidOperationException naming the queue and the agent state) instead of dereferencing.
  3. Drop the ! on both Channel! uses in CreateAsync so the compiler enforces whichever contract is chosen.

Worth auditing the other Channel! sites in the RabbitMQ transport for the same shape.

Related: #3832 (also from Bug_189), #3817 (CIRabbitMQ variance), #3171 (the channel-only-shutdown case the disposed check protects).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions