5.2.0 -> 6.0.0-6.5.0 Performance Regression?!

[ProIcons]

Describe the bug

So I'm experimenting with a DLQ Analyzer tool, and i tried consuming messages from a DLQ with over 2.5million messages.

As the first step and PoC I wanted to setup a dummy consuming mechanism to see how it works.

This dummy consumer, consumed messages with a capped rate of 50/s.

This didn't make any sense because i was using another tool called QueueExplorer, and it was consuming from my computer to this rabbitmq with over 300 messages/s.

Downgrading the the version of the client this tool is using (5.2.0) seemed to increase the rate to approx 500/s

I might be missing something really obvious here, is there a configuration that might change this behavior? am i missing something?

I tried all versions and this occurs from 6.0.0 all the way up to 6.5.0.
On 5.x this doesn't seem to be the case.

Reproduction steps

        using var connection = _connectionFactory.CreateConnection();
        var channel = connection.CreateModel();
        string uriString =
            $"{(config.SSL ? "amqps" : "amqp")}://{Uri.EscapeDataString(config.Username)}:{Uri.EscapeDataString(
                config.Password)}@{Uri.EscapeDataString(config.Host)}:{config.Port}";
        return new ConnectionFactory()
        {
            Uri = new Uri(uriString),
            AutomaticRecoveryEnabled = true,
            TopologyRecoveryEnabled = true
        };
        var consumer = new EventingBasicConsumer(channel);
        consumer.Received += (model, eventArgs) =>
        {
        };

        channel.BasicConsume(queueInfo.Name, false, "", false, true, null, consumer);

Expected behavior

Better consuming performance.

Additional context

No response

[michaelklishin]

50, 300, 500 deliveries a second are all trivial numbers for any client, and in particular this one, as long as you use long lived connections.

There are workloads that are affected by various changes in 6.x, for example, #860 or #1354.

They usually do not affect consumer deliveries, although peak concurrency of consumer processing can be affected.

#1252, #1355, and #1321 are the most relevant issues or discussions that I recall.

Perhaps your workload could benefit from a customer scheduler.
We need profiler and traffic capture data to suggest anything more specific without guessing.

[ProIcons]

I'm content with the 400/s peak because i'm located on EU East, and the RabbitMQ Broker is located on US East, and i'm also accessing it through OpenVPN which also adds latency. My maximum speed with the servers, located on US East, is at around 300MBps. With 6.x i never reached that, with 5.2 i reached that no problem. I Tested that on the same queue with the same data inside. (it's Shoveled and no more data are being added)

I actually did profile the software with the 6.x with Rider's Profiler, and the biggest hotspot was apparently the low level networking. But i haven't kept the result file.

The project is running with .NET 7.

.NET:

> dotnet --list-runtimes                                                                                                                                                                                                                                                                                                                               
Microsoft.AspNetCore.App 6.0.19 [/usr/lib/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.8 [/usr/lib/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 6.0.19 [/usr/lib/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.8 [/usr/lib/dotnet/shared/Microsoft.NETCore.App]

> dotnet --list-sdks                                                                                                                                                                                                                                                                                                                                   
6.0.119 [/usr/lib/dotnet/sdk]
7.0.108 [/usr/lib/dotnet/sdk]

My OS Is:

> lsb_release -a                                                                                                                                                                                                                                                                                                                                       
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 23.04
Release:	23.04
Codename:	lunar

> uname -a                                                                                                                                                                                                                                                                                                                                             
Linux hades 6.2.0-20-generic #20-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr  6 07:48:48 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

My CPU is 5950X,
RAM: 128GB @ 3200MHz.

I have switched off the Ubuntu's stub resolver.. because... problems.

I cannot provide you the traffic capture data, but later this week i can provide you data from the profilings for both the 5.2 and 6.x.

[MarcoZuiderwijkBS]

Hello,

We experience the same issue like you had. Have you been able to find a solution for this? I also tested this with a Python and Java consumer which were working fine. This only happens when using the RabbitMQ.Client version 6.x.x.

Kind regards,
Marco Zuiderwijk

[lukebakken]

@ProIcons @MarcoZuiderwijkBS - PLEASE test the latest version 7 alpha release:

https://www.nuget.org/packages/RabbitMQ.Client/7.0.0-alpha.6