IModel BasicPublish/WaitForConfirmsOrDie function never completes

[ArcaneDiver]

In our production environment we often have spikes of thousands of data to read, process and publish individually to a rabbit broker.
Recently, as the data peaks have increased, it has happened that our publish function completely locks it self.

Log("Start publish");

_channel.BasicPublish(_configuration.Exchange, string.Empty, properties, messageBytes);
_channel.WaitForConfirmsOrDie(new TimeSpan(0, 0, 2));

Log("Completed publish");

The client version is 6.8.1

We receive only the first log but the second never comes out. Every operation that uses the channel is protected by semaphores, so since the publish never completes, the semaphore is never released and everything is locked.

The strange thing is that there no exception, but we noticed, analyzing the rabbit server logs, that after like 2/3 seconds the connection get closed and reopened and still the function has never gets completed or faults. Also we have some other background service that only consumes and after the reconnection they seems to be able to consume messages but since the channel is protected by semaphore they aren't able to ack.

Sadly in our local enviroment, we were unable to reproduce the problem. We also tried to manually close the connection but that has genereted some exception as also trying to set to 0 the watermark.

Has somebody any ideas on which could be the problem?

[lukebakken]

Hello!

I strongly suggest starting the process of migrating your code to the latest version 7 release candidate:

https://www.nuget.org/packages/RabbitMQ.Client/7.0.0-rc.12

In addition, the next RC will include a new feature by which this library will track confirmations for you. The WaitForConfirms* methods are removed in version of 7 this library. Here is an example of what the new publisher tracking API provides:

https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/main/projects/Test/Applications/PublisherConfirms/PublisherConfirms.cs

Having said all that, in your case, I recommend that you DO NOT use the WaitForConfirms* methods. Instead, add event handlers to the BasicAck, BasicNack and BasicReturn events and handle them yourself. That way, you can implement a timeout if needed. This tutorial shows how to use those events:

https://www.rabbitmq.com/tutorials/tutorial-seven-dotnet

[ArcaneDiver]

Which is the ETA of the stable release?

[DPDmancul]

It is a bad advice recommending using a release candidate, especially because the OP is using Rabbit in production, as stated. And in such environments one must suggest using stable releases only.

Moreover the event handlers approach may not be a valid alternative to the WaitForConfirms approach in all scenarios: for some use cases it is indeed required to wait for the confirm. This scenario is not covered by the event-driven approach, and even the tutorial you linked shows to use the WaitForConfirms one.

[lukebakken]

It is really a pity an upgrade removes useful features, leading to support fewer use cases.

v7 removes WaitForConfirms* in the API because the same functionality is implemented in a much better fashion.

[DPDmancul]

It is very debatable if reducing the supported scenarios is indeed an improvement. It is possible to continue supplying old way to do things and in documentation illustrate the pro and cons of each one, so when the end user can have a choice, he can do the right one. But not every time you can have a choice: in some scenarios you are forced to use one or the other.

[lukebakken]

It is very debatable if reducing the supported scenarios is indeed an improvement.

Hey, any time I can delete code, it's an improvement. Like I said, v7 supports "synchronously" waiting on a confirm. Per-message, even, if you'd like.

The nice thing about a major version release is that we get to change the API for the better.

There have been years of discussions about v7. The community has had plenty of time to give input. 7.0.0 will ship this month.

[DPDmancul]

Like I said, v7 supports "synchronously" waiting on a confirm

I missed that part, unless it is the reimplement it yourself suggestion

[lukebakken]

Take some time to review the code to which I linked in this earlier comment:

#1705 (comment)

It pretty much demonstrates the new API around publisher confirms:

https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/main/projects/Test/Applications/PublisherConfirms/PublisherConfirms.cs

[ArcaneDiver]

I'm aligned with my collegue @DPDmancul, we can't use a version that isn't production ready. Since the application requires that every message is sent inside a transaction we must be certain that the single message is received by the broken, since we can't lose any message.

Also i have gave a look to the source code and the publish just writes into the socket, and the waitForConfirm uses a CountdownEvent to wait that asynchronously until it times out or the ack is received and _deliveryTagsCountdown is released.

So i don't really understand what could lead to such issues.

[lukebakken]

So i don't really understand what could lead to such issues.

Neither do I, without a way to reproducee.