Concerns with RabbitMQ Readiness Probe in Parallel Pod Management Policy

[uk1988]

Hello,

We are using the RabbitMQ Cluster Operator to deploy our RabbitMQ cluster.
While reviewing the readiness probe configuration, we noticed the following:

.spec.podManagementPolicy is set to Parallel, which brings all three RabbitMQ pods online simultaneously.
The readiness probe only checks the local pod's readiness and does not verify whether the full RabbitMQ cluster has been formed.

As a result, traffic may be routed to a pod before it has successfully joined the cluster. For example, pods 1 and 2 may be in the process of forming a cluster, while pod 3 passes its readiness probe and begins accepting connections—even though it has not yet joined the cluster.
In certain edge cases, we have observed that this behavior poses a risk to our setup.
Are there any recommended options to enhance the readiness probe so that it only allows traffic once the entire RabbitMQ cluster is fully formed and all nodes are synchronized?

[Zerpet]

The operator configures rabbitmq to perform peer discovery and automatically form a cluster. RabbitMQ does not serve traffic until the cluster is formed. There were situation in RabbitMQ 3.13.x that a node may start as a standalone node, instead of joining the cluster. This change is included in RabbitMQ 4.1+ rabbitmq/rabbitmq-server#13050

My best guess is that you are in a rabbitmq version that doesn't include this fix. We used to have the pod management policy set to ordered ready, but this caused a severe issue where RabbitMQ won't start at all if all Pods were restarted or deleted at once (e.g. k8s node drain). This comment summarises it pretty well: #989 (comment)

[uk1988]

@Zerpet thank you for the input.
Yes, we were running 3.11.x when we had these issues in production.
We are now switch to 4.0.9. But I guess we should for 4.1 if we do not want to have this issue anymore.

Yes, we also had similar issue with startups on OrderedReady with pg Zalando operator...