etcd peer discovery logs an error

[sinawic]

Describe the bug

i have an etcd instance
and 3 rabbitmq instances with rabbitmq_peer_discovery_etcd plugin enabled

on initial setup when rabbitmq instances are coming up they can't or dont register themselves on etcd
then fail to stanalone mode and then register on etcd

part of the logs that is showing rabbitmq instance cant find any peers bec no one has registered on etcd yet

rabbit-3    | 2024-07-01 09:41:53.095026+00:00 [error] <0.353.0> etcd peer discovery: successfully extracted nodes:
rabbit-3    | 2024-07-01 09:41:53.095135+00:00 [info] <0.254.0> Peer discovery: discovered no peer nodes to cluster with. Some discovery backends can filter nodes out based on a readiness criteria. Enabling debug logging might help troubleshoot.

then after like 30 seconds i get these on the logs of each rabbitmq instance:

rabbit-3    | =PROGRESS REPORT==== 1-Jul-2024::09:42:07.861301 ===
rabbit-3    |     supervisor: {local,inet_gethost_native_sup}
rabbit-3    |     started: [{pid,<0.98.0>},{mfa,{inet_gethost_native,init,[[]]}}]
rabbit-3    | 
rabbit-3    | =PROGRESS REPORT==== 1-Jul-2024::09:42:07.882081 ===
rabbit-3    |     supervisor: {local,kernel_safe_sup}
rabbit-3    |     started: [{pid,<0.97.0>},
rabbit-3    |               {id,inet_gethost_native_sup},
rabbit-3    |               {mfargs,{inet_gethost_native,start_link,[]}},
rabbit-3    |               {restart_type,temporary},
rabbit-3    |               {significant,false},
rabbit-3    |               {shutdown,1000},
rabbit-3    |               {child_type,worker}]
rabbit-3    | 
rabbit-3    | 2024-07-01 09:42:07.903687+00:00 [error] <0.254.0> Peer discovery: could not discover and join another node; proceeding as a standalone node
rabbit-3    | 2024-07-01 09:42:07.947982+00:00 [notice] <0.44.0> Application mnesia exited with reason: stopped

and in the meantime i try fetching data from etcd and verify that no data has been written "yet"

then after proceeding as a standalone node i can verify that each instance writes to etcd as below:

etcdctl get --prefix=true "/rabbitmq"
/rabbitmq/discovery/rabbitmq/clusters/default/nodes/rabbit@rabbit-1
{"lease_id":7587879747894505220,"node":"rabbit@rabbit-1","ttl":61}
/rabbitmq/discovery/rabbitmq/clusters/default/nodes/rabbit@rabbit-2
{"lease_id":7587879747894505226,"node":"rabbit@rabbit-2","ttl":61}
/rabbitmq/discovery/rabbitmq/clusters/default/nodes/rabbit@rabbit-3
{"lease_id":7587879747894505223,"node":"rabbit@rabbit-3","ttl":61}

despite being able to connect to etcd and writing to it
if i restart rabbitmq instances still they come up in standalone mode

unless i remove all instances volumes but not the data from etcd
then bring up all instances and then the cluster will be setup successfully!

and here is my rabbitmq.conf file:

# cluster_formation.peer_discovery_backend = etcd

# the backend can also be specified using its module name
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_etcd

# etcd endpoints. This property is required or peer discovery won't be performed.
cluster_formation.etcd.endpoints.1 = etcd:2379

am i doing anything wrong anywhere?
am i missing something like any configurations?

Reproduction steps

1.setup etcd instance with no authentication enabled
2.setup 3 instances of rabbitmq using the configuration i provided on describe section
3.then bring up the instances and see through the logs
...

Expected behavior

i expect instances of rabbitmq to be able to register themselves before falling to standalone mode, then discover the registered instances and create the cluster

exactly how it works when there is already the data of registered instances on etcd!

Additional context

No response

[sinawic]

so if i join instances of rabbitmq one by one
still i get the following error in logs:

rabbit-2    | 2024-07-01 11:29:23.808535+00:00 [error] <0.353.0> etcd peer discovery: successfully extracted nodes: rabbit@rabbit-1
rabbit-2    | =PROGRESS REPORT==== 1-Jul-2024::11:29:24.074018 ===
rabbit-2    |     supervisor: {local,inet_gethost_native_sup}
rabbit-2    |     started: [{pid,<0.98.0>},{mfa,{inet_gethost_native,init,[[]]}}]
rabbit-2    | 
rabbit-2    | =PROGRESS REPORT==== 1-Jul-2024::11:29:24.079205 ===
rabbit-2    |     supervisor: {local,kernel_safe_sup}
rabbit-2    |     started: [{pid,<0.97.0>},
rabbit-2    |               {id,inet_gethost_native_sup},
rabbit-2    |               {mfargs,{inet_gethost_native,start_link,[]}},
rabbit-2    |               {restart_type,temporary},
rabbit-2    |               {significant,false},
rabbit-2    |               {shutdown,1000},
rabbit-2    |               {child_type,worker}]
rabbit-2    | 
rabbit-2    | 2024-07-01 11:29:25.085123+00:00 [error] <0.353.0> etcd peer discovery: successfully extracted nodes: rabbit@rabbit-1
rabbit-2    | =PROGRESS REPORT==== 1-Jul-2024::11:29:25.362963 ===
rabbit-2    |     supervisor: {local,inet_gethost_native_sup}
rabbit-2    |     started: [{pid,<0.98.0>},{mfa,{inet_gethost_native,init,[[]]}}]
rabbit-2    | 
rabbit-2    | =PROGRESS REPORT==== 1-Jul-2024::11:29:25.370864 ===
rabbit-2    |     supervisor: {local,kernel_safe_sup}
rabbit-2    |     started: [{pid,<0.97.0>},
rabbit-2    |               {id,inet_gethost_native_sup},
rabbit-2    |               {mfargs,{inet_gethost_native,start_link,[]}},
rabbit-2    |               {restart_type,temporary},
rabbit-2    |               {significant,false},
rabbit-2    |               {shutdown,1000},
rabbit-2    |               {child_type,worker}]
rabbit-2    | 
rabbit-2    | 2024-07-01 11:29:25.374677+00:00 [error] <0.254.0> Peer discovery: could not discover and join another node; proceeding as a standalone node
rabbit-2    | 2024-07-01 11:29:25.393571+00:00 [notice] <0.44.0> Application mnesia exited with reason: stopped

i dont understand
as it says successfully extracted nodes: rabbit@rabbit-1 but then Peer discovery: could not discover and join another node; proceeding as a standalone node
so i dont know eventually it could detect any peers or not! and if yes why fall back to standalone mode

[lukebakken]

@sinawic thanks for using RabbitMQ.

Team RabbitMQ uses discussions to assist users until a bug or other actionable work is discovered. In this case, you do not tell us the following essential information:

• RabbitMQ and Erlang version
• etcd version

Please see Team RabbitMQ's community support policy. You must be using the latest version of RabbitMQ, AND provide a way for us to reproduce this issue, before we will assist you.

It should be very simple to provide a script, or docker compose project, to demonstrate this issue.

[michaelklishin]

@sinawic we don't guess in this community and you haven't even provided any details on the RabbitMQ version used and your config file.

[error] <0.254.0> Peer discovery: could not discover and join another node; proceeding as a standalone node

likely should not be an error because one of the nodes, the first one to boot, will not discover any peers and will continue as a standalone node. That message likely should have been an info one.

The best way to know if it can "detect any peers or not" is to form a cluster with more than one node (I know, this is such a brilliant idea).

[michaelklishin]

Also, enabling debug logging will particularly help with troubleshooting peer discovery, if there's anything to troubleshoot at all.

[sinawic]

hi thanks for the quick response!

here's my docker-compose.yml file:

version: '3'
services:
  rabbit-1:
    image: rabbitmq:3.13-management
    volumes:
      - ./config/rabbitmq.conf:/config/rabbitmq.conf
      - ./data/rabbit-1:/var/lib/rabbitmq/
    environment:
      - RABBITMQ_CONFIG_FILE=/config/rabbitmq
      - RABBITMQ_ERLANG_COOKIE=QWERTYUIOP
      - RABBITMQ_DEFAULT_USER=user
      - RABBITMQ_DEFAULT_PASS=password
    hostname: rabbit-1
    container_name: rabbit-1
    ports:
      - 5671:5672
      - 15671:15672
    depends_on:
      - etcd
  rabbit-2:
    image: rabbitmq:3.13-management
    volumes:
      - ./config/rabbitmq.conf:/config/rabbitmq.conf
      - ./data/rabbit-2:/var/lib/rabbitmq/
    environment:
      - RABBITMQ_CONFIG_FILE=/config/rabbitmq
      - RABBITMQ_ERLANG_COOKIE=QWERTYUIOP
      - RABBITMQ_DEFAULT_USER=user
      - RABBITMQ_DEFAULT_PASS=password
    hostname: rabbit-2
    container_name: rabbit-2
    ports:
      - 5672:5672
      - 15672:15672
    depends_on:
      - etcd
  rabbit-3:
    image: rabbitmq:3.13-management
    volumes:
      - ./config/rabbitmq.conf:/config/rabbitmq.conf
      - ./data/rabbit-3:/var/lib/rabbitmq/
    environment:
      - RABBITMQ_CONFIG_FILE=/config/rabbitmq
      - RABBITMQ_ERLANG_COOKIE=QWERTYUIOP
      - RABBITMQ_DEFAULT_USER=user
      - RABBITMQ_DEFAULT_PASS=password
    hostname: rabbit-3
    container_name: rabbit-3
    ports:
      - 5673:5672
      - 15673:15672
    depends_on:
      - etcd
  etcd:
    image: bitnami/etcd:3.5.14
    container_name: etcd
    environment:
      - ALLOW_NONE_AUTHENTICATION=yes
      - ETCD_ADVERTISE_CLIENT_URLS=http://etcd:2379
    volumes:
      - ./etcd:/bitnami/etcd
    ports:
      - 2379:2379
      - 2380:2380

and here's the config file that is mapped to the rabbitmq containers as their primary config file to boot up:

# cluster_formation.peer_discovery_backend = etcd

# the backend can also be specified using its module name
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_etcd

# etcd endpoints. This property is required or peer discovery won't be performed.
cluster_formation.etcd.endpoints.1 = etcd:2379

so the versions and envs are clear in the docker-compose

[sinawic]

so when i hit docker-compose up -d etcd comes up i can connect to it and get keys of rabbitmq with:
etcdctl get --prefix=true "/rabbitmq"
this doesn't return anything
not before all instances fall into standalone mode
then i get this as the result at that exact moment when all of the instances fall into standalone mode:

/rabbitmq/discovery/rabbitmq/clusters/default/nodes/rabbit@rabbit-1
{"lease_id":7587879767041522951,"node":"rabbit@rabbit-1","ttl":61}
/rabbitmq/discovery/rabbitmq/clusters/default/nodes/rabbit@rabbit-2
{"lease_id":7587879767041522954,"node":"rabbit@rabbit-2","ttl":61}
/rabbitmq/discovery/rabbitmq/clusters/default/nodes/rabbit@rabbit-3
{"lease_id":7587879767041522948,"node":"rabbit@rabbit-3","ttl":61}

here's the log file for the whole docker-compose up -d:

cluster.log

[sinawic]

I've reached to this finally:
if the instance information exist in the etcd before it tries to bootstrap
it will bootstrap into cluster mode connecting with other peers successfully
if not it will fall into standalone mode

like if these data exists in etcd before instances come up:

/rabbitmq/discovery/rabbitmq/clusters/default/nodes/rabbit@rabbit-1
{"lease_id":7587879767151170499,"node":"rabbit@rabbit-1","ttl":61}
/rabbitmq/discovery/rabbitmq/clusters/default/nodes/rabbit@rabbit-2
{"lease_id":7587879767151170492,"node":"rabbit@rabbit-2","ttl":61}
/rabbitmq/discovery/rabbitmq/clusters/default/nodes/rabbit@rabbit-3
{"lease_id":7587879767151170496,"node":"rabbit@rabbit-3","ttl":61}

then things work as expected
if not each instance will fall into standalone mode

@michaelklishin
@lukebakken

[sinawic]

damn this is driving me crazy 🤣
seems like version rabbitmq:3.13-management is the problematic one
i downgraded to rabbitmq:3.12-management and everything seems to be working as expected

only i have another problem with setting policy now [{<<"federation-upstream-set">>,<<"all">>}] are not recognised policy settings
and im just gonna google it now!

[lukebakken]

Ensure that the correct plugins are enabled -

https://github.com/lukebakken/rabbitmq-server-11595/blob/main/config/enabled_plugins

[lukebakken]

@sinawic - I've taken the information you have provided and imported here - https://github.com/lukebakken/rabbitmq-server-11595

[lukebakken]

@sinawic - You're correct, etcd peer discover is broken in RabbitMQ 3.13.3, but it has since been fixed by this PR - #11439

In order to use that fix, you must use a docker image built on the main branch of the RabbitMQ server source code. I updated my project to use that and peer discovery works:

https://github.com/lukebakken/rabbitmq-server-11595/blob/main/docker-compose.yml#L9

The fix will ship in RabbitMQ 3.13.4.

Thanks, and sorry for this hassle!

[sinawic]

hey thanks for the reply
and thanks for the effort
i hope i could help in any way possible!
👍