Clustering - Not working?

[uktricky]

I've installed a new Gravity (0.19.0+d971e3b5) in Docker using the default docker compose from the documentation on a Raspberry Pi ARM64 and this is running fine. I've tried to create a cluster by going into Nodes and then Join - I've entered the IP of the host I want to add it onto too and created the docker-compose file for that node. The original node was 10.2.20.24.

When I spin up the docker compose up -d and then check the logs all I have is (this is another Raspberry ARM64 device):
gravity-1 | {"level":"info","ts":1735507101.6110024,"logger":"instance","msg":"Gravity starting","instance":"10.2.20.22","version":"0.19.0+d971e3b5","instance":"10.2.20.22","version":"0.19.0+d971e3b5"}

[tompaah]

This probably should go into the discussions instead, but I'll try to answer here.

The clustering is very sensitive to any changes, I've noticed. Change an IP-adress? It breaks. Scrap a node and rebuild it? It breaks. So my best tip after fiddling around with Gravity for some months now, is to just scrap everything (remove containers, compose files and persistent storage) you did and do it all again from beginning; deploy the first node and then the others. It'll probably work fine.

While this "scrap everything and begin all over again" approach might appear unconventional I found it to be of not much bother; the cluster is very fast to set up if you have pre-made compose-files and then you can populate the database with dhcp/dns-entries with the API or Gravity CLI. It's a matter of less than a minute to deploy the complete cluster if you have all the files prepared in advance.

[uktricky]

Thanks @tompaah you got me on the right track whilst removing and reinstalling, I spotted the issue ... When creating the config for the 2nd/3rd node, even though I had all options (roles) selected it defaults to just the API in the generated docker-compose ...

BOOTSTRAP_ROLES: api

Once I amended this to include the extra features it connected and sync'd up fine.

gravity-1  | {"level":"debug","ts":1735559531.1699057,"logger":"extconfig","msg":"failed to get IPs from interface","instance":"","version":"0.19.0+d971e3b5","error":"interface is loopback","if":"lo"}
gravity-1  | {"level":"debug","ts":1735559531.170136,"logger":"extconfig","msg":"Detected IP of instance","instance":"","version":"0.19.0+d971e3b5","ip":"10.2.20.22"}
gravity-1  | {"level":"debug","ts":1735559531.1705265,"logger":"extconfig","msg":"failed to get IPs from interface","instance":"","version":"0.19.0+d971e3b5","error":"interface is loopback","if":"lo"}
gravity-1  | {"level":"info","ts":1735559531.1898947,"logger":"instance","msg":"Gravity starting","instance":"10.2.20.22","version":"0.19.0+d971e3b5","instance":"10.2.20.22","version":"0.19.0+d971e3b5"}
gravity-1  | {"level":"info","ts":1735559531.1900058,"logger":"instance","msg":"'etcd' in bootstrap roles, starting embedded etcd","instance":"10.2.20.22","version":"0.19.0+d971e3b5","instance":"10.2.20.22"}
gravity-1  | {"level":"info","ts":1735559531.1902702,"logger":"role.etcd","msg":"joining etcd cluster","instance":"10.2.20.22","version":"0.19.0+d971e3b5","peer":"http://10.2.20.24:8008"}
gravity-1  | {"level":"info","ts":1735559538.2512488,"logger":"role.etcd","msg":"joining etcd cluster","instance":"10.2.20.22","version":"0.19.0+d971e3b5","initialCluster":"10.2.20.24=https://10.2.20.24:2380,10.2.20.22=https://10.2.20.22:2380"}
gravity-1  | {"level":"info","ts":1735559538.2513063,"logger":"role.etcd","msg":"starting embedded etcd","instance":"10.2.20.22","version":"0.19.0+d971e3b5"}
gravity-1  | {"level":"warn","ts":1735559538.2513804,"logger":"role.etcd","msg":"Running http and grpc server on single port. This is not recommended for production.","instance":"10.2.20.22","version":"0.19.0+d971e3b5"}
gravity-1  | {"level":"warn","ts":1735559538.2520723,"logger":"role.etcd","msg":"automatically generate certificates","instance":"10.2.20.22","version":"0.19.0+d971e3b5","certificate-validity-bound-not-after":1767095538.2520635}