etcd errors running swarm local example

[yayitswei]

I'm following the swarm-local instructions and getting some errors on this step:

$ eval `docker-machine env swarm-manager`
$ ETCD_TOKEN=aaaa make swarm-stolon

Here are the failure logs:

$ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                         PORTS
uuzxpehhr3g8        etcd-00             replicated          1/1                 quay.io/coreos/etcd:v3.0.15   
4o0wy75g5cpp        etcd-01             replicated          1/1                 quay.io/coreos/etcd:v3.0.15   
xqr4zxes8kwr        etcd-02             replicated          1/1                 quay.io/coreos/etcd:v3.0.15   
zkcsb4rr8hnf        keeper              replicated          3/3                 isim/stolon-keeper:0.5.0      
qhu26zp0l2ab        proxy               replicated          1/1                 isim/stolon-proxy:0.5.0       *:30000->25432/tcp
urgkxkl0iyen        sentinel            replicated          1/1                 isim/stolon-sentinel:0.5.0    

$ docker service logs -f uuzxpehhr3g8
...
etcd-00.1.6icbla764imb@swarm-worker-00    | 2018-02-11 21:40:32.226154 W | etcdserver: cannot get the version of member 36f6e20eb1659d86 (Get http://etcd-02:2380/version: dial tcp: lookup etcd-02 on 127.0.0.11:53: no such host)
etcd-00.1.6icbla764imb@swarm-worker-00    | 2018-02-11 21:40:36.254532 W | etcdserver: failed to reach the peerURL(http://etcd-02:2380) of member 36f6e20eb1659d86 (Get http://etcd-02:2380/version: dial tcp: lookup etcd-02 on 127.0.0.11:53: no such host)
etcd-00.1.6icbla764imb@swarm-worker-00    | 2018-02-11 21:40:36.257664 W | etcdserver: cannot get the version of member 36f6e20eb1659d86 (Get http://etcd-02:2380/version: dial tcp: lookup etcd-02 on 127.0.0.11:53: no such host)
etcd-00.1.6icbla764imb@swarm-worker-00    | 2018-02-11 21:40:38.205087 I | rafthttp: peer 36f6e20eb1659d86 became active
etcd-00.1.6icbla764imb@swarm-worker-00    | 2018-02-11 21:40:38.205297 I | rafthttp: established a TCP streaming connection with peer 36f6e20eb1659d86 (stream Message writer)
etcd-00.1.6icbla764imb@swarm-worker-00    | 2018-02-11 21:40:38.205869 I | rafthttp: established a TCP streaming connection with peer 36f6e20eb1659d86 (stream MsgApp v2 writer)
...
etcd-00.1.6icbla764imb@swarm-worker-00    | 2018-02-11 21:41:16.880168 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp [::]:2379: connect: cannot assign requested address"; Reconnecting to {"[::]:2379" <nil>}
etcd-00.1.6icbla764imb@swarm-worker-00    | 2018-02-11 21:41:27.018667 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp [::]:2379: connect: cannot assign requested address"; Reconnecting to {"[::]:2379" <nil>}
etcd-00.1.6icbla764imb@swarm-worker-00    | 2018-02-11 21:41:30.420764 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp [::]:2379: connect: cannot assign requested address"; Reconnecting to {"[::]:2379" <nil>}
etcd-00.1.6icbla764imb@swarm-worker-00    | 2018-02-11 21:41:33.271163 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp [::]:2379: connect: cannot assign requested address"; Reconnecting to {"[::]:2379" <nil>}
etcd-00.1.6icbla764imb@swarm-worker-00    | 2018-02-11 21:41:37.434911 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp [::]:2379: connect: cannot assign requested address"; Reconnecting to {"[::]:2379" <nil>}
etcd-00.1.6icbla764imb@swarm-worker-00    | 2018-02-11 21:41:40.241785 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp [::]:2379: connect: cannot assign requested address"; Reconnecting to {"[::]:2379" <nil>}
etcd-00.1.6icbla764imb@swarm-worker-00    | 2018-02-11 21:41:52.866965 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 75.080072ms)
etcd-00.1.6icbla764imb@swarm-worker-00    | 2018-02-11 21:41:52.870762 W | etcdserver: server is likely overloaded
etcd-00.1.6icbla764imb@swarm-worker-00    | 2018-02-11 21:41:52.870824 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 78.955981ms)
etcd-00.1.6icbla764imb@swarm-worker-00    | 2018-02-11 21:41:52.870849 W | etcdserver: server is likely overloaded
etcd-00.1.6icbla764imb@swarm-worker-00    | 2018-02-11 21:43:08.364375 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp [::]:2379: connect: cannot assign requested address"; Reconnecting to {"[::]:2379" <nil>}
etcd-00.1.6icbla764imb@swarm-worker-00    | 2018-02-11 21:43:09.616386 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp [::]:2379: connect: cannot assign requested address"; Reconnecting to {"[::]:2379" <nil>}
etcd-00.1.6icbla764imb@swarm-worker-00    | 2018-02-11 21:43:19.430980 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp [::]:2379: connect: cannot assign requested address"; Reconnecting to {"[::]:2379" <nil>}
etcd-00.1.6icbla764imb@swarm-worker-00    | 2018-02-11 21:43:20.353878 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp [::]:2379: connect: cannot assign requested address"; Reconnecting to {"[::]:2379" <nil>}
etcd-00.1.6icbla764imb@swarm-worker-00    | 2018-02-11 21:43:22.741231 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp [::]:2379: connect: cannot assign requested address"; Reconnecting to {"[::]:2379" <nil>}
etcd-00.1.6icbla764imb@swarm-worker-00    | 2018-02-11 21:43:45.296846 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp [::]:2379: connect: cannot assign requested address"; Reconnecting to {"[::]:2379" <nil>}

Any tips on how to debug this? Thanks!

[ihcsim]

Can you check whether your etcd cluster is healthy with something like:

$ docker exec <etcd-container-name> etcdctl cluster-health

I found two similar issues at:

1. Cannot assign requested address after switching from 3.1.0 to 3.2.0 etcd-io/etcd#8151
2. Running the official docker image for etcd will listen only on localhost. etcd-io/etcd#8230

You can try use a newer version of etcd.

[yayitswei]

Seems like upgrading etcd worked, thanks for your help! Looks healthy when running the cluster-health command.

-ETCD_VERSION=v3.0.15
+ETCD_VERSION=v3.3.0