[kubeadm control plane]: etcd communication errors are being swallowed #2454
Description
What steps did you take and what happened:
While testing an upgrade the etcd health checks were failing repeatedly. With the code from #2451 in place I could resolve it down one level:
failed to create etcd client: unable to create etcd client: context deadline exceeded
After some work, I found that my etcd ca secret was regenerated, changing the private key (see: #2454). It seems that GRPC has exactly one error message when the connection is misconfigured, and that's "context deadline exceeded." I haven't yet found a way to get more information on what happened via the API, but I'm continuing to dig.
What did you expect to happen:
When I set up the same condition with etcdctl and k port-forward I got a helpful error message:
{"level":"warn","ts":"2020-02-25T20:50:29.757-0800","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-f67a0407-f684-4682-bb79-ec33c94b2178/127.0.0.1:63477","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: remote error: tls: bad certificate\""}
Error: context deadline exceeded
Note the Error: context deadline exceeded is what came back from clientv3.New, and the other is a log statement being printed to stderr.
Anything else you would like to add:
I found that any error sent back from the proxy dial function was being swallowed in the same way. It also looks like we're not using the errorStream we set up with the API Server, so it's possible that we'd miss important information about the proxy connection.
Environment:
- Cluster-api version: master
- Minikube/KIND version: kind v0.7.0 go1.13.6 darwin/amd64
- Kubernetes version: (use
kubectl version): a mix of v1.15 and v1.16 control plane nodes - OS (e.g. from
/etc/os-release): ubuntu
/kind bug
/assign
/lifecycle active