Skip to content
This repository has been archived by the owner on Jun 20, 2024. It is now read-only.

Test flake in 870_weave_recovers_unreachable_ips #3690

Closed
bboreham opened this issue Aug 14, 2019 · 2 comments · Fixed by #3716
Closed

Test flake in 870_weave_recovers_unreachable_ips #3690

bboreham opened this issue Aug 14, 2019 · 2 comments · Fixed by #3716
Labels
chore

Comments

@bboreham
Copy link
Contributor

Failing fairly often, e.g. https://circleci.com/gh/weaveworks/weave/12212.
Generally succeeds on re-run, e.g. 12213 is the re-run of 12212.

>>> Test /home/circleci/src/github.com/weaveworks/weave/test/870_weave_recovers_unreachable_ips_on_relaunch_3_test.sh finished after 193.8 secs with error: exit status 1
Test weave-net deallocates from IPAM on node failure
Tearing down kubernetes cluster
test-12212-1-0.us-central1-a.weave-net> [preflight] running pre-flight checks
test-12212-1-0.us-central1-a.weave-net> [reset] stopping the kubelet service
test-12212-1-0.us-central1-a.weave-net> [reset] unmounting mounted directories in "/var/lib/kubelet"
test-12212-1-0.us-central1-a.weave-net> [reset] removing kubernetes-managed containers
test-12212-1-0.us-central1-a.weave-net> [reset] cleaning up running containers using crictl with socket /var/run/dockershim.sock
test-12212-1-0.us-central1-a.weave-net> [reset] failed to list running pods using crictl: exit status 1. Trying to use docker instead[reset] no etcd manifest found in "/etc/kubernetes/manifests/etcd.yaml". Assuming external etcd
test-12212-1-0.us-central1-a.weave-net> [reset] deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
test-12212-1-0.us-central1-a.weave-net> [reset] deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
test-12212-1-0.us-central1-a.weave-net> [reset] deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
test-12212-1-1.us-central1-a.weave-net> [preflight] running pre-flight checks
test-12212-1-1.us-central1-a.weave-net> [reset] stopping the kubelet service
test-12212-1-1.us-central1-a.weave-net> [reset] unmounting mounted directories in "/var/lib/kubelet"
test-12212-1-1.us-central1-a.weave-net> [reset] removing kubernetes-managed containers
test-12212-1-1.us-central1-a.weave-net> [reset] cleaning up running containers using crictl with socket /var/run/dockershim.sock
test-12212-1-1.us-central1-a.weave-net> [reset] failed to list running pods using crictl: exit status 1. Trying to use docker instead[reset] no etcd manifest found in "/etc/kubernetes/manifests/etcd.yaml". Assuming external etcd
test-12212-1-1.us-central1-a.weave-net> [reset] deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
test-12212-1-1.us-central1-a.weave-net> [reset] deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
test-12212-1-1.us-central1-a.weave-net> [reset] deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
test-12212-1-2.us-central1-a.weave-net> [preflight] running pre-flight checks
test-12212-1-2.us-central1-a.weave-net> [reset] stopping the kubelet service
test-12212-1-2.us-central1-a.weave-net> [reset] unmounting mounted directories in "/var/lib/kubelet"
test-12212-1-2.us-central1-a.weave-net> [reset] removing kubernetes-managed containers
test-12212-1-2.us-central1-a.weave-net> [reset] cleaning up running containers using crictl with socket /var/run/dockershim.sock
test-12212-1-2.us-central1-a.weave-net> [reset] failed to list running pods using crictl: exit status 1. Trying to use docker instead[reset] no etcd manifest found in "/etc/kubernetes/manifests/etcd.yaml". Assuming external etcd
test-12212-1-2.us-central1-a.weave-net> [reset] deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
test-12212-1-2.us-central1-a.weave-net> [reset] deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
test-12212-1-2.us-central1-a.weave-net> [reset] deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
Setting up kubernetes cluster
I0814 14:11:35.707956    3994 feature_gate.go:230] feature gates: &{map[]}
test-12212-1-0.us-central1-a.weave-net> [init] using Kubernetes version: v1.11.0
test-12212-1-0.us-central1-a.weave-net> [preflight] running pre-flight checks
I0814 14:11:35.733298    3994 kernel_validator.go:81] Validating kernel version
I0814 14:11:35.733412    3994 kernel_validator.go:96] Validating kernel config
	[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 17.06.2-ce. Max validated version: 17.03
test-12212-1-0.us-central1-a.weave-net> [preflight/images] Pulling images required for setting up a Kubernetes cluster
test-12212-1-0.us-central1-a.weave-net> [preflight/images] This might take a minute or two, depending on the speed of your internet connection
test-12212-1-0.us-central1-a.weave-net> [preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
test-12212-1-0.us-central1-a.weave-net> [kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
test-12212-1-0.us-central1-a.weave-net> [kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
test-12212-1-0.us-central1-a.weave-net> [preflight] Activating the kubelet service
test-12212-1-0.us-central1-a.weave-net> [certificates] Generated ca certificate and key.
test-12212-1-0.us-central1-a.weave-net> [certificates] Generated apiserver certificate and key.
test-12212-1-0.us-central1-a.weave-net> [certificates] apiserver serving cert is signed for DNS names [test-12212-1-0 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.128.0.71]
test-12212-1-0.us-central1-a.weave-net> [certificates] Generated apiserver-kubelet-client certificate and key.
test-12212-1-0.us-central1-a.weave-net> [certificates] Generated sa key and public key.
test-12212-1-0.us-central1-a.weave-net> [certificates] Generated front-proxy-ca certificate and key.
test-12212-1-0.us-central1-a.weave-net> [certificates] Generated front-proxy-client certificate and key.
test-12212-1-0.us-central1-a.weave-net> [certificates] Generated etcd/ca certificate and key.
test-12212-1-0.us-central1-a.weave-net> [certificates] Generated etcd/server certificate and key.
test-12212-1-0.us-central1-a.weave-net> [certificates] etcd/server serving cert is signed for DNS names [test-12212-1-0 localhost] and IPs [127.0.0.1 ::1]
test-12212-1-0.us-central1-a.weave-net> [certificates] Generated etcd/peer certificate and key.
test-12212-1-0.us-central1-a.weave-net> [certificates] etcd/peer serving cert is signed for DNS names [test-12212-1-0 localhost] and IPs [10.128.0.71 127.0.0.1 ::1]
test-12212-1-0.us-central1-a.weave-net> [certificates] Generated etcd/healthcheck-client certificate and key.
test-12212-1-0.us-central1-a.weave-net> [certificates] Generated apiserver-etcd-client certificate and key.
test-12212-1-0.us-central1-a.weave-net> [certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
test-12212-1-0.us-central1-a.weave-net> [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
test-12212-1-0.us-central1-a.weave-net> [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
test-12212-1-0.us-central1-a.weave-net> [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
test-12212-1-0.us-central1-a.weave-net> [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
test-12212-1-0.us-central1-a.weave-net> [controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
test-12212-1-0.us-central1-a.weave-net> [controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
test-12212-1-0.us-central1-a.weave-net> [controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
test-12212-1-0.us-central1-a.weave-net> [etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
test-12212-1-0.us-central1-a.weave-net> [init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
test-12212-1-0.us-central1-a.weave-net> [init] this might take a minute or longer if the control plane images have to be pulled
test-12212-1-0.us-central1-a.weave-net> [apiclient] All control plane components are healthy after 45.019342 seconds
test-12212-1-0.us-central1-a.weave-net> [uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
test-12212-1-0.us-central1-a.weave-net> [kubelet] Creating a ConfigMap "kubelet-config-1.11" in namespace kube-system with the configuration for the kubelets in the cluster
test-12212-1-0.us-central1-a.weave-net> [markmaster] Marking the node test-12212-1-0 as master by adding the label "node-role.kubernetes.io/master=''"
test-12212-1-0.us-central1-a.weave-net> [markmaster] Marking the node test-12212-1-0 as master by adding the taints [node-role.kubernetes.io/master:NoSchedule]
test-12212-1-0.us-central1-a.weave-net> [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "test-12212-1-0" as an annotation
test-12212-1-0.us-central1-a.weave-net> [bootstraptoken] using token: 112233.4455667788990000
test-12212-1-0.us-central1-a.weave-net> [bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
test-12212-1-0.us-central1-a.weave-net> [bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
test-12212-1-0.us-central1-a.weave-net> [bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
test-12212-1-0.us-central1-a.weave-net> [bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
test-12212-1-0.us-central1-a.weave-net> [addons] Applied essential addon: CoreDNS
test-12212-1-0.us-central1-a.weave-net> [addons] Applied essential addon: kube-proxy
test-12212-1-0.us-central1-a.weave-net> 
test-12212-1-0.us-central1-a.weave-net> Your Kubernetes master has initialized successfully!
test-12212-1-0.us-central1-a.weave-net> 
test-12212-1-0.us-central1-a.weave-net> To start using your cluster, you need to run the following as a regular user:
test-12212-1-0.us-central1-a.weave-net> 
test-12212-1-0.us-central1-a.weave-net> mkdir -p $HOME/.kube
test-12212-1-0.us-central1-a.weave-net> sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
test-12212-1-0.us-central1-a.weave-net> sudo chown $(id -u):$(id -g) $HOME/.kube/config
test-12212-1-0.us-central1-a.weave-net> 
test-12212-1-0.us-central1-a.weave-net> You should now deploy a pod network to the cluster.
test-12212-1-0.us-central1-a.weave-net> Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
test-12212-1-0.us-central1-a.weave-net> https://kubernetes.io/docs/concepts/cluster-administration/addons/
test-12212-1-0.us-central1-a.weave-net> 
test-12212-1-0.us-central1-a.weave-net> You can now join any number of machines by running the following on each node
test-12212-1-0.us-central1-a.weave-net> as root:
test-12212-1-0.us-central1-a.weave-net> 
test-12212-1-0.us-central1-a.weave-net> kubeadm join 10.128.0.71:6443 --token 112233.4455667788990000 --discovery-token-ca-cert-hash sha256:333b69a994b194c5f8aad732cd16aa7bf8b9f00a4efe17cf513e5feeec24b41d
test-12212-1-0.us-central1-a.weave-net> 
test-12212-1-1.us-central1-a.weave-net> [preflight] running pre-flight checks
	[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs_wrr ip_vs_sh ip_vs ip_vs_rr] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]
you can solve this problem with following methods:
 1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

I0814 14:12:35.440398    3844 kernel_validator.go:81] Validating kernel version
I0814 14:12:35.440513    3844 kernel_validator.go:96] Validating kernel config
	[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 17.06.2-ce. Max validated version: 17.03
test-12212-1-1.us-central1-a.weave-net> [discovery] Trying to connect to API Server "10.128.0.71:6443"
test-12212-1-1.us-central1-a.weave-net> [discovery] Created cluster-info discovery client, requesting info from "https://10.128.0.71:6443"
test-12212-1-1.us-central1-a.weave-net> [discovery] Failed to connect to API Server "10.128.0.71:6443": token id "112233" is invalid for this cluster or it has expired. Use "kubeadm token create" on the master node to creating a new valid token
test-12212-1-1.us-central1-a.weave-net> [discovery] Trying to connect to API Server "10.128.0.71:6443"
test-12212-1-1.us-central1-a.weave-net> [discovery] Created cluster-info discovery client, requesting info from "https://10.128.0.71:6443"
test-12212-1-1.us-central1-a.weave-net> [discovery] Cluster info signature and contents are valid and no TLS pinning was specified, will use API Server "10.128.0.71:6443"
test-12212-1-1.us-central1-a.weave-net> [discovery] Successfully established connection with API Server "10.128.0.71:6443"
test-12212-1-1.us-central1-a.weave-net> [kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.11" ConfigMap in the kube-system namespace
test-12212-1-1.us-central1-a.weave-net> [kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
test-12212-1-1.us-central1-a.weave-net> [kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
test-12212-1-1.us-central1-a.weave-net> [preflight] Activating the kubelet service
test-12212-1-1.us-central1-a.weave-net> [tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
test-12212-1-1.us-central1-a.weave-net> [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "test-12212-1-1" as an annotation
test-12212-1-1.us-central1-a.weave-net> 
test-12212-1-1.us-central1-a.weave-net> This node has joined the cluster:
test-12212-1-1.us-central1-a.weave-net> * Certificate signing request was sent to master and a response
test-12212-1-1.us-central1-a.weave-net> was received.
test-12212-1-1.us-central1-a.weave-net> * The Kubelet was informed of the new secure connection details.
test-12212-1-1.us-central1-a.weave-net> 
test-12212-1-1.us-central1-a.weave-net> Run 'kubectl get nodes' on the master to see this node join the cluster.
test-12212-1-2.us-central1-a.weave-net> [preflight] running pre-flight checks
	[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] or no builtin kernel ipvs support: map[nf_conntrack_ipv4:{} ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{}]
you can solve this problem with following methods:
 1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

I0814 14:12:44.130369    3904 kernel_validator.go:81] Validating kernel version
I0814 14:12:44.130508    3904 kernel_validator.go:96] Validating kernel config
	[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 17.06.2-ce. Max validated version: 17.03
test-12212-1-2.us-central1-a.weave-net> [discovery] Trying to connect to API Server "10.128.0.71:6443"
test-12212-1-2.us-central1-a.weave-net> [discovery] Created cluster-info discovery client, requesting info from "https://10.128.0.71:6443"
test-12212-1-2.us-central1-a.weave-net> [discovery] Cluster info signature and contents are valid and no TLS pinning was specified, will use API Server "10.128.0.71:6443"
test-12212-1-2.us-central1-a.weave-net> [discovery] Successfully established connection with API Server "10.128.0.71:6443"
test-12212-1-2.us-central1-a.weave-net> [kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.11" ConfigMap in the kube-system namespace
test-12212-1-2.us-central1-a.weave-net> [kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
test-12212-1-2.us-central1-a.weave-net> [kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
test-12212-1-2.us-central1-a.weave-net> [preflight] Activating the kubelet service
test-12212-1-2.us-central1-a.weave-net> [tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
test-12212-1-2.us-central1-a.weave-net> [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "test-12212-1-2" as an annotation
test-12212-1-2.us-central1-a.weave-net> 
test-12212-1-2.us-central1-a.weave-net> This node has joined the cluster:
test-12212-1-2.us-central1-a.weave-net> * Certificate signing request was sent to master and a response
test-12212-1-2.us-central1-a.weave-net> was received.
test-12212-1-2.us-central1-a.weave-net> * The Kubelet was informed of the new secure connection details.
test-12212-1-2.us-central1-a.weave-net> 
test-12212-1-2.us-central1-a.weave-net> Run 'kubectl get nodes' on the master to see this node join the cluster.
test-12212-1-0.us-central1-a.weave-net> serviceaccount/weave-net created
test-12212-1-0.us-central1-a.weave-net> clusterrole.rbac.authorization.k8s.io/weave-net created
test-12212-1-0.us-central1-a.weave-net> clusterrolebinding.rbac.authorization.k8s.io/weave-net created
test-12212-1-0.us-central1-a.weave-net> role.rbac.authorization.k8s.io/weave-net created
test-12212-1-0.us-central1-a.weave-net> rolebinding.rbac.authorization.k8s.io/weave-net created
test-12212-1-0.us-central1-a.weave-net> daemonset.extensions/weave-net created
Checking unreachable IPs
Dropping node test-12212-1-1.us-central1-a.weave-net with 'sudo kubectl delete node'
test-12212-1-0.us-central1-a.weave-net> node "test-12212-1-1" deleted
Checking unreachable IPs
Tearing down kubernetes cluster
test-12212-1-0.us-central1-a.weave-net> [preflight] running pre-flight checks
test-12212-1-0.us-central1-a.weave-net> [reset] stopping the kubelet service
test-12212-1-0.us-central1-a.weave-net> [reset] unmounting mounted directories in "/var/lib/kubelet"
test-12212-1-0.us-central1-a.weave-net> [reset] removing kubernetes-managed containers
test-12212-1-0.us-central1-a.weave-net> [reset] cleaning up running containers using crictl with socket /var/run/dockershim.sock
test-12212-1-0.us-central1-a.weave-net> [reset] failed to list running pods using crictl: exit status 1. Trying to use docker instead[reset] deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes /var/lib/etcd]
test-12212-1-0.us-central1-a.weave-net> [reset] deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
test-12212-1-0.us-central1-a.weave-net> [reset] deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
test-12212-1-1.us-central1-a.weave-net> [preflight] running pre-flight checks
test-12212-1-1.us-central1-a.weave-net> [reset] stopping the kubelet service
test-12212-1-1.us-central1-a.weave-net> [reset] unmounting mounted directories in "/var/lib/kubelet"
test-12212-1-1.us-central1-a.weave-net> [reset] removing kubernetes-managed containers
test-12212-1-1.us-central1-a.weave-net> [reset] cleaning up running containers using crictl with socket /var/run/dockershim.sock
test-12212-1-1.us-central1-a.weave-net> [reset] failed to list running pods using crictl: exit status 1. Trying to use docker instead[reset] no etcd manifest found in "/etc/kubernetes/manifests/etcd.yaml". Assuming external etcd
test-12212-1-1.us-central1-a.weave-net> [reset] deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
test-12212-1-1.us-central1-a.weave-net> [reset] deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
test-12212-1-1.us-central1-a.weave-net> [reset] deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
test-12212-1-2.us-central1-a.weave-net> [preflight] running pre-flight checks
test-12212-1-2.us-central1-a.weave-net> [reset] stopping the kubelet service
test-12212-1-2.us-central1-a.weave-net> [reset] unmounting mounted directories in "/var/lib/kubelet"
test-12212-1-2.us-central1-a.weave-net> [reset] removing kubernetes-managed containers
test-12212-1-2.us-central1-a.weave-net> [reset] cleaning up running containers using crictl with socket /var/run/dockershim.sock
test-12212-1-2.us-central1-a.weave-net> [reset] failed to list running pods using crictl: exit status 1. Trying to use docker instead[reset] no etcd manifest found in "/etc/kubernetes/manifests/etcd.yaml". Assuming external etcd
test-12212-1-2.us-central1-a.weave-net> [reset] deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
test-12212-1-2.us-central1-a.weave-net> [reset] deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
test-12212-1-2.us-central1-a.weave-net> [reset] deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
test #5 "unreachable_ip_addresses_count test-12212-1-0.us-central1-a.weave-net" failed:
	expected "0"
	got "1"
test #6 "unreachable_ip_addresses_count test-12212-1-2.us-central1-a.weave-net" failed:
	expected "0"
	got "1"
2 of 9 tests failed in 191.354s.
@bboreham bboreham added the chore label Aug 14, 2019
@bboreham
Copy link
Contributor Author

I got some Weave Net pod logs for a fail: the interesting thing is that HOST2 seems to reconnect after the node delete notification.

HOST1:

INFO: 2019/10/11 14:48:11.965897 Command line options: map[http-addr:127.0.0.1:6784 metrics-addr:0.0.0.0:6782 expect-npc:true ipalloc-init:consensus=2 port:6783 conn-limit:100 datapath:datapath ipalloc-range:10.32.0.0/12 nickname:test-12587-0-0 no-dns:true db-prefix:/weavedb/weave-net docker-api: host-root:/host name:d6:c0:35:6b:e7:4b]
INFO: 2019/10/11 14:48:11.966251 weave  unreleased
INFO: 2019/10/11 14:48:12.063254 Bridge type is bridged_fastdp
INFO: 2019/10/11 14:48:12.063272 Communication between peers is unencrypted.
INFO: 2019/10/11 14:48:12.067329 Our name is d6:c0:35:6b:e7:4b(test-12587-0-0)
INFO: 2019/10/11 14:48:12.067358 Launch detected - using supplied peer list: [10.128.0.105 10.128.0.106]
INFO: 2019/10/11 14:48:12.143382 Unable to fetch ConfigMap kube-system/weave-net to infer unique cluster ID
INFO: 2019/10/11 14:48:12.143422 Checking for pre-existing addresses on weave bridge
INFO: 2019/10/11 14:48:12.176741 [allocator d6:c0:35:6b:e7:4b] No valid persisted data
INFO: 2019/10/11 14:48:12.187334 [allocator d6:c0:35:6b:e7:4b] Initialising via deferred consensus
INFO: 2019/10/11 14:48:12.187387 Sniffing traffic on datapath (via ODP)
INFO: 2019/10/11 14:48:12.189301 ->[10.128.0.105:6783] attempting connection
INFO: 2019/10/11 14:48:12.192990 ->[10.128.0.106:6783] attempting connection
INFO: 2019/10/11 14:48:12.200517 ->[10.128.0.106:6783|ba:2e:11:69:a5:e2(test-12587-0-2)]: connection ready; using protocol version 2
INFO: 2019/10/11 14:48:12.200604 overlay_switch ->[ba:2e:11:69:a5:e2(test-12587-0-2)] using fastdp
INFO: 2019/10/11 14:48:12.200632 ->[10.128.0.106:6783|ba:2e:11:69:a5:e2(test-12587-0-2)]: connection added (new peer)
INFO: 2019/10/11 14:48:12.214164 ->[10.128.0.105:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: connection ready; using protocol version 2
INFO: 2019/10/11 14:48:12.214256 overlay_switch ->[52:9b:77:28:8d:4f(test-12587-0-1)] using fastdp
INFO: 2019/10/11 14:48:12.214295 ->[10.128.0.105:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: connection added (new peer)
INFO: 2019/10/11 14:48:12.218161 Listening for HTTP control messages on 127.0.0.1:6784
INFO: 2019/10/11 14:48:12.226111 Listening for metrics requests on 0.0.0.0:6782
INFO: 2019/10/11 14:48:12.230320 ->[10.128.0.106:6783|ba:2e:11:69:a5:e2(test-12587-0-2)]: connection fully established
INFO: 2019/10/11 14:48:12.230733 sleeve ->[10.128.0.106:6783|ba:2e:11:69:a5:e2(test-12587-0-2)]: Effective MTU verified at 1398
INFO: 2019/10/11 14:48:12.239238 overlay_switch ->[52:9b:77:28:8d:4f(test-12587-0-1)] using sleeve
INFO: 2019/10/11 14:48:12.239263 ->[10.128.0.105:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: connection fully established
INFO: 2019/10/11 14:48:12.239372 overlay_switch ->[52:9b:77:28:8d:4f(test-12587-0-1)] using fastdp
INFO: 2019/10/11 14:48:12.240481 sleeve ->[10.128.0.105:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: Effective MTU verified at 1398
INFO: 2019/10/11 14:48:12.342832 Discovered remote MAC ba:2e:11:69:a5:e2 at ba:2e:11:69:a5:e2(test-12587-0-2)
INFO: 2019/10/11 14:48:12.386941 Discovered remote MAC 52:9b:77:28:8d:4f at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:12.440924 Discovered remote MAC 0a:24:dd:13:04:03 at ba:2e:11:69:a5:e2(test-12587-0-2)
INFO: 2019/10/11 14:48:12.642827 Discovered remote MAC 12:c2:af:c9:69:7c at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:12.706906 Discovered remote MAC c6:06:f1:95:ea:59 at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:12.854272 Discovered remote MAC 2e:11:03:d5:22:91 at ba:2e:11:69:a5:e2(test-12587-0-2)
INFO: 2019/10/11 14:48:13.090702 [kube-peers] Added myself to peer list &{[{ba:2e:11:69:a5:e2 test-12587-0-2} {52:9b:77:28:8d:4f test-12587-0-1} {d6:c0:35:6b:e7:4b test-12587-0-0}]}
DEBU: 2019/10/11 14:48:13.132990 [kube-peers] Nodes that have disappeared: map[d6:c0:35:6b:e7:4b:{d6:c0:35:6b:e7:4b test-12587-0-0}]
WARN: 2019/10/11 14:48:13.133137 [kube-peers] not removing myself {d6:c0:35:6b:e7:4b test-12587-0-0}
DEBU: 2019/10/11 14:48:13.139167 [kube-peers] Nodes that have disappeared: map[d6:c0:35:6b:e7:4b:{d6:c0:35:6b:e7:4b test-12587-0-0}]
WARN: 2019/10/11 14:48:13.139420 [kube-peers] not removing myself {d6:c0:35:6b:e7:4b test-12587-0-0}
DEBU: 2019/10/11 14:48:13.147498 [kube-peers] Nodes that have disappeared: map[d6:c0:35:6b:e7:4b:{d6:c0:35:6b:e7:4b test-12587-0-0}]
WARN: 2019/10/11 14:48:13.147703 [kube-peers] not removing myself {d6:c0:35:6b:e7:4b test-12587-0-0}
10.40.0.0
10.128.0.105
10.128.0.106
DEBU: 2019/10/11 14:48:13.265264 registering for updates for node delete events
INFO: 2019/10/11 14:48:17.214124 Discovered remote MAC 8e:83:29:e2:60:0e at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:17.312144 Discovered remote MAC 8a:a3:fd:36:eb:b4 at 52:9b:77:28:8d:4f(test-12587-0-1)
DEBU: 2019/10/11 14:48:19.560619 [kube-peers] Nodes that have disappeared: map[52:9b:77:28:8d:4f:{52:9b:77:28:8d:4f test-12587-0-1} d6:c0:35:6b:e7:4b:{d6:c0:35:6b:e7:4b test-12587-0-0}]
DEBU: 2019/10/11 14:48:19.560644 [kube-peers] Preparing to remove disappeared peer 52:9b:77:28:8d:4f
DEBU: 2019/10/11 14:48:19.563497 [kube-peers] Noting I plan to remove  52:9b:77:28:8d:4f
DEBU: 2019/10/11 14:48:19.569059 weave DELETE to http://127.0.0.1:6784/peer/52:9b:77:28:8d:4f with map[]
INFO: 2019/10/11 14:48:19.574313 [kube-peers] rmpeer of 52:9b:77:28:8d:4f: 262144 IPs taken over from 52:9b:77:28:8d:4f

INFO: 2019/10/11 14:48:19.583977 ->[10.128.0.105:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: connection shutting down due to error: read tcp4 10.128.0.107:56677->10.128.0.105:6783: read: connection reset by peer
INFO: 2019/10/11 14:48:19.584188 ->[10.128.0.105:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: connection deleted
INFO: 2019/10/11 14:48:19.586630 Removed unreachable peer 52:9b:77:28:8d:4f(test-12587-0-1)
WARN: 2019/10/11 14:48:19.592494 [kube-peers] not removing myself {d6:c0:35:6b:e7:4b test-12587-0-0}
DEBU: 2019/10/11 14:48:19.599114 [kube-peers] Nodes that have disappeared: map[d6:c0:35:6b:e7:4b:{d6:c0:35:6b:e7:4b test-12587-0-0}]
WARN: 2019/10/11 14:48:19.599201 [kube-peers] not removing myself {d6:c0:35:6b:e7:4b test-12587-0-0}
DEBU: 2019/10/11 14:48:19.952364 [kube-peers] Nodes that have disappeared: map[d6:c0:35:6b:e7:4b:{d6:c0:35:6b:e7:4b test-12587-0-0}]
WARN: 2019/10/11 14:48:19.952615 [kube-peers] not removing myself {d6:c0:35:6b:e7:4b test-12587-0-0}
DEBU: 2019/10/11 14:48:20.153754 weave POST to http://127.0.0.1:6784/connect with map[replace:[true] peer:[10.128.0.106]]
INFO: 2019/10/11 14:48:21.237874 ->[10.128.0.105:39755] connection accepted
INFO: 2019/10/11 14:48:21.238789 ->[10.128.0.105:6783] attempting connection
INFO: 2019/10/11 14:48:21.273076 ->[10.128.0.105:39755|52:9b:77:28:8d:4f(test-12587-0-1)]: connection ready; using protocol version 2
INFO: 2019/10/11 14:48:21.273351 overlay_switch ->[52:9b:77:28:8d:4f(test-12587-0-1)] using fastdp
INFO: 2019/10/11 14:48:21.273411 ->[10.128.0.105:39755|52:9b:77:28:8d:4f(test-12587-0-1)]: connection added
INFO: 2019/10/11 14:48:21.274418 ->[10.128.0.105:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: connection ready; using protocol version 2
INFO: 2019/10/11 14:48:21.274606 overlay_switch ->[52:9b:77:28:8d:4f(test-12587-0-1)] using fastdp
INFO: 2019/10/11 14:48:21.274660 ->[10.128.0.105:39755|52:9b:77:28:8d:4f(test-12587-0-1)]: connection deleted
INFO: 2019/10/11 14:48:21.274745 ->[10.128.0.105:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: connection added
INFO: 2019/10/11 14:48:21.275029 ->[10.128.0.105:39755|52:9b:77:28:8d:4f(test-12587-0-1)]: connection shutting down due to error: Multiple connections to 52:9b:77:28:8d:4f(test-12587-0-1) added to d6:c0:35:6b:e7:4b(test-12587-0-0)
INFO: 2019/10/11 14:48:21.276800 ->[10.128.0.105:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: connection fully established
INFO: 2019/10/11 14:48:21.280127 sleeve ->[10.128.0.105:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: Effective MTU verified at 1398
INFO: 2019/10/11 14:48:21.338833 Discovered remote MAC c6:06:f1:95:ea:59 at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:21.384071 Discovered remote MAC 52:9b:77:28:8d:4f at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:21.954871 Discovered remote MAC 12:c2:af:c9:69:7c at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:23.875303 ->[10.128.0.105:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: connection shutting down due to error: read tcp4 10.128.0.107:38063->10.128.0.105:6783: read: connection reset by peer
INFO: 2019/10/11 14:48:23.875720 ->[10.128.0.105:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: connection deleted
INFO: 2019/10/11 14:48:23.876413 Removed unreachable peer 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:24.393358 Discovered remote MAC 4e:de:38:ef:c3:d7 at ba:2e:11:69:a5:e2(test-12587-0-2)

HOST3:

INFO: 2019/10/11 14:48:11.774943 Command line options: map[db-prefix:/weavedb/weave-net docker-api: ipalloc-range:10.32.0.0/12 metrics-addr:0.0.0.0:6782 no-dns:true conn-limit:100 datapath:datapath ipalloc-init:consensus=2 host-root:/host name:ba:2e:11:69:a5:e2 port:6783 expect-npc:true http-addr:127.0.0.1:6784 nickname:test-12587-0-2]
INFO: 2019/10/11 14:48:11.775139 weave  unreleased
INFO: 2019/10/11 14:48:11.849084 Bridge type is bridged_fastdp
INFO: 2019/10/11 14:48:11.849102 Communication between peers is unencrypted.
INFO: 2019/10/11 14:48:11.854314 Our name is ba:2e:11:69:a5:e2(test-12587-0-2)
INFO: 2019/10/11 14:48:11.854346 Launch detected - using supplied peer list: [10.128.0.107 10.128.0.105]
INFO: 2019/10/11 14:48:11.987378 Unable to fetch ConfigMap kube-system/weave-net to infer unique cluster ID
INFO: 2019/10/11 14:48:11.987481 Checking for pre-existing addresses on weave bridge
INFO: 2019/10/11 14:48:12.014364 [allocator ba:2e:11:69:a5:e2] No valid persisted data
INFO: 2019/10/11 14:48:12.019424 [allocator ba:2e:11:69:a5:e2] Initialising via deferred consensus
INFO: 2019/10/11 14:48:12.019637 Sniffing traffic on datapath (via ODP)
INFO: 2019/10/11 14:48:12.020265 ->[10.128.0.105:6783] attempting connection
INFO: 2019/10/11 14:48:12.020486 ->[10.128.0.107:6783] attempting connection
INFO: 2019/10/11 14:48:12.021385 ->[10.128.0.107:6783] error during connection attempt: dial tcp4 :0->10.128.0.107:6783: connect: connection refused
INFO: 2019/10/11 14:48:12.021873 ->[10.128.0.105:6783] error during connection attempt: dial tcp4 :0->10.128.0.105:6783: connect: connection refused
INFO: 2019/10/11 14:48:12.023099 Listening for HTTP control messages on 127.0.0.1:6784
INFO: 2019/10/11 14:48:12.023393 Listening for metrics requests on 0.0.0.0:6782
INFO: 2019/10/11 14:48:12.135510 ->[10.128.0.105:42053] connection accepted
INFO: 2019/10/11 14:48:12.140825 ->[10.128.0.105:42053|52:9b:77:28:8d:4f(test-12587-0-1)]: connection ready; using protocol version 2
INFO: 2019/10/11 14:48:12.141110 overlay_switch ->[52:9b:77:28:8d:4f(test-12587-0-1)] using fastdp
INFO: 2019/10/11 14:48:12.141177 ->[10.128.0.105:42053|52:9b:77:28:8d:4f(test-12587-0-1)]: connection added (new peer)
INFO: 2019/10/11 14:48:12.151662 ->[10.128.0.105:42053|52:9b:77:28:8d:4f(test-12587-0-1)]: connection fully established
INFO: 2019/10/11 14:48:12.175300 sleeve ->[10.128.0.105:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: Effective MTU verified at 1398
INFO: 2019/10/11 14:48:12.193481 ->[10.128.0.107:54651] connection accepted
INFO: 2019/10/11 14:48:12.200750 ->[10.128.0.107:54651|d6:c0:35:6b:e7:4b(test-12587-0-0)]: connection ready; using protocol version 2
INFO: 2019/10/11 14:48:12.201065 overlay_switch ->[d6:c0:35:6b:e7:4b(test-12587-0-0)] using fastdp
INFO: 2019/10/11 14:48:12.201125 ->[10.128.0.107:54651|d6:c0:35:6b:e7:4b(test-12587-0-0)]: connection added (new peer)
INFO: 2019/10/11 14:48:12.204539 ->[10.128.0.107:54651|d6:c0:35:6b:e7:4b(test-12587-0-0)]: connection fully established
INFO: 2019/10/11 14:48:12.240779 sleeve ->[10.128.0.107:6783|d6:c0:35:6b:e7:4b(test-12587-0-0)]: Effective MTU verified at 1398
INFO: 2019/10/11 14:48:12.386919 Discovered remote MAC 52:9b:77:28:8d:4f at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:12.561471 Discovered remote MAC 52:20:0f:40:c4:8a at d6:c0:35:6b:e7:4b(test-12587-0-0)
INFO: 2019/10/11 14:48:12.625454 Discovered remote MAC 86:38:7c:a6:0b:4e at d6:c0:35:6b:e7:4b(test-12587-0-0)
INFO: 2019/10/11 14:48:12.642866 Discovered remote MAC 12:c2:af:c9:69:7c at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:12.708530 Discovered remote MAC c6:06:f1:95:ea:59 at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:12.836941 [kube-peers] Added myself to peer list &{[{ba:2e:11:69:a5:e2 test-12587-0-2}]}
DEBU: 2019/10/11 14:48:12.845751 [kube-peers] Nodes that have disappeared: map[ba:2e:11:69:a5:e2:{ba:2e:11:69:a5:e2 test-12587-0-2}]
WARN: 2019/10/11 14:48:12.845886 [kube-peers] not removing myself {ba:2e:11:69:a5:e2 test-12587-0-2}
DEBU: 2019/10/11 14:48:12.851978 [kube-peers] Nodes that have disappeared: map[ba:2e:11:69:a5:e2:{ba:2e:11:69:a5:e2 test-12587-0-2}]
WARN: 2019/10/11 14:48:12.852094 [kube-peers] not removing myself {ba:2e:11:69:a5:e2 test-12587-0-2}
DEBU: 2019/10/11 14:48:12.857592 [kube-peers] Nodes that have disappeared: map[ba:2e:11:69:a5:e2:{ba:2e:11:69:a5:e2 test-12587-0-2}]
WARN: 2019/10/11 14:48:12.857707 [kube-peers] not removing myself {ba:2e:11:69:a5:e2 test-12587-0-2}
10.32.0.1
INFO: 2019/10/11 14:48:13.010188 Discovered remote MAC d6:c0:35:6b:e7:4b at d6:c0:35:6b:e7:4b(test-12587-0-0)
DEBU: 2019/10/11 14:48:13.112876 registering for updates for node delete events
10.128.0.107
10.128.0.105
INFO: 2019/10/11 14:48:17.214010 Discovered remote MAC 8e:83:29:e2:60:0e at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:17.312161 Discovered remote MAC 8a:a3:fd:36:eb:b4 at 52:9b:77:28:8d:4f(test-12587-0-1)
DEBU: 2019/10/11 14:48:19.573833 [kube-peers] Nodes that have disappeared: map[ba:2e:11:69:a5:e2:{ba:2e:11:69:a5:e2 test-12587-0-2} 52:9b:77:28:8d:4f:{52:9b:77:28:8d:4f test-12587-0-1}]
WARN: 2019/10/11 14:48:19.574563 [kube-peers] not removing myself {ba:2e:11:69:a5:e2 test-12587-0-2}
DEBU: 2019/10/11 14:48:19.574651 [kube-peers] Preparing to remove disappeared peer 52:9b:77:28:8d:4f
DEBU: 2019/10/11 14:48:19.578360 [kube-peers] Existing annotation d6:c0:35:6b:e7:4b
INFO: 2019/10/11 14:48:19.584272 ->[10.128.0.105:42053|52:9b:77:28:8d:4f(test-12587-0-1)]: connection shutting down due to error: read tcp4 10.128.0.106:6783->10.128.0.105:42053: read: connection reset by peer
INFO: 2019/10/11 14:48:19.584402 ->[10.128.0.105:42053|52:9b:77:28:8d:4f(test-12587-0-1)]: connection deleted
INFO: 2019/10/11 14:48:19.585810 Removed unreachable peer 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:19.586118 ->[10.128.0.105:6783] attempting connection
INFO: 2019/10/11 14:48:19.587108 ->[10.128.0.105:6783] error during connection attempt: dial tcp4 :0->10.128.0.105:6783: connect: connection refused
DEBU: 2019/10/11 14:48:19.595230 [kube-peers] Nodes that have disappeared: map[ba:2e:11:69:a5:e2:{ba:2e:11:69:a5:e2 test-12587-0-2} 52:9b:77:28:8d:4f:{52:9b:77:28:8d:4f test-12587-0-1}]
WARN: 2019/10/11 14:48:19.595349 [kube-peers] not removing myself {ba:2e:11:69:a5:e2 test-12587-0-2}
DEBU: 2019/10/11 14:48:19.595440 [kube-peers] Preparing to remove disappeared peer 52:9b:77:28:8d:4f
DEBU: 2019/10/11 14:48:19.600546 [kube-peers] Noting I plan to remove  52:9b:77:28:8d:4f
DEBU: 2019/10/11 14:48:19.604745 weave DELETE to http://127.0.0.1:6784/peer/52:9b:77:28:8d:4f with map[]
INFO: 2019/10/11 14:48:19.605669 [kube-peers] rmpeer of 52:9b:77:28:8d:4f: 0 IPs taken over from 52:9b:77:28:8d:4f

DEBU: 2019/10/11 14:48:20.150639 [kube-peers] Nodes that have disappeared: map[ba:2e:11:69:a5:e2:{ba:2e:11:69:a5:e2 test-12587-0-2}]
WARN: 2019/10/11 14:48:20.150664 [kube-peers] not removing myself {ba:2e:11:69:a5:e2 test-12587-0-2}
DEBU: 2019/10/11 14:48:20.551138 [kube-peers] Nodes that have disappeared: map[ba:2e:11:69:a5:e2:{ba:2e:11:69:a5:e2 test-12587-0-2}]
WARN: 2019/10/11 14:48:20.551159 [kube-peers] not removing myself {ba:2e:11:69:a5:e2 test-12587-0-2}
DEBU: 2019/10/11 14:48:20.750484 weave POST to http://127.0.0.1:6784/connect with map[peer:[10.128.0.107] replace:[true]]
INFO: 2019/10/11 14:48:21.237178 ->[10.128.0.105:55775] connection accepted
INFO: 2019/10/11 14:48:21.237926 ->[10.128.0.105:55775|52:9b:77:28:8d:4f(test-12587-0-1)]: connection ready; using protocol version 2
INFO: 2019/10/11 14:48:21.238025 overlay_switch ->[52:9b:77:28:8d:4f(test-12587-0-1)] using fastdp
INFO: 2019/10/11 14:48:21.238063 ->[10.128.0.105:55775|52:9b:77:28:8d:4f(test-12587-0-1)]: connection added (new peer)
INFO: 2019/10/11 14:48:21.270916 overlay_switch ->[52:9b:77:28:8d:4f(test-12587-0-1)] using sleeve
INFO: 2019/10/11 14:48:21.271204 ->[10.128.0.105:55775|52:9b:77:28:8d:4f(test-12587-0-1)]: connection fully established
INFO: 2019/10/11 14:48:21.271838 sleeve ->[10.128.0.105:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: Effective MTU verified at 1398
INFO: 2019/10/11 14:48:21.338816 Discovered remote MAC c6:06:f1:95:ea:59 at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:21.385987 Discovered remote MAC 52:9b:77:28:8d:4f at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:21.739486 overlay_switch ->[52:9b:77:28:8d:4f(test-12587-0-1)] using fastdp
INFO: 2019/10/11 14:48:21.954937 Discovered remote MAC 12:c2:af:c9:69:7c at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:23.875162 ->[10.128.0.105:55775|52:9b:77:28:8d:4f(test-12587-0-1)]: connection shutting down due to error: read tcp4 10.128.0.106:6783->10.128.0.105:55775: read: connection reset by peer
INFO: 2019/10/11 14:48:23.875228 ->[10.128.0.105:55775|52:9b:77:28:8d:4f(test-12587-0-1)]: connection deleted
INFO: 2019/10/11 14:48:23.875526 ->[10.128.0.105:6783] attempting connection
INFO: 2019/10/11 14:48:23.875802 ->[10.128.0.105:6783] error during connection attempt: dial tcp4 :0->10.128.0.105:6783: connect: connection refused
INFO: 2019/10/11 14:48:23.876399 Removed unreachable peer 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:24.494536 Discovered remote MAC 3a:5b:7e:32:ee:de at d6:c0:35:6b:e7:4b(test-12587-0-0)

@bboreham
Copy link
Contributor Author

In the kubelet logs on host2 we can see that it evicts all pods, then starts (some of) them again.
I think shutting down Kubernetes on the node before deleting it would be better.

@bboreham bboreham mentioned this issue Oct 11, 2019
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
chore
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Randomise wait to reclaim IPs on node deletion, and fix flaky test 870 weaveworks/weave
1 participant
@bboreham