Kops fails to join new master when 1 master lost etcd events data volume

[ivnilv]

1. What kops version are you running? The command kops version, will display
this information.

# kops version
Version 1.15.2 (git-ad595825a)
root@bastion:~#

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

# kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.9", GitCommit:"2e808b7cb054ee242b68e62455323aa783991f03", GitTreeState:"clean", BuildDate:"2020-01-18T23:33:14Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.11", GitCommit:"d94a81c724ea8e1ccc9002d89b7fe81d58f89ede", GitTreeState:"clean", BuildDate:"2020-03-12T21:00:06Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

Deleted the etcd-events data volume on 1 of 3 masters.

5. What happened after the commands executed?

New masters in the instanceGroup that had etcd-events data removed no longer joins the cluster.

6. What did you expect to happen?

ETCd-manager to remove the unhealthy member and add the new member without any manual interactions required.

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2020-05-26T12:46:27Z"
  name: kubernetes-disaster-recovery.k8s.organization.com
spec:
  api:
    loadBalancer:
      type: Internal
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: aws
  configBase: s3://AWS-S3-BUCKET
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - instanceGroup: master-eu-central-1a
      name: a
    - instanceGroup: master-eu-central-1b
      name: b
    - instanceGroup: master-eu-central-1c
      name: c
    memoryRequest: 100Mi
    name: main
    version: 3.2.24
  - cpuRequest: 100m
    etcdMembers:
    - instanceGroup: master-eu-central-1a
      name: a
    - instanceGroup: master-eu-central-1b
      name: b
    - instanceGroup: master-eu-central-1c
      name: c
    memoryRequest: 100Mi
    name: events
    version: 3.2.24
  iam:
    allowContainerRegistry: true
    legacy: false
  kubelet:
    anonymousAuth: false
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.15.11
  masterPublicName: api.kubernetes-disaster-recovery.k8s.organization.com
  networkCIDR: 172.23.0.0/16
  networking:
    calico:
      majorVersion: v3
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 0.0.0.0/0
  subnets:
  - cidr: 172.23.32.0/19
    name: eu-central-1a
    type: Private
    zone: eu-central-1a
  - cidr: 172.23.64.0/19
    name: eu-central-1b
    type: Private
    zone: eu-central-1b
  - cidr: 172.23.96.0/19
    name: eu-central-1c
    type: Private
    zone: eu-central-1c
  - cidr: 172.23.0.0/22
    name: utility-eu-central-1a
    type: Utility
    zone: eu-central-1a
  - cidr: 172.23.4.0/22
    name: utility-eu-central-1b
    type: Utility
    zone: eu-central-1b
  - cidr: 172.23.8.0/22
    name: utility-eu-central-1c
    type: Utility
    zone: eu-central-1c
  topology:
    dns:
      type: Private
    masters: private
    nodes: private

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?

In order to fix the issue on our end we had to execute into 1 of the other 2 working etcd-events members, and run the etcdctl member remove command in order to remove the broken node from the list of members.

Next thing, we increased the number of instances for the instanceGroup from 0 to 1 to create a new node, and the rest was taken care of automatically by kops and etcd-manager.. new node joined cluster properly.

The question is, shouldn't this be done automatically by kops or etcd-manager in some step ?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[hakman]

Related issue on the etcd-manager side: kopeio/etcd-manager#339.
/remove-lifecycle stale

[cjbehm]

@ivnilv this was a failure scenario we tested out as well, with the same resolution of manual intervention.
It would be a very welcome improvement for complete single node failure to be automatically recoverable.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[adeelahmadch]

I ran into this issue and after trying different approaches I can say the safest way in case you lost etcd main or etcd events data is to restore it via etcd-manager-ctl. In my case I restore the most recent backup to fix the issue. You can run these command from anywhere as long as you have access to s3 bucket used by kops.

etcd-manager-ctl --backup-store=s3://example-kubernetes-test-194318766199/k8s.test.example.com/backups/etcd/main list-backups
etcd-manager-ctl --backup-store=s3://example-kubernetes-test-194318766199/k8s.test.example.com/backups/etcd/events list-backups

etcd-manager-ctl --backup-store=s3://example-kubernetes-test-194318766199/k8s.test.example.com/backups/etcd/main restore-backup "2021-03-16T23:13:18Z-000008"
etcd-manager-ctl --backup-store=s3://example-kubernetes-test-194318766199/k8s.test.example.com/backups/etcd/events restore-backup "2021-03-16T23:13:18Z-000008"

You can also read more on the kops docs.