Better etcd snapshot/restore in the kubeadm upgrade logic

[luxas]

Right now we exec cp -r, but we should take a snapshot before upgrading, and then actually be able to roll it back seamlessly if something fails. Now we do this on a cp manner using the filesystem, but we should do this via the etcd Go client instead to make it more robust. Might require some re-vendoring/structuring of the etcd Go client

cc @xiang90 @hongchaodeng @jamiehannaford @sbezverk @timothysc @xiangpengzhao @ericchiang

[timothysc]

imo this is a documented process external to kubeadm.

[jamiehannaford]

I'm more of the opinion that reliable rollbacks are an essential component of upgrading etcd, so the user might benefit from having this handled on their behalf. Not fixed to this idea though. Maybe we can discuss in next implementation meeting?

[DGreenstein]

@timothysc - so what was the verdict on this? Is the intent to vendor in the etcd client and use that for making the etcd dump? How does this align with the demo from last week, where we saw the external etcd manager? Should this just be a documented manual effort until etcdmgr is ready? Do we want to proceed with enhancing the functionality of the kubeadm upgrade command?

FWIW: I agree with Jamie - if kubeadm upgrade is going to touch etcd at all, a first-class way of handling the etcd lifecycle in the context of an upgrade should be provided. High-level logic I think may be:

if self-hosted etcd

• backup data
• update deployment object(s)
• restore data

if external

• validate version

test some sane basic functionality like is cluster reporting healthy?

[timothysc]

There already exists logic right now that will snap and restore, but it's pretty brute force atm.

[luxas]

We now at least have the possibility (when we upgrade godeps in v1.12) to do the snapshotting properly etcd-io/etcd#9118

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[timothysc]

I'm going to close this and wait to see the community asks or reopens. This seems to be a non-issue.