Init KEP to migrate in-tree dockershim to out-of-tree

[resouer]

A initial KEP to discuss about criteria and plan to migrate in-tree dockershim to out-of-tree.

Ref: kubernetes/kubernetes#73933

Please note this KEP is not target on 1.14 or any specific release recently, it's designed to be a join effort spanning on many future releases and requires carefully revise and thinking.

/assign @yujuhong @dchen1107

/cc @dims @kubernetes/sig-node-pr-reviews

[dims]

build-in -> built-in

[dims]

Existing instead of Existed

[dims]

cri-dockerd will still be vendoring kubernetes/kubernetes for a while during the transition. vendoring kubernetes/kubernetes is not easy

[resouer]

ACK, it's a good point

[dims]

Can we add another goal of writing up a criteria for adopting one of the existing CRI runtimes (for example containerd) that can eventually replace cri-dockerd?

[zhangxiaoyu-zidif]

considering that there are too many docker users, it be better not to cease cri-dockerd here.

[dchen1107]

SIG Node doesn't want to make a call which CRI implementation the vendor / the user should use. Instead, we design the API to define how the runtime engaging with Kubelet and nodes, a test suite to verify / qualify the functionalities, a portable toolset (cri-tool) for debuggability and introspect.

[dims]

we will continue updating the docker/docker dependency. right?

[zhangxiaoyu-zidif]

Perhaps continue updating the docker/docker dependency is a must.

[yujuhong]

Yes, we still need to update Docker when needed.

[dims]

@yujuhong @derekwaynecarr @PatrickLang @dchen1107 does this seem ready? (can we merge and iterate?)

[dims]

Another benefit as noted by @liggitt in the dependency doc[1], docker has just 7 month support[2]. So the quicker we get off dockerd shim the better we will be.

[1] https://docs.google.com/document/d/1WA8N7C48nkJmme9a96DU0o9jBpeycPhht8WF-Eam9QQ/edit?ts=5c9a308b#heading=h.46nlk76xzfpg
[2] https://docs.docker.com/install/#support

[dims]

@resouer let's please add a "new binary named cri-dockerd" with code being in-tree (line number 113)

[dims]

We will also need to ensure that this binary gets into the release artifacts

[zhangxiaoyu-zidif]

@dims do you mean that cri-dockerd could be compiled to an independent binary like kubelet here?

[dims]

@zhangxiaoyu-zidif yep.

[liggitt]

Long-term, do we want to maintain a dockershim CRI implementation? If not, would it make more sense to focus efforts on getting CRI to v1, then immediately start the deprecation period for dockershim (6 months, according to https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecating-a-flag-or-cli) in favor of existing out of tree CRI impls?

[liggitt]

does this not already exist in the form of containerd?

[resouer]

I assume they are different. During recent sig-node meeting, folks from Docker Inc are willing to maintain a CRI-docker.

[tallclair]

All recent versions of docker are built on top of containerd, so I'm unclear on what functionality dockershim has that cri-containerd does not.

@Random-Liu do you know if there is any reason to use dockershim over cri-containerd with recent versions of docker?

[Random-Liu]

I think in the long run, we do expect people to move to CRI conformant container runtimes like containerd and cri-o to be able to leverage new Kubernetes features.

However, people may still need more time to make that move, one of the biggest reason is that with dockershim, you can actually see all your Kubernetes containers in Docker, and people's tools may have been relying on that. For example, people may have built security tools, monitoring tools, logging tools around that.

With containerd or cri-o, you won't see Kubernetes containers in Docker any more. It takes time for people to migrate their tooling, that's why we need to keep dockershim for a while.

[Random-Liu]

Another reason is that today's Kubernetes windows support is still based on dockershim. cri-containerd Windows is WIP, but not finished yet.

[zhangxiaoyu-zidif]

so we cloud remain both mechanisms currently and depreciated the old way gradually.

[yujuhong]

Yes, there are still features that are supported by docker/dockershim, but not with containerd.

[liggitt]

I strongly support progressing CRI to v1, for the sake of all the runtime implementations. I'm less clear on the benefit of extracting dockershim to a standalone CRI. Did you consider deprecating dockershim in favor of existing CRI implementations instead?

[resouer]

I will try to confirm if sig-node or Docker Inc has a plan to support a CRI-docker for long time (which is what I heard during the sig meeting). If not, deprecating dockershim directly will be the goal.

[dchen1107]

We talked about this at SIG Node, including the folks from Docker Inc. Docker folks plan to maintain cri-dockershim.

[liggitt]

this would be extremely helpful for existing CRI impls to test/verify compatibility

[yujuhong]

Cluster/node e2e doesn't care what CRI implementation you use under the hood. I think maybe I've missed the point of this statement.

[resouer]

There're two points in my mind:

1. Ensure cluster/node e2e are 100% CRI focused. For instance: DockerValidator should use the CRI, not the Docker API kubernetes#55414
2. Ensure test-infra install CRI-docker or cri-containerd binary in e2e machines. Currently, they install Docker binary only.

[dims]

@resouer WDYT of @liggitt 's alternative? I am inclined to support that as it is less work overall

[resouer]

@dims I agree, will update the KEP to reflect this proposal.

[tedyu]

What percentage of degradation is acceptable (10%) ?

[neolit123]

10% might not be acceptable by the wider audience.
but my guess is that it's going to end up being less than that 🤔

[tedyu]

Currently starting dockershim is governed by a flag:

			if kubeletServer.KubeletFlags.ExperimentalDockershim {
				if err := RunDockershim(&kubeletServer.KubeletFlags, kubeletConfig, stopCh); err != nil {

Would there be another flag or enum which is used to choose the effective out-of-tree container runtime ?
My assumption is that the kubeletServer.KubeletFlags, kubeletConfig and stopCh parameters provide sufficient information for starting the chosen out-of-tree container runtime.

[bg-chun]

/cc @bg-chun

[bg-chun]

How is it going?
For my work(container isolation of hugepages).
I updated CRI to have hugepages limit for containers then I am working for Container Runtimes.
Containerd and cri-o doesn't has any issue for my work But, Docker has tiny issue due to the dockershim dependency :( .
I jsut hope I will see this changes in 2020.

[liggitt]

cc @dashpole - what does kubelet/cadvisor do differently when configured to speak to a CRI instead of dockershim? does cadvisor still contact the container runtime directly or would this let us stop enabling runtime-specific integrations in cadvisor?

[dashpole]

what does kubelet/cadvisor do differently when configured to speak to a CRI instead of dockershim? does cadvisor still contact the container runtime directly or would this let us stop enabling runtime-specific integrations in cadvisor?

The only difference is that it disables metrics provided by the CRI. It still contacts the container runtime directly to get metadata about containers.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[warmchang]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[frittentheke]

/remove-lifecycle stale

[BenTheElder]

If @resouer is working on this PR it won't go stale, but just removing the stale label is not going to get this merged. If they're not working on it someone should probably start a forked PR.

[saschagrunert]

@resouer there are ongoing discussions about deprecating dockershim in Kubernetes. Are you around to drive this KEP forward or would you prefer if someone else takes over? 😇

[dims]

@saschagrunert after talking to @liggitt and @derekwaynecarr this plan is kinda unworkable. Please see prototype in https://github.com/dims/cri-dockerd

Problem is that this external CRI impl will need to vendor in k8s.io/kubernetes which is verboten! ( https://github.com/dims/cri-dockerd/blob/main/go.mod#L21 )

[saschagrunert]

@dims I'm fine with that as well as the planned deprecation. I guess we can still recycle some parts like the hostport manager.

[dims]

@saschagrunert for sure! if you see containerd/cri repo, it no longer depends on k/k :) we should do the same for CRI-O

[saschagrunert]

@saschagrunert for sure! if you see containerd/cri repo, it no longer depends on k/k :) we should do the same for CRI-O

Yes, thanks for supporting us on that front! ❤️

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[warmchang]

target release is v1.20 now. 😄

[SergeyKanzhelev]

I think this is outdated after #2221 and #1985

/close

[k8s-ci-robot]

@SergeyKanzhelev: Closed this PR.

In response to this:

I think this is outdated after #2221 and #1985

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.