-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
A KEP proposing supporting building a Kubelet without docker, via the use of build tags.
- Loading branch information
1 parent
8f238f5
commit 3586aa0
Showing
1 changed file
with
291 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,291 @@ | ||
| --- | ||
| title: Building a Dockerless Kubelet | ||
| authors: | ||
| - "@mattjmcnaughton" | ||
| owning-sig: sig-node | ||
| participating-sigs: | ||
| - sig-testing | ||
| - sig-release | ||
| reviewers: | ||
| - "@dims" | ||
| - "@BenTheElder" | ||
| - TBD | ||
| approvers: | ||
| - TBD | ||
| editor: TBD | ||
| creation-date: 2020-02-05 | ||
| last-updated: 2020-02-09 | ||
| status: provisional | ||
| see-also: | ||
| - "/keps/sig-cloud-provider/20190729-building-without-in-tree-providers.md" | ||
| --- | ||
|
|
||
| # Building a Dockerless Kubelet | ||
|
|
||
| ## Table of Contents | ||
|
|
||
| <!-- toc --> | ||
| - [Release Signoff Checklist](#release-signoff-checklist) | ||
| - [Summary](#summary) | ||
| - [Motivation](#motivation) | ||
| - [History](#history) | ||
| - [Returning to Motivation](#returning-to-motivation) | ||
| - [Goals](#goals) | ||
| - [Non-Goals](#non-goals) | ||
| - [Proposal](#proposal) | ||
| - [User Stories](#user-stories) | ||
| - [Story 1](#story-1) | ||
| - [Story 2](#story-2) | ||
| - [Implementation Details/Notes/Constraints](#implementation-detailsnotesconstraints) | ||
| - [Risks and Mitigations](#risks-and-mitigations) | ||
| - [Design Details](#design-details) | ||
| - [Test Plan](#test-plan) | ||
| - [Graduation Criteria](#graduation-criteria) | ||
| - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) | ||
| - [Version Skew Strategy](#version-skew-strategy) | ||
| - [Implementation History](#implementation-history) | ||
| - [Drawbacks](#drawbacks) | ||
| - [Alternatives](#alternatives) | ||
| <!-- /toc --> | ||
|
|
||
| ## Release Signoff Checklist | ||
|
|
||
| - [ ] kubernetes/enhancements issue in release milestone, which links to KEP (this should be a link to the KEP location in kubernetes/enhancements, not the initial KEP PR) | ||
| - [ ] KEP approvers have set the KEP status to `implementable` | ||
| - [ ] Design details are appropriately documented | ||
| - [ ] Test plan is in place, giving consideration to SIG Architecture and SIG Testing input | ||
| - [ ] Graduation criteria is in place | ||
| - [ ] "Implementation History" section is up-to-date for milestone | ||
| - [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] | ||
| - [ ] Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes | ||
|
|
||
| ## Summary | ||
|
|
||
| This proposal outlines a plan to enable building a dockerless Kubelet. We | ||
| define a dockerless Kubelet as a Kubelet with no "Docker-specific" code and | ||
| no dependency on the `docker/docker` Golang package. We define "Docker-specific" | ||
| code as code which only serves a purpose when Docker is the container runtime. | ||
| "Docker-specific" code is never executed when the Kubelet uses a remote | ||
| container runtime (i.e. containerd or CRI-O). | ||
|
|
||
| Supporting building a dockerless Kubelet is a precursor to moving all | ||
| "Docker-specific" Kubelet code out-of-tree, in the name of treating Docker like | ||
| any other container runtime. | ||
|
|
||
| At a high level, this undertaking is similar to the efforts of | ||
| sig-cloud-provider to support [Building Kubernetes Without In-Tree Cloud Providers](/keps/sig-cloud-provider/20190729-building-without-in-tree-providers.md). | ||
| A big thanks to them for their leadership; much of this KEP is based off their great work :) | ||
|
|
||
| ## Motivation | ||
|
|
||
| For this KEP to be worthwhile, we must believe the following two statements. | ||
|
|
||
| First, we must see a benefit to a dockerless Kubelet. | ||
| Second, we must believe supporting the ability to compile a dockerless Kubelet is | ||
| a useful first step towards a truly dockerless Kubelet. | ||
|
|
||
| A quick review of recent Kubernetes history provides context when considering | ||
| whether we agree with the statements in question. | ||
|
|
||
| ### History | ||
|
|
||
| With 1.5, Kubernetes introduced the [Container Runtime Interface](https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/) | ||
| (CRI). The CRI defines a standard interface for the Kubelet to communicate with container | ||
| runtimes. At the time of the CRI's release, Kubernetes supported only the Docker | ||
| and rkt container runtimes. Kubernetes introduced the CRI to avoid needing | ||
| provider specific code for each new container runtime. | ||
|
|
||
| Since the CRI's release, the Kubelet only interacts with container runtimes via | ||
| the CRI. For increasingly popular CRI implementations like containerd or CRI-O, | ||
| the singular focus on the CRI poses no obstacles, as these container runtimes | ||
| supported the CRI from the start. However, the Kubelet still needed a solution for | ||
| Docker and rkt, in-tree runtimes which did not support the CRI. In-tree support | ||
| for Rkt was deprecated, leaving only Docker as an issue. | ||
|
|
||
| Ultimately, the Kubelet introduced the `dockershim` to address Docker's lack of | ||
| CRI support. When a cluster operator chooses to use Docker as the container | ||
| runtime, the Kubelet starts running the `dockershim` in a separate go | ||
| routine within the main `kubelet` process; it is not currently possible to run | ||
| the `dockershim` as a standalone binary/process. The `dockershim` supports the | ||
| CRI, so the Kubelet communicates with `dockershim` as if it was any other remote | ||
| container runtime implementing the CRI. The `dockershim` makes the appropriate | ||
| calls to the Docker daemon via a heavy dependence on the `docker/docker` client library. | ||
|
|
||
| The `docker/docker` client library is a particularly painful dependency because | ||
| its pulls in code from many different open source libraries. For those managing | ||
| k8s dependencies, it can be extremely difficult to keep up with the changes to | ||
| all these dependencies. Additionally, all the open source libraries required by | ||
| `docker/docker` bloat the Kubelet binary. | ||
|
|
||
| ## Returning to Motivation | ||
|
|
||
| We can anticipate the following benefits from the Kubelet having no in-tree | ||
| "Docker-specific" code and no dependency on the `docker/docker` Golang package. | ||
|
|
||
| First, a dockerless Kubelet would truly treat all container runtimes the same. | ||
| Second, the Kubelet's scope of responsibility would decrease: it would no longer | ||
| be responsible for making Docker conform to the CRI. Finally, the painful | ||
| `docker/docker` dependency would be gone. | ||
|
|
||
| While the aforementioned benefits are desirable, they do not outweigh the cons | ||
| of completely dropping support for Docker as a container runtime, as | ||
| Docker remains popular. In order for Kubernetes | ||
| to both support Docker as a container runtime, and for the Kubelet do have no | ||
| in-tree "Docker-specific" code and no dependency on `docker/docker`, one of two | ||
| following paths must be followed: either Docker must begin implementing the CRI natively | ||
| or the `dockershim` must be moved out-of-tree into a standalone component. | ||
|
|
||
| Clearly, both of these paths forward require significant work. Either effort would | ||
| require finding an owner, making non-trivial code changes, and updating patterns | ||
| of cluster management/operation. | ||
|
|
||
| Faced with a hefty chunk of work, we naturally try to break it up into smaller | ||
| components. This desire leads us to our second question: is supporting | ||
| compiling a dockerless Kubelet an appropriate first step? | ||
|
|
||
| We argue yes. First, the work to support compiling a dockerless Kubelet will | ||
| be useful regardless of which path forward we chose. First, to compile a dockerless | ||
| Kubelet we must consolidate all "Docker-specific" Kubelet code into specific | ||
| locations, which are easier to move out-of-tree or delete entirely when the time comes. Furthermore, | ||
| after this initial consolidation, we can create tooling to impose limitations on | ||
| where "Docker-specific" code can/cannot live. Second, allowing developers to | ||
| compile a dockerless Kubelet assists in testing either proposed solution. | ||
| Finally, allowing compiling a dockerless Kubelet allows projects/cluster | ||
| operators which already do not depend on Docker to obtain the dockerless | ||
| Kubelet's benefits (i.e. smaller binaries) without waiting for the | ||
| completion of the longer term projects to make Docker support the CRI or | ||
| move `dockershim` out-of-tree. | ||
|
|
||
| ### Goals | ||
|
|
||
| Our goals follow from our motivation: | ||
|
|
||
| 1. Support building Kubelet, from the `master` branch, without any "Docker-specific" code and without any | ||
| dependency on `docker/docker`. As mentioned | ||
| previously, we imagine the resulting binaries to be used to test the | ||
| different paths for deleting/moving out-of-tree all "Docker-specific" code. | ||
| 2. Draw clear delineations, with CI support, for what code in Kubelet can and | ||
| cannot be "Docker-specific" and depend on `docker/docker`. | ||
|
|
||
| ### Non-Goals | ||
|
|
||
| Our non-goals also follow from our motivation: | ||
|
|
||
| 1. Either making Docker support the CRI or moving `dockershim` out-of-tree. | ||
| 2. Removing uses of the `docker/docker` Golang library outside of the Kubelet. | ||
| 3. Changing the official Kubernetes release builds. | ||
|
|
||
| ## Proposal | ||
|
|
||
| We will undertake the following steps to obtain our goals. First, we will ensure | ||
| that all "Docker-specific" code in the Kubelet lives in `dockershim`. Then, we | ||
| will add a [build constraint](https://golang.org/pkg/go/build/#hdr-Build_Constraints) | ||
| to the Kubelet for a pseudo "build tag" specifying not to include | ||
| any in-tree "Docker-specific" code. If builds do not specify this build tag, the Go | ||
| compiler will compile the Kubelet as normal. If we do, the Go compiler will | ||
| compile the Kubelet without the "Docker-specific" code, and as a result, without | ||
| the dependency on `docker/docker`. In other words, it will simulate the aforementioned | ||
| code/dependency's removal. | ||
|
|
||
| A prototype is available in [kubernetes/kubernetes#87746](https://github.com/kubernetes/kubernetes/pull/87746). | ||
|
|
||
| To ensure that this dockerless Kubelet continues to function we will add CI building in this mode, | ||
| and CI running end to end tests against it (to be elaborated on in the test plan). Additionally, to | ||
| ensure the Kubelet doesn't introduce new dependencies on the `docker/docker` | ||
| Golang library, we will add automated tooling enforcing that only the | ||
| `dockershim` can depend on `docker/docker`. | ||
|
|
||
| One quick additional note - currently `cadvisor` also depends on the | ||
| `docker/docker` client library. Since `kubelet` depends on `cadvisor`, the | ||
| `kubelet` can not truly be rid of the `docker/docker` client library until `cadvisor` no | ||
| longer depends on `docker/docker`. Work to remove the `docker/docker` dependency | ||
| from `cadvisor` is being dealt with in separate workstreams. | ||
|
|
||
| This proposal follows the previous patterns for similar work, namely the efforts | ||
| of sig-cloud-provider to support [Building Without In-Tree Cloud Providers](/keps/sig-cloud-provider/20190729-building-without-in-tree-providers.md). | ||
|
|
||
| ### User Stories | ||
|
|
||
| #### Story 1 | ||
|
|
||
| As a developer working to make Docker function like any other remote container | ||
| runtime, I am attempting to validate that my proposed solution | ||
| functions correctly. Using this dockerless build ensures the Kubelet contains no | ||
| "Docker-specific" code, and any success/failures can be attributed to my | ||
| implementation. | ||
|
|
||
| #### Story 2 | ||
|
|
||
| As a Kubernetes developer/user, for example a maintainer of [kind](https://github.com/kubernetes-sigs/kind), | ||
| I want to use Kubelet binaries which only contain the code I need. Since kind does not use | ||
| Docker as its container runtime, compiling a dockerless Kubelet gives me a | ||
| smaller binary. | ||
|
|
||
| ### Implementation Details/Notes/Constraints | ||
|
|
||
| A couple high level notes (to be extended over time): | ||
|
|
||
| - We implement this functionality using a synthetic `dockerless` tag in go build | ||
| constraints on relevant sources. If `GOFLAGS=-tags=dockerless` during build, | ||
| then "Docker-specific" code will be excluded from the Kubelet build. With no | ||
| "Docker-specific" code, we should also be excluding the dependency on | ||
| `docker/docker`. | ||
|
|
||
| ### Risks and Mitigations | ||
|
|
||
| This feature is only developer facing, which removes a large class of risks. | ||
|
|
||
| The largest remaining risk is that the build tags fall out of date, or are | ||
| burdensome to continue updating, which leads to the dockerless Kubelet build | ||
| breaking and/or being costly to maintain. This risk grows the longer the | ||
| dockerless Kubelet exists (i.e. the longer it takes to move `dockershim` out of | ||
| tree/have Docker support the CRI). Fortunately, these risks can be mitigated via | ||
| the CI tooling discussed earlier. | ||
|
|
||
| ## Design Details | ||
|
|
||
| ### Test Plan | ||
|
|
||
| **Note:** *Section not required until targeted at a release.* | ||
|
|
||
| TBD | ||
|
|
||
| ### Graduation Criteria | ||
|
|
||
| **Note:** *Section not required until targeted at a release.* | ||
|
|
||
| TBD | ||
|
|
||
| ### Upgrade / Downgrade Strategy | ||
|
|
||
| N/A | ||
|
|
||
| ### Version Skew Strategy | ||
|
|
||
| N/A | ||
|
|
||
| ## Implementation History | ||
|
|
||
| - original prototype [kubernetes/kubernetes#87746](https://github.com/kubernetes/kubernetes/pull/87746) | ||
| - original KEP PR [TBD] | ||
|
|
||
| ## Drawbacks | ||
|
|
||
| One drawback is the opportunity cost of pursuing this workstream as opposed to | ||
| other possible workstreams. | ||
|
|
||
| Another drawback is the slight additional cost of the CI tooling we propose | ||
| adding. | ||
|
|
||
| ## Alternatives | ||
|
|
||
| One alternative would be to do nothing. | ||
|
|
||
| Another alternative could be waiting to address "Docker-specific" code in the | ||
| Kubelet until we have more momentum around one of the longer-term solutions | ||
| discussed above. If we waited, we could delete "Docker-specific" code entirely, | ||
| instead of just compiling without it. | ||
|
|
||
| Finally, we could attempt to have a long-running branch in which all | ||
| "Docker-specific" code has been deleted, instead of attempting to support | ||
| compiling a dockerless Kubelet from master. |