MEP: Change from custom Buildroot ISO to standard Kubernetes OS

[afbjorklund]

Currently we are using a custom image, which is built using the Buildroot tool...

It is a replacement for the original Boot2Docker image, based on Tiny Core Linux.
(The board config also draws some inspiration from the original CoreOS images)

For the KIC base image, we have instead changed to using a standard ubuntu:20.04

We could do the same for the ISO image, base it on a standard Linux distribution.
(Preferrably one of the official ones, that are supported and tested by kubeadm)

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin

Ubuntu 16.04+
Debian 9+
CentOS 7
Red Hat Enterprise Linux (RHEL) 7
Fedora 25+
HypriotOS v1.0.1+
Flatcar Container Linux (tested with 2512.3.0)

Something like:

• Ubuntu 16.04
• Ubuntu 18.04
• Ubuntu 20.04
• CentOS 7
• CentOS 8
• Fedora 32

Comparison table:

Year	Debian	Ubuntu	CentOS	Fedora
2010	6	10.04	6	12
2012	7	12.04		15
2014	8	14.04	7	18
2016	9	16.04		24
2018	10	18.04	8	28
2020		20.04		32

It's easier to use Ubuntu and CentOS (than Debian and RHEL), due to licensing issues etc.

Suggestion to use the same OS version as the KIC base image, which is Ubuntu 20.04 LTS.

---

It only needs the kernel capability to run containers, the container runtime can be provisioned.

Using a package manager makes it easier to add/remove software, compared to rebuilding.

When using one of cloud drivers for docker-machine, it uses an Ubuntu 16.04 image.
You can see how it works, by checking out the minikube "generic" driver (with any VM).

https://docs.docker.com/machine/get-started-cloud/

#9545 (comment)

The package list could be the same as the docker image, to start with (minus kind-specific)
Currently the ISO image works by copying everything from the rootfs on the CD into a tmpfs.

---

The motivation for this is that it takes considerable engineering effort to keep the image updated.

It also introduces differences between the minikube environment, and a production environment.

The downsides would be less control over the image, and possibly also a larger image to download.
And most likely we still would have to do our own image adaption/build - or add cloud-init support.

But maybe it would be easier to let someone else handle the distribution, rather than making our own.
This is similar to letting kubeadm handle the installation, rather than do our own bespoke "localkube".

... Dude, Let's Go Bowling
--Walter Sobchak

[afbjorklund]

TODO: Write the actual MEP

[medyagh]

@afbjorklund I wonder how larger our ISO becomes if we move off buildroot ?

[afbjorklund]

I wonder how larger our ISO becomes if we move off buildroot ?

The guess is "double" (< 500M), but then again nobody actually cares...

The software that we add on top of the base OS is so much bigger than the underlying image, so that it almost "disappears".

• You can see that the ubuntu:focal image is < 100M, but the kicbase image is almost a gigabyte (941M)

• Changing from the cache to the preload, adds another +176M. Or even yet another +412M with containerd.

So it is similar to when we optimized the size of the minikube binary - only to pull in a gigabyte of k8s container images ?

Hopefully the total (compressed) download can be kept like 1G or so:

54M	/usr/bin/minikube
213M	iso/minikube-v1.16.0.iso
492M	preloaded-tarball/preloaded-images-k8s-v8-v1.20.0-docker-overlay2-amd64.tar.lz4
758M	total

Like I mentioned above, this MEP is similar to when we changed the bootstrapper from "localkube" to "kubeadm"...

The image does become bigger (in both cases), but you get a more standard installation and easier maintenance.

---

Anyway, the vagrant image (ubuntu/focal64) is 499M:

499M	ubuntu-focal-20.04-cloudimg.vmdk
72K	ubuntu-focal-20.04-cloudimg-configdrive.vmdk

https://cloud-images.ubuntu.com/

The docker image is well below that, without the kernel etc.

REPOSITORY   TAG       IMAGE ID       CREATED       SIZE
ubuntu       focal     f643c72bc252   3 weeks ago   72.9MB

https://hub.docker.com/_/ubuntu

The Buildroot OS base is around that size as well, i.e. 72M.
Main bloat was adding systemd, which also dragged in glibc.

I'm still interested in the smaller distributions, like docker-machine or podman-machine or localkube or k3s....

But I think the custom distribution is more of a side project, and stealing focus from the minikube development ?

[afbjorklund]

I thought this presentation was interesting: https://landley.net/aboriginal/presentation.html

Package Management and Accidental Distros

Linux From Scratch intentionally does not cover package management.
Package management two parts: Using a package manager (rpm, dpkg, ipkg/opkg, portage) to install/uninstall/track packages, and creating a repository of build scripts and package dependencies.
Creating a repository of package dependencies and build options is a huge undertaking and requires constant maintenance as new packages are released upstream.
Fedora, Debian, Gentoo, and others already have perfectly acceptable repositories you can leverage. Creating your own is a huge time sink.
Despite this, it's very easy to fall into the trap of creating your own Linux distro by mistake, at which point complexity explodes out of control.
Maintaining your own distro is enormously time consuming.
The more packages your build system builds, the more likely this is to happen.
Defending yourself from accidental distros.
Divide the hairball into orthogonal layers. Toolchain selection has nothing to do with package management
Needing to change one is no excuse for accepting responsibility to maintain the other.
Delegate everything you can to existing projects.
Push your changes upstream, which makes them somebody else's problem.
Figure out what your goals are and what you're not going to do
You can't stay focused if you can't say no

Buildroot Example

The buildroot project was originally just a uClibc toolchain creator and test harness. The easy way to test that a package built against uClibc was to add it to buildroot, and since it never had a clear design boundary allowing it to say "no" to new features, this quickly grew out of hand.
The project maintainer (Erik Andersen) and several of his senior developers had so much of their time taken up by buildroot they stopped contributing to the original project, uClibc.
The uClibc mailing list was swamped by buildroot traffic for years until Rob created a new buildroot list and kicked the traffic over there, but uClibc development still hasn't fully recovered.
This is a significant contributing factor to uClibc's TLS/ NPTL support being feature complete for its first architecture in 2006 (there was an OLS paper on it) but still not having shown up in an actual uClibc release 3 years later.
They do the distro thing badly: no package manager, no repository.
Buildroot didn't use a package manager (like rpm, dpkg, ipkg/opkg, or portage), instead it encoded its dependency resolution in menuconfig's kconfig files, and the package build logic in a series of nested makefiles, then stored the whole thing in source control. Thus the builtroot develpers took a while to notice it was becoming a distro.
Lots of open source build systems forked off of buildroot, just as lots of in-house build systems forked off of board support packages. They quickly diverge enough to become independent projects capable of absorbing enormous amounts of time.
Moral: Original project stalled because its developers sucked away, wound up doing the distro thing badly with no package manager, no repository, no stable releases.
Don't let this happen to you.

Or in other words, "OS development is hard, let's go bowling". See also https://distrowatch.com/

[medyagh]

But I think the custom distribution is more of a side project, and stealing focus from the minikube development ?

it would be cool if we could have kuberentes org adopt our Buildroot OS and invest it as community , and minikube could be one of the users of it ? could we start that conversation ?

It has been good work put into it for years and it has been battletested in many environments for years, wdyt?

[afbjorklund]

I see there is more interest in the vanilla images, or in some proprietary LinuxKit image like with Docker Desktop.

But the discussion should definitely be had, that was (more or less) why I opened this issue in the first place...

I like the Tiny Core Linux (for docker-machine) or Buildroot (for minikube) images, mostly on technical merits.

But if has been hard to sell that, since people (and kubeadm) seem to prefer using Ubuntu or Fedora distributions ?

[afbjorklund]

@priyawadhwa

I fail to see how this is a support question ?

[afbjorklund]

@medyagh :

But I think the custom distribution is more of a side project, and stealing focus from the minikube development ?

it would be cool if we could have kuberentes org adopt our Buildroot OS and invest it as community , and minikube could be one of the users of it ? could we start that conversation ?

It has been good work put into it for years and it has been battletested in many environments for years, wdyt?

I like this idea, and I think that we could start such a project. But it is still rather different from the rest of Minikube.

In my own experiments I also wanted to include everything needed for kubeadm, which conflicts somewhat with:

1. Being able to choose container runtime
2. Being able to choose kubernetes version

Since Buildroot doesn't have a package manager, it would mean there would be a lot of different images around...

One workaround would be to use tarballs, but then one would still have to invent a whole ecosystem around that ?

---

There is a separate discussion about creating a new Tiny Core Linux distro, for running docker-machine/podman-machine.

It's like a smaller version of the same or similar project, with a different upstream and with slightly different design goals:

Machine:

• 1 vCPU, 1G RAM
• rootfs
• init
• Docker / Podman

Minikube:

• 2 vCPU, 2G RAM
• tmpfs
• systemd
• Kubernetes (kubeadm)

One alternative to the current tmpfs implementation on an .iso would be to just use a standard ext4 file system and an .img.

Another variation would be to switch from the current isolinux over to a standard boot loader like GRUB for better portability.

[rgordill]

There are work for most of the distros to create a minimal one in the container space. For example, in Fedora there are different alternatives created with kickstart: https://pagure.io/fedora-kickstarts/tree/main, or in ubuntu https://cdimage.ubuntu.com/netboot/.

From them, adding packages through package-manager is quite more easy to do and maintain.

[afbjorklund]

From them, adding packages through package-manager is quite more easy to do and maintain.

@rgordill : The coreos installation is supposed to be ephemeral/immutable, instead it uses toolbox for debugging purposes

https://github.com/coreos/toolbox

For the prototype we are using the "cloud" images (as available through vagrant), rather than kickstart and netboot and such.

[rgordill]

Usually "cloud" images are built with those alternatives. Anyway, the fact is that some low level workloads requires some OOB capabilities on the OS to be able to be deployed, like for example nmstate (https://nmstate.io/kubernetes-nmstate/user-guide/102-configuration), or they need a particular kernel version in the host.

I know this is not the general use case, but if a simple way of building the iso were in place, we may select easily the one that we need, and not looking for an alternate solution to it.

[afbjorklund]

I'm not sure updating or building the ISO was complex, it was more like "maybe it shouldn't be handled by minikube project"

[afbjorklund]

But we said that we would do one more iteration of the ISO, based on Buildroot 2021.02 (which Linux kernel version is still unclear)

We will cover up the lack of a package manager by using tarballs, will only be used for the container runtime and for kubernetes...
The generic/ssh driver should give us plenty of opportunity to test other installations, and to fix the non-functional provisioner.

But a drop-in replacement for the minikube.iso would be nice, it would be easier to compare if there was a concrete example.

[afbjorklund]

Possibly this could be used https://github.com/tianon/boot2docker-debian as a base, for the ubuntu alternative

[afbjorklund]

https://github.com/kubernetes/minikube/blob/master/enhancements/proposed/20201230-standard-distro/standard-distro.md

/lifecycle frozen

[afbjorklund]

There have been no volunteers for this project, so any prototyping would have to use an external VM and the SSH driver.