Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build KIC base image for ARM #9227

Closed
afbjorklund opened this issue Sep 12, 2020 · 11 comments
Closed

Build KIC base image for ARM #9227

afbjorklund opened this issue Sep 12, 2020 · 11 comments
Assignees
Labels
co/docker-driver Issues related to kubernetes in container co/kic-base kind/feature Categorizes issue or PR as related to a new feature. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.

Comments

@afbjorklund
Copy link
Collaborator

Once we have cleaned up the kicbase image and started to use regular packages, we can build for other architectures too.

So we can have both amd64 and arm64 variants.

Currently it is quite the mess to do it, since the docker daemon doesn't support architectures (only buildx and registry does)

But it is doable, Ubuntu 20.04 LTS supports both arch.

@afbjorklund afbjorklund added co/kic-base priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. kind/feature Categorizes issue or PR as related to a new feature. co/docker-driver Issues related to kubernetes in container labels Sep 12, 2020
@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Sep 12, 2020

This is needed for docker driver support on raspberry pi 4 and on apple silicon macs, as well as being good for portability.

More and more cloud providers offer arm-based servers now: https://www.arm.com/blogs/blueprint/optimizing-data-center

See #6159 and #9224

@afbjorklund
Copy link
Collaborator Author

Once the custom packages are out of the way and the images are streamlined, this won't be too hard to do (except buildx)

There is one image for "arm32v7", like the Raspberry Pi OS (32-bit) a.k.a. arm

And one image for "arm64v8", like the Raspberry Pi 4 hardware (64-bit) arm64

@afbjorklund
Copy link
Collaborator Author

I was able to get it starting on the Raspberry Pi, which is ARM hardware. Found some other weird bugs in the process...

https://github.com/afbjorklund/minikube/tree/kicbase-armv7

https://github.com/afbjorklund/minikube/tree/kicbase-arm64

pi@raspberrypi:~/minikube $ ./out/minikube start --driver docker --base-image=local/kicbase:v0.0.12-snapshot3-arm-snapshot
😄  minikube v1.13.0 on Raspbian 10.4 (arm)
✨  Using the docker driver based on existing profile

⛔  Requested memory allocation (1887MB) is less than the recommended minimum 1907MB. Deployments may fail.


🧯  The requested memory allocation of 1887MiB does not leave room for system overhead (total system memory: 1887MiB). You may face stability issues.
💡  Suggestion: Start minikube with less memory allocated: 'minikube start --memory=1887mb'

👍  Starting control plane node minikube in cluster minikube
🏃  Updating the running docker "minikube" container ...
E0914 22:33:46.738251    2878 cache.go:63] save image to file "gcr.io/k8s-minikube/storage-provisioner-arm:v3" -> "/home/pi/.minikube/cache/images/gcr.io/k8s-minikube/storage-provisioner-arm_v3" failed: nil image for gcr.io/k8s-minikube/storage-provisioner-arm:v3: GET https://gcr.io/v2/k8s-minikube/storage-provisioner-arm/manifests/v3: MANIFEST_UNKNOWN: Failed to fetch "v3" from request "/v2/k8s-minikube/storage-provisioner-arm/manifests/v3".

🧯  Docker is nearly out of disk space, which may cause deployments to fail! (87% of capacity)
💡  Suggestion: 

    Try at least one of the following to free up space on the device:
    
    1. Run "docker system prune" to remove unused docker data
    2. Increase the amount of memory allocated to Docker for Desktop via
    Docker icon > Preferences > Resources > Disk Image Size
    3. Run "minikube ssh -- docker system prune" if using the docker container runtime
🍿  Related issue: https://github.com/kubernetes/minikube/issues/9024

E0914 22:33:52.184651    2878 cache.go:191] Error caching images:  Caching images for kubeadm: caching images: caching image "/home/pi/.minikube/cache/images/gcr.io/k8s-minikube/storage-provisioner-arm_v3": nil image for gcr.io/k8s-minikube/storage-provisioner-arm:v3: GET https://gcr.io/v2/k8s-minikube/storage-provisioner-arm/manifests/v3: MANIFEST_UNKNOWN: Failed to fetch "v3" from request "/v2/k8s-minikube/storage-provisioner-arm/manifests/v3".
🐳  Preparing Kubernetes v1.19.0 on Docker 19.03.8 ...
❌  Unable to load cached images: loading cached images: stat /home/pi/.minikube/cache/images/gcr.io/k8s-minikube/storage-provisioner-arm_v3: no such file or directory
🔎  Verifying Kubernetes components...
🌟  Enabled addons: default-storageclass, storage-provisioner

❗  /usr/local/bin/kubectl is version 1.17.0, which may have incompatibilites with Kubernetes 1.19.0.
💡  Want kubectl v1.19.0? Try 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" by default

First have to "allow" running on non-amd64, and then work around issues #9248 and #9252

        if runtime.GOARCH != "amd64" {
                return registry.State{Error: fmt.Errorf("docker driver is not supported on %q systems yet", runtime.GOARCH), Installed: false, Healthy: false, Fix: "Try other drivers", Doc: docURL}
        }

@afbjorklund
Copy link
Collaborator Author

Also the emoji are missing from the default terminal (LXTerminal). Installing "fonts-noto-color-emoji" didn't fix it.

The emoji do work in e.g. a text editor, but I think the terminal emulator from the LXDE desktop doesn't provide.

minikube-raspberry

@ilya-zuyev
Copy link
Contributor

AWS provides arm64 instances.
Tried to use one to build KICbase image:

ubuntu@ip-172-31-39-30:~/src/minikube$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.1 LTS"
ubuntu@ip-172-31-39-30:~/src/minikube$ 

ubuntu@ip-172-31-39-30:~/src/minikube$ uname -a
Linux ip-172-31-39-30 5.4.0-1024-aws #24-Ubuntu SMP Sat Sep 5 06:17:48 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
ubuntu@ip-172-31-39-30:~/src/minikube$ 
ubuntu@ip-172-31-39-30:~/src/minikube$ make kic-base-image
[skip]

Get:20 http://ports.ubuntu.com/ubuntu-ports focal-security/restricted arm64 Packages [1139 B]
Get:21 http://ports.ubuntu.com/ubuntu-ports focal-security/main arm64 Packages [310 kB]
Fetched 15.4 MB in 2s (6375 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package podman
The command '/bin/sh -c sh -c "echo 'deb https://dl.bintray.com/afbjorklund/podman focal main' > /etc/apt/sources.list.d/podman.list" &&     curl -L https://bintray.com/user/downloadSubjectPublicKey?username=afbjorklund -o afbjorklund-public.key.asc &&     apt-key add - < afbjorklund-public.key.asc &&     clean-install podman=1.9.3~1' returned a non-zero code: 100
make: *** [Makefile:587: kic-base-image] Error 100

looks like we miss arm64 podman package

@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Oct 20, 2020

looks like we miss arm64 podman package

6ab74c2

Yeah, I deleted that pinning an just installed podman2 instead. We need to get that working sometime anyway.

But it’s not a major issue to build podman1 (and cri-o if needed) for ubuntu 20.04 arm64. I used pbuilder to do it.

See #8063 (comment) for how it was built

@afbjorklund
Copy link
Collaborator Author

@ilya-zuyev published a podman_1.9.3~1_arm64.deb

@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Nov 15, 2020

Will do some rebasing of the current images/Dockerfile from the latest one...

We dropped the kindbase (and forked it), so that is not needed anymore.


We can do some tagging hack, to just use the regular name of the base:

FROM ubuntu:focal-20201008

And then make sure that it is actually pulling the right architecture image:

Digest OS/ARCH Compressed Size
1d7b639619bd linux/amd64 27.24 MB
ebc8925ee519 linux/arm/v7 22.93 MB
65cd340c0735 linux/arm64/v8 25.91 MB

https://hub.docker.com/_/ubuntu?tab=tags&page=1&name=focal
https://hub.docker.com/r/arm32v7/ubuntu/tags?page=1&name=focal
https://hub.docker.com/r/arm64v8/ubuntu/tags?page=1&name=focal

This is so that we can cross-build everything from a regular amd64 machine.

You will need buildx / buildkit and a local registry in order to do that, though...

@afbjorklund
Copy link
Collaborator Author

Most things seem to working, after patching some built-in "exit" calls and so on.

😄  minikube v1.15.1 on Ubuntu 20.04 (arm64)
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating docker container (CPUs=2, Memory=32100MB) ...
🐳  Preparing Kubernetes v1.19.4 on Docker 19.03.13 ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: default-storageclass, storage-provisioner
💡  kubectl not found. If you need it, try: 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
😄  minikube v1.15.1 on Ubuntu 20.04 (arm64)
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating docker container (CPUs=2, Memory=32100MB) ...
🎁  Preparing Kubernetes v1.19.4 on CRI-O 1.18.4 ...
🔗  Configuring CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: default-storageclass
💡  kubectl not found. If you need it, try: 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

KIND is also working on this (support for ARM): kubernetes-sigs/kind#166

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 16, 2021
@tstromberg
Copy link
Contributor

This was released earlier this month.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
co/docker-driver Issues related to kubernetes in container co/kic-base kind/feature Categorizes issue or PR as related to a new feature. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Projects
None yet
Development

No branches or pull requests

5 participants