WIP: Add riscv64 support

[brandond]

Proposed Changes

Get K3s working on riscv64

Types of Changes

hack

Verification

Testing

Linked Issues

• Add support for riscv64 architecture #7151

User-Facing Change

NONE

Further Comments

Lots of caveats:

• Only tested this on VisionFive2
• Requires a custom kernel, as the one shipped by StarFive is missing pretty much everything - cgroups, iptables, vxlan, and so on. It is easy enough to rebuild a custom kernel and pack it into a flattened image tree that can be dropped in place of the existing one on the boot partition, using the sources at https://github.com/starfive-tech/VisionFive2
• Don't have a static buildroot for riscv64 yet; at the moment I'm just using the riscv64/alpine:edge base image.

[brandond]

Currently working, with both docker and containerd:

root@k3s-server-1:/ # kubectl get node -o wide
NAME           STATUS   ROLES                  AGE   VERSION                INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                   KERNEL-VERSION   CONTAINER-RUNTIME
k3s-server-1   Ready    control-plane,master   80s   v1.26.3+k3s-73fec085   172.17.0.2    <none>        K3s v1.26.3+k3s-73fec085   5.15.0           containerd://1.6.19-k3s1

root@k3s-server-1:/ # kubectl run busybox --restart=Never --rm -it --image docker.io/library/busybox:latest -- uname -a
Linux busybox 5.15.0 #4 SMP Thu Apr 13 00:16:53 UTC 2023 riscv64 GNU/Linux

root@riscv01:~# kubectl get node -o wide
NAME                STATUS   ROLES                       AGE   VERSION                INTERNAL-IP   EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION   CONTAINER-RUNTIME
riscv01.lan.khaus   Ready    control-plane,etcd,master   78s   v1.26.3+k3s-398089cd   10.0.1.202    <none>        Ubuntu 22.10   5.15.0           docker://20.10.21

root@riscv01:~# kubectl run busybox --restart=Never --rm -it --image docker.io/library/busybox:latest -- uname -a
Linux busybox 5.15.0 #4 SMP Thu Apr 13 00:16:53 UTC 2023 riscv64 GNU/Linux

[brandond]

Here's a copy of the kernel config I'm currently using, in case anyone wants to play with this on their own:
config.gz

[dominch]

@brandond - check out ubuntu official starfive vision five 2 image, k3s-check script reports almost all-green lines and it's much easier to start with this build than build on your own. This kernel (6.2) contains some fixes from upstream and don't have issues like limited to 4GB memory on 8GB boards.
So far I found that only green led and nvme is problematic, that may be fixed soon. If You already updated u-boot then just burn eMMC or sd card.

[chazapis]

This is implemented using cross-compiling in #7778.

[brandond]

I'm going to close this out - this was mostly just for me to tinker with, and the linked PR is much more comprehensive.

[dominch]

@brandond thanks for information, should we see first binaries in next releases? I'll test them out :)

[brandond]

No, we don't have a solid timeline for even experimental binaries out of our CI pipeline. We will likely have to deal with some issues around riscv64 versions of our required images before we can move this forward.