Proposal: Add ARMv7 and ARM64 Binaries

[karlskewes]

Understand the maintainer catch up happening at the moment.

I run a multi-arch lab cluster and many people are using ARM SBC's as a Kubernetes training platform. I think it would increase adoption of the project if ARM arch's were supported to some extent.

I have verified kubeseal and controller built with GOARCH works correctly on my armv7 and arm64 SBC's. arm64 docker images also work in my k8s cluster.

As part of the GoReleaser PR #140 I initially added ARMv7 and ARM64 archives and docker images then backed it out to simplify the PR per #140 (comment).
It would be trivial to add basic ARM support as a PR on top of #150 .

Three niggles:

1. Travis doesn't support ARM - GitHub issue - so testing artifacts wouldn't be possible. Blocker?
2. Building multi-arch docker images:
   We would still (?) need to use the docker cli experimental features env var to create a single manifest with multi-arch. Or add buildx as a dependency and use in Makefile. Not a blocker?
3. Makefile and Dockerfiles (multi-stage) would be written for building on amd64. Those wanting to build on arm could modify local copies easy enough though. Not a blocker?

The release artifacts would look something like:

1. kubeseal archives for windows, darwin and per arch (amd64/armv7/arm64) for linux.
2. controller docker images for linux with arch as suffix -amd64, -armv7, -arm64, plus the existing <no suffix == amd64> unless we do multi-arch somehow.
3. Docs install instructions to add arch suffix to image tag if installing in ARM cluster, plus potentially a nodeAffinity yaml or jsonnet snippet too.

Suggestions/improvements welcome.

[Waterdrips]

Hey, Any chance of including these ARM builds.

We have mult-arch binaries and want users to be able to use kubeseal on ARM based devices
openfaas/faas-cli#749

[mkmik]

Sure we can easily add them (even without switching to goreleaser).

[alexellis]

Thank you. How would you like to see this done? Do you have capacity at your end to kick it off?

[mkmik]

A PR is welcome

[alexellis]

I don't have a lot of time to devote to churn and getting things wrong, can you give an outline of how you would like to see this done in specific, broad-brush steps? You mentioned not wanting to use goreleaser and not wanting to use other tools, so your direction, or a PR from your team would be great to move things forward.

cc @Waterdrips

[mkmik]

We already run cross-builds for several OSs, see Travis config file. I think it would be trivial to do the same thing for a few selected cpu archs

[alexellis]

👍 thank you

[alexellis]

I'm looking at the .travis.yml and I can't see the section that should be updated. What other arches and OSes are you building for - can you link to an example?

https://github.com/bitnami-labs/sealed-secrets/blob/master/.travis.yml

Also looked at the Makefile https://github.com/bitnami-labs/sealed-secrets/blob/master/Makefile

I was looking for something like this, where I could add the other option to a list or set of commands:

https://github.com/inlets/inletsctl/blob/master/Makefile#L9
https://github.com/openfaas/faas-cli/blob/master/Dockerfile.redist#L19

Are you driving it by this list?

os:
  - linux
  - osx

If that's how it's done, then it doesn't cover windows, which appears to be hardcoded, can we add the armhf variant in the same way?

https://github.com/bitnami-labs/sealed-secrets/blob/master/.travis.yml#L118

[mkmik]

Yes Travis native Linux and osx host environments are useful because they can be used to actually run the tests.

We decided to build the windows binary without without also setting up a Windows ci host environments and I'd use the same approach for ARM binaries, so yes, let's just build them ad hoc like we do for the windows binary (by setting the GOARCH env var)

[alexellis]

Output from the kubeseal-arm binary built locally with the command from Travis:

pi@faasd-pi:~ $ cat /etc/os-release 
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
pi@faasd-pi:~ $ ./kubeseal-arm --version
kubeseal version: 92709b235508da8cf41264dac4a58322c2eecac3+dirty
pi@faasd-pi:~ $ 

[mkmik]

Do you also need docker images for arm?

[alexellis]

We're just using the binaries for the time being, as a client for a Kubernetes cluster running on x86_64.

At some point we may come back and suggest multi-arch for the controller. Here's a couple of files from the inlets-operator we put together with @matevzmihalic - https://github.com/inlets/inlets-operator/blob/master/Dockerfile - https://github.com/inlets/inlets-operator/blob/master/Makefile

If you can let me know when you've cut a patch release, we'll update the CLI we use with kubeseal.

[onedr0p]

@mkmik can this issue be re-opened? The initial ask of providing ARM64 or multi-arch Docker image isn't complete. Thanks.

[mkmik]

@onedr0p could you please open another issue?

[onedr0p]

Done.