feat: Add build support for ARM images (#2167)

[alinbalutoiu]

Since ArgoCD is written in Go, the code can be compiled on other architectures as well.

Unfortunately, I haven't found an option for CircleCI to run the build on the ARM architecture, so there are no automated builds for the ARM images.

For the convenience, I have created a Drone pipeline which supports both arm and arm64 to show the build status for the ARM images for this PR: https://cloud.drone.io/alinbalutoiu/argo-cd/13

Images are being pushed automatically to https://hub.docker.com/r/alinbalutoiu/argocd/tags

To build the ARM images, you need to run make armimage from an arm device (such as a raspberry pi for example).

It is currently running on my home Raspberry Pi 4 without any issues.

Closes #2167
Closes #3120

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Optional. My organization is added to USERS.md.
• I've signed the CLA and my build is green (troubleshooting builds).

[codecov]

Codecov Report

Merging #3554 into master will decrease coverage by 0.22%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3554      +/-   ##
==========================================
- Coverage   41.58%   41.35%   -0.23%     
==========================================
  Files         115      115              
  Lines       16463    16598     +135     
==========================================
+ Hits         6846     6864      +18     
- Misses       8731     8846     +115     
- Partials      886      888       +2     

Impacted Files	Coverage Δ
controller/cache/cache.go	0.00% <0.00%> (ø)
server/application/application.go	28.31% <0.00%> (+0.15%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bc83719...8e03ac4. Read the comment docs.

[alinbalutoiu]

@paulfantom thanks a lot for the feedback! I've updated the PR.

Seems like Kustomize was recently updated to v3.5.5 in Argo-CD. This breaks the installation from source (kubernetes-sigs/kustomize#2462) which is currently used to install Kustomize in the arm systems. The build works fine after updating Kustomize to v3.6.1 (https://cloud.drone.io/alinbalutoiu/argo-cd/16/1/1)

Note that I haven't included the Kustomize version update in this PR yet, I believe that would be more suitable as a different PR. Need to look more into how to do that, currently tried with (alinbalutoiu@1d3704b)

[jannfis]

Thanks a lot for this awesome PR @alinbalutoiu!

I personally think ARM is an often under-appreciated platform, and this PR is a great step forward for the reach of ArgoCD. I have locally build & tested a new image, and it all seems to be well so far!

I would love to merge this PR, however, what do you think of also having the ability to download the ARM CLIs from the server, as it is possible with the other binaries? You can have a look at #3015 - which introduced the Windows CLI - what needs to be done for that. I'd love to see this, but we can also do it in another PR - I guess that'd be fine as well for me. Maybe a seperate PR would be even better, because there's another build toolchain related PR in the queue I want to merge timely, which touches some of the files contained in this PR as well.

Please let me know what you think about it and whether you'd like to implement it.

[alinbalutoiu]

Thanks a lot for the review @jannfis ! To be honest it didn't seem obvious why the ArgoCD CLIs for Windows / Darwin were uploaded in the Linux container image, so I got them removed from the ARM image (https://github.com/argoproj/argo-cd/pull/3554/files#diff-b67911656ef5d18c4ae36cb6741b7965R245-R246).
So the ARM image won't actually have that functionality.

In my opinion it would be better to move out the CLIs from the container image (the ARM CLI would add an extra ~50MB to the size of the image), and replace that with direct links to the GitHub release page. It should also decrease the container image size making the image pull faster. What do you think about that?

Also please note that this PR requires a version bump for kustomize from 3.5.5 to 3.6.1 (I guess it should be similar to this one #3619). I don't really understand how to update the argoproj/argocd-test-tools image, but I assume it contains the updated kustomize binary. Should the kustomize version update be part of this PR as well?

Regarding the build process, would you be open to integrate Drone CI (free for Open Source projects) as it comes with ARM support? I can help with that if you're interested.

[jannfis]

In my opinion it would be better to move out the CLIs from the container image (the ARM CLI would add an extra ~50MB to the size of the image), and replace that with direct links to the GitHub release page. It should also decrease the container image size making the image pull faster. What do you think about that?

Yes, that's probably the right to head to. Our image got quite bloated lately, as was also mentioned in #3674. Probably moving out the CLI binaries to their own Docker images would be the way to go, and I like the idea of linking from the downloads page in UI to a GitHub release too.

Also please note that this PR requires a version bump for kustomize from 3.5.5 to 3.6.1 (I guess it should be similar to this one #3619). I don't really understand how to update the argoproj/argocd-test-tools image, but I assume it contains the updated kustomize binary. Should the kustomize version update be part of this PR as well?

The argocd-test-tools image is deprecated for the CI toolchain after the PR I already mentioned is ready & merged. We'll probably stop maintaining it as Docker image too, but let developers build it locally so they have a clean build environment that's up-to-date with what the CI does, so there'll be less nasty surprises after submitting a PR. :)

Regarding the build process, would you be open to integrate Drone CI (free for Open Source projects) as it comes with ARM support? I can help with that if you're interested.

We're just about to migrate the CI from CircleCI to GitHub actions. Lets see how that works out, and then check if and how we can use Drone for some of the steps. I have not yet had a thorough look at Drone, but I figured it's the next cool kid on the block for doing CI :) But tbh, at this point in time I'd rather would like to connect ARM runners to GH actions. Is it possible to use Drone as external GitHub actions runner?

Anyway, I'll merge this PR anytime soon so I can further prepare the PR for migration of CI to GH actions. When that's also merged, we can do the upgrade to Kustomze v3.6.x (there's already #3655) without having to also update & publish the argocd-test-tools image. Then, people can at least start building their own ARM images from the official tree, and from there we'll look further how to automate their builds.

Does it sound good to you?

[alinbalutoiu]

Yes, that's probably the right to head to. Our image got quite bloated lately, as was also mentioned in #3674. Probably moving out the CLI binaries to their own Docker images would be the way to go, and I like the idea of linking from the downloads page in UI to a GitHub release too.

Should I remove the ARM CLIs from the image in that case? https://github.com/argoproj/argo-cd/pull/3554/files#diff-b67911656ef5d18c4ae36cb6741b7965R170-R171
This will add an extra ~100MB to the image size otherwise.

We're just about to migrate the CI from CircleCI to GitHub actions. Lets see how that works out, and then check if and how we can use Drone for some of the steps. I have not yet had a thorough look at Drone, but I figured it's the next cool kid on the block for doing CI :) But tbh, at this point in time I'd rather would like to connect ARM runners to GH actions. Is it possible to use Drone as external GitHub actions runner?

Connecting an ARM runner to GH actions would definitely be a better option if you have the possibility to provide some ARM systems for that. Here's the .drone.yml I've used to build the ARM images.

Anyway, I'll merge this PR anytime soon so I can further prepare the PR for migration of CI to GH actions. When that's also merged, we can do the upgrade to Kustomze v3.6.x (there's already #3655) without having to also update & publish the argocd-test-tools image. Then, people can at least start building their own ARM images from the official tree, and from there we'll look further how to automate their builds.

Does it sound good to you?

I'd be more than happy to see the support for ARM builds added to the official tree. Sounds good to me!

[jannfis]

Should I remove the ARM CLIs from the image in that case? https://github.com/argoproj/argo-cd/pull/3554/files#diff-b67911656ef5d18c4ae36cb6741b7965R170-R171
This will add an extra ~100MB to the image size otherwise.

Probably a good idea :) I just noticed it too while inspecting the image built from your PR. So yeah, would be great if you could adapt the build process to not copy the images into the final image. Afterwards, I'll approve & merge this PR.

In the meanwhile, we should figure out how to proceed with handling the CLIs in the future.

/cc @alexmt

[alinbalutoiu]

I've removed copying the ARM CLIs in the final image. If anyone needs the binaries, they would need to build the image and fetch the CLI from inside the container image. Note that I'm still avoiding building the Darwin and Windows CLIs on ARM systems to improve the build speed.

Note that I've added as a temporary workaround this: https://github.com/argoproj/argo-cd/pull/3554/files#diff-54d4bdaa54a651a05076c4dfffacfed1R14-R18
It will use the latest version of kustomize when building on ARM systems, so that after the PR gets merged the ARM builds will work without having to wait for the kustomize version bump. Let me know if you think that's not a good idea or if it's better to wait until the kustomize version update (ARM builds will not work with current 3.5.5 kustomize version).

Build output can be seen here: https://cloud.drone.io/alinbalutoiu/argo-cd/18
If that ends up green then this is good to go in my opinion.

[jannfis]

I've removed copying the ARM CLIs in the final image. If anyone needs the binaries, they would need to build the image and fetch the CLI from inside the container image. Note that I'm still avoiding building the Darwin and Windows CLIs on ARM systems to improve the build speed.

I think we should integrate their build into the release process and publish the binaries from the releases page, so that people wanting to use the ARM binaries for the CLI do not have to build a complete image themselves.

Note that I've added as a temporary workaround this: https://github.com/argoproj/argo-cd/pull/3554/files#diff-54d4bdaa54a651a05076c4dfffacfed1R14-R18
It will use the latest version of kustomize when building on ARM systems, so that after the PR gets merged the ARM builds will work without having to wait for the kustomize version bump. Let me know if you think that's not a good idea or if it's better to wait until the kustomize version update (ARM builds will not work with current 3.5.5 kustomize version).

I've already seen that, but thanks for the additional hint. I guess we can fix that until the next release. On a second thought, maybe it might even make sense to setup a Drone pipeline for building all Docker images without pushing them. Thanks for providing your drone.yml configuration, I'll see what I can do from it! :)

Build output can be seen here: https://cloud.drone.io/alinbalutoiu/argo-cd/18
If that ends up green then this is good to go in my opinion.

It's green, as well as the CircleCI checks. 🎉

[jannfis]

LGTM, thanks for this PR and the discussions!

[alinbalutoiu]

I've already seen that, but thanks for the additional hint. I guess we can fix that until the next release. On a second thought, maybe it might even make sense to setup a Drone pipeline for building all Docker images without pushing them. Thanks for providing your drone.yml configuration, I'll see what I can do from it! :)

If you're interested in the Drone pipeline which builds the Docker images for ARM without pushing them here it is: alinbalutoiu@c54e5d9

Glad to see this being merged, thanks for the review and merging the PR!

[LS80]

@alinbalutoiu Using alinbalutoiu/argocd:v1.6.1 on ARM with Kustomize I have the error ComparisonError: rpc error: code = Unknown desc = fork/exec /usr/local/bin/kustomize: no such file or directory.

[alinbalutoiu]

@LS80 thanks for noticing that, I built yesterday the images for the latest version and it worked fine, but I have checked only with helm. I'll look into the issue with kustomize.

[alinbalutoiu]

@LS80 I've created the issue #3908 to further track this.

[ppouliot]

Are you still looking for a home to build for arm and arm64?
I suggest you apply for hosting here:
https://osuosl.org/services/aarch64/request_hosting/

[slmingol]

Was able to use @alinbalutoiu ARM built containers to deploy argo-cd quite easily:

     helm repo add argo https://argoproj.github.io/argo-helm
     helm install my-argocd argo/argo-cd \
         --namespace argocd \
         --create-namespace \
         --set global.image.repository="alinbalutoiu/argocd" \
         --set installCRDs="false" \
         --wait

[paulfantom]

I am maintaining my fork of argocd in which the only addition is a custom CI pipeline which builds images for aarch64 and amd64, and pushes them to quay.io/paulfantom/argocd. This uses github actions and doesn't need any external hosting. Personally, I don't have time to make this part of argocd project but if anyone wants to integrate it into argo-cd project and needs help in understanding how it works feel free to reach out.

[alinbalutoiu]

That looks great @paulfantom , thanks for sharing it. I will look into integrating it in the ArgoCD project when I get the time.

[geowalrus4gh]

We're just about to migrate the CI from CircleCI to GitHub actions. Lets see how that works out, and then check if and how we can use Drone for some of the steps. I have not yet had a thorough look at Drone, but I figured it's the next cool kid on the block for doing CI :) But tbh, at this point in time I'd rathe

Any plans to create new CI which builds ARM arch too ?