Add ADR about failed multus integration

[manuelbuil]

Proposed Changes

I worked for a while in the integration of k3s and Multus but it was pretty complicated. This ADR tries to explain what was tried and why I finally gave up

Types of Changes

New feature

Verification

Testing

Linked Issues

User-Facing Change

Further Comments

[brandond]

LGTM. I usually just write it up in the final format the ADR will be accepted in, so marked as Approved, and with the Design suggestion section as if it was the Decision... so that people can approve and you don't need to update it again and get another round of approvals just to do minor rewording.

[brandond]

I would probably do like we do with disabled components in RKE2, and ship these as a packaged manifest, and then just add them to the --disable list if --multus isn't set.

[brandond]

I'm not sure a Job is the best way to do this, given that that the process for assigning job pods to specific nodes is a bit clumsy. We would need to reinvent much of the logic around creating and managing job-node assignment, and this logic already exists in the daemonset controller. I think we should either drop the required files directly from the k3s supervisor process, or use a daemonset to ensure that the install pod gets run on every node.

[manuelbuil]

if it is too clumsy, then perhaps it is best for k3s to do that job but then, I don't want to increase the size of k3s by adding too many binaries that most people won't use... I'm starting to wonder if perhaps the idea of that user that suggested to add "macvlan" to the cni multi-exec might be the best one :D

[manuelbuil]

macvlan + multus + whereabouts +.... I'll have fun in the next days

[manuelbuil]

Adding multus+whereabouts to the multi-exec is not worth it.:

• Multus binary increases the size of k3s significantly. Given that only a handful of users will benefit from this, I don't think it makes sense
• whereabouts binary has very very old dependencies. That project actually looks sort of unmaintained. Anyway, it would creep in a lot of CVEs and we don't want that

[brandond]

Open question: do we want to add the multus images to the core k3s airgap image list, or should we have an optional image tarball, like we do for rke2?

[manuelbuil]

Good point, probably the latter makes more sense. What would be the disadvantages of it?

[brandond]

Disadvantage to adding an optional tarball is that we have to update build scripts, QA processes, and documentation to handle the optional tarballs. We have never had more than 1 for k3s. How big are the additional images?

[manuelbuil]

yeah. To minimize that pain, I'd leave the same name for the current tarball k3s-airgap-images-${ARCH}.tar and the images file k3s-images.txt

[codecov]

Codecov Report

Attention: Patch coverage is 33.33333% with 14 lines in your changes are missing coverage. Please review.

Project coverage is 43.77%. Comparing base (06b6444) to head (59ef7c7).

❗ Current head 59ef7c7 differs from pull request most recent head 916b122. Consider uploading reports for the commit 916b122 to get more accurate results

Files	Patch %	Lines
pkg/deploy/zz_generated_bindata.go	0.00%	12 Missing ⚠️
pkg/server/server.go	60.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #9434      +/-   ##
==========================================
- Coverage   48.66%   43.77%   -4.89%     
==========================================
  Files         158      151       -7     
  Lines       14017    13580     -437     
==========================================
- Hits         6821     5945     -876     
- Misses       5906     6416     +510     
+ Partials     1290     1219      -71     

Flag	Coverage Δ
e2etests	39.29% <33.33%> (-7.21%)	⬇️
inttests	39.38% <33.33%> (?)
unittests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[brandond]

the k3s-multus chart has the same amd64 nodeSelector issue as the rke2 chart. We should fix that before we add the chart to k3s, since we do fully support arm64.

https://github.com/k3s-io/k3s-charts/blob/main/charts/multus/4.0.201%2Bupv4.0.2-build2024020801/values.yaml#L81

[manuelbuil]

This PR is postponed for the April release