Use k3d instead of kind

[benbz]

k3d integrates a LB, metrics-server and a registry so we can simplify our config a fair bit.

It also appears faster to startup, at the cost of needing to retry

resp = <Response [429 Too Many Requests]>

    @staticmethod
    def raise_for_status(resp):
        try:
            resp.raise_for_status()
        except httpx.HTTPError as e:
>           raise transform_exception(e)
E           lightkube.core.exceptions.ApiError: storage is (re)initializing

[github-actions]

dyff of changes in rendered templates of CI manifests

Full contents of manifests and dyffs are available in https://github.com/element-hq/ess-helm/actions/runs/19371712297/artifacts/4570957734

pytest-matrix-rtc-synapse-wellknown-values.yaml

@@ Ingress/ess-ci/release-name-matrix-rtc - metadata @@
- annotations:
-   nginx.ingress.kubernetes.io/proxy-buffering: off
-   nginx.ingress.kubernetes.io/proxy-read-timeout: 120
-   nginx.ingress.kubernetes.io/proxy-send-timeout: 120



@@ Ingress/ess-ci/release-name-synapse - metadata @@
- annotations:
-   nginx.ingress.kubernetes.io/proxy-body-size: 100M



@@ Ingress/ess-ci/release-name-well-known - spec.rules.0.http.paths.0.pathType @@
- ImplementationSpecific
+ Prefix

[gaelgatelement]

We are loosing kubernetes system logs and pod logs with this move to k3d. It's quite useful to understand what happened in CI. We should have an equivalent with k3d before merging this I believe.

[benbz]

Yup I've run into the lack of this whilst debugging the CI failures. Oustanding things are

• Pod logs in CI
• Wait for 443/TCP to be available locally before ingress_ready returns
• Fix the flakiness with the pod networking tests which is now much frequent
• Fix whatever is causing HAProxy failures?
• Possibly do things with DockerHub in the k3d config itself