feat: CommunityToolkit.Aspire.Hosting.K3s — k3s Kubernetes cluster hosting integration

[edmondshtogu]

Closes #1321

Overview of changes

Adds CommunityToolkit.Aspire.Hosting.K3s, a hosting integration that runs a lightweight Kubernetes cluster as an Aspire resource tree. Developers can declare a local Kubernetes cluster in Program.cs — with Helm charts, manifests, and exposed service endpoints — the same way they add Redis or PostgreSQL. No external tooling beyond a compatible container runtime (Docker or Podman) is required.

What's included

New package: src/CommunityToolkit.Aspire.Hosting.K3s/

File	Responsibility
K3sClusterResource.cs	ContainerResource — k3s server; holds kubeconfig directory path and image settings
K3sClusterOptions.cs	Configuration (pod/service CIDR, disabled components, k3s image tag, helm/kubectl image overrides)
K3sBuilderExtensions.cs	AddK3sCluster, WithDataVolume, WithLifetime, WithReference(cluster), WithK3sVersion, …
K3sReadinessHealthCheck.cs	File-based health check — polls cluster/kubeconfig.yaml, writes local/ + container/ variants, probes nodes via KubernetesClient
HelmReleaseResource.cs	ContainerResource — runs alpine/helm; child of cluster; exits 0 on success
K3sBuilderExtensions.Helm.cs	AddHelmRelease, WithHelmValue, WithHelmValuesFile
K8sManifestResource.cs	ContainerResource — runs alpine/k8s; child of cluster; exits 0 on success
K3sBuilderExtensions.Manifest.cs	AddK8sManifest with auto-detected Kustomize support
K3sServiceEndpointResource.cs	Resource — in-process port-forward; M1 passive health via IsReady flag
K3sBuilderExtensions.ServiceEndpoint.cs	AddServiceEndpoint, WithReference(endpoint)
K3sInProcessPortForwarder.cs	KubernetesClient WebSocket TCP forwarder; binds 0.0.0.0:{port}
K3sAgentResource.cs	Worker node support (K3sClusterOptions.AgentCount)
HelmContainerImageTags.cs / KubectlContainerImageTags.cs	Pinned image defaults; overridable via K3sClusterOptions

New Tests & Examples:

• Unit Tests: tests/CommunityToolkit.Aspire.Hosting.K3s.Tests/ — 87 unit tests covering resource registration, script generation, kubeconfig variants, Kustomize detection, values file injection, and public API null guards.
• Examples: examples/k3s/CommunityToolkit.Aspire.Hosting.K3s.AppHost/
• TypeScript playground: playground/polyglot/TypeScript/CommunityToolkit.Aspire.Hosting.K3s/ValidationAppHost/

Key design decisions

• Health check via bind-mount, not docker exec. k3s writes its kubeconfig to K3S_KUBECONFIG_OUTPUT=/tmp/k3s-kubeconfig/kubeconfig.yaml, bind-mounted to AppHostDirectory/.k3s/{name}/cluster/ on the host. The health check polls File.Exists, rewrites server URLs into local/ and container/ variants, then confirms node readiness via IKubernetes.CoreV1.ListNodeAsync. No shell access, no docker exec, works with any container runtime.

• Helm and kubectl run as containers. HelmReleaseResource and K8sManifestResource extend ContainerResource and are shown as children of the cluster in the Aspire dashboard. The install/apply script is injected via WithContainerFiles. They cannot use WaitFor(cluster) (Aspire forbids a child waiting for its parent), so their scripts poll for /root/.kube/kubeconfig.yaml — which only appears after the cluster health check passes — before proceeding. Consumers use WaitForCompletion(helmRelease) since these are run-to-completion containers.

• Kubeconfig delivered via bind-mount to all containers. WithReference(cluster) on containers, the helm installer, and the kubectl applier all bind-mount AppHostDirectory/.k3s/{name}/container/ (server: https://{name}:6443) at a known in-container path and set KUBECONFIG. Bind-mount is used uniformly so the kubeconfig updates automatically if the cluster is recreated without restarting dependent containers. No KUBECONFIG_DATA base64 encoding — all standard Kubernetes tooling (kubectl, helm, KubernetesClient SDK) works without custom bootstrap code.

• Kustomize auto-detected. AddK8sManifest checks for kustomization.yaml at configuration time: if present, it bind-mounts the directory (preserving relative base references) and uses kubectl apply -k; otherwise it uses an async WithContainerFiles callback to copy only the YAML files at container-start time and applies with kubectl apply -f --server-side. The callback approach avoids Aspire's build-time path validation on the string overload. The script auto-detects the mode at runtime.

• Service exposure without NodePort. K3sServiceEndpointResource starts an in-process KubernetesClient WebSocket port-forward bound to 0.0.0.0:{hostPort}. Host resources receive services__{name}__url=http(s)://localhost:{port}; container resources receive services__{name}__url=http(s)://host.docker.internal:{port} with --add-host=host.docker.internal:host-gateway injected automatically via ContainerRuntimeArgsCallbackAnnotation (DCP does not inject this on Linux). The forwarder resolves targetPort from the service spec (not the service port) before opening the pod WebSocket, and only signals ready after a running pod is confirmed — not when the TCP listener starts.

• Image overrides via K3sClusterOptions. The helm and kubectl container images are configurable via HelmImage/HelmTag/HelmRegistry and KubectlImage/KubectlTag/KubectlRegistry on K3sClusterOptions. Defaults: docker.io/alpine/helm:3.17.3 and docker.io/alpine/kubectl:1.36.0.

• Robustness fixes from review. WithK3sVersion propagates the image tag to all agent nodes (prevents server/agent version skew). AddServiceEndpoint validates the port is in the range 1–65535. Helm values files are indexed as {i}-{filename} so declaration order is preserved and basename collisions are safe. All helm --set values and --values paths are POSIX single-quote escaped via ShellEscape().

Usage

var cluster = builder.AddK3sCluster("k8s")
    .WithDataVolume()
    .WithK3sVersion("v1.36.0-k3s1");

var widgetCrd = cluster.AddK8sManifest("widget-crd", "./k8s/crds/");

var argocd = cluster.AddHelmRelease("argocd", "argo-cd",
    repo: "https://argoproj.github.io/argo-helm",
    version: "7.8.0",
    @namespace: "argocd")
    .WithHelmValuesFile("./deploy/argocd-values.yaml");

var ui = cluster.AddServiceEndpoint("argocd-ui", "argocd-server", 443, "argocd")
    .WaitForCompletion(argocd);

builder.AddProject<Projects.WidgetOperator>("operator")
    .WaitForCompletion(widgetCrd)
    .WithReference(cluster);

builder.AddProject<Projects.Api>("api")
    .WaitFor(ui)
    .WithReference(ui);

Example

PR Checklist

• Created a feature/dev branch in your fork (vs. submitting directly from a commit on main)
• Based off latest main branch of toolkit
• PR doesn't include merge commits (always rebase on top of our main, if needed)
• New integration
  • Docs are written
  • Added description of major feature to project description for NuGet package (4000 total character limit, so don't push entire description over that)
• Tests for the changes have been added (for bug fixes / features) (if applicable)
• Contains NO breaking changes
• Every new API (including internal ones) has full XML docs
• Code follows all style conventions

Other information

Security Assumptions
This change assumes local development only. k3s runs in privileged mode (required by k3s/containerd). The bind-mounted kubeconfig directory is readable only by the AppHost user. KubernetesClientConfiguration uses the embedded CA cert from the kubeconfig; no DangerousAcceptAnyServerCertificateValidator is used.

Remaining Follow-up Work

• Publish-time diagnostic when a K3sClusterResource has no production counterpart configured.

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/CommunityToolkit/Aspire/main/eng/scripts/dogfood-pr.sh | bash -s -- 1322

Or

• Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/CommunityToolkit/Aspire/main/eng/scripts/dogfood-pr.ps1) } 1322"

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[edmondshtogu]

@dotnet-policy-service agree

[Copilot]

Pull request overview

Copilot reviewed 37 out of 38 changed files in this pull request and generated 13 comments.

[Copilot]

Pull request overview

Copilot reviewed 37 out of 38 changed files in this pull request and generated 15 comments.

Comments suppressed due to low confidence (4)

src/CommunityToolkit.Aspire.Hosting.K3s/K3sBuilderExtensions.Manifest.cs:95

• This async callback has no await, which will produce CS1998 and be treated as an error in this repo. Use a completed Task return (or a synchronous overload) so the project builds cleanly.

            resourceBuilder.WithContainerFiles("/k8s-manifests", async (ctx, ct) =>
            {
                if (Directory.Exists(absolutePath))

src/CommunityToolkit.Aspire.Hosting.K3s/K3sBuilderExtensions.Helm.cs:91

• This async expression-bodied callback has no await, so CS1998 will be emitted and treated as an error. Return a completed task (or use a synchronous overload) instead of marking the lambda async.

            .WithContainerFiles("/helm-values", async (ctx, ct) =>
                release.ValuesFiles
                    .Select((hostPath, i) => (ContainerFileSystemItem)new ContainerFile

src/CommunityToolkit.Aspire.Hosting.K3s/K3sInProcessPortForwarder.cs:152

• The same empty-selector case here can select the first ready pod in the namespace and forward traffic to an unrelated workload. Services without selectors are valid in Kubernetes, so the forwarder should not treat an empty selector as "all pods".

            var selector = string.Join(",",
                (svc.Spec.Selector ?? new Dictionary<string, string>()).Select(kv => $"{kv.Key}={kv.Value}"));

            var pods = await k8sClient.CoreV1
                .ListNamespacedPodAsync(@namespace, labelSelector: selector, cancellationToken: ct)

tests/CommunityToolkit.Aspire.Hosting.K3s.Tests/K3sClusterResourceTests.cs:253

• This test name says it verifies the pod subnet argument, but the assertion only checks that a cluster resource exists. It would still pass if WithPodSubnet stopped adding --cluster-cidr, so it should assert the command-line args.

[Copilot]

Pull request overview

Copilot reviewed 37 out of 38 changed files in this pull request and generated 7 comments.

[aaronpowell]

I've just tried running the TypeScript app host on my devcontainer and the podinfo-web resource doesn't seem to be starting.

Here's the logs for it:

Port-forward: 0.0.0.0:40629 → svc/podinfo.podinfo:9898
Service podinfo/podinfo not yet ready; retrying…
k8s.Autorest.HttpOperationException: Operation returned an invalid status code 'NotFound', response body {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services \"podinfo\" not found","reason":"NotFound","details":{"name":"podinfo","kind":"services"},"code":404}

   at k8s.Kubernetes.SendRequestRaw(String requestContent, HttpRequestMessage httpRequest, CancellationToken cancellationToken) in /_/src/KubernetesClient/Kubernetes.cs:line 180
   at k8s.AbstractKubernetes.ICoreV1Operations_ReadNamespacedServiceWithHttpMessagesAsync[T](String name, String namespaceParameter, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken) in /_/src/KubernetesClient/obj/Release/net10.0/generated/LibKubernetesGenerator/LibKubernetesGenerator.KubernetesClientSourceGenerator/CoreV1Operations.g.cs:line 4018
   at k8s.AbstractKubernetes.k8s.ICoreV1Operations.ReadNamespacedServiceWithHttpMessagesAsync(String name, String namespaceParameter, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken) in /_/src/KubernetesClient/obj/Release/net10.0/generated/LibKubernetesGenerator/LibKubernetesGenerator.KubernetesClientSourceGenerator/CoreV1Operations.g.cs:line 4028
   at k8s.CoreV1OperationsExtensions.ReadNamespacedServiceAsync(ICoreV1Operations operations, String name, String namespaceParameter, Nullable`1 pretty, CancellationToken cancellationToken) in /_/src/KubernetesClient/obj/Release/net10.0/generated/LibKubernetesGenerator/LibKubernetesGenerator.KubernetesClientSourceGenerator/CoreV1OperationsExtensions.g.cs:line 24966
   at CommunityToolkit.Aspire.Hosting.K3sInProcessPortForwarder.WaitForServiceReadyAsync(ILogger logger, CancellationToken ct) in /workspaces/aspire-communitytoolkit/src/CommunityToolkit.Aspire.Hosting.K3s/K3sInProcessPortForwarder.cs:line 147
Service podinfo/podinfo not yet ready; retrying…
k8s.Autorest.HttpOperationException: Operation returned an invalid status code 'NotFound', response body {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services \"podinfo\" not found","reason":"NotFound","details":{"name":"podinfo","kind":"services"},"code":404}

   at k8s.Kubernetes.SendRequestRaw(String requestContent, HttpRequestMessage httpRequest, CancellationToken cancellationToken) in /_/src/KubernetesClient/Kubernetes.cs:line 180
   at k8s.AbstractKubernetes.ICoreV1Operations_ReadNamespacedServiceWithHttpMessagesAsync[T](String name, String namespaceParameter, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken) in /_/src/KubernetesClient/obj/Release/net10.0/generated/LibKubernetesGenerator/LibKubernetesGenerator.KubernetesClientSourceGenerator/CoreV1Operations.g.cs:line 4018
   at k8s.AbstractKubernetes.k8s.ICoreV1Operations.ReadNamespacedServiceWithHttpMessagesAsync(String name, String namespaceParameter, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken) in /_/src/KubernetesClient/obj/Release/net10.0/generated/LibKubernetesGenerator/LibKubernetesGenerator.KubernetesClientSourceGenerator/CoreV1Operations.g.cs:line 4028
   at k8s.CoreV1OperationsExtensions.ReadNamespacedServiceAsync(ICoreV1Operations operations, String name, String namespaceParameter, Nullable`1 pretty, CancellationToken cancellationToken) in /_/src/KubernetesClient/obj/Release/net10.0/generated/LibKubernetesGenerator/LibKubernetesGenerator.KubernetesClientSourceGenerator/CoreV1OperationsExtensions.g.cs:line 24966
   at CommunityToolkit.Aspire.Hosting.K3sInProcessPortForwarder.WaitForServiceReadyAsync(ILogger logger, CancellationToken ct) in /workspaces/aspire-communitytoolkit/src/CommunityToolkit.Aspire.Hosting.K3s/K3sInProcessPortForwarder.cs:line 147

It just keeps repeating that. Is there any additional things that need to be installed to run it?

Edit: I left it running for ~30 minutes. The resource started but never became healthy - same error as above in the logs.

[aaronpowell]

I also tried running the .NET app host and had the same outcome (although I only let that one run for 10 minutes 🤣)

[edmondshtogu]

@aaronpowell I found the issue!

On native Linux, Docker binds files using their inode number on start time, not their path name. EnsureKubeconfigPlaceholder created an empty placeholder file (inode X), which the Helm/kubectl containers bind-mount as /tmp/k3s-kubeconfig.yaml. When K3sReadinessHealthCheck later ran WriteAtomicAsync (write to temp → File.Move → rename), that rename() created a new inode Y at the same path. The containers kept tracking the old empty inode X, so [ -f /tmp/k3s-kubeconfig.yaml ] was true but the file was always empty, therefor helm install never run, leaving the service endpoint in waiting state. Additionally I added a timeout as the safety net for helm/manifest health check to prevent hanging indefinitely.

In my macOS it worked as I have configuring Colima with sshfs which resolves bind-mounted files by path (not inode), so the rename was visible inside containers.

Tests in codespaces are now passing!

[edmondshtogu]

K3s tests passed 🎉