Enable Gateway API and AGC in AddAzureKubernetesEnvironment - #16990
Merged
Mitch Denny (mitchdenny) merged 27 commits intoMay 14, 2026
Merged
Enable Gateway API and AGC in AddAzureKubernetesEnvironment#16990Mitch Denny (mitchdenny) merged 27 commits into
Mitch Denny (mitchdenny) merged 27 commits into
Conversation
Contributor
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 16990Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 16990" |
…dAzureKubernetesEnvironment
Adds opt-in Azure Application Gateway for Containers (AGC) ingress to
AddAzureKubernetesEnvironment via a new public surface:
- AddLoadBalancer(name, subnet) on the AKS env returns a new
AzureKubernetesLoadBalancerResource and flips internal cluster-level
flags so the emitted Bicep enables the AKS-managed Gateway API
installation (ingressProfile.gatewayAPI.installation = 'Standard') and
the AGC ALB controller add-on (ingressProfile.applicationLoadBalancer.
enabled = true) on the 2025-09-02-preview API. Also applies the AGC
Microsoft.ServiceNetworking/trafficControllers subnet delegation to the
supplied subnet.
- WithLoadBalancer on KubernetesGatewayResource / KubernetesIngressResource
attaches the alb.networking.azure.io association annotations and defaults
the gatewayClassName/ingressClassName to azure-alb-external when unset.
Each LB owns its own apply-alb-crd-{name} pipeline step that waits for
the azure-alb-external GatewayClass and applies an ApplicationLoadBalancer
CR pointing at the supplied subnet. Per-LB resources are required because
each AGC ALB caps at 5 frontends.
Implementation note: the typed ingressProfile.gatewayAPI / .applicationLoadBalancer
properties are not exposed by Azure.Provisioning.ContainerService 1.0.0-beta.6
(and the parent ManagedClusterIngressProfile type is internal), so the
two values are injected via reflection on the lazily-created internal
IngressProfile instance. The narrow reflection surface is contained in
AksPreviewIngressProfileInjector and clearly documented.
Fixes #16971
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds a playground at playground/AksDemo demonstrating the new AKS AddLoadBalancer + WithLoadBalancer surface. The AppHost provisions a VNet with two AGC frontend subnets, an AzureKubernetesEnvironment, two ApplicationLoadBalancer resources, and two gateways routing to a simple Web API. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AddAzureKubernetesEnvironment does not add its resource to the model
in run mode, so the per-env aks-get-credentials-{name} pipeline step
is never collected. AddLoadBalancer was unconditionally adding the LB
resource, whose apply-alb-crd-{name} step depends on
aks-get-credentials-{name}, causing pipeline validation to fail at
'aspire run' time:
Step 'apply-alb-crd-public' depends on unknown step 'aks-get-credentials-aks'
Mirror the AddGateway/AddIngress run-mode pattern: in run mode return
a builder without registering the resource; in publish mode register
and ExcludeFromManifest.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Both the second LB and second gateway were named 'admin', which collide in the application model in publish mode. Suffix the gateway names with '-gw' to keep them distinct. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Match the AMD SKU used across the AKS deployment E2E tests so this playground deploys consistently across regions and quotas. Adds a 'workload' node pool sized the same as the system pool. Also extends the root .gitignore so playground aspire-output/ and aspire-manifest.json publish artifacts are not tracked. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The AKS managed cluster defaults its serviceCIDR to 10.0.0.0/16, which collides with the playground VNet's 10.0.0.0/16 and fails provisioning with ServiceCidrOverlapExistingSubnetsCidr. Move the VNet (and all subnets) to the 10.100.0.0/16 range. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The AKS-managed AGC ALB add-on auto-creates a managed identity
(`applicationloadbalancer-{cluster}`) inside the cluster's MC_*
resource group. AKS auto-grants this identity permissions on resources
inside MC_* but does not grant it any permissions on user-supplied
subnets that live outside MC_*. The controller then fails with
LinkedAuthorizationFailed on `Microsoft.Network/virtualNetworks/subnets/join/action`
when it tries to create the AGC association, leaving Gateways stuck
at PROGRAMMED=Unknown forever.
The AKS REST schema marks
`ingressProfile.applicationLoadBalancer.identity` as readOnly so we
can't pre-create the identity and hand it in. Instead we read the
auto-created identity's principalId back from the cluster outputs and
emit a Network Contributor role assignment per LB subnet, scoped to
the subnet, in the same Bicep module that owns the cluster. This fills
the cross-RG gap without changing the addon path.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The AKS gateways forward /api -> storefront-gw and /admin -> admin-gw without rewriting the path, so the API needs matching prefixes for end-to-end requests through AGC to return 200. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds an aks.AddHelmChart(...) call for cert-manager v1.18.2 with the Helm values required to enable its Gateway API integration. The ClusterIssuer (ACME HTTP-01 via gatewayHTTPRoute) and the TLS listener patch on storefront-gw are not yet expressible in the Aspire app model, so they live under playground/AksDemo/k8s/ as hand-applied manifests with a README walking through the post-deploy steps. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pivot from the manual kubectl-patch flow to the existing WithTls() + WithGatewayAnnotation() + tls-fqdn-discovery pipeline step in Aspire.Hosting.Kubernetes. The pipeline step polls the Gateway's status.addresses for the AGC-assigned <random>.fz<n>.alb.azure.com FQDN, patches the listener hostname, creates a self-signed bootstrap TLS secret, and transfers field ownership back to Helm via SSA. cert-manager's Gateway API integration then sees the listener with TLS configuration plus the cert-manager.io/cluster-issuer annotation and issues a Let's Encrypt cert via HTTP-01 against that FQDN, replacing the bootstrap secret in place. The only remaining manual step is one kubectl apply of the ClusterIssuer (cert-manager doesn't ship with default issuers). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds WithForceUpgrade() to KubernetesHelmChartResource. When set, the helm-install pipeline step passes --force to 'helm upgrade --install', causing Helm to replace (delete + create) any object that cannot be patched in place during upgrade. This is the documented escape hatch for charts that ship admission webhooks (cert-manager, kyverno, gatekeeper, opa) on AKS clusters where the Azure Policy add-on / Deployment Safeguards installs an admissionsenforcer field manager. That field manager mutates the chart's ValidatingWebhookConfiguration after install, and Helm 3.18+ SSA then refuses to overwrite the conflicting field on the next upgrade with: conflict with "admissionsenforcer" using admissionregistration.k8s.io/v1: .webhooks[*].namespaceSelector --force is destructive (recreates resources) but for cert-manager specifically this is benign: deployments, services, and webhooks all recreate within seconds, the AGC data plane keeps serving traffic, and the cert-manager-issued TLS Secrets live outside the helm release so they're untouched. Wires WithForceUpgrade() into the AksDemo cert-manager install since the playground typically runs on clusters that have the policy add-on enabled. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Helm 3.18 deprecated --force in favor of --force-replace, and neither is allowed alongside server-side apply (which cert-manager and most modern charts opt into): invalid operation: cannot use server-side apply and force replace together The correct flag for SSA conflict resolution in helm 3.18+ is --take-ownership, which tells SSA to take over fields owned by another field manager non-destructively (no resources recreated). This is better than the deprecated --force in every way for the admissionsenforcer scenario the API is documented for. Updates the install args, XML doc, comment, and playground guidance accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Helm 4 added a proper SSA conflict-resolution flag: --force-conflicts. This is what we actually want for the admissionsenforcer scenario: - --take-ownership transfers helm release ownership of resources, not field ownership; doesn't address SSA field conflicts. - --force / --force-replace are deprecated and incompatible with SSA (helm errors with 'cannot use server-side apply and force replace together'). - --force-conflicts forces SSA to take over conflicting fields non-destructively, matching kubectl's --force-conflicts semantic. Min Helm version for this flag is 4.x; tracking proper version validation in #16977. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Non-TLS sibling of KubernetesGatewayTlsDeploymentTests. Uses AddAzureKubernetesEnvironment + AddLoadBalancer + AddGateway and lets the Aspire pipeline provision AKS, ACR, VNet, AGC ingress profile, and the AGC ApplicationLoadBalancer + Gateway/HTTPRoute. Verifies the gateway gets an AGC FQDN and curl http://<fqdn>/api returns 200. TLS issuance via cert-manager is intentionally out of scope here and will be covered by a follow-up test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Mitch Denny (mitchdenny)
force-pushed
the
mitchdenny/aks-gateway-api-and-app-load-balancer
branch
from
May 12, 2026 23:51
463f5b4 to
90dcebf
Compare
Member
Author
|
/deployment-test |
Contributor
|
🚀 Deployment tests starting on PR #16990... This will deploy to real Azure infrastructure. Results will be posted here when complete. |
Contributor
|
🚀 Deployment tests starting on PR #16990... This will deploy to real Azure infrastructure. Results will be posted here when complete. |
Contributor
|
❌ Deployment E2E Tests failed — 34 passed, 2 failed, 0 cancelled View test results and recordings
|
Contributor
|
🎬 CLI E2E Test Recordings — 80 recordings uploaded (commit View all recordings
📹 Recordings uploaded automatically from CI run #25831768277 |
Contributor
|
Pull request created: #945
|
Contributor
|
📝 Documentation has been drafted in microsoft/aspire.dev#945 targeting Updated
Note This draft PR needs human review before merging. |
This was referenced May 14, 2026
This was referenced May 22, 2026
Open
This was referenced May 30, 2026
Closed
14 tasks
This was referenced Jun 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This wires up Azure Application Gateway for Containers (AGC) end-to-end in
AddAzureKubernetesEnvironment(...), so Aspire can stand up a fully working ingress story on AKS in oneaspire deploy. The motivation is issue #16971: todayAddLoadBalancer(...)exists in the surface but the AKS environment doesn't actually provision the AGC ingress profile, grant the controller identity rights on the delegated subnet, or expose Gateway API. This change closes those gaps and adds the playground that proved the flow.Approach
ingressProfile.webApplicationRouting/ preview API) when anAddLoadBalancer(...)is in the model, and emit a Network Contributor role assignment so the AGC managed identity can update the delegated subnet. Without that role assignment, the ALB controller never finishes data plane provisioning.AddLoadBalancer(...)registers anAzureKubernetesLoadBalancerResourceonly in publish mode (run mode skips registration so the dashboard stays clean), and the AKS pipeline applies the matchingApplicationLoadBalancerCR alongside the Helm release.WithForceUpgrade()on Helm charts: New extension onIResourceBuilder<KubernetesHelmChartResource>that appends--force-conflictstohelm upgrade --install. This is the SSA conflict-resolution flag (mirrorskubectl apply --force-conflicts); needed for charts whose webhooks get mutated by AKS Azure Policy / Deployment Safeguards (admissionsenforcerfield manager). Non-destructive and opt-in.AddAzureKubernetesEnvironment,AddLoadBalancer, two gateways (storefront with TLS, admin without), routes mapped viaWithRoute, cert-manager installed viaAddHelmChart(...).WithForceUpgrade(), andWithTls()+cert-manager.io/cluster-issuerannotation for automatic cert issuance via the existingtls-fqdn-discoverypipeline step. Includes ak8s/cluster-issuer.yamlthat the user applies once (cert-manager intentionally ships no default ClusterIssuers).Validated end-to-end
Two clean deploys to brand new resource groups (
aksingress4,aksingress5):--force-conflicts-> AGC data plane programs the gateways with FQDNs (~5 min) ->tls-fqdn-discoverypatches HTTPS listener ->kubectl applyofcluster-issuer.yaml-> Let's Encrypt prod cert issued in ~90 s ->https://<agc-fqdn>/apireturns HTTP/2 200 with a real LE R12-issued cert.Things to flag for review
--force-conflictsflag requires Helm 4. We hit three different Helm flag errors landing on the right one (--force->--take-ownership->--force-conflicts). Validating the installed Helm version is filed as follow-up Validate Helm is installed and assert compatible version for Kubernetes deploy #16977 (milestone 13.4) and intentionally out of scope here.kubectl applystep. cert-manager does not ship default issuers; this is expected and documented inplayground/AksDemo/k8s/README.md.10.100.0.0/16to avoid colliding with the default AKS service CIDR.Standard_D2as_v5is used for node pools because the default size has constrained quota in many subscriptions and this size is broadly available.Follow-on work
AddCertManagerforAspire.Hosting.Kubernetes: A follow-on PR layered on top of this one adds a first-classAddCertManager(...)API plus typedClusterIssuerresources, replacing the rawAddHelmChart("cert-manager", ...).WithForceUpgrade()+ manualkubectl apply -f cluster-issuer.yamlpattern used by the AksDemo playground here. It works with this AGC + Gateway API integration unchanged —WithTls()on a routed resource still triggers thetls-fqdn-discoverypipeline step, only the cert-manager wiring becomes typed instead of string-template Helm. See Add typed cert-manager API for Kubernetes/AKS environments #17008 for the typed API shape.Fixes: #16971
Checklist
WithForceUpgrade+ AksDemo playground as the scenario test).<remarks />and<code />elements on your triple slash comments?