Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
3a65605
Enable Gateway API and Azure Application Gateway for Containers in Ad…
May 12, 2026
d315bd5
Add AksDemo playground
May 12, 2026
9bee831
Skip AddLoadBalancer model registration in run mode
May 12, 2026
1ba08a8
Avoid name collision between LB and gateway in AksDemo
May 12, 2026
aff9fb3
Use Standard_D2as_v5 for AksDemo node pools
May 12, 2026
ab7eb33
Shift AksDemo VNet to 10.100.0.0/16 to avoid AKS service CIDR conflict
May 12, 2026
ef69867
Auto-grant AGC controller identity Network Contributor on ALB subnets
May 12, 2026
3c79cf1
AksDemo: add /api and /admin endpoints for AGC smoke tests
May 12, 2026
6474a22
AksDemo: install cert-manager via AddHelmChart + TLS manifests
May 12, 2026
c06617c
AksDemo: use WithTls + cert-manager annotation for auto HTTPS
May 12, 2026
699f6a9
AksDemo: use letsencrypt-prod ClusterIssuer
May 12, 2026
d83fe50
Aspire.Hosting.Kubernetes: add WithForceUpgrade for helm charts
May 12, 2026
86ccfc0
Aspire.Hosting.Kubernetes: WithForceUpgrade uses --take-ownership
May 12, 2026
83688d3
Aspire.Hosting.Kubernetes: WithForceUpgrade uses --force-conflicts
May 12, 2026
90dcebf
Add E2E deployment test for AddAzureKubernetesEnvironment + Gateway
May 12, 2026
a5691a9
Remove playground/AksDemo/k8s folder
May 13, 2026
1459cf9
Use westus3 for AKS gateway deployment test
May 13, 2026
ff812c7
Fix apiService variable casing in AKS gateway deployment test
May 13, 2026
970fc55
Route gateway / and verify /weatherforecast in AKS gateway test
May 13, 2026
3ee6883
Skip AddHelmChart model registration in run mode
May 13, 2026
0a03c5c
Pre-create bootstrap TLS secret before Gateway FQDN discovery
May 13, 2026
60a4fa0
Update aks-support.md with AGC TLS, AksDemo, and E2E test status
May 13, 2026
ec2f41f
Rename WithForceUpgrade to WithForceConflicts to match --force-confli…
May 13, 2026
a89df2f
Address PR #16990 review findings
May 13, 2026
1d18d66
Document why AKS preview ingress profile injection requires reflection
May 13, 2026
4883406
Pass --server-side alongside --force-conflicts to helm upgrade --install
May 13, 2026
315279b
Address IEvangelist PR review feedback (1, 3, 4, 5)
May 13, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,8 @@ tests/PolyglotAppHosts/**/Java/**/*.class

# Publisher Outputs
playground/**/publish/
playground/**/aspire-output/
playground/**/aspire-manifest.json

# TypeScript AppHost
playground/**/dist/
Expand Down
4 changes: 4 additions & 0 deletions Aspire.slnx
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,10 @@
<Project Path="playground/AzureServiceBus/ServiceBus.AppHost/ServiceBus.AppHost.csproj" />
<Project Path="playground/AzureServiceBus/ServiceBusWorker/ServiceBusWorker.csproj" />
</Folder>
<Folder Name="/playground/AksDemo/">
<Project Path="playground/AksDemo/AksDemo.ApiService/AksDemo.ApiService.csproj" />
<Project Path="playground/AksDemo/AksDemo.AppHost/AksDemo.AppHost.csproj" />
</Folder>
<Folder Name="/playground/AzureStorageEndToEnd/">
<Project Path="playground/AzureStorageEndToEnd/AzureStorageEndToEnd.ApiService/AzureStorageEndToEnd.ApiService.csproj" />
<Project Path="playground/AzureStorageEndToEnd/AzureStorageEndToEnd.AppHost/AzureStorageEndToEnd.AppHost.csproj" />
Expand Down
107 changes: 105 additions & 2 deletions docs/specs/aks-support.md
Original file line number Diff line number Diff line change
Expand Up @@ -589,7 +589,109 @@ var aks = builder.AddAzureKubernetesService("aks")
- 🔲 AKS resource does not implement `IAzureContainerRegistry` (ACR outputs not exposed via standard interface)

#### Ingress controller
- 🔲 Application Gateway Ingress Controller (AGIC) or other ingress support
- ✅ Azure Application Gateway for Containers (AGC) via `AddLoadBalancer()` + `WithLoadBalancer()` (see below)
- ✅ Cert-manager auto-TLS via `WithTls(issuer)` on Gateway resources (bootstrap secret + post-FQDN cert-manager swap)
- ✅ Helm `WithForceConflicts()` (`--force-conflicts`) for cert-manager / AGC controller SSA field-manager conflicts
- ✅ AGC controller identity auto-granted `Network Contributor` on each ALB subnet
- ✅ `AddHelmChart` / `AddLoadBalancer` skip model registration in run mode (matches `AddIngress` / `AddGateway`)
- 🔲 Cluster-level no-arg `AddLoadBalancer()` (auto VNet + delegated subnet) — left as future work

### Application Gateway for Containers (AGC) ingress ✅

Opt-in, multi-LB ingress wired into Aspire's existing `AddGateway`/`AddIngress` model. The single public entry point is `AddLoadBalancer` on the AKS environment:

```csharp
var vnet = builder.AddAzureVirtualNetwork("vnet", "10.100.0.0/16");
var aksSubnet = vnet.AddSubnet("aks", "10.100.0.0/22");
var alb1Subnet = vnet.AddSubnet("alb1", "10.100.4.0/24");
var alb2Subnet = vnet.AddSubnet("alb2", "10.100.5.0/24");

var aks = builder.AddAzureKubernetesEnvironment("aks").WithSubnet(aksSubnet);

var lb1 = aks.AddLoadBalancer("public", alb1Subnet);
var lb2 = aks.AddLoadBalancer("admin", alb2Subnet);

aks.AddGateway("storefront").WithLoadBalancer(lb1);
aks.AddIngress("api").WithLoadBalancer(lb1);
aks.AddGateway("admin-portal").WithLoadBalancer(lb2);
```

What `AddLoadBalancer` does:
- Flips internal flags on the AKS environment so its emitted Bicep uses the
`2025-09-02-preview` API version and includes both
`properties.ingressProfile.gatewayAPI.installation: 'Standard'` and
`properties.ingressProfile.applicationLoadBalancer.enabled: true`. The preview API
shape is injected via `AksPreviewIngressProfileInjector`, which uses a small reflection
shim onto `ProvisionableConstruct.DefineProperty<T>` because the underlying
`ManagedClusterIngressProfile` type is `internal` in `Azure.Provisioning.ContainerService`
and cannot be subclassed like the other preview-Bicep injectors do. (These flags are
internal-only — the cluster-level toggles are intentionally not exposed as separate
public extensions to keep the surface small and prevent users from opting into the
preview API by accident.)
- Applies an idempotent `AzureSubnetServiceDelegationAnnotation` for
`Microsoft.ServiceNetworking/trafficControllers` to the supplied subnet (multiple LBs
may share the same subnet).
- Auto-grants `Network Contributor` on each ALB subnet to the AGC controller's
user-assigned identity (the cluster's `applicationLoadBalancer` ingress profile
identity), so the controller can program the subnet without manual role wiring.
- Skips registering the resource in the model in run mode (mirrors `AddIngress` /
`AddGateway` / `AddHelmChart`) so the helm/k8s pipeline isn't pulled in when running
locally.
- Returns an `AzureKubernetesLoadBalancerResource` whose own pipeline step
(`apply-alb-crd-{name}`) waits for the `azure-alb-external` GatewayClass to appear,
then `kubectl apply -f -` an `ApplicationLoadBalancer` CR named `alb-{name}` in the
`default` namespace, pointing at the supplied subnet.

What `WithLoadBalancer` does on a `KubernetesGatewayResource` / `KubernetesIngressResource`:
- Adds the AGC association annotations
(`alb.networking.azure.io/alb-name: alb-{lb}`, `alb.networking.azure.io/alb-namespace: default`)
to the rendered Helm template.
- Defaults the `gatewayClassName` / `ingressClassName` to `azure-alb-external` if the
user did not set one explicitly.

Why multi-LB by design: each AGC `ApplicationLoadBalancer` caps at five frontends, so
larger apps need to spread Gateways/Ingresses across multiple LBs. Each
`AzureKubernetesLoadBalancerResource` owns its own pipeline step so apply / wait-ready /
future-destroy lifecycle is per-LB.

#### TLS via cert-manager (`WithTls`) ✅

Gateways can opt into auto-managed HTTPS via cert-manager + Let's Encrypt:

```csharp
aks.AddHelmChart("cert-manager", "https://charts.jetstack.io", "cert-manager")
.WithValues(new { crds = new { enabled = true } });

aks.AddGateway("storefront")
.WithLoadBalancer(lb1)
.WithTls("letsencrypt-prod"); // ClusterIssuer name
```

`WithTls(issuer)` does:
- Adds an HTTPS listener (port 443) referencing a TLS secret named `{gateway}-tls`.
- Annotates the Gateway with `cert-manager.io/cluster-issuer: {issuer}` so cert-manager
watches the Gateway and mints a cert into the referenced secret once the AGC frontend
FQDN is discoverable.
- Pre-creates a self-signed bootstrap TLS secret with placeholder hostname
`bootstrap.invalid` **before** waiting for the Gateway FQDN. AGC refuses to program a
Gateway whose HTTPS listener references a non-existent secret, but FQDN discovery
doesn't run until the Gateway is programmed — pre-creating the bootstrap secret breaks
this chicken-and-egg deadlock. Once the FQDN is known, the existing patch logic adds
the discovered hostname so cert-manager can swap in the real cert.

Force-conflicts for SSA conflicts: helm chart resources support `WithForceConflicts()` which
adds `--force-conflicts` to `helm upgrade`. Required because cert-manager
and AGC controllers patch fields on the same Gateway/secret resources, producing
server-side-apply field-manager conflicts that would otherwise fail subsequent helm
upgrades.

### AksDemo playground ✅

`playground/AksDemo/` exercises the full AKS environment + AGC + Gateway API + cert-manager
TLS path against a live AKS cluster. AppHost uses `AddAzureKubernetesEnvironment` +
`AddLoadBalancer` + multiple `AddGateway` resources sharing one ALB, with
`Standard_D2as_v5` system node pool and VNet at `10.100.0.0/16` (avoids the default AKS
service CIDR `10.0.0.0/16`).

#### Managed Prometheus/Grafana
- 🔲 Azure Monitor workspace for managed Prometheus
Expand Down Expand Up @@ -617,4 +719,5 @@ var aks = builder.AddAzureKubernetesService("aks")

- 31 AKS unit tests passing (extensions + infrastructure)
- 88 K8s base tests passing
- Manual E2E validation against live Azure clusters
- `playground/AksDemo/` validated end-to-end against a live AKS cluster (multi-Gateway, AGC, cert-manager TLS)
- `tests/Aspire.Deployment.EndToEnd.Tests/AksAzureKubernetesEnvironmentGatewayDeploymentTests` — automated deployment test that provisions an AKS environment with AGC + Gateway, deploys an API, and verifies an HTTP 200 from `<gateway-fqdn>/weatherforecast`. Passes against live Azure (`westus3`, `Standard_D2as_v5`).
13 changes: 13 additions & 0 deletions playground/AksDemo/AksDemo.ApiService/AksDemo.ApiService.csproj
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
<Project Sdk="Microsoft.NET.Sdk.Web">

<PropertyGroup>
<TargetFramework>$(DefaultTargetFramework)</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
</PropertyGroup>

<ItemGroup>
<ProjectReference Include="..\..\Playground.ServiceDefaults\Playground.ServiceDefaults.csproj" />
</ItemGroup>

</Project>
52 changes: 52 additions & 0 deletions playground/AksDemo/AksDemo.ApiService/Program.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

var builder = WebApplication.CreateBuilder(args);

builder.AddServiceDefaults();

var app = builder.Build();

app.MapDefaultEndpoints();

// Simple endpoints to validate the Gateway -> HTTPRoute -> Service path through AGC.
// / - returns identifying info so it's obvious which pod handled the request
// /hello/{name} - echoes a name so route prefixes can be exercised
// /info - returns environment metadata useful for debugging

static object BuildIdentity(string surface) => new
{
service = "AksDemo.ApiService",
surface,
machineName = Environment.MachineName,
podIp = Environment.GetEnvironmentVariable("POD_IP"),
timestampUtc = DateTimeOffset.UtcNow
};

app.MapGet("/", () => Results.Ok(BuildIdentity("root")));

// The AKS gateways route /api -> storefront-gw and /admin -> admin-gw without
// rewriting the path, so the API needs endpoints under those exact prefixes
// for end-to-end smoke tests through AGC to return 200.
app.MapGet("/api", () => Results.Ok(BuildIdentity("storefront")));
app.MapGet("/admin", () => Results.Ok(BuildIdentity("admin")));

app.MapGet("/hello/{name}", (string name) => Results.Ok(new
{
message = $"Hello, {name}!",
machineName = Environment.MachineName
}));

app.MapGet("/info", () => Results.Ok(new
{
machineName = Environment.MachineName,
osVersion = Environment.OSVersion.ToString(),
processorCount = Environment.ProcessorCount,
dotnetVersion = Environment.Version.ToString(),
aspnetcoreEnv = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT"),
podIp = Environment.GetEnvironmentVariable("POD_IP"),
podName = Environment.GetEnvironmentVariable("POD_NAME"),
nodeName = Environment.GetEnvironmentVariable("NODE_NAME")
}));

app.Run();
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"$schema": "http://json.schemastore.org/launchsettings.json",
"profiles": {
"http": {
"commandName": "Project",
"dotnetRunMessages": true,
"launchBrowser": false,
"applicationUrl": "http://localhost:5197",
"environmentVariables": {
"ASPNETCORE_ENVIRONMENT": "Development"
}
}
}
}
9 changes: 9 additions & 0 deletions playground/AksDemo/AksDemo.ApiService/appsettings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft.AspNetCore": "Warning"
}
},
"AllowedHosts": "*"
}
21 changes: 21 additions & 0 deletions playground/AksDemo/AksDemo.AppHost/AksDemo.AppHost.csproj
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>$(DefaultTargetFramework)</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<IsAspireHost>true</IsAspireHost>
<UserSecretsId>3f6d6bf5-9d5e-45b1-86b3-629372b14c0e</UserSecretsId>
</PropertyGroup>

<ItemGroup>
<AspireProjectOrPackageReference Include="Aspire.Hosting.AppHost" />
<AspireProjectOrPackageReference Include="Aspire.Hosting.Azure" />
<AspireProjectOrPackageReference Include="Aspire.Hosting.Azure.Kubernetes" />
<AspireProjectOrPackageReference Include="Aspire.Hosting.Azure.Network" />

<ProjectReference Include="..\AksDemo.ApiService\AksDemo.ApiService.csproj" />
</ItemGroup>

</Project>
78 changes: 78 additions & 0 deletions playground/AksDemo/AksDemo.AppHost/AppHost.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

#pragma warning disable ASPIREAZURE003 // AddSubnet / AzureSubnetResource are evaluation-only

var builder = DistributedApplication.CreateBuilder(args);

// VNet layout:
// 10.100.0.0/16 - vnet (chosen to avoid the AKS default service CIDR 10.0.0.0/16)
// 10.100.0.0/22 - aks node pool subnet (1024 IPs - room for pods/nodes)
// 10.100.4.0/24 - public AGC frontend subnet (delegated to ServiceNetworking by AddLoadBalancer)
// 10.100.5.0/24 - admin AGC frontend subnet
//
// AGC requires the ALB frontend subnet to be /24 or larger and to be delegated to
// Microsoft.ServiceNetworking/trafficControllers. AddLoadBalancer applies the delegation
// for us; we just need to make sure the AKS subnet and ALB subnets do not overlap.
var vnet = builder.AddAzureVirtualNetwork("vnet", "10.100.0.0/16");
var aksSubnet = vnet.AddSubnet("aks-nodes", "10.100.0.0/22");
var publicSubnet = vnet.AddSubnet("alb-public", "10.100.4.0/24");
var adminSubnet = vnet.AddSubnet("alb-admin", "10.100.5.0/24");

var aks = builder.AddAzureKubernetesEnvironment("aks")
.WithSubnet(aksSubnet)
// Use the same AMD-based SKU as our AKS deployment E2E tests so this
// playground deploys consistently across regions and quotas.
.WithSystemNodePool("Standard_D2as_v5");

aks.AddNodePool("workload", "Standard_D2as_v5", minCount: 1, maxCount: 3);

// Two AGC ApplicationLoadBalancers. Each AGC ALB caps at 5 frontends, so production apps
// often need to spread Gateways/Ingresses across multiple LBs. This playground uses two
// just to exercise the multi-LB code path.
var publicLb = aks.AddLoadBalancer("public", publicSubnet);
var adminLb = aks.AddLoadBalancer("admin", adminSubnet);

var api = builder.AddProject<Projects.AksDemo_ApiService>("api")
.WithExternalHttpEndpoints();

// Public gateway: serves /api -> the api service, attached to the public AGC ALB.
// WithLoadBalancer attaches the alb.networking.azure.io association annotations and
// defaults the gatewayClassName to "azure-alb-external".
//
// WithTls() (no hostname) creates an HTTPS listener that gets its hostname patched in
// by Aspire's tls-fqdn-discovery pipeline step once AGC assigns the gateway its
// <random>.fz<n>.alb.azure.com FQDN. The cert-manager.io/cluster-issuer annotation
// then triggers cert-manager to issue a real Let's Encrypt cert via HTTP-01 against
// that FQDN. A `letsencrypt-prod` ClusterIssuer needs to exist in the cluster.
aks.AddGateway("storefront-gw")
.WithLoadBalancer(publicLb)
.WithRoute("/api", api.GetEndpoint("http"))
.WithTls()
.WithGatewayAnnotation("cert-manager.io/cluster-issuer", "letsencrypt-prod");

// Admin gateway: serves the same backend but on a separate AGC ALB so a different set of
// network policies, frontends, or DNS names can hang off it.
aks.AddGateway("admin-gw")
.WithLoadBalancer(adminLb)
.WithRoute("/admin", api.GetEndpoint("http"));

// cert-manager installed via Helm so we can issue Let's Encrypt certificates for the AGC
// gateways via the HTTP-01 challenge. Gateway API support is enabled so cert-manager will
// watch Gateway listeners for TLS configuration and auto-issue Certificates.
//
// WithForceConflicts is needed because AKS clusters with the Azure Policy add-on (or
// Deployment Safeguards) install an `admissionsenforcer` field manager that mutates the
// cert-manager ValidatingWebhookConfiguration after the first install. Helm's SSA then
// fails the next upgrade with a conflict on .webhooks[*].namespaceSelector.
// WithForceConflicts adds --force-conflicts which tells SSA to take over the conflicting
// field non-destructively (no resources recreated).
aks.AddHelmChart("cert-manager", "oci://quay.io/jetstack/charts/cert-manager", "v1.18.2")
.WithHelmValue("crds.enabled", "true")
.WithHelmValue("config.apiVersion", "controller.config.cert-manager.io/v1alpha1")
.WithHelmValue("config.kind", "ControllerConfiguration")
.WithHelmValue("config.enableGatewayAPI", "true")
.WithForceConflicts()
.WithDestroy();

builder.Build().Run();
44 changes: 44 additions & 0 deletions playground/AksDemo/AksDemo.AppHost/Properties/launchSettings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
{
"$schema": "http://json.schemastore.org/launchsettings.json",
"profiles": {
"https": {
"commandName": "Project",
"dotnetRunMessages": true,
"launchBrowser": true,
"applicationUrl": "https://localhost:16140;http://localhost:16141",
"environmentVariables": {
"ASPNETCORE_ENVIRONMENT": "Development",
"DOTNET_ENVIRONMENT": "Development",
"ASPIRE_DASHBOARD_OTLP_ENDPOINT_URL": "https://localhost:17060",
"ASPIRE_RESOURCE_SERVICE_ENDPOINT_URL": "https://localhost:17060",
"ASPIRE_SHOW_DASHBOARD_RESOURCES": "true"
}
},
"http": {
"commandName": "Project",
"dotnetRunMessages": true,
"launchBrowser": true,
"applicationUrl": "http://localhost:16141",
"environmentVariables": {
"ASPNETCORE_ENVIRONMENT": "Development",
"DOTNET_ENVIRONMENT": "Development",
"ASPIRE_DASHBOARD_OTLP_ENDPOINT_URL": "http://localhost:17061",
"ASPIRE_RESOURCE_SERVICE_ENDPOINT_URL": "http://localhost:17061",
"ASPIRE_SHOW_DASHBOARD_RESOURCES": "true",
"ASPIRE_ALLOW_UNSECURED_TRANSPORT": "true"
}
},
"generate-manifest": {
"commandName": "Project",
"launchBrowser": true,
"dotnetRunMessages": true,
"commandLineArgs": "--publisher manifest --output-path aspire-manifest.json",
"applicationUrl": "http://localhost:16141",
"environmentVariables": {
"ASPNETCORE_ENVIRONMENT": "Development",
"DOTNET_ENVIRONMENT": "Development",
"ASPIRE_DASHBOARD_OTLP_ENDPOINT_URL": "http://localhost:17061"
}
}
}
}
9 changes: 9 additions & 0 deletions playground/AksDemo/AksDemo.AppHost/appsettings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft.AspNetCore": "Warning",
"Aspire.Hosting.Dcp": "Warning"
}
}
}
5 changes: 5 additions & 0 deletions playground/AksDemo/aspire.config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"appHost": {
"path": "AksDemo.AppHost/AksDemo.AppHost.csproj"
}
}
Loading
Loading