Gateway TLS without hostname (FQDN discovery) + TS AppHost endpoint fix - #16551
Conversation
When WithTls() is called without WithHostname(), the Gateway now: 1. Generates an HTTPS listener without a hostname restriction 2. After Helm deploy, polls Gateway status for the assigned address 3. Patches the HTTPS listener to add the discovered hostname 4. Creates a bootstrap self-signed TLS secret with the discovered FQDN 5. cert-manager then detects the hostname and issues a real certificate This enables a single-deploy TLS workflow for controllers like AGC that assign FQDNs automatically (e.g., *.alb.azure.com), without requiring users to deploy once to discover the FQDN and then redeploy with it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
After patching the Gateway hostname via JSON patch, re-apply the full Gateway YAML with --server-side --field-manager=helm --force-conflicts to transfer field ownership back to Helm. This prevents SSA conflicts when the user later redeploys with an explicit hostname via WithHostname(). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 16551Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 16551" |
|
Sébastien Ros (@sebastienros) David Fowler (@davidfowl) — Heads up on a resource identity issue with the TS AppHost RPC bridge that I've worked around in this PR. The hack: Lines 653-668 and 872-890 in The problem: When Impact: This likely affects any deployment target that does resource lookups by reference — Docker Compose has the same Repro: TS AppHost with The real fix should be in the ATS bridge to preserve resource identity so that |
|
Repro AppHost (TypeScript): import { createBuilder } from "./.modules/aspire.js";
const builder = await createBuilder();
// This creates a NodeAppResource on the C# side and registers a handle in the ATS HandleRegistry.
// The "app" resource is added to the app model (appModel.GetComputeResources() will include it).
const app = await builder
.addNodeApp("app", "./api", "src/index.ts")
.withHttpEndpoint({ env: "PORT" })
.withExternalHttpEndpoints();
// At this point, "app" is a resolved NodeAppResource (the promise was awaited).
// Calling getEndpoint("http") sends an RPC call to C# which does:
// builder.Resource.GetEndpoint("http")
// This creates an EndpointReference whose .Resource property should be the same
// NodeAppResource instance that is in the app model. But it appears to be a
// DIFFERENT instance — possibly because the ATS handle resolution wraps or
// re-creates the resource when crossing the RPC boundary.
const endpoint = await app.getEndpoint("http");
const k8s = await builder.addKubernetesEnvironment("env");
// When this route is processed during publish, the code does:
// deploymentTargets.TryGetValue(endpointRef.Resource, out var k8sResource)
//
// deploymentTargets was populated from appModel.GetComputeResources() — keyed
// by the ORIGINAL resource instance. But endpointRef.Resource is a DIFFERENT
// instance (same name "app", different object reference), so TryGetValue returns
// false and the route is silently skipped. No HTTPRoute is generated.
await k8s.addGateway("ingress")
.withGatewayClass("azure-alb-external")
.withGatewayPathRoute("/", endpoint) // <-- endpoint.Resource != appModel resource
.withHostname("myapp.example.com")
.withGatewayTlsAuto();
await builder.build().run();The same AppHost works perfectly in C# because |
There was a problem hiding this comment.
Pull request overview
Adds Kubernetes Gateway TLS support for cases where the Gateway hostname isn’t known at publish time by generating hostless HTTPS listeners and introducing a post-deploy pipeline step that discovers the assigned address and patches the Gateway/listeners, plus a workaround for TypeScript AppHost endpoint resource identity mismatches.
Changes:
- Generate HTTPS Gateway listeners even when
WithTls()is used withoutWithHostname()(hostless listener) and add a new deploy pipeline step to discover and patch the listener hostname post-deploy. - Add name-based fallback matching when resolving
EndpointReference.Resourceto Kubernetes deployment targets (to unblock TS AppHost/Gateway route generation). - Add a new unit test asserting TLS-without-hostname produces an HTTPS listener without a
hostnamefield in the rendered Gateway YAML.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| tests/Aspire.Hosting.Kubernetes.Tests/KubernetesGatewayTests.cs | Adds coverage for generating an HTTPS listener when TLS is configured without an explicit hostname. |
| src/Aspire.Hosting.Kubernetes/KubernetesEnvironmentResource.cs | Implements hostless HTTPS listener generation, adds FQDN discovery + patching pipeline step, and introduces TS AppHost deployment-target lookup fallback. |
| string? discoveredFqdn = null; | ||
| var maxAttempts = 60; // 5 minutes with 5s intervals | ||
| for (var attempt = 0; attempt < maxAttempts; attempt++) | ||
| { | ||
| var jsonPathArgs = $"get gateway {gatewayName} --namespace {@namespace} -o jsonpath=\"{{.status.addresses[0].value}}\""; | ||
| if (environment.KubeConfigPath is not null) | ||
| { | ||
| jsonPathArgs += $" --kubeconfig \"{environment.KubeConfigPath}\""; | ||
| } |
There was a problem hiding this comment.
The Gateway address discovery reads only .status.addresses[0].value and treats it as an FQDN, but Gateway status addresses can be IPAddress as well as Hostname and ordering isn’t guaranteed. If an IP is returned here, patching it into spec.listeners[].hostname (and using it as the cert CN) will be invalid. Consider selecting the first address with type == "Hostname" (or validating the value as a DNS name) and skipping discovery with a clear warning when only IPs are available.
There was a problem hiding this comment.
My agent found this problem as well. 😄
| // Find HTTPS listeners without a hostname and patch them with the discovered FQDN. | ||
| // We query the Gateway spec to find the correct listener indices dynamically. | ||
| var listenerJsonArgs = $"get gateway {gatewayName} --namespace {@namespace} -o jsonpath=\"{{.spec.listeners}}\""; | ||
| if (environment.KubeConfigPath is not null) | ||
| { | ||
| listenerJsonArgs += $" --kubeconfig \"{environment.KubeConfigPath}\""; |
There was a problem hiding this comment.
kubectl ... -o jsonpath="{.spec.listeners}" output is not guaranteed to be valid JSON, but the code immediately feeds it to JsonDocument.Parse. This will likely throw for many clusters/controllers, and the fallback index assumption can patch the wrong listener. Prefer kubectl get gateway ... -o json and then parse spec.listeners from the JSON to reliably locate HTTPS listeners missing hostname.
| var patchOps = string.Join(",", httpsListenerIndices.Select( | ||
| idx => $"{{\"op\":\"add\",\"path\":\"/spec/listeners/{idx}/hostname\",\"value\":\"{discoveredFqdn}\"}}")); | ||
| var patchJson = $"[{patchOps}]"; | ||
|
|
||
| var patchArgs = $"patch gateway {gatewayName} --namespace {@namespace} --type=json -p=\"{patchJson.Replace("\"", "\\\"")}\""; | ||
| if (environment.KubeConfigPath is not null) | ||
| { | ||
| patchArgs += $" --kubeconfig \"{environment.KubeConfigPath}\""; | ||
| } | ||
|
|
||
| var (patchResult, patchDisposable) = ProcessUtil.Run(new ProcessSpec("kubectl") | ||
| { | ||
| Arguments = patchArgs, | ||
| ThrowOnNonZeroReturnCode = false, | ||
| InheritEnv = true, | ||
| OnOutputData = line => context.Logger.LogDebug("{Line}", line), | ||
| OnErrorData = line => context.Logger.LogDebug("{Line}", line) | ||
| }); | ||
|
|
||
| await using (patchDisposable.ConfigureAwait(false)) | ||
| { | ||
| var patchExitResult = await patchResult.WaitAsync(context.CancellationToken).ConfigureAwait(false); | ||
| if (patchExitResult.ExitCode != 0) | ||
| { | ||
| context.Logger.LogWarning( | ||
| "Failed to patch Gateway '{GatewayName}' with hostname '{Hostname}' (exit code {ExitCode}). " + | ||
| "You may need to redeploy with an explicit hostname via WithHostname().", | ||
| gatewayName, discoveredFqdn, patchExitResult.ExitCode); | ||
| continue; | ||
| } | ||
| } |
There was a problem hiding this comment.
The JSON patch payload is assembled via string interpolation with discoveredFqdn embedded directly. If the discovered value contains quotes/backslashes/whitespace (or is otherwise unexpected), this can break the JSON patch and can also alter the kubectl argument parsing. It would be safer to build the patch ops using JsonSerializer (so the hostname value is JSON-escaped) and avoid manual Replace-based escaping.
| var patchOps = string.Join(",", httpsListenerIndices.Select( | |
| idx => $"{{\"op\":\"add\",\"path\":\"/spec/listeners/{idx}/hostname\",\"value\":\"{discoveredFqdn}\"}}")); | |
| var patchJson = $"[{patchOps}]"; | |
| var patchArgs = $"patch gateway {gatewayName} --namespace {@namespace} --type=json -p=\"{patchJson.Replace("\"", "\\\"")}\""; | |
| if (environment.KubeConfigPath is not null) | |
| { | |
| patchArgs += $" --kubeconfig \"{environment.KubeConfigPath}\""; | |
| } | |
| var (patchResult, patchDisposable) = ProcessUtil.Run(new ProcessSpec("kubectl") | |
| { | |
| Arguments = patchArgs, | |
| ThrowOnNonZeroReturnCode = false, | |
| InheritEnv = true, | |
| OnOutputData = line => context.Logger.LogDebug("{Line}", line), | |
| OnErrorData = line => context.Logger.LogDebug("{Line}", line) | |
| }); | |
| await using (patchDisposable.ConfigureAwait(false)) | |
| { | |
| var patchExitResult = await patchResult.WaitAsync(context.CancellationToken).ConfigureAwait(false); | |
| if (patchExitResult.ExitCode != 0) | |
| { | |
| context.Logger.LogWarning( | |
| "Failed to patch Gateway '{GatewayName}' with hostname '{Hostname}' (exit code {ExitCode}). " + | |
| "You may need to redeploy with an explicit hostname via WithHostname().", | |
| gatewayName, discoveredFqdn, patchExitResult.ExitCode); | |
| continue; | |
| } | |
| } | |
| var patchOperations = httpsListenerIndices.Select(idx => new | |
| { | |
| op = "add", | |
| path = $"/spec/listeners/{idx}/hostname", | |
| value = discoveredFqdn | |
| }); | |
| var patchJson = System.Text.Json.JsonSerializer.Serialize(patchOperations); | |
| var patchFilePath = System.IO.Path.GetTempFileName(); | |
| await System.IO.File.WriteAllTextAsync(patchFilePath, patchJson, context.CancellationToken).ConfigureAwait(false); | |
| try | |
| { | |
| var patchArgs = $"patch gateway {gatewayName} --namespace {@namespace} --type=json --patch-file \"{patchFilePath}\""; | |
| if (environment.KubeConfigPath is not null) | |
| { | |
| patchArgs += $" --kubeconfig \"{environment.KubeConfigPath}\""; | |
| } | |
| var (patchResult, patchDisposable) = ProcessUtil.Run(new ProcessSpec("kubectl") | |
| { | |
| Arguments = patchArgs, | |
| ThrowOnNonZeroReturnCode = false, | |
| InheritEnv = true, | |
| OnOutputData = line => context.Logger.LogDebug("{Line}", line), | |
| OnErrorData = line => context.Logger.LogDebug("{Line}", line) | |
| }); | |
| await using (patchDisposable.ConfigureAwait(false)) | |
| { | |
| var patchExitResult = await patchResult.WaitAsync(context.CancellationToken).ConfigureAwait(false); | |
| if (patchExitResult.ExitCode != 0) | |
| { | |
| context.Logger.LogWarning( | |
| "Failed to patch Gateway '{GatewayName}' with hostname '{Hostname}' (exit code {ExitCode}). " + | |
| "You may need to redeploy with an explicit hostname via WithHostname().", | |
| gatewayName, discoveredFqdn, patchExitResult.ExitCode); | |
| continue; | |
| } | |
| } | |
| } | |
| finally | |
| { | |
| try | |
| { | |
| System.IO.File.Delete(patchFilePath); | |
| } | |
| catch (System.IO.IOException) | |
| { | |
| } | |
| catch (UnauthorizedAccessException) | |
| { | |
| } | |
| } | |
| } |
| // Transfer field ownership to Helm using server-side apply so subsequent | ||
| // Helm deploys don't encounter SSA conflicts on the patched hostname field. | ||
| var currentGatewayArgs = $"get gateway {gatewayName} --namespace {@namespace} -o yaml"; | ||
| if (environment.KubeConfigPath is not null) | ||
| { | ||
| currentGatewayArgs += $" --kubeconfig \"{environment.KubeConfigPath}\""; | ||
| } | ||
|
|
||
| var gatewayYamlLines = new List<string>(); | ||
| var (getGwResult, getGwDisposable) = ProcessUtil.Run(new ProcessSpec("kubectl") | ||
| { | ||
| Arguments = currentGatewayArgs, | ||
| ThrowOnNonZeroReturnCode = false, |
There was a problem hiding this comment.
Re-applying the output of kubectl get gateway ... -o yaml risks including server-populated/read-only fields (e.g., status, metadata.resourceVersion, metadata.managedFields). Depending on kubectl/version and CRD schema, kubectl apply can fail or behave unpredictably. Consider generating a minimal manifest containing only apiVersion/kind/metadata(name,namespace)/spec (or re-applying the Helm-rendered YAML) before doing server-side apply for field ownership transfer.
| { | ||
| var tempFile = Path.GetTempFileName(); | ||
| try |
There was a problem hiding this comment.
Temp file creation uses Path.GetTempFileName(). In this repo, prefer creating temp files under a securely created temp directory (e.g., Directory.CreateTempSubdirectory() and then Path.Combine) rather than using GetTempFileName, to align with the temporary directory guidance and avoid leaving pre-created temp files behind on failures.
| context.Logger.LogInformation("Creating bootstrap TLS secret '{SecretName}' for '{Hostname}'.", secretName, discoveredFqdn); | ||
|
|
||
| using var ecdsa = ECDsa.Create(ECCurve.NamedCurves.nistP256); | ||
| var certRequest = new CertificateRequest($"CN={discoveredFqdn}", ecdsa, HashAlgorithmName.SHA256); |
There was a problem hiding this comment.
The bootstrap self-signed cert is created with only a CN. Modern TLS clients generally require the hostname to be present in the Subject Alternative Name (SAN) extension; CN-only certs can be rejected. Consider adding a SAN DNS name extension for the discovered hostname so the temporary certificate is usable during the window before cert-manager replaces it.
| var certRequest = new CertificateRequest($"CN={discoveredFqdn}", ecdsa, HashAlgorithmName.SHA256); | |
| var certRequest = new CertificateRequest($"CN={discoveredFqdn}", ecdsa, HashAlgorithmName.SHA256); | |
| var subjectAlternativeNameBuilder = new SubjectAlternativeNameBuilder(); | |
| subjectAlternativeNameBuilder.AddDnsName(discoveredFqdn); | |
| certRequest.CertificateExtensions.Add(subjectAlternativeNameBuilder.Build()); |
Use ResourceNameComparer on the deploymentTargets dictionary so that endpoint references created through the TypeScript AppHost RPC bridge (which may use a different resource instance) resolve correctly by resource name. This matches the pattern already used in KubernetesEnvironmentContext._kubernetesComponents. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
5cd9269 to
0aba6af
Compare
The YAML serializer quotes string values like protocol: "HTTPS" on CI. Use a more flexible assertion that matches both quoted and unquoted forms. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Re-running the failed jobs in the CI workflow for this pull request because 1 job was identified as retry-safe transient failures in the CI run attempt.
|
1. Address type validation: Parse full Gateway JSON status, prefer Hostname-type addresses, fall back to DNS-like values, skip IPs. 2. Use -o json instead of jsonpath: Parse full Gateway JSON for both address discovery and listener index detection. More reliable. 3. Use JsonSerializer + --patch-file: Build JSON patch operations with proper serialization, write to temp file to avoid shell escaping. 4. Minimal manifest for field ownership: Read current Gateway JSON, strip server fields (status, resourceVersion, managedFields), keep only apiVersion/kind/metadata(name,namespace)/spec for SSA apply. 5. Temp file pattern: Use CreateTempSubdirectory consistently. 6. Add SAN to bootstrap certs: Add SubjectAlternativeNameBuilder with DNS name in both DiscoverFqdnAndBootstrapTlsAsync and the existing BootstrapTlsSecretsAsync. Modern TLS clients require SAN. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The minimal manifest used for server-side apply field ownership transfer was missing annotations and labels, causing AGC annotations like alb.networking.azure.io/alb-name to be stripped. Now copies annotations and labels from the current Gateway metadata. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
All automated review feedback addressed (commit ba38721 + 070bc04)1. Address type validation (comment on line 988) 2. jsonpath output reliability (comment on line 1038) 3. JSON patch via string interpolation (comment on line 1132) 4. Full YAML re-apply for field ownership (comment on line 1146) 5. Temp file pattern (comment on line 1158) 6. Bootstrap cert missing SAN (comment on line 1227) |
|
/deployment-test |
|
🚀 Deployment tests starting on PR #16551... This will deploy to real Azure infrastructure. Results will be posted here when complete. |
…ncer Per the official AGC quickstart docs, use the correct flags: - --enable-gateway-api: enables Gateway API CRDs - --enable-application-load-balancer: enables ALB controller addon - --network-plugin azure: required Azure CNI - Standard_D2as_v5: AMD VM SKU for quota compatibility - Use the add-on's auto-created aks-appgateway subnet instead of creating one manually Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
/deployment-test |
|
🚀 Deployment tests starting on PR #16551... This will deploy to real Azure infrastructure. Results will be posted here when complete. |
The ALB controller pods need time to initialize after cluster creation. Poll until pods are Running and GatewayClass azure-alb-external exists, with up to 10 minutes timeout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
/deployment-test |
|
🚀 Deployment tests starting on PR #16551... This will deploy to real Azure infrastructure. Results will be posted here when complete. |
cert-manager requires parentRefs to include a name. Use 'ingress' to
match the Gateway name from AddGateway('ingress'), and include the
namespace to match the Helm deploy namespace.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
/deployment-test |
|
🚀 Deployment tests starting on PR #16551... This will deploy to real Azure infrastructure. Results will be posted here when complete. |
The injected AppHost code needs: - using Aspire.Hosting.Kubernetes for AddGateway extension methods - #pragma warning disable ASPIRECOMPUTE003 for AddContainerRegistry Both prepended to the top of the file alongside ASPIREPIPELINES001. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
/deployment-test |
|
🚀 Deployment tests starting on PR #16551... This will deploy to real Azure infrastructure. Results will be posted here when complete. |
The starter template generates builder.AddProject('webfrontend') without
assigning to a variable. The Gateway route needs a reference to it, so
inject 'var webfrontend =' before the AddProject call.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
/deployment-test |
|
🚀 Deployment tests starting on PR #16551... This will deploy to real Azure infrastructure. Results will be posted here when complete. |
|
|
||
| string? discoveredFqdn = null; | ||
| var maxAttempts = 60; // 5 minutes with 5s intervals | ||
| for (var attempt = 0; attempt < maxAttempts; attempt++) |
There was a problem hiding this comment.
Would using Polly help here?
There was a problem hiding this comment.
Good idea. Polly isn't currently referenced in Aspire.Hosting.Kubernetes so it would be a new dependency for this one retry loop. Happy to refactor if you think it's worth the dependency, or we could use the Microsoft.Extensions.Resilience helpers if those are available transitively. For now the manual loop matches the pattern used elsewhere in this file (e.g. the ALB ready polling in the E2E test).
There was a problem hiding this comment.
Aspire.Hosting already references Polly.Core.
We don't need this. If you are happy with a loop - I just figured I'd ask.
Replace the manual for-loop retry with a Polly ResiliencePipeline using constant 5s backoff, 60 max attempts, and result-based retry (retries when result is null). Polly.Core is already a transitive dependency via Aspire.Hosting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Re-running the failed jobs in the CI workflow for this pull request because 1 job was identified as retry-safe transient failures in the CI run attempt.
|
|
🎬 CLI E2E Test Recordings — 76 recordings uploaded (commit View all recordings
📹 Recordings uploaded automatically from CI run #25140333612 |
|
/deployment-test |
|
🚀 Deployment tests starting on PR #16551... This will deploy to real Azure infrastructure. Results will be posted here when complete. |
|
✅ Deployment E2E Tests passed — 33 passed, 0 failed, 0 cancelled View test results and recordings
|
|
/backport to release/13.3 |
|
Started backporting to |
|
Pull request created: #782
|
|
A draft documentation PR has been opened on microsoft/aspire.dev targeting the Branch: What was documentedUpdated
The draft PR needs human review before merging.
|
Summary
This PR adds support for Gateway TLS without a pre-known hostname and fixes endpoint resource lookup for TypeScript AppHosts.
1. Gateway TLS without pre-known hostname (FQDN auto-discovery)
When
WithTls()is called withoutWithHostname(), the Gateway now:tls-fqdn-discoverypipeline step pollskubectl get gatewayfor the assigned address (up to 5 min)This enables single-deploy TLS for controllers like AGC that assign FQDNs automatically (e.g.,
*.alb.azure.com).2. Helm field manager conflict fix
After patching the Gateway hostname, re-applies a minimal Gateway manifest (apiVersion, kind, metadata with annotations/labels, spec — no server fields) with
--server-side --field-manager=helm --force-conflictsto transfer field ownership back to Helm. This prevents SSA conflicts when the user later redeploys with an explicit hostname viaWithHostname().3. TypeScript AppHost endpoint resource lookup fix
Uses
ResourceNameCompareron thedeploymentTargetsdictionary so that endpoint references created through the TypeScript AppHost RPC bridge (which may use a different resource instance) resolve correctly by resource name. This matches the pattern already used inKubernetesEnvironmentContext._kubernetesComponents.Testing
AddGateway_WithTls_NoHostname_GeneratesHttpsListenerWithoutHostname