Skip to content

OCPBUGS-100184: Wait for Azure ignition DNS before creating workers - #9172

Merged
openshift-merge-bot[bot] merged 4 commits into
openshift:mainfrom
stbenjam:agent/gate-azure-nodepools-on-ignition-dns
Aug 4, 2026
Merged

OCPBUGS-100184: Wait for Azure ignition DNS before creating workers#9172
openshift-merge-bot[bot] merged 4 commits into
openshift:mainfrom
stbenjam:agent/gate-azure-nodepools-on-ignition-dns

Conversation

@stbenjam

@stbenjam stbenjam commented Jul 29, 2026

Copy link
Copy Markdown
Member

These bugs and fixes were automatically generated by a payload-agent experiment to improve resilience and diagnostics for infrastructure failures. Please review the PR and either shepherd it to merge or close it. If the work is incorrect or unhelpful, a brief comment would help us improve. Thanks, and apologies if we missed the mark.

Bug: OCPBUGS-100184

What this PR does / why we need it:

On public Azure HostedClusters with an explicit ignition DNS hostname, wait for that hostname to resolve before continuing NodePool reconciliation into CAPI machine creation.

Azure starts a finite VM provisioning deadline when the machine is created. In this AKS conformance run, Azure DNS API throttling delayed the ignition A record until after that deadline, leading to OSProvisioningTimedOut. The NodePool now reports IgnitionEndpointAvailable=False with reason ExternalDNSHostNotReachable, requeues after ten seconds, and automatically proceeds when DNS becomes available.

The gate is limited to Azure and to explicit public external-DNS ignition hostnames. Private Azure and other platform paths are unchanged.

Which issue(s) this PR fixes:

Fixes: OCPBUGS-100184

Special notes for your reviewer:

The resolver is injected only to keep the condition logic deterministic in unit tests. Tests cover unresolvable and resolvable Azure hostnames, private Azure, non-Azure, missing external DNS configuration, and a missing ignition endpoint.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs. (No documentation change is needed.)
  • This change includes unit tests.

Summary by CodeRabbit

  • Bug Fixes
    • Improved Azure node pool validation by checking whether external Ignition endpoint hostnames are DNS-resolvable.
    • Node pools now report a clear availability condition and retry automatically when the Ignition endpoint hostname cannot be reached.
    • Existing Ignition certificate validation continues after successful DNS resolution.
    • Non-Azure node pools and Azure configurations without an external hostname retain their existing behavior.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 29, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@stbenjam: This pull request explicitly references no jira issue.

Details

In response to this:

What this PR does / why we need it:

On public Azure HostedClusters with an explicit ignition DNS hostname, wait for that hostname to resolve before continuing NodePool reconciliation into CAPI machine creation.

Azure starts a finite VM provisioning deadline when the machine is created. In this AKS conformance run, Azure DNS API throttling delayed the ignition A record until after that deadline, leading to OSProvisioningTimedOut. The NodePool now reports IgnitionEndpointAvailable=False with reason ExternalDNSHostNotReachable, requeues after ten seconds, and automatically proceeds when DNS becomes available.

The gate is limited to Azure and to explicit public external-DNS ignition hostnames. Private Azure and other platform paths are unchanged.

Which issue(s) this PR fixes:

Fixes: N/A (NO-JIRA; observed in the linked payload job)

Special notes for your reviewer:

The resolver is injected only to keep the condition logic deterministic in unit tests. Tests cover unresolvable and resolvable Azure hostnames, private Azure, non-Azure, missing external DNS configuration, and a missing ignition endpoint.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced. (NO-JIRA; failing CI is linked above.)
  • This change includes docs. (No documentation change is needed.)
  • This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 29, 2026
@openshift-ci

openshift-ci Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 30eaeccc-f460-4f50-8d54-ef2ea30f8b04

📥 Commits

Reviewing files that changed from the base of the PR and between 9772cd1 and c201e41.

📒 Files selected for processing (3)
  • hypershift-operator/controllers/nodepool/conditions.go
  • hypershift-operator/controllers/nodepool/conditions_test.go
  • hypershift-operator/controllers/nodepool/nodepool_controller.go
🚧 Files skipped from review as they are similar to previous changes (3)
  • hypershift-operator/controllers/nodepool/nodepool_controller.go
  • hypershift-operator/controllers/nodepool/conditions.go
  • hypershift-operator/controllers/nodepool/conditions_test.go

📝 Walkthrough

Walkthrough

The NodePool reconciler now checks Azure Ignition external DNS resolution before Ignition CA secret validation. The resolver can be injected for tests. Resolution failures set IgnitionEndpointAvailable=False and requeue reconciliation after 10 seconds. Tests cover platform type, endpoint configuration, resolution results, condition details, and requeue behavior.

Sequence Diagram(s)

sequenceDiagram
  participant NodePoolReconciler
  participant ignitionEndpointAvailableCondition
  participant DNSResolver
  NodePoolReconciler->>ignitionEndpointAvailableCondition: evaluate endpoint availability
  ignitionEndpointAvailableCondition->>DNSResolver: resolve Azure external DNS hostname
  DNSResolver-->>ignitionEndpointAvailableCondition: resolution result
  ignitionEndpointAvailableCondition-->>NodePoolReconciler: condition status and optional 10-second requeue
Loading

Suggested reviewers: devguyio, sdminonne

🚥 Pre-merge checks | ✅ 11
✅ Passed checks (11 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: waiting for Azure Ignition DNS resolution before creating workers.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The added test uses only literal, static t.Run names and no Ginkgo title calls; names contain no generated identifiers, timestamps, namespaces, IPs, or runtime interpolation.
Test Structure And Quality ✅ Passed The added test uses focused table subtests, in-memory fake objects, and no cluster waits; no timeout is needed, and the error assertion has diagnostic context.
Topology-Aware Scheduling Compatibility ✅ Passed The branch changes only NodePool condition logic, resolver injection, and tests; it adds no pod affinity, topology spread, selectors, tolerations, replica, or PDB scheduling constraints.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The added test is a standard Go TestIgnitionEndpointAvailableCondition unit test, not a Ginkgo e2e test; DNS calls use an injected fake resolver and no external connectivity.
No-Weak-Crypto ✅ Passed The complete PR diff adds DNS resolution, condition handling, and tests only; it contains no weak-crypto APIs, custom cryptography, or secret/token comparisons.
Container-Privileges ✅ Passed The PR changes only Go controller and test files; the added diff contains no privileged, host namespace, SYS_ADMIN, allowPrivilegeEscalation, or root execution settings.
No-Sensitive-Data-In-Logs ✅ Passed The DNS gate logs only a constant wait message; it does not log the hostname, resolver error, credentials, tokens, or customer data. The hostname appears only in a NodePool status condition.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot added area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release and removed do-not-merge/needs-area labels Jul 29, 2026
@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.33333% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 44.91%. Comparing base (b6a286e) to head (c201e41).
⚠️ Report is 28 commits behind head on main.

Files with missing lines Patch % Lines
...rshift-operator/controllers/nodepool/conditions.go 83.33% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9172      +/-   ##
==========================================
+ Coverage   44.90%   44.91%   +0.01%     
==========================================
  Files         775      775              
  Lines       97267    97285      +18     
==========================================
+ Hits        43673    43696      +23     
+ Misses      50581    50573       -8     
- Partials     3013     3016       +3     
Files with missing lines Coverage Δ
...erator/controllers/nodepool/nodepool_controller.go 43.47% <ø> (ø)
...rshift-operator/controllers/nodepool/conditions.go 59.88% <83.33%> (+1.40%) ⬆️
Flag Coverage Δ
cmd-support 38.42% <ø> (ø)
cpo-hostedcontrolplane 47.32% <ø> (ø)
cpo-other 45.67% <ø> (ø)
hypershift-operator 54.98% <83.33%> (+0.05%) ⬆️
other 34.32% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@stbenjam stbenjam changed the title NO-JIRA: Wait for Azure ignition DNS before creating workers OCPBUGS-100184: Wait for Azure ignition DNS before creating workers Jul 29, 2026
@openshift-ci-robot openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Jul 29, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@stbenjam: This pull request references Jira Issue OCPBUGS-100184, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

These bugs and fixes were automatically generated by a payload-agent experiment to improve resilience and diagnostics for infrastructure failures. Please review the PR and either shepherd it to merge or close it. If the work is incorrect or unhelpful, a brief comment would help us improve. Thanks, and apologies if we missed the mark.

Bug: OCPBUGS-100184

What this PR does / why we need it:

On public Azure HostedClusters with an explicit ignition DNS hostname, wait for that hostname to resolve before continuing NodePool reconciliation into CAPI machine creation.

Azure starts a finite VM provisioning deadline when the machine is created. In this AKS conformance run, Azure DNS API throttling delayed the ignition A record until after that deadline, leading to OSProvisioningTimedOut. The NodePool now reports IgnitionEndpointAvailable=False with reason ExternalDNSHostNotReachable, requeues after ten seconds, and automatically proceeds when DNS becomes available.

The gate is limited to Azure and to explicit public external-DNS ignition hostnames. Private Azure and other platform paths are unchanged.

Which issue(s) this PR fixes:

Fixes: OCPBUGS-100184

Special notes for your reviewer:

The resolver is injected only to keep the condition logic deterministic in unit tests. Tests cover unresolvable and resolvable Azure hostnames, private Azure, non-Azure, missing external DNS configuration, and a missing ignition endpoint.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs. (No documentation change is needed.)
  • This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@stbenjam
stbenjam marked this pull request as ready for review July 29, 2026 20:22
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 29, 2026
@openshift-ci
openshift-ci Bot requested review from devguyio and sdminonne July 29, 2026 20:24
Prevent Azure machines from starting their provisioning deadline before the public ignition hostname resolves.

Surface DNS readiness on the NodePool and retry automatically.

Keep failure details in NodePool status without logging the customer hostname.

Signed-off-by: Stephen Benjamin <stephen@redhat.com>
@stbenjam
stbenjam force-pushed the agent/gate-azure-nodepools-on-ignition-dns branch from 7c6daf4 to 21fc24b Compare July 29, 2026 20:29
@stbenjam

Copy link
Copy Markdown
Member Author

/pipeline required

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke
/test unit
/test verify

@stbenjam

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@openshift-ci-robot

Copy link
Copy Markdown

@stbenjam: This pull request references Jira Issue OCPBUGS-100184, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

These bugs and fixes were automatically generated by a payload-agent experiment to improve resilience and diagnostics for infrastructure failures. Please review the PR and either shepherd it to merge or close it. If the work is incorrect or unhelpful, a brief comment would help us improve. Thanks, and apologies if we missed the mark.

Bug: OCPBUGS-100184

What this PR does / why we need it:

On public Azure HostedClusters with an explicit ignition DNS hostname, wait for that hostname to resolve before continuing NodePool reconciliation into CAPI machine creation.

Azure starts a finite VM provisioning deadline when the machine is created. In this AKS conformance run, Azure DNS API throttling delayed the ignition A record until after that deadline, leading to OSProvisioningTimedOut. The NodePool now reports IgnitionEndpointAvailable=False with reason ExternalDNSHostNotReachable, requeues after ten seconds, and automatically proceeds when DNS becomes available.

The gate is limited to Azure and to explicit public external-DNS ignition hostnames. Private Azure and other platform paths are unchanged.

Which issue(s) this PR fixes:

Fixes: OCPBUGS-100184

Special notes for your reviewer:

The resolver is injected only to keep the condition logic deterministic in unit tests. Tests cover unresolvable and resolvable Azure hostnames, private Azure, non-Azure, missing external DNS configuration, and a missing ignition endpoint.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs. (No documentation change is needed.)
  • This change includes unit tests.

Summary by CodeRabbit

  • Bug Fixes
  • Improved Azure node pool validation by checking whether external Ignition endpoint hostnames are DNS-resolvable.
  • Node pools now report a clear availability condition and retry automatically when the Ignition endpoint hostname cannot be reached.
  • Existing Ignition certificate validation continues after successful DNS resolution.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@redhat-chai-bot

redhat-chai-bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

TRT Review — Payload Infrastructure Resilience

Assessment: High value. This addresses the primary root cause of the AKS conformance failures identified by the payload agent analysis.

Context from payload agent analysis

The payload agent classified the aggregated-hypershift-aks-conformance-5.0 failure as infrastructure (failure_type: "infra") with this precise causal chain:

"Azure DNS API 429 throttling delayed ignition A-record creation past the VM provisioning deadline, causing OSProvisioningTimedOut; Azure Disk CSI recovery PUTs then failed LocationRequired and blocked PVC attachments."

This PR breaks the causal chain at the first link — the race between DNS propagation and VM provisioning.

Code review

The implementation is well-scoped and follows HyperShift patterns:

  1. DNS readiness gate — Before the NodePool reconciler clears IgnitionEndpointAvailable, it now resolves the external DNS hostname for Azure public clusters. If DNS isn't ready, it sets a clear condition (ExternalDNSHostNotReachable), logs the error, and requeues after 10 seconds. This prevents CAPI from creating VMs that will immediately timeout waiting for an ignition endpoint that doesn't exist in DNS yet.

  2. Platform scoping — The gate is correctly limited to AzurePlatform with an explicit external DNS hostname. Private Azure topologies and all other platforms are untouched. This is important — DNS readiness is specifically an Azure-ExternalDNS timing issue, not a general problem.

  3. Testability — The resolveDNSHostname function is injected via the reconciler struct, making unit tests deterministic. The test matrix covers six cases: unresolvable Azure, resolvable Azure, private Azure (skipped), non-Azure (skipped), missing external DNS config (skipped), and missing ignition endpoint.

  4. netutil.ServiceExternalDNSHostnameByHC — Uses the existing utility to extract the hostname, which is the right approach rather than parsing the service publishing strategy inline.

CodeRabbit's sensitive-data flag

CodeRabbit flagged the log line "Ignition endpoint DNS hostname is not resolvable, waiting" as potentially exposing customer hostnames. In the context of HyperShift operator logs (management cluster only, not guest-visible), logging the hostname and error is standard practice and essential for debugging Azure DNS issues. This is not a customer-facing log surface.

Impact

This addresses the initiating cause in the payload agent's failure chain. Without this fix, Azure DNS throttling creates a race: VMs are provisioned before their ignition endpoint is resolvable, leading to OSProvisioningTimedOut. The downstream effects (failed VM states, CSI LocationRequired errors) are then addressed by the companion cloud-provider-azure PR #199.

Together, these two PRs form a defense-in-depth pair: this one prevents the bad state from occurring, and #199 makes recovery work when it does.

Recommendation: Merge-worthy. Clear, well-tested fix for a documented Azure infrastructure timing issue. The HyperShift team should review the interaction with their reconciler requeue patterns, but the approach is sound.

@openshift openshift deleted a comment from redhat-chai-bot Jul 29, 2026
@cwbotbot

cwbotbot commented Jul 29, 2026

Copy link
Copy Markdown

Test Results

e2e-aws

e2e-aks

@redhat-chai-bot

Copy link
Copy Markdown
Contributor

/retest-required

CI failures are Azure/AWS infrastructure issues (TLS handshake timeouts, EOF, DNS no such host), unrelated to the NodePool DNS readiness gate changes in this PR.

@redhat-chai-bot

Copy link
Copy Markdown
Contributor

/retest-required

@redhat-chai-bot

redhat-chai-bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Question: Should the DNS gate narrow to ARO HCP instead of all Azure?

A question came up during review — this PR gates on hcluster.Spec.Platform.Type == hyperv1.AzurePlatform, but ARO HCP and self-managed Azure are architecturally quite different. Should the scope be tightened to ARO HCP specifically?

How the gate scopes today

The check calls ServiceExternalDNSHostnameByHC(hcluster, Ignition), which filters on:

  1. IsPublicHC(hc) — for Azure, returns true when topology is Public, PublicAndPrivate, or empty (default). This matches both ARO HCP and self-managed Azure if they're public.
  2. Explicit Route.Hostname or LoadBalancer.Hostname in the service publishing strategy — returns empty string if no explicit external DNS hostname is configured.

Effective behavior:

Scenario DNS check fires?
ARO HCP (public, with external DNS) ✅ Yes (intended)
Self-managed Azure (public, with external DNS) ✅ Yes (possibly unintended)
Self-managed Azure (public, no external DNS hostname) ❌ No — ServiceExternalDNSHostnameByHC returns ""
Private Azure (either flavor) ❌ No — IsPublicHC returns false

Suggestion

There's an existing IsAroHCPByHC(hc) utility in support/netutil/visibility.go that distinguishes ARO HCP from self-managed Azure by checking AzureAuthenticationConfigType == ManagedIdentities. The gate could be narrowed to:

if netutil.IsAroHCPByHC(hcluster) {

Arguments for narrowing:

  • The observed failure was specifically in ARO HCP (AKS conformance run). ARO HCP always deploys external DNS for ignition; self-managed Azure typically doesn't.
  • ARO HCP and self-managed Azure have different DNS architectures. A self-managed user who configures external DNS might have different propagation characteristics where a 10s fixed requeue is inappropriate.
  • Reduces risk of unintended side effects for the self-managed Azure path.

Arguments for keeping AzurePlatform (current scope):

  • The root cause is Azure DNS API throttling (429s) — that affects all Azure DNS, not just ARO HCP. A self-managed Azure cluster with external DNS would benefit from the same protection.
  • The ServiceExternalDNSHostnameByHC filter already prevents any behavior change for self-managed clusters without explicit external DNS hostnames, which is the common case.
  • Broader scope is more defensive — if self-managed Azure ever hits the same 429 throttling, the fix is already in place.

Bottom line

The ServiceExternalDNSHostnameByHC filter does most of the scoping work — self-managed Azure without external DNS is unaffected regardless. But if the intent is specifically to fix an ARO HCP problem with ARO HCP DNS infrastructure, narrowing to IsAroHCPByHC would be more precise. If the broader AzurePlatform scope is intentional for defense-in-depth, documenting that reasoning here would be helpful.


Posted on behalf of a reviewer from the #forum-ocp-hypershift discussion.

@stbenjam

Copy link
Copy Markdown
Member Author

Thanks for the thorough analysis — great question.

I think keeping the broader AzurePlatform scope is the right call here, and ServiceExternalDNSHostnameByHC is already doing the heavy lifting for scoping:

  1. IsPublicHC(hc) — private Azure is excluded regardless
  2. Explicit hostname requiredServiceExternalDNSHostnameByHC returns "" (and the check is skipped) when no LoadBalancer.Hostname or Route.Hostname is configured, which is the common case for self-managed Azure

So self-managed Azure clusters without external DNS are already completely unaffected — the only clusters that hit this code path are ones that have explicitly configured an external DNS hostname for ignition, on a public Azure topology.

The root cause is Azure DNS API throttling (429s on the Azure DNS API), which is not ARO HCP-specific — it affects any Azure DNS zone. A self-managed Azure cluster that configures external DNS for ignition would be equally vulnerable to this race, so the broader scope provides defense-in-depth at no cost to the unaffected path.

Happy to add an inline comment explaining this rationale if that would be helpful.

@stbenjam

Copy link
Copy Markdown
Member Author

Added an inline comment explaining the broader AzurePlatform scope rationale in bc0901e. Thanks for the suggestion — should be clearer for future readers now.

Address review: document why the DNS readiness gate uses AzurePlatform
rather than narrowing to ARO HCP. Azure DNS API throttling (429s)
affects all Azure DNS zones, and ServiceExternalDNSHostnameByHC already
scopes the check to public clusters with an explicit external DNS
hostname.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@stbenjam
stbenjam force-pushed the agent/gate-azure-nodepools-on-ignition-dns branch from bc0901e to 9772cd1 Compare July 31, 2026 14:37
@stbenjam

Copy link
Copy Markdown
Member Author

Addressed the Test Structure And Quality warning in c201e41 by adding a diagnostic message to the ignitionEndpointAvailableCondition error assertion.

@bryan-cox bryan-cox left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

@openshift-ci

openshift-ci Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bryan-cox, stbenjam

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 3, 2026
@Nirshal

Nirshal commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Aug 3, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-azure-self-managed
/test e2e-v2-gke

@stbenjam

stbenjam commented Aug 3, 2026

Copy link
Copy Markdown
Member Author

/verified by CI

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Aug 3, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@stbenjam: This PR has been marked as verified by CI.

Details

In response to this:

/verified by CI

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@bryan-cox

Copy link
Copy Markdown
Member

/retest

@bryan-cox

Copy link
Copy Markdown
Member

/retest-required

@bryan-cox

Copy link
Copy Markdown
Member

/test e2e-kubevirt-aws-ovn-reduced

@bryan-cox

Copy link
Copy Markdown
Member

/override "ci/prow/e2e-kubevirt-aws-ovn-reduced"

This was failing due to an outside reason outside this PR

@openshift-ci

openshift-ci Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

@bryan-cox: Overrode contexts on behalf of bryan-cox: ci/prow/e2e-kubevirt-aws-ovn-reduced

Details

In response to this:

/override "ci/prow/e2e-kubevirt-aws-ovn-reduced"

This was failing due to an outside reason outside this PR

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-merge-bot
openshift-merge-bot Bot merged commit 0cd0254 into openshift:main Aug 4, 2026
43 checks passed
@openshift-ci-robot

Copy link
Copy Markdown

@stbenjam: Jira Issue Verification Checks: Jira Issue OCPBUGS-100184
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-100184 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

These bugs and fixes were automatically generated by a payload-agent experiment to improve resilience and diagnostics for infrastructure failures. Please review the PR and either shepherd it to merge or close it. If the work is incorrect or unhelpful, a brief comment would help us improve. Thanks, and apologies if we missed the mark.

Bug: OCPBUGS-100184

What this PR does / why we need it:

On public Azure HostedClusters with an explicit ignition DNS hostname, wait for that hostname to resolve before continuing NodePool reconciliation into CAPI machine creation.

Azure starts a finite VM provisioning deadline when the machine is created. In this AKS conformance run, Azure DNS API throttling delayed the ignition A record until after that deadline, leading to OSProvisioningTimedOut. The NodePool now reports IgnitionEndpointAvailable=False with reason ExternalDNSHostNotReachable, requeues after ten seconds, and automatically proceeds when DNS becomes available.

The gate is limited to Azure and to explicit public external-DNS ignition hostnames. Private Azure and other platform paths are unchanged.

Which issue(s) this PR fixes:

Fixes: OCPBUGS-100184

Special notes for your reviewer:

The resolver is injected only to keep the condition logic deterministic in unit tests. Tests cover unresolvable and resolvable Azure hostnames, private Azure, non-Azure, missing external DNS configuration, and a missing ignition endpoint.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs. (No documentation change is needed.)
  • This change includes unit tests.

Summary by CodeRabbit

  • Bug Fixes
  • Improved Azure node pool validation by checking whether external Ignition endpoint hostnames are DNS-resolvable.
  • Node pools now report a clear availability condition and retry automatically when the Ignition endpoint hostname cannot be reached.
  • Existing Ignition certificate validation continues after successful DNS resolution.
  • Non-Azure node pools and Azure configurations without an external hostname retain their existing behavior.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

@stbenjam: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-robot

Copy link
Copy Markdown
Contributor

Fix included in release 5.0.0-0.nightly-2026-08-05-033721

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants