OCPSTRAT-1677: fix(cpo): increase NTH default workers from 10 to 20

[enxebre]

Summary

• Increases the AWS Node Termination Handler default WORKERS from 10 to 20

Scale evaluation showed that 10 workers caps graceful spot termination handling at ~80-100 concurrent interruptions. At 100 nodes, the SQS-to-drain pipeline reaches the 2-minute AWS notice window boundary — any pods with non-zero terminationGracePeriodSeconds cause ungraceful terminations.

Increasing to 20 extends safe coverage to ~150-200 concurrent spot interruptions with negligible resource overhead (idle goroutines), better matching common ROSA HCP cluster sizes. Clusters running 300+ spot nodes should tune WORKERS explicitly per the scale evaluation sizing formula.

Related aws/aws-node-termination-handler#1278

Test plan

• Deploy a HostedCluster with spot NodePools and verify the NTH deployment has WORKERS=20
• Simulate concurrent spot interruptions at ~100 nodes and verify all drains complete within the 2-minute window

🤖 Generated with Claude Code

Summary by CodeRabbit

• Chores
  • Updated worker configuration for AWS Node Termination Handler, increasing capacity from 10 to 20 workers.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci-robot]

@enxebre: This pull request references OCPSTRAT-1677 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the feature to target either version "5.0." or "openshift-5.0.", but it targets "openshift-4.22" instead.

Details

In response to this:

Summary

• Increases the AWS Node Termination Handler default WORKERS from 10 to 20

Scale evaluation showed that 10 workers caps graceful spot termination handling at ~80-100 concurrent interruptions. At 100 nodes, the SQS-to-drain pipeline reaches the 2-minute AWS notice window boundary — any pods with non-zero terminationGracePeriodSeconds cause ungraceful terminations.

Increasing to 20 extends safe coverage to ~150-200 concurrent spot interruptions with negligible resource overhead (idle goroutines), better matching common ROSA HCP cluster sizes. Clusters running 300+ spot nodes should tune WORKERS explicitly per the scale evaluation sizing formula.

Test plan

• Deploy a HostedCluster with spot NodePools and verify the NTH deployment has WORKERS=20
• Simulate concurrent spot interruptions at ~100 nodes and verify all drains complete within the 2-minute window

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[coderabbitai]

📝 Walkthrough

Walkthrough

The WORKERS environment variable in the aws-node-termination-handler Deployment manifest has been updated from a value of "10" to "20". This change affects the configuration of the aws-node-termination-handler component deployed in the hosted control plane. The modification is contained within a single Kubernetes Deployment resource definition in the manifest file.

🚥 Pre-merge checks | ✅ 12

✅ Passed checks (12 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically describes the main change: increasing NTH default workers from 10 to 20 in the deployment manifest.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	This check is not applicable to the PR. The PR only modifies a Kubernetes Deployment manifest file (changing WORKERS env var from 10 to 20). It contains no Ginkgo tests or test name definitions.
Test Structure And Quality	✅ Passed	The PR adds a test file using standard Go testing (testing.T), not Ginkgo. Ginkgo tests require Describe/Context/It blocks. This check targets Ginkgo tests specifically, so it does not apply.
Microshift Test Compatibility	✅ Passed	The PR modifies only a Kubernetes Deployment manifest (deployment.yaml), not any Ginkgo e2e tests. The custom check applies only to new e2e tests, making it not applicable to this change.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	PR changes only the WORKERS environment variable in a deployment manifest (10→20), not Ginkgo e2e tests. SNO compatibility check applies only to new Ginkgo test additions.
Topology-Aware Scheduling Compatibility	✅ Passed	PR modifies only the WORKERS environment variable (10→20), an app configuration parameter. No scheduling constraints, affinity rules, or topology-dependent selectors are introduced.
Ote Binary Stdout Contract	✅ Passed	PR modifies only a static YAML deployment manifest. No OTE test binaries, test suites, or process-level code with stdout writes were changed. OTE Binary Stdout Contract check is not applicable.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The PR adds new Ginkgo e2e tests but they do not contain IPv4 assumptions or external connectivity requirements that would fail in IPv6-only disconnected environments.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: enxebre

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [enxebre]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 40.34%. Comparing base (1eddaf8) to head (01a43de).
⚠️ Report is 198 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8500      +/-   ##
==========================================
+ Coverage   40.00%   40.34%   +0.34%     
==========================================
  Files         751      755       +4     
  Lines       92838    93167     +329     
==========================================
+ Hits        37137    37587     +450     
+ Misses      53014    52877     -137     
- Partials     2687     2703      +16     

see 19 files with indirect coverage changes

Flag	Coverage Δ
cmd-support	34.30% <ø> (+0.21%)	⬆️
cpo-hostedcontrolplane	41.76% <ø> (+1.19%)	⬆️
cpo-other	40.14% <ø> (ø)
hypershift-operator	50.72% <ø> (+0.19%)	⬆️
other	31.54% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[devguyio]

/lgtm
/test images

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

[cblecker]

/uncc

[enxebre]

/retest

[enxebre]

/retest

[enxebre]

/pipeline required
/retest

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

[hypershift-jira-solve-ci]

I now have all the evidence needed. Adjacent PRs (#8497, #8498) have passing Konflux checks, PR #8501 also has passing Konflux build pipelines, and the PR diff only changes a YAML value from "10" to "20". The root cause is confirmed.

Test Failure Analysis Complete

Job Information

• Prow Job: Red Hat Konflux / control-plane-operator-main-on-pull-request, hypershift-operator-main-on-pull-request, hypershift-release-mce-50-on-pull-request, hypershift-cli-mce-50-on-pull-request
• Build ID: control-plane-operator-main-on-pull-request-hpfc4, hypershift-operator-main-on-pull-request-lj2fg, hypershift-release-mce-50-on-pull-request-vm79z, hypershift-cli-mce-50-on-pull-request-zt8dk
• PR: #8500 — OCPSTRAT-1677: fix(cpo): increase NTH default workers from 10 to 20
• Namespace: crt-redhat-acm-tenant
• Failure Time: 2026-06-01T14:44:44Z (all 4 pipelines failed within 1 second)

Test Failure Analysis

Error

Pipeline crt-redhat-acm-tenant/<pipeline-name> can't be Run; it contains Tasks that don't exist:
Couldn't retrieve Task "resolver type bundles\nname = init\n": error requesting remote resource:
error getting "bundleresolver" "crt-redhat-acm-tenant/bundles-<hash>":
cannot retrieve the oci image: GET https://quay.io/v2/: unexpected status code 502 Bad Gateway

Summary

All four Konflux pipeline failures are caused by a transient Quay.io registry outage (HTTP 502 Bad Gateway) that occurred at 2026-06-01T14:44:44Z. The Konflux/Tekton pipelines could not resolve their Tekton Task bundles from quay.io, so none of the pipelines ever started executing — they failed at task resolution time within 1 second. This is entirely unrelated to the PR's code changes, which only modify a YAML constant (WORKERS: "10" → "20") in the AWS Node Termination Handler deployment configuration and associated test fixtures.

Root Cause

The Konflux CI system uses Tekton Bundle Resolvers to fetch pipeline task definitions as OCI images from quay.io. At the time these pipelines were triggered (2026-06-01T14:44:44Z), the Quay.io registry API (https://quay.io/v2/) was returning HTTP 502 Bad Gateway responses. This prevented the Tekton pipeline controller from resolving any tasks, causing all four pipelines to fail immediately before any build steps could execute.

Key evidence confirming this is a transient infrastructure issue:

1. All 4 pipelines failed identically — each hit a different bundle hash but got the same 502 error from quay.io/v2/
2. Execution duration was 1 second — pipelines completed at 14:44:45Z, started at 14:44:44Z. No build or test work was performed.
3. Adjacent PRs have passing Konflux checks — PR OCPBUGS-81544: requeue when AutoNodeEnabled is progressing #8497 and AUTOSCALE-681: remove TechPreviewNoUpgrade gate from karpenter upgrade test #8498 both show all Konflux pipeline checks passing (10-30 minute durations, indicating real builds). PR WIP: CNTRLPLANE-3443: add new ExternalOIDCExternalClaimsSourcing feature gate #8501 also has passing Konflux build pipelines.
4. PR changes cannot cause build failures — the diff only modifies a string literal "10" → "20" in 6 YAML fixture/deployment files. No Go code, no imports, no build configuration changed.

Recommendations

1. Re-run the failed checks — Since this was a transient Quay.io outage, simply re-triggering the Konflux pipelines (e.g., by pushing an empty commit or using the Konflux UI to re-run) should resolve all 4 failures.
2. No code changes needed — The PR's changes (NTH WORKERS value bump) are entirely unrelated to the pipeline infrastructure failure.
3. Monitor Quay.io status — If re-runs also fail, check status.quay.io for ongoing incidents before further investigation.

Evidence

Evidence	Detail
Error type	Tekton BundleResolver failed to fetch OCI image from quay.io
HTTP status	502 Bad Gateway from GET https://quay.io/v2/
Failure timestamp	2026-06-01T14:44:44Z → 14:44:45Z (1 second total)
Affected pipelines	All 4 Konflux on-pull-request pipelines (CPO, HO, release-mce-50, cli-mce-50)
Bundle hashes attempted	bundles-f38806cc…, bundles-1a95fee9…, bundles-5b2b5fcc…, bundles-63120626…
PR #8497 Konflux status	✅ All passing (13-29 min build times)
PR #8498 Konflux status	✅ All passing (10-20 min build times)
PR #8501 Konflux status	✅ Build pipelines passing (15-27 min build times)
PR diff scope	6 files changed: YAML value "10" → "20" only (no Go code, no build config)

---

[openshift-ci]

@enxebre: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.