OCPBUGS-62177: revert CRR controller changes - #7784
Conversation
openshift-ci-robot
added
the
jira/valid-reference
|
@sjenning: This pull request explicitly references no jira issue. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Pipeline controller notification For optional jobs, comment This repository is configured in: LGTM mode |
|
@sjenning: This pull request explicitly references no jira issue. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: sjenning The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
WalkthroughRBAC roles are updated across all platform-specific environments and base configurations, removing list and watch verbs from pod permissions. The certificate revocation controller is refactored to remove Pod informer dependency and replace distributed pod verification with inline KAS-connected verification logic. Associated tests are cleaned up to reflect the implementation changes. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~30 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Comment |
openshift-ci
Bot
added
area/control-plane-operator
|
/lgtm |
openshift-ci
Bot
added
area/testing
|
Scheduling tests matching the |
|
@sjenning: This pull request explicitly references no jira issue. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
sjenning
changed the title
openshift-ci-robot
added
the
jira/invalid-bug
|
@sjenning: This pull request references Jira Issue OCPBUGS-62177, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/jira refresh |
openshift-ci-robot
added
jira/valid-bug
|
@sjenning: This pull request references Jira Issue OCPBUGS-62177, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
No GitHub users were found matching the public email listed for the QA contact in Jira (jiezhao@redhat.com), skipping review request. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/verified later by @sjenning |
|
@sjenning: Only users can be targets for the DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (2)
control-plane-pki-operator/certificaterevocationcontroller/certificaterevocationcontroller.go (2)
545-560: RenamesignerSecerttosignerSecretfor clarity.Typo makes the local reads harder to scan and risks copy/paste mistakes later.
♻️ Proposed fix
- signerSecert, signers, err := c.loadCertificateSecret(signer.Namespace, signer.Name) + signerSecret, signers, err := c.loadCertificateSecret(signer.Namespace, signer.Name) if err != nil { return true, nil, false, err } @@ - currentCertPEM, ok := signerSecert.Data[corev1.TLSCertKey] + currentCertPEM, ok := signerSecret.Data[corev1.TLSCertKey] if !ok || len(currentCertPEM) == 0 { - return true, nil, false, fmt.Errorf("signer certificate %s/%s had no data for %s", signerSecert.Namespace, signerSecert.Name, corev1.TLSCertKey) + return true, nil, false, fmt.Errorf("signer certificate %s/%s had no data for %s", signerSecret.Namespace, signerSecret.Name, corev1.TLSCertKey) } - currentKeyPEM, ok := signerSecert.Data[corev1.TLSPrivateKeyKey] + currentKeyPEM, ok := signerSecret.Data[corev1.TLSPrivateKeyKey] if !ok || len(currentKeyPEM) == 0 { - return true, nil, false, fmt.Errorf("signer certificate %s/%s had no data for %s", signerSecert.Namespace, signerSecert.Name, corev1.TLSPrivateKeyKey) + return true, nil, false, fmt.Errorf("signer certificate %s/%s had no data for %s", signerSecret.Namespace, signerSecret.Name, corev1.TLSPrivateKeyKey) }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@control-plane-pki-operator/certificaterevocationcontroller/certificaterevocationcontroller.go` around lines 545 - 560, The variable name signerSecert is a typo; rename it to signerSecret everywhere it’s used (including its declaration and subsequent references in this function and any nearby scope) to improve clarity and avoid mistakes—e.g., in the assignment from c.loadCertificateSecret, in the checks for signerSecret.Data[corev1.TLSCertKey] and signerSecret.Data[corev1.TLSPrivateKeyKey], and in the fmt.Errorf messages that reference signerSecert.Namespace and signerSecert.Name; ensure all occurrences are updated consistently so the code compiles.
580-613: Add timeout to KAS SelfSubjectReview calls to prevent indefinite hangs.Both SelfSubjectReview calls (at these lines and at 852-885) use
ctxdirectly without a timeout. If KAS is unreachable, these requests will block indefinitely sincerest.Config.Timeoutdefaults to 0 (no timeout). Wrap the context withcontext.WithTimeoutbefore passing toCreate().Proposed fix
+ reqCtx, cancel := context.WithTimeout(ctx, 10*time.Second) + defer cancel() - _, err = testClient.AuthenticationV1().SelfSubjectReviews().Create(ctx, &authenticationv1.SelfSubjectReview{}, metav1.CreateOptions{}) + _, err = testClient.AuthenticationV1().SelfSubjectReviews().Create(reqCtx, &authenticationv1.SelfSubjectReview{}, metav1.CreateOptions{})Apply the same fix to the second SSR call at line 878-879.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@control-plane-pki-operator/certificaterevocationcontroller/certificaterevocationcontroller.go` around lines 580 - 613, The SelfSubjectReview Create calls (via testClient.AuthenticationV1().SelfSubjectReviews().Create and the later SSR call around lines 852-885) use the controller's ctx directly and can hang if KAS is unreachable; wrap the ctx with a short context.WithTimeout (e.g. 10s or a configurable constant) before calling Create(), use the derived ctx for the Create() call, and ensure you call the cancel() in a defer immediately after creating the timed context; apply the same change to both SSR invocations so both requests time out instead of blocking indefinitely.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In
`@control-plane-pki-operator/certificaterevocationcontroller/certificaterevocationcontroller.go`:
- Around line 545-560: The variable name signerSecert is a typo; rename it to
signerSecret everywhere it’s used (including its declaration and subsequent
references in this function and any nearby scope) to improve clarity and avoid
mistakes—e.g., in the assignment from c.loadCertificateSecret, in the checks for
signerSecret.Data[corev1.TLSCertKey] and
signerSecret.Data[corev1.TLSPrivateKeyKey], and in the fmt.Errorf messages that
reference signerSecert.Namespace and signerSecert.Name; ensure all occurrences
are updated consistently so the code compiles.
- Around line 580-613: The SelfSubjectReview Create calls (via
testClient.AuthenticationV1().SelfSubjectReviews().Create and the later SSR call
around lines 852-885) use the controller's ctx directly and can hang if KAS is
unreachable; wrap the ctx with a short context.WithTimeout (e.g. 10s or a
configurable constant) before calling Create(), use the derived ctx for the
Create() call, and ensure you call the cancel() in a defer immediately after
creating the timed context; apply the same change to both SSR invocations so
both requests time out instead of blocking indefinitely.
ℹ️ Review info
Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
Review profile: CHILL
Plan: Pro
Cache: Disabled due to data retention organization setting
Knowledge base: Disabled due to data retention organization setting
📒 Files selected for processing (9)
control-plane-operator/controllers/hostedcontrolplane/testdata/control-plane-pki-operator/AROSwift/zz_fixture_TestControlPlaneComponents_control_plane_pki_operator_role.yamlcontrol-plane-operator/controllers/hostedcontrolplane/testdata/control-plane-pki-operator/GCP/zz_fixture_TestControlPlaneComponents_control_plane_pki_operator_role.yamlcontrol-plane-operator/controllers/hostedcontrolplane/testdata/control-plane-pki-operator/IBMCloud/zz_fixture_TestControlPlaneComponents_control_plane_pki_operator_role.yamlcontrol-plane-operator/controllers/hostedcontrolplane/testdata/control-plane-pki-operator/TechPreviewNoUpgrade/zz_fixture_TestControlPlaneComponents_control_plane_pki_operator_role.yamlcontrol-plane-operator/controllers/hostedcontrolplane/testdata/control-plane-pki-operator/zz_fixture_TestControlPlaneComponents_control_plane_pki_operator_role.yamlcontrol-plane-operator/controllers/hostedcontrolplane/v2/assets/control-plane-pki-operator/role.yamlcontrol-plane-pki-operator/certificaterevocationcontroller/certificaterevocationcontroller.gocontrol-plane-pki-operator/certificaterevocationcontroller/certificaterevocationcontroller_test.gotest/e2e/create_cluster_test.go
💤 Files with no reviewable changes (8)
- control-plane-operator/controllers/hostedcontrolplane/testdata/control-plane-pki-operator/AROSwift/zz_fixture_TestControlPlaneComponents_control_plane_pki_operator_role.yaml
- control-plane-operator/controllers/hostedcontrolplane/testdata/control-plane-pki-operator/zz_fixture_TestControlPlaneComponents_control_plane_pki_operator_role.yaml
- test/e2e/create_cluster_test.go
- control-plane-operator/controllers/hostedcontrolplane/testdata/control-plane-pki-operator/GCP/zz_fixture_TestControlPlaneComponents_control_plane_pki_operator_role.yaml
- control-plane-operator/controllers/hostedcontrolplane/v2/assets/control-plane-pki-operator/role.yaml
- control-plane-operator/controllers/hostedcontrolplane/testdata/control-plane-pki-operator/IBMCloud/zz_fixture_TestControlPlaneComponents_control_plane_pki_operator_role.yaml
- control-plane-pki-operator/certificaterevocationcontroller/certificaterevocationcontroller_test.go
- control-plane-operator/controllers/hostedcontrolplane/testdata/control-plane-pki-operator/TechPreviewNoUpgrade/zz_fixture_TestControlPlaneComponents_control_plane_pki_operator_role.yaml
sjenning
added
the
acknowledge-critical-fixes-only
Test Resultse2e-aws
e2e-aks
|
|
/retest |
openshift-ci-robot
added
the
verified
|
@enxebre: This PR has been marked as verified by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@sjenning: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
@sjenning: Jira Issue Verification Checks: Jira Issue OCPBUGS-62177 Jira Issue OCPBUGS-62177 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
Add TestCreateClusterHABreakGlassCredentials to exercise the break-glass credential / CertificateRevocationController flow on a HighlyAvailable control plane (3 KAS replicas). This validates that the CRR controller correctly verifies certificate revocation against each individual KAS pod rather than through the service load balancer. The test creates an HA cluster, asserts all 3 KAS replicas are ready (guarding against false-positive passes on a single-replica cluster), then runs the existing RunTestControlPlanePKIOperatorBreakGlassCredentials integration helper which exercises the full CRR lifecycle. The previous e2e test for this (TestCreateClusterRequestServingIsolation in PR openshift#7405) was reverted in PR openshift#7784 due to a ~50% flake rate. That flakiness was entirely caused by the request serving isolation test infrastructure: 5 dedicated node pools, topology annotation, and node placement validation. None of that is used here. This test is intentionally minimal — HA control plane policy is the only difference from TestCreateCluster. It reuses well-proven building blocks: - ControlPlaneAvailabilityPolicy=HighlyAvailable is exercised by TestUpgradeControlPlane, TestHAEtcdChaos, etc. - RunTestControlPlanePKIOperatorBreakGlassCredentials has been running reliably in TestCreateCluster (SingleReplica) since 4.15. - All async waiting uses the existing EventuallyObject framework with no new polling loops. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add TestCreateClusterHABreakGlassCredentials to exercise the break-glass credential / CertificateRevocationController flow on a HighlyAvailable control plane (3 KAS replicas). This validates that the CRR controller correctly verifies certificate revocation against each individual KAS pod rather than through the service load balancer. The test creates an HA cluster, asserts all 3 KAS replicas are ready (guarding against false-positive passes on a single-replica cluster), then runs the existing RunTestControlPlanePKIOperatorBreakGlassCredentials integration helper which exercises the full CRR lifecycle. The previous e2e test for this (TestCreateClusterRequestServingIsolation in PR openshift#7405) was reverted in PR openshift#7784 due to a ~50% flake rate. That flakiness was entirely caused by the request serving isolation test infrastructure: 5 dedicated node pools, topology annotation, and node placement validation. None of that is used here. This test is intentionally minimal — HA control plane policy is the only difference from TestCreateCluster. It reuses well-proven building blocks: - ControlPlaneAvailabilityPolicy=HighlyAvailable is exercised by TestUpgradeControlPlane, TestHAEtcdChaos, etc. - RunTestControlPlanePKIOperatorBreakGlassCredentials has been running reliably in TestCreateCluster (SingleReplica) since 4.15. - All async waiting uses the existing EventuallyObject framework with no new polling loops. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Fix included in release 5.0.0-0.nightly-2026-05-12-025124 |
6 participants
Reverts #7405 and #7744
The introduction of this net new testcase in
TestCreateClusterRequestServingIsolationhas resulted in a ~50% flake rate.https://testgrid.k8s.io/redhat-hypershift#4.21-aws-ovn&width=20
Note
Medium Risk
Touches certificate revocation gating behavior and RBAC for the PKI operator; while mostly a rollback/simplification, it can change how quickly/accurately revocation completion is detected in HA scenarios.
Overview
Reverts the certificate revocation controller’s per-kube-apiserver-pod verification logic (and related helpers/tests), switching back to a single guest API connectivity check via the service kubeconfig when gating signer trust/revocation.
Updates the control-plane PKI operator
Role(and fixtures) to droppodslist/watchpermissions, and removes the HA request-serving isolation e2e subtest that exercised the break-glass/CRR flow due to flakiness.Written by Cursor Bugbot for commit c8bc247. This will update automatically on new commits. Configure here.
Summary by CodeRabbit
Release Notes
Security
Bug Fixes
Tests