do not merge: OCPBUGS-98213 Verification

[Ben Vesel (bennerv)]

What

Help verify router pod doesn't get deployed until all route are able to be rendered in the router configmap.

Why

Testing

• Build and push OCPBUGS-98213: fix router component ordering to prevent missing HAProxy backends openshift/hypershift#8971 out of band to ACR
• FDescribe on private key vault test so only that e2e runs.
• Look at kusto for the run in question
  • Check events to validate there's only ever a single replicaset of the private-router pod in the hostedcluster namespace
  • Check mgmt-agent logs and kube-apiserver logs to validate the configmap is created, then updated later with the ignition route, then the router pod is created

[Ben Vesel (bennerv)]

/hold will not merge.

[Copilot]

Pull request overview

This PR is a verification-oriented change to help confirm HyperShift router behavior (router pod should not be deployed until routes can be rendered into the router ConfigMap). It does this by (1) forcing a single focused E2E scenario and (2) adding mgmt-agent watchers to emit Kubernetes event timelines (router ConfigMap lifecycle).

Changes:

• Focuses the “Private KeyVault” cluster-create E2E and pins it to an explicitly resolved OCP 4.23 candidate install version, plus an out-of-band CPO image override tag.
• Adds a ConfigMapWatcher in mgmt-agent to log router ConfigMap create/update/delete events (via a field-selector-scoped informer).
• Wires the new watcher into mgmt-agent startup under leader election and adds basic unit tests.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
test/e2e/cluster_create_private_kv.go	Focused E2E test + explicit version resolution + experimental tag to override CPO image
mgmt-agent/pkg/controller/cmwatcher.go	New informer-backed ConfigMap watcher that logs router ConfigMap events
mgmt-agent/pkg/controller/cmwatcher_test.go	Unit tests for the new watcher/logging helper
mgmt-agent/cmd/options.go	Wires ConfigMap watcher/informers into mgmt-agent lifecycle under leader election

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

[Ben Vesel (bennerv)]

KQL queries to show that there is a single revision of router pod replicasets during installation. Can use these for cluster installs if we know the hostedcluster namespace.

let hcNamespace="";
cluster('https://hcp-dev-us-2.eastus2.kusto.windows.net').database('ServiceLogs').kubeAudit
| where timestamp between (datetime(2026-07-13T15:43:01Z) .. datetime(2026-07-14T16:06:00Z))
| where resourceId has 'ci01-j8862720-mgmt-1'
| where stage == 'ResponseComplete'
| where tostring(objectRef.namespace) == 'ocm-arohcpci01-2rhbkb9hj5tgfjeb1avcm9ncp7upjvb1-e1j5q9p7g7s1v2z'
| where objectRef.resource in("deployments", "replicasets")
| project timestamp=requestReceivedTimestamp,
    verb,
    resource = tostring(objectRef.resource),
    name = tostring(objectRef.name),
    userName = tostring(user.username),
    responseCode = toint(responseStatus.code)
| order by timestamp asc

cluster('https://hcp-dev-us-2.eastus2.kusto.windows.net').database('ServiceLogs').kubernetesEvents
| where timestamp between (datetime(2026-07-13T15:43:01Z) .. datetime(2026-07-13T16:06:00Z))
| where cluster in ('ci01-j8862720-svc', 'ci01-j8862720-mgmt-1')
| where eventNamespace == hcNamespace
| where objectKind == 'Deployment' and reason == 'ScalingReplicaSet'
| extend firstSeen = coalesce(firstSeen, todatetime(log.event_time))
| project firstSeen, objectName, message
| order by firstSeen asc

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bennerv

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• test/OWNERS [bennerv]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.

[Ben Vesel (bennerv)]

/test e2e-parallel

[Ben Vesel (bennerv)]

main HyperShift operator updates good on the below scenarios. Only a single replicaset is seen during the lifetime of the cluster.

• non-swift cluster (check for successful pass)
• swift cluster public key vault
• swift cluster private key vault

https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/Azure_ARO-HCP/6033/pull-ci-Azure-ARO-HCP-main-e2e-parallel/2077130989351473152

Data explorer query

let swiftClusterPublicKv="cpo-override-2512-nswttw";
let swiftClusterPrivateKv="private-keyvault-2l2gqg";
let _startTime = datetime(2026-07-14T20:24:07.0000000Z);
let _endTime = datetime(2026-07-14T23:27:29.0000000Z);
let cid = (database('ServiceLogs').table('clustersServiceLogs')
| where timestamp between (_startTime .. _endTime)
| where resource_id has swiftClusterPrivateKv
| where strlen( cid) > 0
| distinct cid);
let hcNamespace=(database('HostedControlPlaneLogs').table('containerLogs')
| where timestamp between (_startTime .. _endTime)
| where namespace_name has_any (cid)
| distinct namespace_name);
database('ServiceLogs').kubernetesEvents
| where eventNamespace  has_any (hcNamespace)
    and objectName contains 'router'
| extend firstSeen = coalesce(firstSeen, todatetime(log.event_time)), lastSeen = coalesce(lastSeen, todatetime(log.event_time))
| summarize arg_max(count, firstSeen, lastSeen) by objectKind, objectName, reason, message
| project objectKind, objectName, reason, message, firstSeen, lastSeen, count
| order by firstSeen asc

[Mike Gahagan (mgahagan73)]

Hi, If you plan on merging the 2 new CPO tests they will need MI Container labels added (labels.MIContainers(n) where n is the number of MI Containers), a rebase will pull in the new verify checks for the MI container labels.

[Ben Vesel (bennerv)]

/test e2e-parallel

should test 4.22 release now based on openshift/hypershift#9212

[Ben Vesel (bennerv)]

/test e2e-parallel

[openshift-ci]

Ben Vesel (@bennerv): The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/test-unit	1dfbac6	link	true	/test test-unit
ci/prow/mega-linter	1dfbac6	link	true	/test mega-linter
ci/prow/integration	1dfbac6	link	true	/test integration
ci/prow/verify	50c4645	link	true	/test verify
ci/prow/e2e-parallel	c84eb71	link	true	/test e2e-parallel

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[Ben Vesel (bennerv)]

/close

Todd will do validation on the HyperShift PRs. This isn't needed but he can cherry-pick out the changes as needed to validate in his own PR if he wants.