Skip to content

CNTRLPLANE-3584: Add kube-scheduler ServiceMonitor with CA-signed serving certs - #8489

Merged
openshift-merge-bot[bot] merged 2 commits into
openshift:mainfrom
dhgautam99:create-kube-scheduler-servicemonitor
Jun 9, 2026
Merged

CNTRLPLANE-3584: Add kube-scheduler ServiceMonitor with CA-signed serving certs#8489
openshift-merge-bot[bot] merged 2 commits into
openshift:mainfrom
dhgautam99:create-kube-scheduler-servicemonitor

Conversation

@dhgautam99

@dhgautam99 dhgautam99 commented May 12, 2026

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Adds a ServiceMonitor for kube-scheduler to enable Prometheus metrics scraping with proper mTLS authentication.

Previously, kube-scheduler auto-generated self-signed serving certificates via --cert-dir=/var/run/kubernetes. Prometheus could not verify the scheduler's identity using the cluster's root CA, and no ServiceMonitor existed.

This PR:

  • Creates a CA-signed serving certificate (scheduler-server) for kube-scheduler, following the KCM pattern
  • Replaces the self-signed cert (emptyDir) with the CA-signed cert (secret volume) in the deployment
  • Adds a kube-scheduler Service exposing port 10259
  • Adds a ServiceMonitor with mTLS config (root-ca for server verification, metrics-client for client auth)
  • Adds SchedulerRelabelConfigs() for SRE metrics set support

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/CNTRLPLANE-3584

Special notes for your reviewer:

Follows the same pattern as kube-controller-manager's ServiceMonitor setup.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Summary by CodeRabbit

  • New Features

    • TLS serving certificate provisioning for the kube-scheduler with secret-backed cert/key mounting, Deployment switched to explicit cert/key flags, and a ClusterIP Service exposing port 10259.
    • ServiceMonitor added to scrape scheduler metrics over TLS using configured CA and client credentials.
  • Metrics

    • Added kube-scheduler relabel config support to metrics configuration.
  • Tests

    • Unit tests covering scheduler certificate reconciliation, ServiceMonitor adaptation, and scheduler component defaults.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 12, 2026
@openshift-ci

openshift-ci Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels May 12, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@dhgautam99: This pull request references Jira Issue OCPBUGS-63328, which is invalid:

  • expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

What this PR does / why we need it:

Adds a ServiceMonitor for kube-scheduler to enable Prometheus metrics scraping with proper mTLS authentication.

Previously, kube-scheduler auto-generated self-signed serving certificates via --cert-dir=/var/run/kubernetes. Prometheus could not verify the scheduler's identity using the cluster's root CA, and no ServiceMonitor existed.

This PR:

  • Creates a CA-signed serving certificate (scheduler-server) for kube-scheduler, following the KCM pattern
  • Replaces the self-signed cert (emptyDir) with the CA-signed cert (secret volume) in the deployment
  • Adds a kube-scheduler Service exposing port 10259
  • Adds a ServiceMonitor with mTLS config (root-ca for server verification, metrics-client for client auth)
  • Adds SchedulerRelabelConfigs() for SRE metrics set support

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPBUGS-63328

Special notes for your reviewer:

Follows the same pattern as kube-controller-manager's ServiceMonitor setup.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. do-not-merge/needs-area labels May 12, 2026
@coderabbitai

coderabbitai Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The controller now reconciles a kube-scheduler serving certificate during PKI reconciliation by creating/updating a scheduler-server Secret via a new pki.ReconcileSchedulerServerSecret function and a SchedulerServerCertSecret manifest helper. The kube-scheduler Deployment was changed to mount that secret and use --tls-cert-file/--tls-private-key-file. A ClusterIP Service exposing port 10259 (named client) and a ServiceMonitor for HTTPS scraping (with TLS and relabeling) were added and wired into the component manifests. Metrics configuration gained kube-scheduler relabel configs and corresponding adapter tests; unit tests cover PKI reconciliation and ServiceMonitor adaptation.

Sequence Diagram(s)

sequenceDiagram
    participant Controller as HostedControlPlaneController
    participant PKI as pki.Reconciler
    participant CA as Root CA Secret
    participant Secret as scheduler-server Secret
    participant KubeSched as kube-scheduler Pod
    participant Service as kube-scheduler Service
    participant Prom as Prometheus (ServiceMonitor)

    Controller->>PKI: ReconcileSchedulerServerSecret(secret, ca, ownerRef)
    PKI->>CA: read root CA secret
    PKI->>Secret: create/update signed serving cert (DNS names)
    PKI-->>Controller: return success / error
    Controller->>KubeSched: ensure Deployment mounts scheduler-server Secret and TLS flags
    KubeSched-->>Service: listen on port 10259 (client)
    Service-->>Prom: ServiceMonitor selects Service (app: kube-scheduler)
    Prom->>KubeSched: scrape HTTPS /metrics using TLS config and relabeling
Loading
🚥 Pre-merge checks | ✅ 10 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Test Structure And Quality ⚠️ Warning Test assertions lack meaningful failure messages. 12+ assertions in scheduler_test.go and component_test.go omit diagnostic messages, violating the stated requirement. Add failure message strings to Gomega assertions: e.g., Expect(err).ToNot(HaveOccurred(), "failed to reconcile scheduler secret") for all assertions.
✅ Passed checks (10 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed All test names in the three new test files are static and deterministic with no dynamic values like timestamps, generated IDs, or variable interpolation.
Topology-Aware Scheduling Compatibility ✅ Passed PR adds ServiceMonitor, Service, and certificate support for kube-scheduler without introducing new scheduling constraints. Pre-existing MultiZoneSpread framework behavior is not changed by this PR.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed No Ginkgo e2e tests added in PR; all three test files are standard Go unit tests, not Ginkgo e2e tests. The custom check applies only to Ginkgo e2e tests.
No-Weak-Crypto ✅ Passed PR introduces no weak cryptographic algorithms, custom crypto implementations, or insecure secret comparisons; uses standard x509 cert infrastructure following established patterns.
Container-Privileges ✅ Passed No privileged container settings found in the PR manifests or code: deployment has no securityContext, privileged mode, hostPID/Network/IPC, allowPrivilegeEscalation, or SYS_ADMIN capabilities.
No-Sensitive-Data-In-Logs ✅ Passed No sensitive data exposed in logs. Only fmt.Sprintf usage creates DNS names with namespace, error messages follow safe patterns matching existing code.
Title check ✅ Passed The PR title clearly and specifically describes the main changes: adding a ServiceMonitor for kube-scheduler with CA-signed serving certificates, which aligns with the core objectives of replacing self-signed certs with CA-signed ones and adding ServiceMonitor functionality.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci Bot added area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release and removed do-not-merge/needs-area labels May 12, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/servicemonitor.go`:
- Around line 12-16: Change the NamespaceSelector to use the hosted control
plane namespace instead of the ServiceMonitor's namespace: replace the current
sm.Spec.NamespaceSelector = prometheusoperatorv1.NamespaceSelector{MatchNames:
[]string{sm.Namespace}} with a selector that uses cpContext.HCP.Namespace (i.e.
MatchNames: []string{cpContext.HCP.Namespace}); update the code around
sm.Spec.NamespaceSelector to reference cpContext.HCP.Namespace so this v2
component follows the same pattern as kas/karpenteroperator/registryoperator.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 7f706c17-7364-4fd2-b06e-12a97677c76c

📥 Commits

Reviewing files that changed from the base of the PR and between b7c62b0 and 2c58e6d.

📒 Files selected for processing (9)
  • control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go
  • control-plane-operator/controllers/hostedcontrolplane/manifests/pki.go
  • control-plane-operator/controllers/hostedcontrolplane/pki/scheduler.go
  • control-plane-operator/controllers/hostedcontrolplane/v2/assets/kube-scheduler/deployment.yaml
  • control-plane-operator/controllers/hostedcontrolplane/v2/assets/kube-scheduler/service.yaml
  • control-plane-operator/controllers/hostedcontrolplane/v2/assets/kube-scheduler/servicemonitor.yaml
  • control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/component.go
  • control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/servicemonitor.go
  • support/metrics/sets.go

Comment on lines +12 to +16
sm.Spec.NamespaceSelector = prometheusoperatorv1.NamespaceSelector{
MatchNames: []string{sm.Namespace},
}
sm.Spec.Endpoints[0].MetricRelabelConfigs = metrics.SchedulerRelabelConfigs(cpContext.MetricsSet)
util.ApplyClusterIDLabel(&sm.Spec.Endpoints[0], cpContext.HCP.Spec.ClusterID)

@coderabbitai coderabbitai Bot May 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify endpoint presence in scheduler ServiceMonitor asset and inspect adapter assumptions.
set -euo pipefail

echo "== kube-scheduler ServiceMonitor asset =="
fd -p 'servicemonitor.yaml' control-plane-operator/controllers/hostedcontrolplane/v2/assets/kube-scheduler \
  -x sed -n '1,220p' {}

echo
echo "== adapter usage =="
rg -n -C3 'adaptServiceMonitor|NamespaceSelector|Endpoints\[0\]' \
  control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/servicemonitor.go

Repository: openshift/hypershift

Length of output: 1409


🏁 Script executed:

# Check how adaptServiceMonitor is called and how the ServiceMonitor is instantiated
grep -rn "adaptServiceMonitor\|NewServiceMonitor" control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/ --include="*.go"

Repository: openshift/hypershift

Length of output: 409


🏁 Script executed:

# Find where ServiceMonitor is instantiated/loaded from assets
rg -B5 -A10 "func.*ServiceMonitor\|load.*ServiceMonitor" control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/

Repository: openshift/hypershift

Length of output: 46


🏁 Script executed:

# Check the pattern in support/controlplane-component for how adapters are typically called
rg -B3 -A3 "Endpoints\[0\]" control-plane-operator/controllers/hostedcontrolplane/v2/ --include="*.go" | head -50

Repository: openshift/hypershift

Length of output: 506


🏁 Script executed:

# Verify what namespace the ServiceMonitor ends up in
rg -B5 -A5 "NamespaceSelector" control-plane-operator/controllers/hostedcontrolplane/v2/ --include="*.go" | head -80

Repository: openshift/hypershift

Length of output: 506


🏁 Script executed:

# Check component.go to understand how ServiceMonitor is instantiated/loaded
sed -n '40,60p' control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/component.go

Repository: openshift/hypershift

Length of output: 377


🏁 Script executed:

# Look at the full servicemonitor.go file
cat -n control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/servicemonitor.go

Repository: openshift/hypershift

Length of output: 916


🏁 Script executed:

# Check support/controlplane-component to understand how adapters are invoked
fd -p 'README.md' support/controlplane-component -x head -100 {}

Repository: openshift/hypershift

Length of output: 2886


🏁 Script executed:

# Search for Endpoints[0] patterns across other component adapters
rg 'Endpoints\[0\]' control-plane-operator/controllers/hostedcontrolplane/v2/ -t go

Repository: openshift/hypershift

Length of output: 9392


🏁 Script executed:

# Check how NamespaceSelector is typically set in other adapters
rg 'NamespaceSelector' control-plane-operator/controllers/hostedcontrolplane/v2/ -t go -B2 -A2

Repository: openshift/hypershift

Length of output: 17302


Use cpContext.HCP.Namespace for NamespaceSelector to align with newer component patterns.

Line 13 should select from cpContext.HCP.Namespace instead of sm.Namespace. Newer v2 components (kas, karpenteroperator, registryoperator) correctly use cpContext.HCP.Namespace; this adapter uses the older pattern inconsistently.

Suggested fix
 func adaptServiceMonitor(cpContext component.WorkloadContext, sm *prometheusoperatorv1.ServiceMonitor) error {
 	sm.Spec.NamespaceSelector = prometheusoperatorv1.NamespaceSelector{
-		MatchNames: []string{sm.Namespace},
+		MatchNames: []string{cpContext.HCP.Namespace},
 	}
 	sm.Spec.Endpoints[0].MetricRelabelConfigs = metrics.SchedulerRelabelConfigs(cpContext.MetricsSet)
 	util.ApplyClusterIDLabel(&sm.Spec.Endpoints[0], cpContext.HCP.Spec.ClusterID)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
sm.Spec.NamespaceSelector = prometheusoperatorv1.NamespaceSelector{
MatchNames: []string{sm.Namespace},
}
sm.Spec.Endpoints[0].MetricRelabelConfigs = metrics.SchedulerRelabelConfigs(cpContext.MetricsSet)
util.ApplyClusterIDLabel(&sm.Spec.Endpoints[0], cpContext.HCP.Spec.ClusterID)
sm.Spec.NamespaceSelector = prometheusoperatorv1.NamespaceSelector{
MatchNames: []string{cpContext.HCP.Namespace},
}
sm.Spec.Endpoints[0].MetricRelabelConfigs = metrics.SchedulerRelabelConfigs(cpContext.MetricsSet)
util.ApplyClusterIDLabel(&sm.Spec.Endpoints[0], cpContext.HCP.Spec.ClusterID)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/servicemonitor.go`
around lines 12 - 16, Change the NamespaceSelector to use the hosted control
plane namespace instead of the ServiceMonitor's namespace: replace the current
sm.Spec.NamespaceSelector = prometheusoperatorv1.NamespaceSelector{MatchNames:
[]string{sm.Namespace}} with a selector that uses cpContext.HCP.Namespace (i.e.
MatchNames: []string{cpContext.HCP.Namespace}); update the code around
sm.Spec.NamespaceSelector to reference cpContext.HCP.Namespace so this v2
component follows the same pattern as kas/karpenteroperator/registryoperator.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@muraee, thanks for the confirmation! @dhgautam99, please update Line 13 to use cpContext.HCP.Namespace instead of sm.Namespace to align with the pattern used by the other v2 components (kas, karpenteroperator, registryoperator):

sm.Spec.NamespaceSelector = prometheusoperatorv1.NamespaceSelector{
    MatchNames: []string{cpContext.HCP.Namespace},
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually nvm, most components use sm.Namespace, we should align the other ones instead

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Check the actual distribution of sm.Namespace vs cpContext.HCP.Namespace in NamespaceSelector adapters
rg -n 'NamespaceSelector' control-plane-operator/controllers/hostedcontrolplane/v2/ --type go -A2 | grep -E 'sm\.Namespace|cpContext\.HCP\.Namespace|MatchNames'

Length of output: 4675



@muraee, you're absolutely right — I was wrong. The data confirms it clearly:

  • ServiceMonitors (9 components): routecm, ocm, olm/catalog_operator, olm/olm_operator, oapi, kcm, nto, etcd, cvo, and now kube_scheduler all use sm.Namespace
  • PodMonitors (all): use cpContext.HCP.Namespace
  • kas/servicemonitor.go is the only ServiceMonitor outlier using cpContext.HCP.Namespace

I incorrectly conflated the PodMonitor pattern with the ServiceMonitor pattern when making the suggestion. The new kube-scheduler ServiceMonitor code using sm.Namespace is correct and consistent with the established pattern. kas would be the one to align in a follow-up.

Apologies for the noise!


✏️ Learnings added
Learnt from: muraee
Repo: openshift/hypershift PR: 8489
File: control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/servicemonitor.go:12-16
Timestamp: 2026-06-01T10:13:52.624Z
Learning: In the HyperShift repository (openshift/hypershift), ServiceMonitor adapters under `control-plane-operator/controllers/hostedcontrolplane/v2/` use `sm.Namespace` for `NamespaceSelector.MatchNames` (e.g., routecm, ocm, olm, oapi, kcm, nto, etcd, cvo, kube_scheduler). PodMonitor adapters use `cpContext.HCP.Namespace` instead. The `kas/servicemonitor.go` is the only ServiceMonitor outlier using `cpContext.HCP.Namespace` and should be considered for alignment to `sm.Namespace`. Do not suggest replacing `sm.Namespace` with `cpContext.HCP.Namespace` in ServiceMonitor adapters.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

@dhgautam99
dhgautam99 force-pushed the create-kube-scheduler-servicemonitor branch from 2c58e6d to b7dc447 Compare May 12, 2026 12:15
@openshift-ci openshift-ci Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 12, 2026
@dhgautam99

Copy link
Copy Markdown
Contributor Author

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels May 12, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@dhgautam99: This pull request references Jira Issue OCPBUGS-63328, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@codecov

codecov Bot commented May 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 46.34146% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.44%. Comparing base (fce95fc) to head (e994412).
⚠️ Report is 90 commits behind head on main.

Files with missing lines Patch % Lines
support/metrics/sets.go 0.00% 14 Missing ⚠️
.../hostedcontrolplane/v2/kube_scheduler/component.go 0.00% 4 Missing ⚠️
...ostedcontrolplane/hostedcontrolplane_controller.go 50.00% 2 Missing and 1 partial ⚠️
...or/controllers/hostedcontrolplane/manifests/pki.go 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8489      +/-   ##
==========================================
+ Coverage   40.69%   41.44%   +0.75%     
==========================================
  Files         755      758       +3     
  Lines       93373    93689     +316     
==========================================
+ Hits        37994    38830     +836     
+ Misses      52646    52137     -509     
+ Partials     2733     2722      -11     
Files with missing lines Coverage Δ
...or/controllers/hostedcontrolplane/pki/scheduler.go 100.00% <100.00%> (ø)
...edcontrolplane/v2/kube_scheduler/servicemonitor.go 100.00% <100.00%> (ø)
...or/controllers/hostedcontrolplane/manifests/pki.go 0.00% <0.00%> (ø)
...ostedcontrolplane/hostedcontrolplane_controller.go 45.71% <50.00%> (+0.68%) ⬆️
.../hostedcontrolplane/v2/kube_scheduler/component.go 26.08% <0.00%> (+26.08%) ⬆️
support/metrics/sets.go 2.25% <0.00%> (-0.10%) ⬇️

... and 48 files with indirect coverage changes

Flag Coverage Δ
cmd-support 34.86% <0.00%> (+0.15%) ⬆️
cpo-hostedcontrolplane 43.59% <70.37%> (+1.78%) ⬆️
cpo-other 42.75% <ø> (+1.35%) ⬆️
hypershift-operator 51.57% <ø> (+0.72%) ⬆️
other 31.64% <ø> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@dhgautam99
dhgautam99 force-pushed the create-kube-scheduler-servicemonitor branch 2 times, most recently from 8606246 to 347cb8d Compare May 13, 2026 06:09
@openshift-ci-robot

Copy link
Copy Markdown

@dhgautam99: This pull request references Jira Issue OCPBUGS-63328, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

What this PR does / why we need it:

Adds a ServiceMonitor for kube-scheduler to enable Prometheus metrics scraping with proper mTLS authentication.

Previously, kube-scheduler auto-generated self-signed serving certificates via --cert-dir=/var/run/kubernetes. Prometheus could not verify the scheduler's identity using the cluster's root CA, and no ServiceMonitor existed.

This PR:

  • Creates a CA-signed serving certificate (scheduler-server) for kube-scheduler, following the KCM pattern
  • Replaces the self-signed cert (emptyDir) with the CA-signed cert (secret volume) in the deployment
  • Adds a kube-scheduler Service exposing port 10259
  • Adds a ServiceMonitor with mTLS config (root-ca for server verification, metrics-client for client auth)
  • Adds SchedulerRelabelConfigs() for SRE metrics set support

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPBUGS-63328

Special notes for your reviewer:

Follows the same pattern as kube-controller-manager's ServiceMonitor setup.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Summary by CodeRabbit

  • New Features

  • TLS serving certificate support for kube-scheduler with automatic reconciliation and mounting.

  • ClusterIP Service exposing kube-scheduler on port 10259.

  • ServiceMonitor with TLS-secured metrics endpoint and scheduler-specific relabeling.

  • Metrics

  • Added kube-scheduler relabel config support to metrics configuration.

  • Tests

  • Added unit tests for scheduler certificate reconciliation, ServiceMonitor adaptation, and scheduler component options.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@dhgautam99

Copy link
Copy Markdown
Contributor Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented May 13, 2026

Copy link
Copy Markdown
Contributor
✅ Actions performed

Full review triggered.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/servicemonitor.go (1)

12-14: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Use HostedControlPlane namespace for ServiceMonitor namespace selection.

Line 13 currently binds MatchNames to sm.Namespace; this should use cpContext.HCP.Namespace so scraping targets the control-plane namespace deterministically.

Proposed fix
 func adaptServiceMonitor(cpContext component.WorkloadContext, sm *prometheusoperatorv1.ServiceMonitor) error {
 	sm.Spec.NamespaceSelector = prometheusoperatorv1.NamespaceSelector{
-		MatchNames: []string{sm.Namespace},
+		MatchNames: []string{cpContext.HCP.Namespace},
 	}

As per coding guidelines, “Follow support/controlplane-component (cpov2) contracts for lifecycle components”.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/servicemonitor.go`
around lines 12 - 14, The ServiceMonitor namespace selector currently sets
MatchNames to sm.Namespace; change it to use the HostedControlPlane namespace by
assigning cpContext.HCP.Namespace to MatchNames (update the
sm.Spec.NamespaceSelector = prometheusoperatorv1.NamespaceSelector{ MatchNames:
[]string{cpContext.HCP.Namespace} }), ensuring scraping targets the
control-plane namespace deterministically; locate this in the ServiceMonitor
construction where sm.Spec.NamespaceSelector is set and replace sm.Namespace
with cpContext.HCP.Namespace.
🧹 Nitpick comments (1)
control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/servicemonitor_test.go (1)

76-96: ⚡ Quick win

Strengthen namespace-selector test to detect wrong source namespace.

Right now the fixture sets hcp.Namespace and sm.Namespace to the same value, so the assertion won’t catch an implementation that incorrectly uses sm.Namespace. Set them differently in at least one case.

Proposed tweak
 			hcp := &hyperv1.HostedControlPlane{
 				ObjectMeta: metav1.ObjectMeta{
 					Name:      "test-hcp",
-					Namespace: "test-namespace",
+					Namespace: "hcp-namespace",
 				},
@@
 			sm := &prometheusoperatorv1.ServiceMonitor{
 				ObjectMeta: metav1.ObjectMeta{
 					Name:      "kube-scheduler",
-					Namespace: "test-namespace",
+					Namespace: "monitoring-namespace",
 				},
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/servicemonitor_test.go`
around lines 76 - 96, The test fixture currently sets hcp.Namespace and
sm.Namespace to the same value so a bug that reads the ServiceMonitor's
Namespace instead of the HCP's Namespace will pass; update the test (the
HostedControlPlane instance named hcp used to build component.WorkloadContext
and the prometheusoperatorv1.ServiceMonitor instance sm) so at least one case
uses different namespaces (e.g., hcp.Namespace = "test-namespace" and
sm.Namespace = "service-namespace") and assert the code under test uses
hcp.Namespace as the source; adjust the test case input for
component.WorkloadContext and the ServiceMonitor setup to ensure the
namespace-selector logic fails if it incorrectly references sm.Namespace.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Duplicate comments:
In
`@control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/servicemonitor.go`:
- Around line 12-14: The ServiceMonitor namespace selector currently sets
MatchNames to sm.Namespace; change it to use the HostedControlPlane namespace by
assigning cpContext.HCP.Namespace to MatchNames (update the
sm.Spec.NamespaceSelector = prometheusoperatorv1.NamespaceSelector{ MatchNames:
[]string{cpContext.HCP.Namespace} }), ensuring scraping targets the
control-plane namespace deterministically; locate this in the ServiceMonitor
construction where sm.Spec.NamespaceSelector is set and replace sm.Namespace
with cpContext.HCP.Namespace.

---

Nitpick comments:
In
`@control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/servicemonitor_test.go`:
- Around line 76-96: The test fixture currently sets hcp.Namespace and
sm.Namespace to the same value so a bug that reads the ServiceMonitor's
Namespace instead of the HCP's Namespace will pass; update the test (the
HostedControlPlane instance named hcp used to build component.WorkloadContext
and the prometheusoperatorv1.ServiceMonitor instance sm) so at least one case
uses different namespaces (e.g., hcp.Namespace = "test-namespace" and
sm.Namespace = "service-namespace") and assert the code under test uses
hcp.Namespace as the source; adjust the test case input for
component.WorkloadContext and the ServiceMonitor setup to ensure the
namespace-selector logic fails if it incorrectly references sm.Namespace.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 4fa36214-c168-4aa6-bfea-b60297adea05

📥 Commits

Reviewing files that changed from the base of the PR and between 1eddaf8 and 347cb8d.

⛔ Files ignored due to path filters (20)
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/AROSwift/zz_fixture_TestControlPlaneComponents_kube_scheduler_controlplanecomponent.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/AROSwift/zz_fixture_TestControlPlaneComponents_kube_scheduler_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/AROSwift/zz_fixture_TestControlPlaneComponents_kube_scheduler_service.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/AROSwift/zz_fixture_TestControlPlaneComponents_kube_scheduler_servicemonitor.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/GCP/zz_fixture_TestControlPlaneComponents_kube_scheduler_controlplanecomponent.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/GCP/zz_fixture_TestControlPlaneComponents_kube_scheduler_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/GCP/zz_fixture_TestControlPlaneComponents_kube_scheduler_service.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/GCP/zz_fixture_TestControlPlaneComponents_kube_scheduler_servicemonitor.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/IBMCloud/zz_fixture_TestControlPlaneComponents_kube_scheduler_controlplanecomponent.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/IBMCloud/zz_fixture_TestControlPlaneComponents_kube_scheduler_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/IBMCloud/zz_fixture_TestControlPlaneComponents_kube_scheduler_service.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/IBMCloud/zz_fixture_TestControlPlaneComponents_kube_scheduler_servicemonitor.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/TechPreviewNoUpgrade/zz_fixture_TestControlPlaneComponents_kube_scheduler_controlplanecomponent.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/TechPreviewNoUpgrade/zz_fixture_TestControlPlaneComponents_kube_scheduler_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/TechPreviewNoUpgrade/zz_fixture_TestControlPlaneComponents_kube_scheduler_service.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/TechPreviewNoUpgrade/zz_fixture_TestControlPlaneComponents_kube_scheduler_servicemonitor.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/zz_fixture_TestControlPlaneComponents_kube_scheduler_controlplanecomponent.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/zz_fixture_TestControlPlaneComponents_kube_scheduler_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/zz_fixture_TestControlPlaneComponents_kube_scheduler_service.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-scheduler/zz_fixture_TestControlPlaneComponents_kube_scheduler_servicemonitor.yaml is excluded by !**/testdata/**
📒 Files selected for processing (12)
  • control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go
  • control-plane-operator/controllers/hostedcontrolplane/manifests/pki.go
  • control-plane-operator/controllers/hostedcontrolplane/pki/scheduler.go
  • control-plane-operator/controllers/hostedcontrolplane/pki/scheduler_test.go
  • control-plane-operator/controllers/hostedcontrolplane/v2/assets/kube-scheduler/deployment.yaml
  • control-plane-operator/controllers/hostedcontrolplane/v2/assets/kube-scheduler/service.yaml
  • control-plane-operator/controllers/hostedcontrolplane/v2/assets/kube-scheduler/servicemonitor.yaml
  • control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/component.go
  • control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/component_test.go
  • control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/servicemonitor.go
  • control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/servicemonitor_test.go
  • support/metrics/sets.go

@dhgautam99

Copy link
Copy Markdown
Contributor Author

the other adapters (for example: etcd, kcm, cvo etc) are using sm.Namespace only. So, skipping coderabbitai's suggestion for now to use cpContext.HCP.Namespace instead of sm.Namespace

@dhgautam99
dhgautam99 marked this pull request as ready for review May 18, 2026 12:28
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 18, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@dhgautam99: This pull request references Jira Issue OCPBUGS-63328, which is invalid:

  • expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is In Progress instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@dhgautam99 dhgautam99 changed the title OCPBUGS-63328: Add kube-scheduler ServiceMonitor with CA-signed serving certs CNTRLPLANE-3584: Add kube-scheduler ServiceMonitor with CA-signed serving certs Jun 5, 2026
@openshift-ci-robot openshift-ci-robot removed the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Jun 5, 2026
@openshift-ci-robot

openshift-ci-robot commented Jun 5, 2026

Copy link
Copy Markdown

@dhgautam99: This pull request references CNTRLPLANE-3584 which is a valid jira issue.

Details

In response to this:

What this PR does / why we need it:

Adds a ServiceMonitor for kube-scheduler to enable Prometheus metrics scraping with proper mTLS authentication.

Previously, kube-scheduler auto-generated self-signed serving certificates via --cert-dir=/var/run/kubernetes. Prometheus could not verify the scheduler's identity using the cluster's root CA, and no ServiceMonitor existed.

This PR:

  • Creates a CA-signed serving certificate (scheduler-server) for kube-scheduler, following the KCM pattern
  • Replaces the self-signed cert (emptyDir) with the CA-signed cert (secret volume) in the deployment
  • Adds a kube-scheduler Service exposing port 10259
  • Adds a ServiceMonitor with mTLS config (root-ca for server verification, metrics-client for client auth)
  • Adds SchedulerRelabelConfigs() for SRE metrics set support

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPBUGS-63328

Special notes for your reviewer:

Follows the same pattern as kube-controller-manager's ServiceMonitor setup.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Summary by CodeRabbit

  • New Features

  • TLS serving certificate provisioning for the kube-scheduler with secret-backed cert/key mounting, Deployment switched to explicit cert/key flags, and a ClusterIP Service exposing port 10259.

  • ServiceMonitor added to scrape scheduler metrics over TLS using configured CA and client credentials.

  • Metrics

  • Added kube-scheduler relabel config support to metrics configuration.

  • Tests

  • Unit tests covering scheduler certificate reconciliation, ServiceMonitor adaptation, and scheduler component defaults.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Comment thread support/metrics/sets.go Outdated
func SchedulerRelabelConfigs(set MetricsSet) []prometheusoperatorv1.RelabelConfig {
switch set {
case MetricsSetTelemetry:
return sreMetricsSetConfig.KubeScheduler

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can't use sreMetricsSetConfig for telemetry. The config might not exist in this case

@dhgautam99
dhgautam99 force-pushed the create-kube-scheduler-servicemonitor branch from 7d26c7b to 160de75 Compare June 5, 2026 08:58
…erts

The kube-scheduler previously auto-generated self-signed serving
certificates via --cert-dir. This change adds a CA-signed serving
certificate, a Service, and a ServiceMonitor to enable Prometheus
metrics scraping with proper mTLS authentication.
Add unit tests for scheduler component options, ServiceMonitor adapter,
and server certificate reconciliation. Regenerate test fixtures after
adding Service, ServiceMonitor, and CA-signed serving certificate support.
@dhgautam99
dhgautam99 force-pushed the create-kube-scheduler-servicemonitor branch from 160de75 to e994412 Compare June 7, 2026 06:58
@muraee

muraee commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jun 8, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-self-managed
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

@hypershift-jira-solve-ci

Copy link
Copy Markdown
Contributor

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-azure-self-managed | Build: 2063890576113143808 | Cost: $4.1198559999999995 | Failed step: hypershift-azure-run-e2e-self-managed

View full analysis report


Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6

@hypershift-jira-solve-ci

hypershift-jira-solve-ci Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Test Failure Analysis Complete

Job Information

Test Failure Analysis

Error

TestNodePool/HostedCluster0/Main/TestRollingUpgrade: Failed to wait for 2 nodes to become ready
for NodePool e2e-clusters-bfntx/node-pool-64cj9-test-rolling-upgrade in 45m0s: context deadline exceeded
  - observed **v1.Node collection invalid: expected 2 nodes, got 0

Azure error: OSProvisioningTimedOut - OS Provisioning for VM
'node-pool-64cj9-test-rolling-upgrade-7wh2t-8ddpl' did not finish in the allotted time.

Summary

The TestRollingUpgrade test failed because both Azure VMs created for the rolling-upgrade NodePool hit the Azure-side OSProvisioningTimedOut error — the guest OS failed to complete provisioning within Azure's 20-minute timeout. The VMs were created at ~08:37 UTC and Azure marked them as Failed at ~08:57 UTC. Since the VMs never completed OS boot, they never registered as Kubernetes Nodes, causing the test to time out after 45 minutes waiting for 2 nodes that would never arrive. This is a transient Azure infrastructure issue unrelated to the PR changes. All 10 other Azure VMs in the same test run provisioned successfully (vmState: Succeeded), and the kube-scheduler pod (the component modified by this PR) was running healthy with 0 restarts.

Root Cause

The root cause is an Azure infrastructure transient failure — specifically, OSProvisioningTimedOut on two VMs in the centralus region.

What happened step by step:

  1. The TestRollingUpgrade subtest created a NodePool requesting 2 worker nodes via a MachineDeployment (node-pool-64cj9-test-rolling-upgrade-7wh2t)
  2. CAPI created two AzureMachines (8ddpl and mzrxz) which successfully provisioned network interfaces and availability sets
  3. Azure began OS provisioning of both VMs at ~08:37 UTC
  4. Both VMs failed to complete OS provisioning within Azure's 20-minute timeout:
    • VM mzrxz: started 08:37:10 → failed 08:57:30 (20m 20s)
    • VM 8ddpl: started 08:37:17 → failed 08:57:34 (20m 17s)
  5. Azure returned OSProvisioningTimedOut error code for both VMs, setting vmState: Failed
  6. Because the VMs never completed booting, kubelet never started, and the nodes never registered with the hosted cluster's API server
  7. The test waited the full 45-minute timeout for 2 nodes, got 0, and failed

Why this is NOT related to the PR:

  • The PR adds a kube-scheduler ServiceMonitor with CA-signed serving certs — it does not modify node provisioning, VM creation, or the CAPI/Azure machine reconciler
  • The kube-scheduler pod was running healthy (Ready=True, restartCount=0, started at 08:32:42)
  • The kube-scheduler ServiceMonitor was successfully deployed in the HCP namespace
  • 10 out of 12 Azure VMs in the same HostedCluster provisioned successfully; only the 2 rolling-upgrade VMs failed
  • All other tests (TestNodePoolReplaceUpgrade, TestNodePoolInPlaceUpgrade, TestNTOMachineConfigGetsRolledOut, etc.) passed on the same cluster
Recommendations
  1. Retest the PR — this is a transient Azure infrastructure failure with no relation to the code changes. A /retest should resolve it.
  2. No code changes needed — the kube-scheduler ServiceMonitor PR is working correctly. The failure is purely in Azure VM OS provisioning.
  3. If the failure recurs, check Azure service health for the centralus region. The OSProvisioningTimedOut error typically indicates Azure-side issues with VM agent startup, disk I/O, or cloud-init execution.
Evidence
Evidence Detail
Failed test TestNodePool/HostedCluster0/Main/TestRollingUpgrade (2709.41s)
Error code Azure OSProvisioningTimedOut on both VMs
VM 1 (mzrxz) Provisioning started 08:37:10, failed 08:57:30 — vmState: Failed
VM 2 (8ddpl) Provisioning started 08:37:17, failed 08:57:34 — vmState: Failed
Other VMs 10/12 AzureMachines in same cluster have vmState: Succeeded
kube-scheduler pod Running, Ready=True, restartCount=0 (started 08:32:42)
kube-scheduler ServiceMonitor Successfully deployed in HCP namespace
Cascading failures 4 total: TestRollingUpgrade → TestNodePool/HostedCluster0/Main → HostedCluster0 → TestNodePool
Test results 314 tests total, 29 skipped, 4 failures (all from single cascade)
Azure region centralus
CAPI event ReconcileError: failed to reconcile AzureMachine service virtualmachine

@dhgautam99

Copy link
Copy Markdown
Contributor Author

/retest

@dhgautam99

Copy link
Copy Markdown
Contributor Author

/verified by @dhgautam99 on lab cluster
Telemetry, SRE and All metric sets are working as expected.

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jun 9, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@dhgautam99: This PR has been marked as verified by @dhgautam99 on lab cluster.

Details

In response to this:

/verified by @dhgautam99 on lab cluster
Telemetry, SRE and All metric sets are working as expected.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

@dhgautam99: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot
openshift-merge-bot Bot merged commit ac1a1c2 into openshift:main Jun 9, 2026
42 checks passed
@dhgautam99
dhgautam99 deleted the create-kube-scheduler-servicemonitor branch June 9, 2026 11:52
@michaelryanmcneill

Copy link
Copy Markdown
Contributor

/cherry-pick release-4.22

@openshift-cherrypick-robot

Copy link
Copy Markdown

@michaelryanmcneill: #8489 failed to apply on top of branch "release-4.22":

Applying: feat(CPO): add kube-scheduler ServiceMonitor with CA-signed serving certs
Using index info to reconstruct a base tree...
M	control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go
M	control-plane-operator/controllers/hostedcontrolplane/manifests/pki.go
M	control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/component.go
Falling back to patching base and 3-way merge...
Auto-merging control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go
CONFLICT (content): Merge conflict in control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go
Auto-merging control-plane-operator/controllers/hostedcontrolplane/manifests/pki.go
Auto-merging control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/component.go
CONFLICT (content): Merge conflict in control-plane-operator/controllers/hostedcontrolplane/v2/kube_scheduler/component.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0001 feat(CPO): add kube-scheduler ServiceMonitor with CA-signed serving certs

Details

In response to this:

/cherry-pick release-4.22

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants