OCPBUGS-91997: Apply MetricsSet relabel configs to KAS ServiceMonitor - #8715
Conversation
|
Pipeline controller notification For optional jobs, comment This repository is configured in: LGTM mode |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis PR imports the metrics package and updates adaptServiceMonitor to populate the ServiceMonitor's first endpoint MetricRelabelConfigs via metrics.KASRelabelConfigs(cpContext.MetricsSet) before injecting the cluster ID label. It also adds TestAdaptServiceMonitor, a table-driven test that runs parallel subtests to verify namespace selector assignment, metric relabel config application for different MetricsSet values, and cluster ID relabel injection when MetricsSetAll is used. Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (2 inconclusive)
✅ Passed checks (9 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
openshift-ci
Bot
added
area/control-plane-operator
63aef41 to
7e6810a
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
control-plane-operator/controllers/hostedcontrolplane/v2/kas/servicemonitor_test.go (1)
36-47: ⚡ Quick winConsider validating metric relabel config content, not just existence.
The test checks that
MetricRelabelConfigsis not nil, but doesn't validate the actual configs returned bymetrics.KASRelabelConfigs. ForMetricsSetTelemetry, you could verify the length is 1 and spot-check theActionorRegexfield to ensure the correct config was applied.💡 Example assertion strengthening
g.Expect(sm.Spec.Endpoints[0].MetricRelabelConfigs).ToNot(BeNil()) +g.Expect(sm.Spec.Endpoints[0].MetricRelabelConfigs).To(HaveLen(1)) +g.Expect(sm.Spec.Endpoints[0].MetricRelabelConfigs[0].Action).To(Equal("keep"))🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@control-plane-operator/controllers/hostedcontrolplane/v2/kas/servicemonitor_test.go` around lines 36 - 47, The test case "When service monitor is adapted, it should apply metric relabel configs" currently only asserts that sm.Spec.Endpoints[0].MetricRelabelConfigs is not nil; update the validate closure to compare the actual relabel configs to the expected output from metrics.KASRelabelConfigs by asserting the slice length equals 1 and spot-checking one or two key fields (e.g., MetricRelabelConfigs[0].Action and MetricRelabelConfigs[0].Regex) against the known values returned by metrics.KASRelabelConfigs (or compare the first element directly) to ensure the correct config was applied.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@control-plane-operator/controllers/hostedcontrolplane/v2/kas/servicemonitor_test.go`:
- Around line 48-67: Update the test's validate closure to also assert the
cluster ID relabel config is present in MetricRelabelConfigs on the
ServiceMonitor endpoints (not just RelabelConfigs): after locating
sm.Spec.Endpoints[0].RelabelConfigs, iterate
sm.Spec.Endpoints[0].MetricRelabelConfigs and verify there is a config with
TargetLabel == "_id" and Replacement == "cluster-abc-123"; reference
util.ApplyClusterIDLabel and the ServiceMonitor endpoint slice
(sm.Spec.Endpoints) when adding this assertion so both RelabelConfigs and
MetricRelabelConfigs are validated.
---
Nitpick comments:
In
`@control-plane-operator/controllers/hostedcontrolplane/v2/kas/servicemonitor_test.go`:
- Around line 36-47: The test case "When service monitor is adapted, it should
apply metric relabel configs" currently only asserts that
sm.Spec.Endpoints[0].MetricRelabelConfigs is not nil; update the validate
closure to compare the actual relabel configs to the expected output from
metrics.KASRelabelConfigs by asserting the slice length equals 1 and
spot-checking one or two key fields (e.g., MetricRelabelConfigs[0].Action and
MetricRelabelConfigs[0].Regex) against the known values returned by
metrics.KASRelabelConfigs (or compare the first element directly) to ensure the
correct config was applied.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: d73fe37b-a2da-42f3-af73-b1908d4d4dae
📒 Files selected for processing (2)
control-plane-operator/controllers/hostedcontrolplane/v2/kas/servicemonitor.gocontrol-plane-operator/controllers/hostedcontrolplane/v2/kas/servicemonitor_test.go
muraee
left a comment
muraee
left a comment
There was a problem hiding this comment.
Nice catch — the production fix is correct and minimal. KAS was the only component missing the KASRelabelConfigs() call.
A few issues with the test, and a cleanup suggestion — see inline comments.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8715 +/- ##
=======================================
Coverage 41.54% 41.55%
=======================================
Files 758 758
Lines 93838 93839 +1
=======================================
+ Hits 38986 38994 +8
+ Misses 52107 52100 -7
Partials 2745 2745
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
|
/approve |
openshift-ci
Bot
added
the
approved
81a8677 to
8147217
Compare
|
Thanks for the review @muraee! Addressed all feedback:
|
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
cat control-plane-operator/controllers/hostedcontrolplane/v2/kas/servicemonitor_test.goLength of output: 4825
The test suite looks solid. Thanks for the thorough follow-up! |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@control-plane-operator/controllers/hostedcontrolplane/v2/kas/servicemonitor_test.go`:
- Around line 45-48: The test comment says there should be two
MetricRelabelConfigs (a "keep" rule plus a cluster ID relabel) but the assertion
currently only checks >= 1; update the assertions in servicemonitor_test.go to
assert the expected count and presence of both rules by checking
len(sm.Spec.Endpoints[0].MetricRelabelConfigs) is >= 2 (or == 2 if exact) and
then verify the second config corresponds to the cluster ID relabel (inspect
sm.Spec.Endpoints[0].MetricRelabelConfigs[1].Action/Regex or the relabel target)
in addition to the existing check of MetricRelabelConfigs[0] being the "keep"
rule.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: b925ef48-0a99-4744-b22a-f9d4e63d7bc2
⛔ Files ignored due to path filters (5)
control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/AROSwift/zz_fixture_TestControlPlaneComponents_kube_apiserver_servicemonitor.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/GCP/zz_fixture_TestControlPlaneComponents_kube_apiserver_servicemonitor.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/IBMCloud/zz_fixture_TestControlPlaneComponents_kube_apiserver_servicemonitor.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/TechPreviewNoUpgrade/zz_fixture_TestControlPlaneComponents_kube_apiserver_servicemonitor.yamlis excluded by!**/testdata/**control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/zz_fixture_TestControlPlaneComponents_kube_apiserver_servicemonitor.yamlis excluded by!**/testdata/**
📒 Files selected for processing (2)
control-plane-operator/controllers/hostedcontrolplane/v2/kas/servicemonitor.gocontrol-plane-operator/controllers/hostedcontrolplane/v2/kas/servicemonitor_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
- control-plane-operator/controllers/hostedcontrolplane/v2/kas/servicemonitor.go
The KAS adaptServiceMonitor() function was missing the line that applies MetricRelabelConfigs from the configured MetricsSet (Telemetry/SRE/All). Every other component (etcd, KCM, CVO, scheduler, OAPI, OCM, routecm, NTO, OLM, catalog-operator) correctly calls its corresponding metrics.XxxRelabelConfigs(cpContext.MetricsSet), but KAS did not. This caused the KAS ServiceMonitor to always use the hardcoded Telemetry default from the YAML asset (3 metrics: apiserver_storage_objects, apiserver_request_total, apiserver_current_inflight_requests), ignoring the sre-metric-set ConfigMap entirely when METRICS_SET=SRE. The fix adds the missing call to metrics.KASRelabelConfigs() and a unit test matching the pattern used by KCM (kcm/servicemonitor_test.go).
8147217 to
707e294
Compare
|
/jira ARO-25913 |
Add apiserver_envelope_encryption_* and apiserver_encryption_config_* metrics to the kubeAPIServer section of the sre-metrics-set ConfigMap. These metrics are needed for the KMS SLI (ARO-25913) but are currently dropped by the KAS ServiceMonitor's metricRelabelings. A HyperShift bug fix (openshift/hypershift#8715) is required for the ConfigMap to take effect — until then, a workaround ServiceMonitor must be created manually per HCP namespace.
Add apiserver_envelope_encryption_* and apiserver_encryption_config_* metrics to the kubeAPIServer section of the sre-metrics-set ConfigMap. These metrics are needed for the KMS SLI (ARO-25913) but are currently dropped by the KAS ServiceMonitor's metricRelabelings. A HyperShift bug fix (openshift/hypershift#8715) is required for the ConfigMap to take effect — until then, a workaround ServiceMonitor must be created manually per HCP namespace.
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: muraee, swiencki, wanghaoran1988 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/lgtm |
|
/verified bypass |
openshift-ci-robot
added
the
verified
|
@wanghaoran1988: The DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
HyperShift ServiceMonitor bug fix (openshift/hypershift#8715) will take months to roll out. Without it, KAS pod metrics are not scraped, causing false positives from error, saturation, and traffic alerts. - Remove hcp-kms-errors-rules, hcp-kms-saturation-rules, hcp-kms-traffic-rules alert groups (blocked by upstream fix) - Downgrade all availability alerts to severity: info (Sev4) - Keep recording rules and dashboard intact for when fix lands
|
@wanghaoran1988: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
@wanghaoran1988: Jira Issue Verification Checks: Jira Issue OCPBUGS-91997 Jira Issue OCPBUGS-91997 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Fix included in release 5.0.0-0.nightly-2026-06-25-122140 |
HyperShift ServiceMonitor bug fix (openshift/hypershift#8715) will take months to roll out. Without it, KAS pod metrics are not scraped, causing false positives from error, saturation, and traffic alerts. - Remove hcp-kms-errors-rules, hcp-kms-saturation-rules, hcp-kms-traffic-rules alert groups (blocked by upstream fix) - Downgrade all availability alerts to severity: info (Sev4) - Keep recording rules and dashboard intact for when fix lands
|
/cherry-pick release-4.18 |
|
/cherry-pick release-4.19 |
|
/cherry-pick release-4.20 |
|
/cherry-pick release-4.21 |
|
/cherry-pick release-4.22 |
|
@redhat-chai-bot: #8715 failed to apply on top of branch "release-4.18": DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@redhat-chai-bot: #8715 failed to apply on top of branch "release-4.19": DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@redhat-chai-bot: #8715 failed to apply on top of branch "release-4.20": DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@redhat-chai-bot: new pull request created: #8900 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@redhat-chai-bot: new pull request created: #8901 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
HyperShift ServiceMonitor bug fix (openshift/hypershift#8715) will take months to roll out. Without it, KAS pod metrics are not scraped, causing false positives from error, saturation, and traffic alerts. - Remove hcp-kms-errors-rules, hcp-kms-saturation-rules, hcp-kms-traffic-rules alert groups (blocked by upstream fix) - Downgrade all availability alerts to severity: info (Sev4) - Keep recording rules and dashboard intact for when fix lands
HyperShift ServiceMonitor bug fix (openshift/hypershift#8715) will take months to roll out. Without it, KAS pod metrics are not scraped, causing false positives from error, saturation, and traffic alerts. - Remove hcp-kms-errors-rules, hcp-kms-saturation-rules, hcp-kms-traffic-rules alert groups (blocked by upstream fix) - Downgrade all availability alerts to severity: info (Sev4) - Keep recording rules and dashboard intact for when fix lands
HyperShift ServiceMonitor bug fix (openshift/hypershift#8715) will take months to roll out. Without it, KAS pod metrics are not scraped, causing false positives from error, saturation, and traffic alerts. - Remove hcp-kms-errors-rules, hcp-kms-saturation-rules, hcp-kms-traffic-rules alert groups (blocked by upstream fix) - Downgrade all availability alerts to severity: info (Sev4) - Keep recording rules and dashboard intact for when fix lands
HyperShift ServiceMonitor bug fix (openshift/hypershift#8715) will take months to roll out. Without it, KAS pod metrics are not scraped, causing false positives from error, saturation, and traffic alerts. - Remove hcp-kms-errors-rules, hcp-kms-saturation-rules, hcp-kms-traffic-rules alert groups (blocked by upstream fix) - Downgrade all availability alerts to severity: info (Sev4) - Keep recording rules and dashboard intact for when fix lands
8 participants
Summary
adaptServiceMonitor()to applyMetricRelabelConfigsfrom the configuredMetricsSet, matching the pattern used by every other component (etcd, KCM, CVO, scheduler, OAPI, OCM, routecm, NTO, OLM, catalog-operator)TestAdaptServiceMonitorfollowing the same pattern askcm/servicemonitor_test.goProblem
The KAS ServiceMonitor always uses the hardcoded Telemetry default from the YAML asset file (3 metrics only:
apiserver_storage_objects,apiserver_request_total,apiserver_current_inflight_requests), regardless of theMETRICS_SETenvironment variable.When
METRICS_SET=SREis set and thesre-metric-setConfigMap is configured with additional metrics (e.g.,apiserver_envelope_encryption_*for KMS monitoring), the KAS ServiceMonitor ignores them becauseadaptServiceMonitor()never callsmetrics.KASRelabelConfigs(cpContext.MetricsSet).Every other component correctly applies its metrics set:
MetricRelabelConfigslinemetrics.EtcdRelabelConfigs(cpContext.MetricsSet)metrics.KCMRelabelConfigs(cpContext.MetricsSet)metrics.CVORelabelConfigs(cpContext.MetricsSet)metrics.SchedulerRelabelConfigs(cpContext.MetricsSet)metrics.OpenShiftAPIServerRelabelConfigs(cpContext.MetricsSet)metrics.OpenShiftControllerManagerRelabelConfigs(cpContext.MetricsSet)metrics.OpenShiftRouteControllerManagerRelabelConfigs(cpContext.MetricsSet)metrics.NTORelabelConfigs(cpContext.MetricsSet)metrics.OLMRelabelConfigs(cpContext.MetricsSet)metrics.CatalogOperatorRelabelConfigs(cpContext.MetricsSet)Fix
Add the missing line to
v2/kas/servicemonitor.go:Test plan
TestAdaptServiceMonitoradded (mirrorskcm/servicemonitor_test.go)go test ./control-plane-operator/controllers/hostedcontrolplane/v2/kas/ -run TestAdaptServiceMonitor -vSummary by CodeRabbit
New Features
Tests