feat: add KMS etcd encryption SLI observability stack - #5690
openshift-merge-bot[bot] merged 1 commit into
Conversation
Dashboard Screenshots
|
f580183 to
c26672b
Compare
There was a problem hiding this comment.
⚠️ Not ready to approve
The KAS SRE metrics allow-list change would drop apiserver_request_sli_duration_seconds_{bucket,count} (breaking existing HCP latency SLIs), and the new dashboard’s histogram p99 queries are currently incorrect without rate(...[5m]).
Pull request overview
Adds an observability stack (metrics, recording rules, alerts, and a Grafana dashboard) to monitor the customer-managed etcd encryption (Azure KMS) user journey across HCPs.
Changes:
- Exposes new HostedCluster condition metrics via kube-state-metrics (
ValidAzureKMSConfig,EtcdAvailable) and adds new KMS-focused recording rules. - Introduces multi-window/multi-burn-rate alerting for KMS availability and KMS operation error budget burn, plus threshold alerts for latency/saturation/freshness.
- Adds an SRE “user journey” Grafana dashboard for KMS/etcd encryption SLIs.
File summaries
| File | Description |
|---|---|
| observability/recording-rules-hcps.yaml | Registers new KMS recording rule groups into the HCP recording-rules bundle. |
| observability/prometheus/values-mgmt.yaml | Adds KSM customResourceState metrics for ValidAzureKMSConfig and EtcdAvailable HostedCluster conditions. |
| observability/prometheus/testdata/zz_fixture_TestHelmTemplate_helmtest_mgmt_resources.yaml | Updates Helm golden fixture for the new KSM condition metrics. |
| observability/prometheus/testdata/zz_fixture_TestHelmTemplate_helmtest_mgmt_resources_unset.yaml | Updates Helm golden fixture for the new KSM condition metrics (unset variant). |
| observability/grafana-dashboards/sre/user-journey/kms-etcd-encryption.json | Adds the new SRE user-journey dashboard (11 panels) for KMS SLIs. |
| observability/alerts/HCPkmsMonitor-prometheusRule.yaml | Adds KMS alert rules (availability burn-rate tiers, errors burn-rate tiers, and threshold alerts). |
| observability/alerts/HCPkmsMonitor-prometheusRule_test.yaml | Adds promtool tests for the KMS alert rules. |
| observability/alerts/HCPkasRecord-prometheusRule-kms.yaml | Adds recording rules for KSM-based KMS availability SLI windows. |
| observability/alerts/HCPkasRecord-prometheusRule-kms-envelope.yaml | Adds recording rules for envelope-encryption KMS errors/latency/saturation SLIs. |
| observability/alerts/HCPkasRecord-prometheusRule-kms-envelope_test.yaml | Adds promtool tests for the envelope-encryption KMS recording rules. |
| observability/alerts/HCPkasRecord-prometheusRule-kms_test.yaml | Adds promtool tests for the KSM-based KMS recording rules. |
| observability/alerts-rp-services.yaml | Registers the new KMS alert rule file into the RP lane bundle. |
| hypershiftoperator/deploy/templates/sre-metrics-set.configmap.yaml | Updates the HyperShift SRE metrics-set allow-list for KAS metrics (now a restrictive regex). |
| hypershiftoperator/zz_fixture_TestHelmTemplate_dev_westus3_mgmt_1_hypershift.yaml | Updates the HyperShiftOperator Helm fixture to reflect the new metrics-set allow-list. |
| dev-infrastructure/zz_fixture_TestHelmTemplate_dev_westus3_mgmt_1_arohcp_monitor.yaml | Updates monitor Helm fixture for the new KSM condition metrics (rendered config). |
| dev-infrastructure/modules/metrics/rules/generatedRPPrometheusAlertingRules.bicep | Regenerates RP alerting rule groups to include the new KMS alerts. |
| dev-infrastructure/modules/metrics/rules/generatedHCPRecordingRules.bicep | Regenerates HCP recording rule groups to include the new KMS recording rules. |
Copilot's findings
- Files reviewed: 17/17 changed files
- Comments generated: 4
Note
Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.
c26672b to
e272e85
Compare
There was a problem hiding this comment.
⚠️ Not ready to approve
The generated RP alerting rules Bicep appears to have dropped existing nodepool rule groups and the new dashboard contains PromQL bugs (histogram_quantile without rate and NaN-on-idle error-rate expressions).
Copilot's findings
- Files reviewed: 15/15 changed files
- Comments generated: 6
Note
Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.
e272e85 to
fb81950
Compare
Simon Wiencki (swiencki)
left a comment
Simon Wiencki (swiencki)
left a comment
There was a problem hiding this comment.
Reviewed in detail against the alerting ADR (openshift-online/architecture#79), the existing KAS availability pattern (#5527), and current prod metric availability. The KMS observability stack is well-constructed: the availability SLI uses the correct stateless kube-state-metrics pattern with deleted-cluster gating and full label preservation, the burn-rate math is right, promtool tests pass, and the dashboard is in the correct SRE user-journey location and registered.
The one blocking issue is wiring: the alerts are deployed to the services Azure Monitor Workspace while the recording rules they consume are produced in the HCP workspace, so as wired the alerts cannot fire. There are also several ADR-conformance and metric-availability gaps noted inline. Details below.
fb81950 to
04e4649
Compare
There was a problem hiding this comment.
⚠️ Not ready to approve
The KMS envelope recording rules drop the _id label on the error-series, which prevents the error-rate alert expressions from matching errors/total and will stop KMS error alerts from firing.
Copilot's findings
- Files reviewed: 15/15 changed files
- Comments generated: 2
Note
Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.
2a11782 to
f5b6e97
Compare
There was a problem hiding this comment.
⚠️ Not ready to approve
There are correctness issues in the new PromQL recording/alerting setup (label-set mismatch risk) and the implemented kube-state-metrics conditions don’t match the PR description (missing EtcdAvailable).
Copilot's findings
- Files reviewed: 17/17 changed files
- Comments generated: 2
Note
Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.
|
Was there a historical metrics report ran on the alerts that I can review? If not you will want to reach out to Mariusz Mazur (@mmazur) on what that looks like. |
90c6b6e to
07d2c59
Compare
a819220 to
fa98435
Compare
fa98435 to
77d9031
Compare
Add KSM custom resource state config to expose the ValidAzureKMSConfig HostedCluster condition as hostedClusterAPI_valid_azure_kms_config gauge. This is the metrics collection foundation for the KMS etcd encryption user journey (ARO-25913). Recording rules, alerts, and a Grafana dashboard will follow in a separate PR after metrics have soaked in production and thresholds can be validated.
00ae54a to
7bee246
Compare
|
/retest e2e-parallel |
|
/test e2e-parallel |
|
/retest e2e-parallel |
|
/lgtm |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: mmazur, swiencki, wanghaoran1988 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Haoran Wang (@wanghaoran1988): The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Summary
Expose the
ValidAzureKMSConfigHostedCluster condition as a Prometheus gauge via kube-state-metrics custom resource state config (ARO-25913).This is the metrics collection foundation for the KMS etcd encryption user journey. It ships only the KSM metric — recording rules, alerts, and a Grafana dashboard will follow in a separate PR after metrics have soaked in production.
What this PR adds
observability/prometheus/values-mgmt.yamlValidAzureKMSConfigcondition ashostedClusterAPI_valid_azure_kms_configPrometheus gauge (StateSet: True/False/Unknown)What's deferred (follow-up PR)
apiserver_envelope_encryption_*)E2E verification
pers-usw3haow-mgmt-1)hostedClusterAPI_valid_azure_kms_config{status="True"}confirmed flowing to Azure Monitor WorkspaceTest plan
make testpasses (helm template fixtures updated)haowangtest HCP cluster