feat(observability): hcp kube-apiserver error rate and latency alerts to SRE - #6080
Conversation
Restore alert, recording rule, and lane config files to their pre-revert state (before PR Azure#5754). Content is verbatim from commit 167ca21. Follow-up commits fix path prefixes, add severityCeiling, wire into deployment infrastructure, and address review feedback. Original work: PR Azure#5527 (SudoBrendan)
… alert files Just to keep the github UI clean for review, we rename premptively. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Rename UJ prefix to userJourney (camelCase) per ADR naming convention - Downgrade all alert severities to warning for initial rollout, use CEN - Add severityCeiling parameter to rp-hcp-rules.bicep - Wire rpHcpAlerts module into monitoring.bicep - Add run-rp-hcps build target to prometheus-rules Makefile - Fix alerts-rp-hcps.yaml path prefix to match main conventions - Rename CID to OCM Cluster ID in descriptions - Separate Management Cluster and Namespace into distinct lines - Add Environment and Region to descriptions (external labels) - Change summary format to [HCPKASAvailableBurn] with namespace - Add resource_id label to recording rule aggregation chains - Update all test expectations to match Co-Authored-By: Simon Wiencki <swiencki@redhat.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Generated from `make alerts recording-rules` in observability/. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 6 multi-window burn-rate alerts for kube-apiserver grounded in the ARO 99.95% SLA: 3 error rate and 3 latency, each with fast (14.4x, 1h/5m), medium (6x, 6h/30m), and slow (1x, 3d/6h) tiers. Each alert includes traffic guards (>1 req/s), provisioning guards (offset 15m to exclude newly created clusters), correlationId for IcM dedup, and severity "3". Promtool test files cover 7 scenarios each. Updates the KAS Golden Signals dashboard to reference the actual alert names and corrects the SLO target from 99.9% to 99.95% in all text and PromQL expressions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: SudoBrendan The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Skipping CI for Draft Pull Request. |
Brendan Bergen (SudoBrendan)
changed the title
| correlationId: "userJourneyKubeApiserverErrors/{{ $labels.cluster }}/{{ $labels.namespace }}" | ||
| - alert: userJourneyKubeApiserverErrors6h30m | ||
| expr: | | ||
| ( |
There was a problem hiding this comment.
The comment says a follow-up will align slow-burn to severity "4"; add a tracking Jira reference so it doesn't get lost.
| ( | ||
| kas:apiserver_request_5xx:rate_avg_1h / kas:apiserver_request_total:rate_avg_1h | ||
| > (14.4 * (1 - 0.9995)) | ||
| and on (namespace, cluster) |
There was a problem hiding this comment.
The correlationId uses namespace while the availability alerts use _id; consider aligning for consistent IcM incident grouping across all KAS alert types.
There was a problem hiding this comment.
I like the idea, but this is actually a bit complicated. To get _id requires a join of some kind, because the KAS itself (where this metric comes from) is _id (hypershift) unaware. To do this, we'd introduce dependencies across our various SLIs.
As an alternative, we could update the other HCP alert to use namespace as our identifier instead of _id. It's worth knowing that those alerts would have correlation IDs like cluster-1/ocm-env-abc123-foo and these would have cluster-1/ocm-env-abc123, but the prefixes already distinguish different alerts from each other, and honestly I think this identifies the alerts a bit better (where is the SLI coming from, rather than some random GUID from OCM that I don't see SRE relying on) anyways? WDYT?
There was a problem hiding this comment.
Using namespace is more inline with what SRE would require. Question is do we want to make this the standard?
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
| - alert: userJourneyKubeApiserverLatency1h5m | ||
| expr: | | ||
| ( | ||
| (1 - kas:apiserver_request_latency:sli_ratio:rate5m) |
There was a problem hiding this comment.
All the kas:apiserver_request_latency:sli_ratio: are dependent on and blockd by upstream bug.
The bug has been merged and backported till 4.20 (our default version). Once rolled out for our HCp clusters, should emit the metrics for use.
3 participants
What
Why
Testing
Testing is required for feature completion and tests should be part of the pull
request along with the feature changes.
Describe the testing provided. If you did not add tests, provide a clear
justification.
Special notes for your reviewer
PR Checklist
If E2E tests are included:
demonstrate that the test is able to detect a defect/error and fail with
proper error message and logs which communicates nature of the problem.