feat(base-cluster/monitoring): non-critical alerts aren't routed to on-call

[cwrau]

Period WorkingHours prevents this from waking someone up, but it will
still be sent out during the day.

Summary by CodeRabbit

• New Features
  • Updated several monitoring alerts to use a higher severity level ("critical" instead of "warning") for improved visibility.
  • Added a new "period: WorkingHours" label to multiple alerts to indicate active monitoring during business hours.

[coderabbitai]

Walkthrough

Alert rules across multiple PrometheusRule templates were updated to change the severity label from "warning" to "critical" where applicable, and to add a new label "period: WorkingHours" to all affected alerts. No modifications were made to alert expressions, durations, or other metadata.

Changes

File(s)	Change Summary
charts/base-cluster/templates/backup/velero.yaml	Changed Velero backup failure alert severity from "warning" to "critical"; added "period: WorkingHours" label.
charts/base-cluster/templates/cert-manager/rules/certificate-expiration.yaml	Changed certificate expiration alerts' severity to "critical"; added "period: WorkingHours" label to both.
charts/base-cluster/templates/flux/rules/flux-status.yaml	Changed "ResourcesFailing" alert severity to "critical"; added "period: WorkingHours" label to both alerts.
charts/base-cluster/templates/monitoring/kdave/rules/releases-with-deprecation.yaml	Changed deprecation alert severity to "critical"; added "period: WorkingHours" label to both alerts.
charts/base-cluster/templates/nfs-server-provisioner/rules/storage-size.yaml	Changed NFS over-provisioned alert severity to "critical"; added "period: WorkingHours" label.

Suggested labels

base-cluster

Suggested reviewers

• marvinWolff
• tasches
• teutonet-bot

Poem

🐇
Alerts now shout, not softly warn,
"Critical!" they cry from dusk to dawn.
With "WorkingHours" they wear a tag,
Ensuring no alert will lag.
The cluster’s safe, the rabbits cheer—
For warnings loud, we all can hear!

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between b8b3f80 and 2553aab.

📒 Files selected for processing (5)

• charts/base-cluster/templates/backup/velero.yaml (1 hunks)
• charts/base-cluster/templates/cert-manager/rules/certificate-expiration.yaml (2 hunks)
• charts/base-cluster/templates/flux/rules/flux-status.yaml (2 hunks)
• charts/base-cluster/templates/monitoring/kdave/rules/releases-with-deprecation.yaml (2 hunks)
• charts/base-cluster/templates/nfs-server-provisioner/rules/storage-size.yaml (1 hunks)

🔇 Additional comments (8)

charts/base-cluster/templates/backup/velero.yaml (1)

105-106: Elevate VeleroBackupFailures to critical and limit routing to working hours.
Severity label updated to "critical" and "period: WorkingHours" ensures this alert only routes during defined working hours.

charts/base-cluster/templates/nfs-server-provisioner/rules/storage-size.yaml (1)

28-29: Promote NFSOverProvisioned alert to critical during working hours.
Changing severity to "critical" and adding the "period" label correctly restricts notification to working hours.

charts/base-cluster/templates/flux/rules/flux-status.yaml (2)

27-28: Promote ResourcesFailing to critical and restrict to working hours.
Updating severity and adding the working-hours label aligns this alert with on-call schedules.

---

38-38: Constrain MetricsMissing alert notifications to working hours.
Adding "period: WorkingHours" ensures this critical alert only fires during the intended window.

charts/base-cluster/templates/monitoring/kdave/rules/releases-with-deprecation.yaml (2)

27-28: Escalate ReleasesHaveDeprecatedAPIs and apply working-hours constraint.
Converting severity to "critical" and adding the period label correctly scopes this alert.

---

38-38: Add working-hours period to ReleasesHaveRemovedAPIs alert.
This label ensures the existing critical alert only routes to on-call during working hours.

charts/base-cluster/templates/cert-manager/rules/certificate-expiration.yaml (2)

27-28: Elevate CertificateExpiringSoon (<14d) to critical in working hours.
Updating severity and adding "period: WorkingHours" aligns with standard alert routing practices.

---

38-38: Restrict CertificateExpiringSoon (<7d) notifications to working hours.
Adding the period label scopes this existing critical alert to the appropriate window.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[Copilot]

Pull Request Overview

This PR updates several alert rules to ensure non-critical notifications are only sent during defined working hours and treats them as critical alerts.

• Changed severity from warning to critical in multiple PrometheusRule templates
• Added a period: WorkingHours label to relevant alert definitions
• Applied updates across NFS provisioner, Kubernetes deprecation, Flux, cert-manager, and Velero charts

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
charts/base-cluster/templates/nfs-server-provisioner/rules/storage-size.yaml	Bumped severity and added working-hours period
charts/base-cluster/templates/monitoring/kdave/rules/releases-with-deprecation.yaml	Updated severity and period for deprecation alerts
charts/base-cluster/templates/flux/rules/flux-status.yaml	Adjusted severity and added working-hours period
charts/base-cluster/templates/cert-manager/rules/certificate-expiration.yaml	Changed severity and added period to expiration rule
charts/base-cluster/templates/backup/velero.yaml	Updated Velero backup alert labels

Comments suppressed due to low confidence (2)

charts/base-cluster/templates/nfs-server-provisioner/rules/storage-size.yaml:29

• [nitpick] These severity and period label blocks are duplicated across multiple rule files. Consider extracting them into a Helm partial (e.g., _working_hours_labels) to reduce repetition and simplify future updates.

period: WorkingHours

[teutonet-bot]

🤖 I have diffed this beep boop

"/$namespace/$kind/$name.yaml" for normal resources
"/$namespace/HelmRelease/$name/$namespace/$kind/$name.yaml" for HelmReleases <- this is recursive
'null' means it's either cluster-scoped or it's in the default namespace for the HelmRelease

charts/base-cluster/ci/monitoring-ingress-unauthenticated-values.yaml

charts/base-cluster/ci/rbac-values.yaml

charts/base-cluster/ci/pagerduty-values.yaml

charts/base-cluster/ci/velero-backupStorageLocations-gen-values.yaml

charts/base-cluster/ci/flux-gitrepositories-gen-values.yaml

charts/base-cluster/ci/traefik-ingress-values.yaml

charts/base-cluster/ci/imagepullsecrets-values.yaml

charts/base-cluster/values.yaml

charts/base-cluster/ci/artifacthub-values.yaml

charts/base-cluster/ci/disabled-ingress-values.yaml

charts/base-cluster/ci/priorityclasses-values.yaml

[charts/base-cluster/ci/deadmansswitch-values.yaml](

<title> Error </title> <script src="https://unpkg.com/htmx.org@2.0.2" integrity="sha384-Y7hw+L/jvKeWIRRkqWYfPcvVxHzVzn5REgzbawhxAuQGwX1XWe70vji+VSeHOThJ" crossorigin="anonymous"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.7.1/jquery.min.js"></script> <script defer src="https://media.ethicalads.io/media/client/ethicalads.min.js"></script> <script> function key(event, letter) { return (event.charCode == letter.charCodeAt() || event.charCode == letter.charCodeAt() + 32) } </script> <script>

    </script>
    <style></style>
    
    
</head>

<body hx-on:keydown="var path=window.location.pathname; if ((event.key=='n' || event.key=='N') && path && path !== '/' && !path.includes('/duplicate/') && !path.includes('/dashboard/') && !path.includes('/accounts/') && !path.includes('/properties/')) { window.location.href = '/'; }">
    <div id="container">
        
        <div class="topbuttons">
            
                <a href="/accounts/login/"
                   title="Personal dashboard, preferences, API token"
                   class="button"><b>Sign in</b></a>
            
            <a href="/" title="Create a new paste (shortcut: 'N')" class=" button">New</a>
            <a href="/api/v2/"
               title="Paste creation API, with code samples"
               class=" activebutton  button">API</a>
            <a href="/help/"
               title="Usage tips, shortcuts"
               class="  button">Help</a>
            <a href="/about/"
               title="Updates, stats, backstory"
               class=" button">About</a>
        </div>
        
<div class="error">
    <h3>Sorry!</h3>
    Sorry, maximum content size is 1,000,000 bytes. Your submission is 2537976 bytes. Your IP will be blocked if you exceed this limit again in the next 60 seconds.
    <hr>
    <p>
        Something unexpected? Please create a <a href="https://dpaste.freshdesk.com/support/tickets/new">support ticket</a>.
    </p>
</div>
<p>
    <a class="button" href="{$ url 'create' %}" onclick="history.back();return false;">Go back</a>
</p>

    </div>
    
    
    <script>
        window.fwSettings={'widget_id':22000000180 };
        !function(){if("function"!=typeof window.FreshworksWidget){var n=function(){n.q.push(arguments)};n.q=[],window.FreshworksWidget=n}}();
    </script>
    <script defer src='https://widget.freshworks.com/widgets/22000000180.js'></script>
</body>

)

charts/base-cluster/ci/monitoring-oidc-values.yaml

charts/base-cluster/ci/basic-values.yaml

charts/base-cluster/ci/monitoring-oidc-ingress-disabled-values.yaml

charts/base-cluster/ci/limitrange-resourcequota-values.yaml