feat(base-cluster/monitoring): add alert about base-cluster updates

[cwrau]

Summary by CodeRabbit

• New Features
  • Added Prometheus monitoring for base-cluster with recording rules to track current chart version.
  • Introduced a critical alert to notify when a base-cluster update is available.
  • Added automatic version tracking (image repository/policy) for the base-cluster chart.
  • Extended metrics to expose image policy latest-version/tag information for monitoring.

[gemini-code-assist]

Summary of Changes

Hello @cwrau, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the base-cluster monitoring infrastructure by integrating automated update detection and alerting. It establishes a mechanism to track both the currently deployed chart version and the latest available version using Flux Image Automation and Prometheus. The system is designed to proactively notify administrators via a critical alert when a new base-cluster chart release is published, streamlining the update management process and improving operational awareness.

Highlights

• Current Version Tracking: Introduced a PrometheusRule to record the currently deployed base-cluster chart version as a metric, enabling comparison with available updates.
• Latest Version Tracking: Added Flux ImageRepository and ImagePolicy resources to automatically track the latest available version of the base-cluster Helm chart from its registry.
• ImagePolicy Metrics: Modified the kube-state-metrics configuration to expose metrics from Flux ImagePolicy resources, specifically making the latest available image version accessible to Prometheus.
• Update Alerting: Implemented a critical Prometheus alert, BaseClusterUpdateAvailable, which triggers when a new base-cluster chart version is detected as available, ensuring timely notification for administrators.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds Helm templates to track base-cluster chart versions and alert on updates: FluxCD ImageRepository/ImagePolicy, a kube-state-metrics config entry to export ImagePolicy metrics, a Prometheus recording rule for current version, and a Prometheus alert rule triggered when a newer chart version is available.

Changes

Cohort / File(s)	Summary
Base-Cluster Update Monitoring charts/base-cluster/templates/base-cluster-update/current-source.yaml, charts/base-cluster/templates/base-cluster-update/update-alert.yaml, charts/base-cluster/templates/base-cluster-update/update-source.yaml	Adds FluxCD resources (ImageRepository, ImagePolicy) for base-cluster-chart, a Prometheus recording rule base_cluster_current_version:info, and a PrometheusRule alert BaseClusterUpdateAvailable (fires after 1w) — all rendered only when values.monitoring.prometheus.enabled is true and labeled for kube-prometheus-stack.
Kube-State-Metrics Configuration charts/base-cluster/templates/monitoring/kube-prometheus-stack/_kube-state-metrics-config.yaml	Appends an ImagePolicy entry to customResourceState.config exporting image_policy_latest_version metrics with latest_version (Info-type) and tag label mapped from status.latestRef.tag; adds customresource_namespace/customresource_name labels and minor whitespace tweak.

Sequence Diagram

sequenceDiagram
    participant IR as ImageRepository / ImagePolicy (FluxCD)
    participant KSM as Kube-State-Metrics
    participant PR as Prometheus Recording Rule
    participant PA as Prometheus Alert Rule

    IR->>IR: Poll registry (24h)
    IR->>KSM: ImagePolicy status (latestRef.tag)
    KSM->>PR: Expose metric image_policy_latest_version{tag}
    PR->>PR: Record base_cluster_current_version:info
    PA->>PR: Query recording rule + image_policy metrics
    PA->>PA: Evaluate alert expression
    PA->>PA: Fire BaseClusterUpdateAvailable if condition met

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• chore(base-cluster/monitoring): adjust metrics syntax #1562: Modifies the same kube-state-metrics config to add customresource label keys used by the new ImagePolicy metrics.
• feat(base-cluster/ingress)!: add option traefik for ingress controller and make it default #1420: Also edits kube-state-metrics config to add/adjust Flux resource metric exports related to ImagePolicy.
• feat(base-cluster/monitoring): non-critical alerts aren't routed to on-call #1533: Adjusts PrometheusRule alert templates and labeling conventions similar to the new alert rule added here.

Suggested reviewers

• tasches
• marvinWolff
• teutonet-bot

Poem

🐰 I nibble charts by moonlit stream,

I count the versions in a dream,
Metrics hum and alerts take flight,
Hop—an update gleams so bright,
Rejoice, the cluster leaps tonight. ✨

🚥 Pre-merge checks | ✅ 2 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Merge Conflict Detection	⚠️ Warning	❌ Merge conflicts detected (20 files): ⚔️ .github/renovate.json (content) ⚔️ .github/workflows/actionlint.yaml (content) ⚔️ .github/workflows/check-licenses.yaml (content) ⚔️ .github/workflows/create-release-prs.yaml (content) ⚔️ .github/workflows/fetch-code-scanning-results.yaml (content) ⚔️ .github/workflows/get-all-charts.yaml (content) ⚔️ .github/workflows/linter.yaml (content) ⚔️ .github/workflows/pr-comment-diff.yaml (content) ⚔️ .github/workflows/release-chart.yaml (content) ⚔️ .github/workflows/release-update-metadata.yaml (content) ⚔️ .github/workflows/scan-for-cves.yaml (content) ⚔️ .github/workflows/update-codeowners.yaml (content) ⚔️ .github/workflows/validate-pullrequest.yaml (content) ⚔️ charts/base-cluster/templates/monitoring/kube-prometheus-stack/_kube-state-metrics-config.yaml (content) ⚔️ charts/base-cluster/values.yaml (content) ⚔️ charts/common/Chart.lock (content) ⚔️ charts/common/Chart.yaml (content) ⚔️ charts/t8s-cluster/templates/management-cluster/autoscaler.yaml (content) ⚔️ charts/t8s-cluster/templates/management-cluster/cluster.yaml (content) ⚔️ charts/t8s-cluster/templates/workload-cluster/cinder-csi-plugin/cinder-csi-plugin.yaml (content) These conflicts must be resolved before merging into main.	Resolve conflicts locally and push changes to this branch.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main objective of the pull request: adding monitoring/alert functionality for base-cluster updates.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch feat/base-cluster/add-alert-about-base-cluster-updates

⚔️ Resolve merge conflicts (beta)

• Auto-commit resolved conflicts to branch feat/base-cluster/add-alert-about-base-cluster-updates
• Create stacked PR with resolved conflicts
• Post resolved changes as copyable diffs in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Code Review

This pull request adds a Prometheus alert to notify about available updates for the base-cluster chart. The implementation has a few critical issues that will prevent the alert from working as intended.

1. The Prometheus alert expression in update-alert.yaml is logically flawed and will fire constantly, regardless of whether an update is available.
2. The mechanism for comparing the current and latest versions in current-source.yaml is not robust, as it doesn't handle common version tag variations (like a 'v' prefix), which will lead to false alerts. This aligns with the repository's rule on semver validation.
3. Resource names and namespaces are hardcoded in update-source.yaml and update-alert.yaml. This will cause conflicts and failures if the chart is deployed multiple times or outside the default namespace.

I've provided detailed comments and suggestions to fix these issues, which involve correcting the PromQL query, making the version comparison more robust, and templating resource names and namespaces for uniqueness and correctness.

[Copilot]

Pull request overview

Adds monitoring support to detect when a newer base-cluster chart version is available, based on Flux ImagePolicy state exposed via kube-state-metrics.

Changes:

• Extend kube-state-metrics custom resource metrics to export the latest ImagePolicy tag.
• Add Flux ImageRepository/ImagePolicy resources to track base-cluster chart versions.
• Add a recording rule for the currently deployed chart version and a Prometheus alert for available updates.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
charts/base-cluster/templates/monitoring/kube-prometheus-stack/_kube-state-metrics-config.yaml	Adds a customResourceState metric to expose ImagePolicy status.latestRef.tag as an Info metric.
charts/base-cluster/templates/base-cluster-update/update-source.yaml	Introduces Flux ImageRepository/ImagePolicy resources to discover the latest base-cluster version.
charts/base-cluster/templates/base-cluster-update/current-source.yaml	Adds a recording rule emitting the current chart version as a Prometheus metric label.
charts/base-cluster/templates/base-cluster-update/update-alert.yaml	Adds a PrometheusRule intended to alert when a newer version is available.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[coderabbitai]

Actionable comments posted: 2

🤖 Fix all issues with AI agents

In `@charts/base-cluster/templates/base-cluster-update/update-alert.yaml`:
- Around line 13-24: The BaseClusterUpdateAvailable alert expression is
incorrect: change the namespace selector from the hard-coded
customresource_namespace="default" to {{ .Release.Namespace }}, remove the
standalone equality check that forces firing, and replace the subtraction with a
label-aware comparison using unless on(tag) between
max(image_policy_latest_version{customresource_namespace="{{ .Release.Namespace
}}",customresource_name="base-cluster-chart"}) by (tag) and
base_cluster_current_version:info so series are matched by tag; update the
BaseClusterUpdateAvailable expr to use this unless-on(tag) pattern to only fire
when a newer tag exists.

In `@charts/base-cluster/templates/base-cluster-update/update-source.yaml`:
- Around line 7-19: The ImagePolicy resource named base-cluster-chart has an
invalid semver value at spec.policy.semver.range ("x.x.x"); update
spec.policy.semver.range in the ImagePolicy (metadata.name: base-cluster-chart)
to a valid Flux semver constraint such as "1.x", ">=1.0.0", or a caret range
like "^1.0.0" (and include "-0" e.g. "^1.x-0" if you need to match prerelease
tags) so the policy will correctly match image tags.

[coderabbitai]

Actionable comments posted: 2

🤖 Fix all issues with AI agents

In @.github/scripts/splitYamlIntoDir:
- Around line 42-49: The check/usage of helmreleaseDir only uses the basename
and tests in the current working directory; change to derive and check the full
target directory path (e.g. build a helmreleaseDirPath by joining dirname
"$helmrelease" and basename -s .yaml "$helmrelease") and then use that full path
for the -d test and as the second argument to splitYamlIntoDir (keep the
existing helmrelease and templating logic intact). Ensure you update references
to helmreleaseDir in the conditional and the splitYamlIntoDir call to use this
full path so existing dirs next to the HelmRelease file are detected correctly.

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In @.github/scripts/splitYamlIntoDir:
- Around line 44-48: The recursive call constructs a malformed path by
prepending dirname to an already computed helmreleaseDir; update the call to
splitYamlIntoDir to pass helmreleaseDir (not "$(dirname
"$helmrelease")/$helmreleaseDir") so it uses the correct directory computed
above (refer to variables helmreleaseDir and helmrelease and the
splitYamlIntoDir invocation).

[cwrau]

Needs to wait until #1948 is merged

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@charts/base-cluster/templates/base-cluster-update/current-source.yaml`:
- Around line 19-21: The current use of {{- with semver .Chart.Version }} can
silently omit the tag label when .Chart.Version isn't valid semver; change the
block so the tag label is always emitted: replace the with-block with an if/else
that tries semver (.Chart.Version) and emits tag: formatted "%d.%d.%d" when
semver succeeds (same code currently inside the with), and in the else branch
emit a safe fallback tag (e.g., the raw .Chart.Version quoted or a fixed "dev"
string) so the recording rule base_cluster_current_version:info always has a tag
label even in development snapshots.

🧹 Nitpick comments (2)

charts/base-cluster/templates/base-cluster-update/update-source.yaml (1)

1-1: Redundant and operator with a single condition.

{{- if and .Values.monitoring.prometheus.enabled }} can be simplified to {{- if .Values.monitoring.prometheus.enabled }}, which is what the other templates in this PR already use.

Proposed fix

-{{- if and .Values.monitoring.prometheus.enabled }}
+{{- if .Values.monitoring.prometheus.enabled }}

charts/base-cluster/templates/base-cluster-update/update-alert.yaml (1)

31-34: severity: critical with for: 1w may confuse responders.

A 1-week pending period implies low urgency, but severity: critical typically signals immediate action. Consider using warning or info severity to align with the actual urgency, or add a comment explaining the rationale if this is intentional (e.g., critical only after being unaddressed for a week).