Skip to content

feat(observability): alert on hermes cron and kopiur backup health #4302

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Tanguille wants to merge 2 commits into from
Closed

feat(observability): alert on hermes cron and kopiur backup health #4302

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e34ea7b
feat(observability): alert on hermes cron and kopiur backup health
Tanguille Aug 2, 2026
6f65de7
Merge branch 'main' into feat/hermes-kopiur-alerting
Tanguille Aug 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@
*.pub
*.key
*.decrypted~*.yaml
/age.key
# `*.key` misses suffixed backups like age.key.bak-x25519-only, which decrypt the
# tracked sops files in this PUBLIC repo. Supersedes the old `/age.key`.
age.key*
/cloudflare-tunnel.json
bootstrap/github-deploy.key
bootstrap/github-deploy.key.pub
Expand Down
1 change: 1 addition & 0 deletions kubernetes/apps/ai/hermes/app/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,5 @@ kind: Kustomization

resources:
- helmrelease.yaml
- prometheusrule.yaml
- secret.sops.yaml
79 changes: 79 additions & 0 deletions kubernetes/apps/ai/hermes/app/prometheusrule.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
---
# yaml-language-server: $schema=https://k8s-schemas.home-operations.com/monitoring.coreos.com/prometheusrule_v1.json
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: hermes-rules
spec:
groups:
- name: hermes.rules
# scripts/cron_health_export.py (on the hermes PVC, not in git) pushes
# cron/jobs.json state every 10m. Pushed, not scraped, so VM's 5m default
# staleness makes a bare instant selector return nothing between pushes:
# absent() would flap and the failure rules would go blind exactly half the
# time. last_over_time() pins every rule to the newest real sample instead of
# the staleness window.
rules:
# The scheduler already records per-job status in cron/jobs.json; nothing read
# it, so a dead cron stayed dark 40h.
- alert: HermesCronJobFailed
expr: |-
last_over_time(hermes_cron_last_status_ok[30m]) == 0
for: 15m
annotations:
summary: >-
Hermes cron {{ $labels.name }} last run failed — it will keep failing on
schedule until the cause is fixed
labels:
severity: warning

# Delivery is a separate failure from execution: the job can succeed and the
# Discord post still vanish, which reads green everywhere else.
- alert: HermesCronDeliveryFailed
expr: |-
last_over_time(hermes_cron_delivery_failed[30m]) == 1
for: 15m
annotations:
summary: >-
Hermes cron {{ $labels.name }} ran but could not deliver its output —
the result is lost, not late
labels:
severity: warning

# Already excludes disabled and paused jobs at export time. One hour of slack
# absorbs a long-running predecessor without flapping.
- alert: HermesCronOverdue
expr: |-
last_over_time(hermes_cron_overdue_seconds[30m]) > 3600
for: 15m
annotations:
summary: >-
Hermes cron {{ $labels.name }} is {{ $value | humanizeDuration }} past its
scheduled run — the scheduler is not firing it
labels:
severity: warning

# The exporter runs inside hermes, so a stale heartbeat means hermes itself is
# down. Without this the other three rules go quiet exactly when they matter.
- alert: HermesCronExporterStale
expr: |-
time() - last_over_time(hermes_cron_export_timestamp[2h]) > 1800
for: 5m
annotations:
summary: >-
Hermes cron health metrics are {{ $value | humanizeDuration }} stale —
cron alerting is blind, check the hermes pod
labels:
severity: warning

# No for:: absent() over a 2h-integrated window cannot flap, and any delay here
# reopens the gap where Stale has already resolved but Absent has not yet fired,
# which reports "recovered" in the middle of an outage.
- alert: HermesCronExporterAbsent
expr: |-
absent(last_over_time(hermes_cron_export_timestamp[2h]))
annotations:
summary: >-
No Hermes cron health metrics at all — the exporter never ran or was removed
labels:
severity: warning
7 changes: 6 additions & 1 deletion kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,10 @@ subjects:
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: flux-web-admin
# The web UI authenticates as Anonymous, so this role is granted to anyone who can
# reach envoy-internal. flux-web-admin adds create on batch/jobs (arbitrary pod
# execution), patch/restart on workloads and delete on pods; flux-web-user is the
# chart's read-only variant. It still carries get/list/watch on *, which includes
# Secrets -- only real auth or dropping web.httpRoute closes that.
name: flux-web-user
Comment on lines +13 to +18

@coderabbitai coderabbitai Bot Aug 2, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

helm template flux-operator \
  oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator \
  --version 0.57.0 \
  --set web.rbac.createRoles=true \
  --set web.networkPolicy.create=true \
  --set serviceMonitor.create=true |
  rg -n -C8 'name: flux-web-user|namespaceSelector: \{\}'

kubectl auth can-i --as=flux --as-group=flux-admin \
  get secrets --all-namespaces

Repository: Tanguille/cluster

Length of output: 193

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '--- clusterrolebinding.yaml ---'
cat -n kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml

printf '%s\n' '--- helmrelease.yaml ---'
cat -n kubernetes/apps/flux-system/flux-operator/app/helmrelease.yaml

printf '%s\n' '--- related flux-operator manifests ---'
find kubernetes/apps/flux-system/flux-operator -maxdepth 3 -type f -print | sort

printf '%s\n' '--- flux web references ---'
rg -n -C4 'flux-web-user|flux-web-admin|web\.rbac|web\.networkPolicy|httpRoute|envoy-internal|flux-operator' kubernetes/apps kubernetes/flux-system 2>/dev/null || true

Repository: Tanguille/cluster

Length of output: 50373

🌐 Web query:

Flux Operator Helm chart 0.57.0 flux-web-user networkPolicy namespaceSelector web RBAC anonymous UI

💡 Result:

In Flux Operator Helm chart 0.57.0, the web interface configuration and security features interact as follows: Web UI and RBAC The Flux Web UI uses Kubernetes RBAC for user authorization [1]. By default, the operator includes the flux-web-user ClusterRole, which provides read-only access to Flux resources and workloads [1][2]. For environments requiring custom, least-privilege access, administrators can bind more restrictive roles to users or groups [2][3]. Anonymous Authentication The UI supports an Anonymous authentication mode [4]. When enabled via the Web Config API (web.config), all users share a fixed identity (username and groups) defined in the configuration [1][4]. Because this identity is shared by all users, this mode is intended only for secure environments with trusted users [4]. Network Policy and Namespace Selection - NetworkPolicy: The Helm chart includes a web.networkPolicy object that defaults to create: true to secure access to the web interface [5][6][7]. - Namespace Filtering: The UI uses the web server's service account to internally list all namespaces and then applies a SelfSubjectAccessReview for the current user to filter the list [8]. Users only see namespaces in the UI for which they have get permissions on the ResourceSet custom resource [1][2][8]. This design prevents the need for cluster-wide namespace list permissions [8]. Helm Chart Configuration (web.serverOnly) When deploying the Web UI as a standalone service (web.serverOnly: true), it acts as a dedicated Helm release separate from the Flux Operator [5][6]. In this mode, if web.userActions.access is set to FineGrained, the web server performs actions using its own service account privileges rather than impersonating the user [6][7]. References: [5][6][7] Official Helm chart documentation for configuration values (web.enabled, web.networkPolicy, etc.). [1][4][8] Flux Operator Web UI documentation on User Management, Web Config API, and RBAC least privilege. [2] Documentation on the flux-web-user ClusterRole. [3] GitHub issue #760 discussing minimal RBAC for the Web UI.

Citations:

🏁 Script executed (no clone):

#!/usr/bin/env bash
set -euo pipefail

api='https://api.github.com/repos/controlplaneio-fluxcd/charts'
for ref in main v0.57.0 0.57.0; do
  printf '%s\n' "--- ref: ${ref} ---"
  curl -fsSL "${api}/git/trees/${ref}?recursive=1" |
    jq -r '.tree[]?.path' |
    rg 'flux-operator/.+(role|network|values|template)|charts/flux-operator' |
    head -80 || true
done

printf '%s\n' '--- documentation references ---'
for url in \
  'https://raw.githubusercontent.com/controlplaneio-fluxcd/charts/main/charts/flux-operator/templates/web/clusterrole.yaml' \
  'https://raw.githubusercontent.com/controlplaneio-fluxcd/charts/main/charts/flux-operator/templates/web/networkpolicy.yaml' \
  'https://raw.githubusercontent.com/controlplaneio-fluxcd/charts/main/charts/flux-operator/values.yaml' \
  'https://raw.githubusercontent.com/controlplaneio-fluxcd/flux-operator/main/docs/web-ui/user-management.md'; do
  printf '%s\n' "### ${url}"
  curl -fsSL "$url" | rg -n -C5 'flux-web-user|networkPolicy|namespaceSelector|Secrets|Anonymous|authentication' || true
done

Length of output: 4199

🏁 Script executed (no clone):

#!/usr/bin/env bash
set -euo pipefail

base='https://raw.githubusercontent.com/controlplaneio-fluxcd/charts/main/charts/flux-operator'
for file in \
  templates/web-clusterrole.yaml \
  templates/web-standard-roles.yaml \
  templates/networkpolicy.yaml \
  values.yaml \
  README.md; do
  printf '%s\n' "### ${file}"
  curl -fsSL "${base}/${file}" |
    rg -n -C8 'flux-web-user|networkPolicy|namespaceSelector|secrets|Secrets|web\.rbac|web\.config|Anonymous' || true
done

printf '%s\n' '--- Flux Operator documentation tree ---'
curl -fsSL 'https://api.github.com/repos/controlplaneio-fluxcd/flux-operator/git/trees/main?recursive=1' |
  jq -r '.tree[]?.path' |
  rg 'web.*(management|rbac|config)|user-management|least-privilege' || true

Length of output: 9506

🏁 Script executed (no clone):

#!/usr/bin/env bash
set -euo pipefail

base='https://raw.githubusercontent.com/controlplaneio-fluxcd/charts/main/charts/flux-operator'
for file in templates/web-standard-roles.yaml templates/web-clusterrole.yaml templates/networkpolicy.yaml; do
  printf '%s\n' "### ${file}"
  curl -fsSL "${base}/${file}" | sed -n '1,240p'
done

for file in docs/web/web-user-management.md docs/web/web-least-privilege-rbac.md docs/web/web-config-api.md; do
  printf '%s\n' "### ${file}"
  curl -fsSL "https://raw.githubusercontent.com/controlplaneio-fluxcd/flux-operator/main/${file}" |
    rg -n -C5 'Anonymous|flux-web-user|Secret|least.?privilege|group|authentication' || true
done

Length of output: 30810

🏁 Script executed (no clone):

#!/usr/bin/env bash
set -euo pipefail

url='https://raw.githubusercontent.com/controlplaneio-fluxcd/flux-operator/main/docs/web/web-user-management.md'
printf '%s\n' '--- predefined role and sensitive-data behavior ---'
curl -fsSL "$url" | sed -n '165,225p'
curl -fsSL "$url" | sed -n '270,300p'

printf '%s\n' '--- least-privilege sensitive-data references ---'
curl -fsSL 'https://raw.githubusercontent.com/controlplaneio-fluxcd/flux-operator/main/docs/web/web-least-privilege-rbac.md' |
  rg -n -C8 'Secret|ConfigMap|sensitive|redact|filter|sanitize' || true

printf '%s\n' '--- OCI chart tags ---'
curl -fsSL 'https://ghcr.io/v2/controlplaneio-fluxcd/charts/flux-operator/tags/list' |
  jq -r '.tags[]?' | rg '^0\.57\.0$|^v?0\.57' || true

Length of output: 11945

🏁 Script executed (no clone):

#!/usr/bin/env bash
set -euo pipefail

repo='controlplaneio-fluxcd/charts/flux-operator'
token="$(
  curl -fsSL \
    "https://ghcr.io/token?service=ghcr.io&scope=repository:${repo}:pull" |
    jq -r .token
)"
test -n "$token" && test "$token" != null

manifest="$(
  curl -fsSL \
    -H "Authorization: Bearer ${token}" \
    -H 'Accept: application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json' \
    "https://ghcr.io/v2/${repo}/manifests/0.57.0"
)"
printf '%s\n' '--- chart manifest ---'
printf '%s' "$manifest" | jq '{schemaVersion, mediaType, config, layers}'

digest="$(printf '%s' "$manifest" | jq -r '.layers[] | select(.mediaType | test("helm.chart.content|gzip")) | .digest' | head -1)"
test -n "$digest" && test "$digest" != null

curl -fsSL \
  -H "Authorization: Bearer ${token}" \
  "https://ghcr.io/v2/${repo}/blobs/${digest}" \
  -o /tmp/flux-operator-0.57.0.tgz

python3 - <<'PY'
import tarfile
from pathlib import Path

archive = Path('/tmp/flux-operator-0.57.0.tgz')
with tarfile.open(archive, 'r:gz') as tf:
    names = tf.getnames()
    print('--- matching chart files ---')
    for name in names:
        if any(x in name for x in ('web-standard-roles.yaml', 'networkpolicy.yaml', 'values.yaml')):
            print(name)
    print('--- relevant templates and values ---')
    for name in names:
        if name.endswith((
            'templates/web-standard-roles.yaml',
            'templates/networkpolicy.yaml',
            'values.yaml',
        )):
            print(f'### {name}')
            text = tf.extractfile(name).read().decode()
            for i, line in enumerate(text.splitlines(), 1):
                if any(term.lower() in line.lower() for term in (
                    'flux-web-user', 'namespaceSelector', 'networkPolicy',
                    'resources:', 'verbs:', 'createRoles',
                )):
                    print(f'{i}:{line}')
PY

Length of output: 176

Restrict anonymous Flux web access.

Every visitor receives the flux-admin identity, which is bound to flux-web-user with cluster-wide get, list, and watch permissions. The NetworkPolicy also permits pods from every namespace to reach port 9080. The UI does not return Secret or ConfigMap data, but this still exposes all other UI-readable resources to unauthenticated callers.

Use OAuth2/OIDC, or bind the anonymous identity to a custom least-privilege role. Restrict port 9080 to the Envoy source and port 8080 to the monitoring source. Apply these changes through the repository manifests and Flux reconciliation.

📍 Affects 2 files
  • kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml#L13-L18 (this comment)
  • kubernetes/apps/flux-system/flux-operator/app/helmrelease.yaml#L30-L34
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml` around
lines 13 - 18, Restrict anonymous Flux web access by configuring OAuth2/OIDC or
binding the anonymous identity to a custom least-privilege role instead of
flux-web-user; update
kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml lines
13-18 accordingly. In
kubernetes/apps/flux-system/flux-operator/app/helmrelease.yaml lines 30-34,
restrict port 9080 to Envoy sources and port 8080 to monitoring sources, then
apply both changes through the repository manifests and Flux reconciliation.

Source: Coding guidelines

@coderabbitai coderabbitai Bot Aug 2, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

kubectl get clusterrolebinding flux-admin \
  -o jsonpath='{.roleRef.name}{"\n"}'

kubectl apply --server-side --dry-run=server \
  -f kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml

Repository: Tanguille/cluster

Length of output: 200

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '--- target manifest ---'
cat -n kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml

printf '%s\n' '--- related Flux operator manifests ---'
rg -n -C 5 'flux-admin|flux-web-admin|flux-web-user|roleRef|ClusterRoleBinding|anonymous|flux-admin' \
  kubernetes/apps/flux-system/flux-operator kubernetes 2>/dev/null || true

printf '%s\n' '--- target history summary ---'
git log --oneline -8 -- kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml

Repository: Tanguille/cluster

Length of output: 13120

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '--- target manifest history and parent state ---'
git show --format=fuller --stat 65e51b09 -- kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml
git show 65e51b09^:kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml 2>/dev/null | cat -n || true
git show 65e51b09:kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml | cat -n

printf '%s\n' '--- Flux Kustomization and app files ---'
fd -t f . kubernetes/apps/flux-system/flux-operator | sort
rg -n -C 6 'kind: Kustomization|prune:|clusterrolebinding.yaml|flux-operator' \
  kubernetes/apps/flux-system/flux-operator

printf '%s\n' '--- all repository references to the binding and roles ---'
rg -n -C 4 'flux-admin|flux-web-admin|flux-web-user' . --glob '!*.lock' || true

printf '%s\n' '--- recent commits affecting the Flux RBAC transition ---'
git log --oneline --all -- kubernetes/apps/flux-system/flux-operator

Repository: Tanguille/cluster

Length of output: 11311

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '--- working-tree change under review ---'
git status --short
git diff -- kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml \
  kubernetes/apps/flux-system/flux-operator/app/helmrelease.yaml \
  kubernetes/apps/flux-system/flux-operator/ks.yaml

printf '%s\n' '--- exact prior binding introduction/change ---'
git log --oneline -S'flux-web-admin' -- kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml
git log --oneline -S'flux-web-user' -- kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml
git show --format=fuller --stat 1981a310 -- kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml kubernetes/apps/flux-system/flux-operator/app/helmrelease.yaml
git show 1981a310 -- kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml kubernetes/apps/flux-system/flux-operator/app/helmrelease.yaml

printf '%s\n' '--- Flux Kustomization controller settings and prune documentation references ---'
rg -n -C 3 'server-side|force|prune|inventory|delet(e|ion)' \
  kubernetes flux-system .github 2>/dev/null | head -250 || true

Repository: Tanguille/cluster

Length of output: 18963

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '--- current revision and target-file commits ---'
git rev-parse HEAD
git log --format='%h %ad %s' --date=short -12 -- kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml
git log --format='%h %ad %s' --date=short -12 -S'flux-web-user' -- kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml

printf '%s\n' '--- commits that contain the current role reference ---'
for c in $(git log --format='%H' -S'flux-web-user' -- kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml | head -5); do
  echo "commit $c"
  git show --format='%h %s' --no-ext-diff "$c" -- kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml | sed -n '1,100p'
done

printf '%s\n' '--- Flux Kustomization resource and controller version context ---'
cat -n kubernetes/apps/flux-system/flux-operator/ks.yaml
cat -n kubernetes/flux/cluster/ks.yaml | sed -n '1,80p'
rg -n 'fluxcd|kustomize-controller|flux-system' kubernetes/apps/flux-system kubernetes/flux | head -120

Repository: Tanguille/cluster

Length of output: 12316

🌐 Web query:

FluxCD Kustomization prune behavior when a resource is renamed, and whether prune deletes the old resource after creating the new resource

💡 Result:

When you rename a resource managed by FluxCD—specifically if you mean renaming the Kustomization object itself—the behavior depends on your prune configuration [1][2]. If spec.prune is set to true, renaming the Kustomization object will cause Flux to treat the original Kustomization as deleted, which triggers the garbage collection (pruning) of all resources previously managed by that Kustomization [1][3][2]. Consequently, those resources will be deleted from the cluster before the new Kustomization (with the new name) attempts to create them [2]. To rename a Flux Kustomization without triggering this deletion, you must perform the following steps [1][2]: 1. Disable garbage collection by setting spec.prune: false in your Kustomization manifest [1]. 2. Commit and push this change to your repository, then wait for Flux to reconcile and acknowledge the change (you can verify this using flux export kustomization) [1]. 3. Rename the Kustomization object in your manifest, then commit and push the change [1]. 4. Flux will delete the old Kustomization object but will not delete the associated workloads because pruning was disabled [1]. 5. Once the new Kustomization is active, you can re-enable pruning by setting spec.prune: true [1]. Flux will then adopt ownership of the existing resources [1]. If you are referring to renaming a specific resource within the manifests being synced by a Kustomization (rather than the Kustomization object itself), Flux will perceive this as a deletion of the old resource and an addition of a new one [3]. Because the old resource is missing from the source revision, Flux will prune it immediately if pruning is enabled [3]. There is no native behavior to ensure the new resource is created before the old one is deleted; the deletion of orphaned resources happens as part of the reconciliation process [3]. To move resources safely between different Kustomizations (or rename them in a way that requires avoiding deletion), the best practice is to temporarily disable pruning while the migration occurs [1].

Citations:

Recreate the binding during the role transition. The flux-operator Kustomization has prune: true, but changing roleRef.name on the existing flux-admin ClusterRoleBinding is rejected because roleRef is immutable. Use a new binding name, then let Flux prune the old binding through a controlled migration.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml` at
line 18, Rename the ClusterRoleBinding identified by `name: flux-web-user` to a
new binding name so the role transition creates a replacement resource instead
of modifying the immutable `roleRef`. Preserve the existing subject and role
references, allowing the `flux-operator` Kustomization’s `prune: true` behavior
to remove the old `flux-admin` binding during migration.

Source: Coding guidelines

apiGroup: rbac.authorization.k8s.io
5 changes: 4 additions & 1 deletion kubernetes/apps/flux-system/flux-operator/app/helmrelease.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,5 +27,8 @@ spec:
parentRefs:
- name: envoy-internal
namespace: network
# Chart default is true. Caps pod ingress to 9080 (web) and 8080 (metrics, kept
# because serviceMonitor.create is on); `from` is namespaceSelector:{}, so reach
# to the anonymous UI is unchanged — the read-only roleRef is what limits it.
networkPolicy:
create: false
create: true
1 change: 1 addition & 0 deletions kubernetes/apps/kopiur-system/kopiur/app/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ kind: Kustomization
resources:
- helmrelease.yaml
- ocirepository.yaml
- prometheusrule.yaml
73 changes: 73 additions & 0 deletions kubernetes/apps/kopiur-system/kopiur/app/prometheusrule.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
---
# yaml-language-server: $schema=https://k8s-schemas.home-operations.com/monitoring.coreos.com/prometheusrule_v1.json
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: kopiur-rules
spec:
groups:
# The chart ships its own `kopiur.rules` group (monitoring.prometheusRule.enabled
# in the HelmRelease); group and alert names here must not collide with it.
- name: kopiur.freshness.rules
rules:
# Backups were down 44h in July before anyone noticed: nothing watched
# freshness. Snapshots run hourly, so 6h is 6 missed runs, not a slow one.
# Named Overdue, not Stale: the chart's KopiurBackupStale fires off the same
# metric with the same severity, so a shared name is one alert fingerprint.
- alert: KopiurBackupOverdue
expr: |-
time() - kopiur_policy_last_backup_success_timestamp_seconds > 6 * 60 * 60
for: 30m
Comment on lines +18 to +20

@coderabbitai coderabbitai Bot Aug 2, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -eu

printf '%s\n' '--- target file ---'
cat -n kubernetes/apps/kopiur-system/kopiur/app/prometheusrule.yaml

printf '%s\n' '--- Kopiur-related files ---'
git ls-files | rg -i 'kopiur|snapshot.?policy|prometheus.?rule|servicemonitor|vmrule|prometheus'
printf '%s\n' '--- Kopiur references ---'
rg -n -i 'kopiur_policy_last_backup_success_timestamp_seconds|kopiur_repository|kopiur.*leader|SnapshotPolicy|prometheusRule|ruleSelector|vmrule|prometheusrule' kubernetes --glob '*.yaml' --glob '*.yml' || true

Repository: Tanguille/cluster

Length of output: 14744

🏁 Script executed:

#!/bin/bash
set -eu

printf '%s\n' '--- target file ---'
cat -n kubernetes/apps/kopiur-system/kopiur/app/prometheusrule.yaml

printf '%s\n' '--- Kopiur-related files ---'
git ls-files | rg -i 'kopiur|snapshot.?policy|prometheus.?rule|servicemonitor|vmrule|prometheus'

printf '%s\n' '--- Kopiur references ---'
rg -n -i 'kopiur_policy_last_backup_success_timestamp_seconds|kopiur_repository|kopiur.*leader|SnapshotPolicy|prometheusRule|ruleSelector|vmrule|prometheusrule' kubernetes --glob '*.yaml' --glob '*.yml' || true

Repository: Tanguille/cluster

Length of output: 14744

🏁 Script executed:

#!/bin/bash
set -eu

printf '%s\n' '--- SnapshotPolicy component ---'
cat -n kubernetes/components/kopiur/backup/snapshotpolicy.yaml
cat -n kubernetes/components/kopiur/backup/kustomization.yaml
cat -n kubernetes/components/kopiur/kustomization.yaml

printf '%s\n' '--- Kopiur application configuration ---'
cat -n kubernetes/apps/kopiur-system/kopiur/app/helmrelease.yaml
cat -n kubernetes/apps/kopiur-system/kopiur/app/kustomization.yaml
cat -n kubernetes/apps/kopiur-system/kopiur/ks.yaml

printf '%s\n' '--- monitoring selectors and PrometheusRule patterns ---'
rg -n -i 'ruleSelector|ruleNamespaceSelector|prometheus:|release:|monitoring.coreos.com/v1|kind: PrometheusRule' kubernetes --glob '*.yaml' --glob '*.yml' | head -250

printf '%s\n' '--- all SnapshotPolicy declarations ---'
rg -n -C 12 'kind: SnapshotPolicy' kubernetes --glob '*.yaml' --glob '*.yml' || true

Repository: Tanguille/cluster

Length of output: 14538

🏁 Script executed:

#!/bin/bash
set -eu
rg -n -i 'kopiur|snapshotpolicy|prometheusrule|ruleSelector|ruleNamespaceSelector' . --glob '*.yaml' --glob '*.yml' --glob '*.md' --glob '*.json' || true

Repository: Tanguille/LLMKube

Length of output: 11859

🌐 Web query:

Kopiur kopiur_policy_last_backup_success_timestamp_seconds SnapshotPolicy metrics

💡 Result:

The metric kopiur_policy_last_backup_success_timestamp_seconds is a component of the Kopiur observability suite, which monitors the backup operations of the Kopiur Kubernetes operator [1]. Kopiur is a Kopia-native operator that manages backups by separating backup recipes, schedules, and invocations into Kubernetes Custom Resource Definitions (CRDs) such as SnapshotPolicy, SnapshotSchedule, and Snapshot [2][3]. Key details regarding this metric and its context include: 1. Function: The metric kopiur_policy_last_backup_success_timestamp_seconds records the Unix timestamp of the last successful backup operation associated with a specific SnapshotPolicy [1]. It allows administrators to track the recency of backups for various PVCs and workloads managed by Kopiur [1][4]. 2. Observability: Metrics are exposed by the Kopiur controller via its /metrics endpoint [1]. Users can enable the metrics.serviceMonitor in the Kopiur Helm chart to scrape these metrics using Prometheus [1]. 3. Configuration and Alerting: The Kopiur Helm chart includes configuration options for these metrics, such as metrics.prometheusRule.backupStaleAfterSeconds, which defines the duration in seconds after which a SnapshotPolicy's last success timestamp is considered "stale" for alerting purposes [1]. 4. Relationship to Kopia: While Kopiur relies on Kopia for the underlying backup storage and movement [2][3], the kopiur_ prefixed metrics are specific to the Kopiur operator's management of its CRDs [1]. Kopia itself also implements native snapshot metrics (often under the kopia_ namespace, such as kopia_last_snapshot_age_seconds) [5][6], but kopiur_policy_last_backup_success_timestamp_seconds is specifically scoped to the operation of the Kopiur SnapshotPolicy resources [1].

Citations:

🌐 Web query:

Kopiur GitHub kopiur_policy_last_backup_success_timestamp_seconds

💡 Result:

The term kopiur_policy_last_backup_success_timestamp_seconds is a Prometheus metric exported by Kopiur [1], a Kopia-native Kubernetes backup operator [2][3]. This metric tracks the Unix timestamp (in seconds) of the last successful backup operation associated with a Kopiur SnapshotPolicy [1]. It is primarily used within Prometheus alerting rules to monitor backup freshness. For example, it is often compared against a threshold (such as backupStaleAfterSeconds, which defaults to 172,800 seconds or 48 hours) to determine if a backup policy is considered stale [1]. Kopiur itself is an open-source project under the Home Operations organization, designed to manage Kopia repositories and snapshots as first-class Kubernetes custom resources (CRDs) [2][3][4].

Citations:

🏁 Script executed:

#!/bin/bash
set -eu

printf '%s\n' '--- Kopiur component consumers ---'
rg -n -C 8 'kopiur/backup|components:|SnapshotSchedule|BACKUP_SNAPSHOTCLASS|KOPIUR_PUID' kubernetes/apps kubernetes/components --glob '*.yaml' --glob '*.yml' | head -400

printf '%s\n' '--- Victoria Metrics and kube-state-metrics configuration ---'
cat -n kubernetes/apps/observability/victoria-metrics/app/helmrelease.yaml | sed -n '1,380p'
fd -i 'helmrelease.yaml' kubernetes/apps/observability/kube-state-metrics kubernetes/apps/observability 2>/dev/null | xargs -r -n1 sh -c 'echo "--- $0"; cat -n "$0"'
rg -n -i 'customResourceState|customresources|kube_customresource|kube-state-metrics|ruleSelector|ruleNamespaceSelector|Prometheus' kubernetes/apps/observability kubernetes/config --glob '*.yaml' --glob '*.yml' || true

Repository: Tanguille/cluster

Length of output: 50376

🏁 Script executed (no clone):

#!/bin/bash
set -eu

base='https://raw.githubusercontent.com/home-operations/kopiur/main'
for path in deploy/helm/kopiur/README.md deploy/helm/kopiur/values.yaml deploy/helm/kopiur/templates/prometheusrule.yaml; do
  echo "--- $path"
  curl -fsSL "$base/$path" | rg -n -C 8 'policy_last_backup|leader_is_leader|repo_snapshot_count|PrometheusRule|stale|serviceMonitor' || true
done

Length of output: 10915

🏁 Script executed (no clone):

#!/bin/bash
set -eu

python3 - <<'PY'
# Read-only semantic model of the relevant PromQL behavior.
# Each vector is represented by its label set and value.
from math import isclose

def absent(vector):
    return [({}, 1)] if not vector else []

def freshness(vector, now, threshold):
    return [(labels, now - value) for labels, value in vector
            if now - value > threshold]

now = 1000
healthy = [
    ({"namespace": "app", "policy": "one"}, 900),
    ({"namespace": "app", "policy": "two"}, 900),
]
one_missing = [
    ({"namespace": "app", "policy": "one"}, 900),
]

print("healthy freshness:", freshness(healthy, now, 360))
print("one-policy-missing freshness:", freshness(one_missing, now, 360))
print("absent(healthy):", absent(healthy))
print("absent(one-policy-missing):", absent(one_missing))
print("absent(all-policies-missing):", absent([]))
PY

Length of output: 271

🏁 Script executed (no clone):

#!/bin/bash
set -eu

repo='home-operations/kopiur'
api="https://api.github.com/repos/$repo/git/trees/main?recursive=1"
curl -fsSL "$api" |
  jq -r '.tree[].path' |
  rg 'observability|metric|prometheus|helm/.+yaml$' |
  head -120

Length of output: 174

🏁 Script executed (no clone):

#!/bin/bash
set -eu

for path in \
  docs/dev/observability.md \
  deploy/helm/kopiur/templates/monitoring/prometheusrule.yaml \
  deploy/helm/kopiur/templates/prometheus-rule.yaml \
  deploy/helm/kopiur/templates/servicemonitor.yaml; do
  echo "--- $path"
  curl -fsSL "https://raw.githubusercontent.com/home-operations/kopiur/main/$path" |
    rg -n -C 10 'policy_last_backup|leader_is_leader|repo_snapshot_count|SnapshotPolicy|labels:|absent|stale' || true
done

Length of output: 18617

🏁 Script executed (no clone):

#!/bin/bash
set -eu

python3 - <<'PY'
# Minimal read-only vector model for the exact failure mode.
# A missing policy has no sample; a healthy remaining policy keeps the metric present.
metric = [
    ({"namespace": "media", "policy": "jellyfin"}, 900),
    ({"namespace": "media", "policy": "sonarr"}, 900),
]
after_one_policy_disappears = [
    series for series in metric if series[0]["policy"] != "sonarr"
]

def global_absent(vector):
    return "alert" if len(vector) == 0 else "no alert"

def stale(vector, now=1000, threshold=360):
    return [labels for labels, timestamp in vector if now - timestamp > threshold]

print("freshness result after sonarr disappears:", stale(after_one_policy_disappears))
print("global absent result after sonarr disappears:", global_absent(after_one_policy_disappears))
print("global absent result after all policies disappear:", global_absent([]))
PY

Length of output: 282

Add per-policy missing-series detection. absent(kopiur_policy_last_backup_success_timestamp_seconds) stays false when any other policy still reports. Add an independent SnapshotPolicy inventory metric and join it with the per-policy metric to alert on missing or stale policies. Keep the global absence rule as a fleet-wide fallback.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@kubernetes/apps/kopiur-system/kopiur/app/prometheusrule.yaml` around lines 18
- 20, Update the Prometheus rules around the existing stale-backup expression to
add an independent SnapshotPolicy inventory metric, then join that inventory
with kopiur_policy_last_backup_success_timestamp_seconds so each policy alerts
when its series is missing or older than six hours for 30 minutes. Retain the
existing global absent rule as the fleet-wide fallback.

annotations:
summary: >-
kopiur {{ $labels.namespace }}/{{ $labels.policy }} has no successful backup
for {{ $value | humanizeDuration }} — restore point is aging
labels:
severity: warning

- alert: KopiurBackupCritical
expr: |-
time() - kopiur_policy_last_backup_success_timestamp_seconds > 24 * 60 * 60
for: 30m
annotations:
summary: >-
kopiur {{ $labels.namespace }}/{{ $labels.policy }} has not backed up
successfully in {{ $value | humanizeDuration }}
labels:
severity: critical

# The July outage removed the CSI snapshot CRDs, so policies stopped reporting
# rather than reporting failure. Absence is the failure mode to catch.
- alert: KopiurPolicyMetricsAbsent
expr: |-
absent(kopiur_policy_last_backup_success_timestamp_seconds)
for: 30m
annotations:
summary: >-
No kopiur policy backup metrics at all — the controller is down or every
policy stopped reporting; backup state is unknown, not healthy
labels:
severity: critical

# A repo whose snapshot count only ever grows means retention is not pruning;
# a drop to zero means the repo was emptied under us.
- alert: KopiurRepoEmpty
expr: |-
kopiur_repo_snapshot_count == 0
for: 1h
annotations:
summary: >-
kopiur repository {{ $labels.name }} reports zero snapshots — the repo is
empty or unreadable
labels:
severity: critical

- alert: KopiurControllerNotLeader
expr: |-
max(kopiur_leader_is_leader) == 0
for: 30m
annotations:
summary: >-
No kopiur controller holds the leader lease — nothing is scheduling backups
labels:
severity: critical
10 changes: 10 additions & 0 deletions kubernetes/components/kopiur/backup/snapshotpolicy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,16 @@ spec:
runAsGroup: ${KOPIUR_PGID:=568}
capabilities:
add: ${KOPIUR_MOVER_CAPS_ADD:=[]}
# Verification is opt-in and nobody had opted in: snapshots ran hourly for 18d with
# LAST-VERIFIED empty on all 18 policies. A repo that stops being restorable reads
# identical to a healthy one until the restore. Fixed hour + 3h jitter spreads the
# fleet wider than `H 3 * * 0` would (a single hashed minute inside one hour), which
# matters because quick verify walks the whole repo.
verification:
quick:
schedule:
cron: 0 3 * * 0
jitter: 3h
compression:
compressor: zstd
retention:
Expand Down