feat(observability): alert on hermes cron and kopiur backup health - #4302
feat(observability): alert on hermes cron and kopiur backup health#4302Tanguille wants to merge 2 commits into
Conversation
Backups ran hourly for 18d with LAST-VERIFIED empty on all 18 policies, and a cron stayed dark 40h, because nothing watched either. The signal already existed on disk and in VictoriaMetrics; only the rules were missing. - hermes-rules: 5 alerts over the cron health metrics pushed from the PVC. Every expr is wrapped in last_over_time(...[30m]) because the series are pushed, not scraped, and VM's 5m staleness makes a bare selector return nothing between pushes -- verified live: the bare selector returns 0 series while the wrapped one returns 14. - kopiur-rules: freshness, metric-absence, empty-repo and leader alerts. Group is kopiur.freshness.rules and the freshness alert is KopiurBackupOverdue: the chart ships its own kopiur.rules group with a KopiurBackupStale on the same metric at the same severity, which would have produced identical alert fingerprints. - SnapshotPolicy: enable weekly quick verification. It is opt-in and nobody had opted in, so a repo that stopped being restorable would have read healthy until a restore was attempted. - flux-operator: bind the anonymous flux-admin group to the chart's read-only flux-web-user instead of flux-web-admin, which granted create on batch/jobs, patch/restart on workloads and delete on pods to anyone reaching the internal gateway. Restore the chart's default networkPolicy. Anonymous read of Secrets remains until the UI gets real auth or the route is dropped. - gitignore: age.key* -- *.key and /age.key both miss suffixed backups like age.key.bak-x25519-only, which decrypts the tracked sops files in this public repo. Rules validated with kubectl apply --dry-run=server and every expression run against live VictoriaMetrics.
📝 WalkthroughWalkthroughThe PR adds Hermes and Kopiur Prometheus alerts, enables Kopiur quick verification, changes Flux Operator RBAC and network policy settings, and broadens the ChangesMonitoring and verification
Flux access and network configuration
Repository ignore rule
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
@@ (root level) @@
# networking.k8s.io/v1/NetworkPolicy/flux-system/flux-operator-web
! + one document added:
+ apiVersion: networking.k8s.io/v1
+ kind: NetworkPolicy
+ metadata:
+ name: flux-operator-web
+ namespace: flux-system
+ labels:
+ app.kubernetes.io/instance: flux-operator
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/name: flux-operator
+ helm.toolkit.fluxcd.io/name: flux-operator
+ helm.toolkit.fluxcd.io/namespace: flux-system
+ spec:
+ ingress:
+ - from:
+ - namespaceSelector: {}
+ ports:
+ - port: 9080
+ protocol: TCP
+ - port: 8080
+ protocol: TCP
+ podSelector:
+ matchLabels:
+ app.kubernetes.io/instance: flux-operator
+ app.kubernetes.io/name: flux-operator
+ policyTypes:
+ - Ingress
|
@@ spec @@
# kopiur.home-operations.com/v1alpha1/SnapshotPolicy/ai/hermes
! + one map entry added:
+ verification:
+ quick:
+ schedule:
+ cron: "0 3 * * 0"
+ jitter: 3h
@@ spec @@
# kopiur.home-operations.com/v1alpha1/SnapshotPolicy/ai/odysseus
! + one map entry added:
+ verification:
+ quick:
+ schedule:
+ cron: "0 3 * * 0"
+ jitter: 3h
@@ spec @@
# kopiur.home-operations.com/v1alpha1/SnapshotPolicy/ai/opencode
! + one map entry added:
+ verification:
+ quick:
+ schedule:
+ cron: "0 3 * * 0"
+ jitter: 3h
@@ spec @@
# kopiur.home-operations.com/v1alpha1/SnapshotPolicy/default/changedetection
! + one map entry added:
+ verification:
+ quick:
+ schedule:
+ cron: "0 3 * * 0"
+ jitter: 3h
@@ spec @@
# kopiur.home-operations.com/v1alpha1/SnapshotPolicy/default/dumbassets
! + one map entry added:
+ verification:
+ quick:
+ schedule:
+ cron: "0 3 * * 0"
+ jitter: 3h
@@ spec @@
# kopiur.home-operations.com/v1alpha1/SnapshotPolicy/default/karakeep
! + one map entry added:
+ verification:
+ quick:
+ schedule:
+ cron: "0 3 * * 0"
+ jitter: 3h
@@ spec @@
# kopiur.home-operations.com/v1alpha1/SnapshotPolicy/default/nextcloud
! + one map entry added:
+ verification:
+ quick:
+ schedule:
+ cron: "0 3 * * 0"
+ jitter: 3h
@@ roleRef.name @@
# rbac.authorization.k8s.io/v1/ClusterRoleBinding/flux-admin
! ± value change
- flux-web-admin
+ flux-web-user
@@ spec.values.web.networkPolicy.create @@
# helm.toolkit.fluxcd.io/v2/HelmRelease/flux-system/flux-operator
! ± value change
- false
+ true
@@ spec @@
# kopiur.home-operations.com/v1alpha1/SnapshotPolicy/media/bazarr
! + one map entry added:
+ verification:
+ quick:
+ schedule:
+ cron: "0 3 * * 0"
+ jitter: 3h
@@ spec @@
# kopiur.home-operations.com/v1alpha1/SnapshotPolicy/media/brrpolice
! + one map entry added:
+ verification:
+ quick:
+ schedule:
+ cron: "0 3 * * 0"
+ jitter: 3h
@@ spec @@
# kopiur.home-operations.com/v1alpha1/SnapshotPolicy/media/fileflows
! + one map entry added:
+ verification:
+ quick:
+ schedule:
+ cron: "0 3 * * 0"
+ jitter: 3h
@@ spec @@
# kopiur.home-operations.com/v1alpha1/SnapshotPolicy/media/jellyfin
! + one map entry added:
+ verification:
+ quick:
+ schedule:
+ cron: "0 3 * * 0"
+ jitter: 3h
@@ spec @@
# kopiur.home-operations.com/v1alpha1/SnapshotPolicy/media/prowlarr
! + one map entry added:
+ verification:
+ quick:
+ schedule:
+ cron: "0 3 * * 0"
+ jitter: 3h
@@ spec @@
# kopiur.home-operations.com/v1alpha1/SnapshotPolicy/media/qbittorrent
! + one map entry added:
+ verification:
+ quick:
+ schedule:
+ cron: "0 3 * * 0"
+ jitter: 3h
@@ spec @@
# kopiur.home-operations.com/v1alpha1/SnapshotPolicy/media/qui
! + one map entry added:
+ verification:
+ quick:
+ schedule:
+ cron: "0 3 * * 0"
+ jitter: 3h
@@ spec @@
# kopiur.home-operations.com/v1alpha1/SnapshotPolicy/media/radarr
! + one map entry added:
+ verification:
+ quick:
+ schedule:
+ cron: "0 3 * * 0"
+ jitter: 3h
@@ spec @@
# kopiur.home-operations.com/v1alpha1/SnapshotPolicy/media/seerr
! + one map entry added:
+ verification:
+ quick:
+ schedule:
+ cron: "0 3 * * 0"
+ jitter: 3h
@@ spec @@
# kopiur.home-operations.com/v1alpha1/SnapshotPolicy/media/sonarr
! + one map entry added:
+ verification:
+ quick:
+ schedule:
+ cron: "0 3 * * 0"
+ jitter: 3h
@@ spec @@
# kopiur.home-operations.com/v1alpha1/SnapshotPolicy/media/wizarr
! + one map entry added:
+ verification:
+ quick:
+ schedule:
+ cron: "0 3 * * 0"
+ jitter: 3h
@@ (root level) @@
# monitoring.coreos.com/v1/PrometheusRule/ai/hermes-rules
! + one document added:
+ apiVersion: monitoring.coreos.com/v1
+ kind: PrometheusRule
+ metadata:
+ name: hermes-rules
+ namespace: ai
+ labels:
+ kustomize.toolkit.fluxcd.io/name: hermes
+ kustomize.toolkit.fluxcd.io/namespace: ai
+ spec:
+ groups:
+ - name: hermes.rules
+ rules:
+ - alert: HermesCronJobFailed
+ annotations:
+ summary: "Hermes cron {{ $labels.name }} last run failed — it will keep failing on schedule until the cause is fixed"
+ expr: "last_over_time(hermes_cron_last_status_ok[30m]) == 0"
+ for: 15m
+ labels:
+ severity: warning
+ - alert: HermesCronDeliveryFailed
+ annotations:
+ summary: "Hermes cron {{ $labels.name }} ran but could not deliver its output — the result is lost, not late"
+ expr: "last_over_time(hermes_cron_delivery_failed[30m]) == 1"
+ for: 15m
+ labels:
+ severity: warning
+ - alert: HermesCronOverdue
+ annotations:
+ summary: "Hermes cron {{ $labels.name }} is {{ $value | humanizeDuration }} past its scheduled run — the scheduler is not firing it"
+ expr: "last_over_time(hermes_cron_overdue_seconds[30m]) > 3600"
+ for: 15m
+ labels:
+ severity: warning
+ - alert: HermesCronExporterStale
+ annotations:
+ summary: "Hermes cron health metrics are {{ $value | humanizeDuration }} stale — cron alerting is blind, check the hermes pod"
+ expr: "time() - last_over_time(hermes_cron_export_timestamp[2h]) > 1800"
+ for: 5m
+ labels:
+ severity: warning
+ - alert: HermesCronExporterAbsent
+ annotations:
+ summary: "No Hermes cron health metrics at all — the exporter never ran or was removed"
+ expr: absent(last_over_time(hermes_cron_export_timestamp[2h]))
+ labels:
+ severity: warning
@@ (root level) @@
# monitoring.coreos.com/v1/PrometheusRule/kopiur-system/kopiur-rules
! + one document added:
+ apiVersion: monitoring.coreos.com/v1
+ kind: PrometheusRule
+ metadata:
+ name: kopiur-rules
+ namespace: kopiur-system
+ labels:
+ kustomize.toolkit.fluxcd.io/name: kopiur
+ kustomize.toolkit.fluxcd.io/namespace: kopiur-system
+ spec:
+ groups:
+ - name: kopiur.freshness.rules
+ rules:
+ - alert: KopiurBackupOverdue
+ annotations:
+ summary: "kopiur {{ $labels.namespace }}/{{ $labels.policy }} has no successful backup for {{ $value | humanizeDuration }} — restore point is aging"
+ expr: "time() - kopiur_policy_last_backup_success_timestamp_seconds > 6 * 60 * 60"
+ for: 30m
+ labels:
+ severity: warning
+ - alert: KopiurBackupCritical
+ annotations:
+ summary: "kopiur {{ $labels.namespace }}/{{ $labels.policy }} has not backed up successfully in {{ $value | humanizeDuration }}"
+ expr: "time() - kopiur_policy_last_backup_success_timestamp_seconds > 24 * 60 * 60"
+ for: 30m
+ labels:
+ severity: critical
+ - alert: KopiurPolicyMetricsAbsent
+ annotations:
+ summary: "No kopiur policy backup metrics at all — the controller is down or every policy stopped reporting; backup state is unknown, not healthy"
+ expr: absent(kopiur_policy_last_backup_success_timestamp_seconds)
+ for: 30m
+ labels:
+ severity: critical
+ - alert: KopiurRepoEmpty
+ annotations:
+ summary: "kopiur repository {{ $labels.name }} reports zero snapshots — the repo is empty or unreadable"
+ expr: "kopiur_repo_snapshot_count == 0"
+ for: 1h
+ labels:
+ severity: critical
+ - alert: KopiurControllerNotLeader
+ annotations:
+ summary: "No kopiur controller holds the leader lease — nothing is scheduling backups"
+ expr: "max(kopiur_leader_is_leader) == 0"
+ for: 30m
+ labels:
+ severity: critical
|
 |
|
Overall Grade |
Security Reliability Complexity Hygiene |
Code Review Summary
| Analyzer | Status | Updated (UTC) | Details |
|---|---|---|---|
| JavaScript |  |
Aug 2, 2026 2:56p.m. | Review ↗ |
| Shell | |
Aug 2, 2026 2:56p.m. | Review ↗ |
Important
AI Review is run only on demand for your team. We're only showing results of static analysis review right now. To trigger AI Review, comment @deepsourcebot review on this thread.
AI Automated ReviewAnalysis engine: qwen-3.6-fast@http://litellm.ai.svc.cluster.local/v1 (openai) Recommendation: Approve with minor context notes. The PR adds robust alerting for Hermes cron jobs and Kopiur backups, hardens Flux Operator RBAC, and improves Change-by-Change Findings
Standards Compliance
Required Checks
Unknowns or Needs Verification
|
|
Split into three focused PRs, and the flux-operator RBAC change is dropped — the anonymous-access risk is accepted.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
kubernetes/components/kopiur/backup/snapshotpolicy.yaml (1)
42-46: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winReject empty verification results when empty backups are invalid.
Without
successExpr, a quick verification can succeed for a zero-file snapshot and advancestatus.lastVerified. That can hide an empty backup. Kopiur documentssuccessExprfor preventing silent zero-file success. (kopiur.home-operations.com)Suggested guard
verification: quick: schedule: cron: 0 3 * * 0 jitter: 3h + successExpr: "stats.files > 0 && stats.errors == 0"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kubernetes/components/kopiur/backup/snapshotpolicy.yaml` around lines 42 - 46, Update the verification.quick configuration in snapshotpolicy.yaml to define the documented successExpr guard, requiring at least one verified file/result so zero-file snapshots are rejected rather than advancing status.lastVerified. Preserve the existing schedule and jitter settings.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml`:
- Line 18: Rename the ClusterRoleBinding identified by `name: flux-web-user` to
a new binding name so the role transition creates a replacement resource instead
of modifying the immutable `roleRef`. Preserve the existing subject and role
references, allowing the `flux-operator` Kustomization’s `prune: true` behavior
to remove the old `flux-admin` binding during migration.
- Around line 13-18: Restrict anonymous Flux web access by configuring
OAuth2/OIDC or binding the anonymous identity to a custom least-privilege role
instead of flux-web-user; update
kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml lines
13-18 accordingly. In
kubernetes/apps/flux-system/flux-operator/app/helmrelease.yaml lines 30-34,
restrict port 9080 to Envoy sources and port 8080 to monitoring sources, then
apply both changes through the repository manifests and Flux reconciliation.
In `@kubernetes/apps/kopiur-system/kopiur/app/prometheusrule.yaml`:
- Around line 18-20: Update the Prometheus rules around the existing
stale-backup expression to add an independent SnapshotPolicy inventory metric,
then join that inventory with
kopiur_policy_last_backup_success_timestamp_seconds so each policy alerts when
its series is missing or older than six hours for 30 minutes. Retain the
existing global absent rule as the fleet-wide fallback.
---
Nitpick comments:
In `@kubernetes/components/kopiur/backup/snapshotpolicy.yaml`:
- Around line 42-46: Update the verification.quick configuration in
snapshotpolicy.yaml to define the documented successExpr guard, requiring at
least one verified file/result so zero-file snapshots are rejected rather than
advancing status.lastVerified. Preserve the existing schedule and jitter
settings.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 452477c4-de9b-4565-ba8f-7787f8fd7847
📒 Files selected for processing (8)
.gitignorekubernetes/apps/ai/hermes/app/kustomization.yamlkubernetes/apps/ai/hermes/app/prometheusrule.yamlkubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yamlkubernetes/apps/flux-system/flux-operator/app/helmrelease.yamlkubernetes/apps/kopiur-system/kopiur/app/kustomization.yamlkubernetes/apps/kopiur-system/kopiur/app/prometheusrule.yamlkubernetes/components/kopiur/backup/snapshotpolicy.yaml
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
Tanguille/LLMKube(auto-detected)Tanguille/toolhive(auto-detected)Tanguille/pr-reviewer-action(auto-detected)Tanguille/2x-R9700-RDNA4-GFX1201-sglang-inference(auto-detected)
| # The web UI authenticates as Anonymous, so this role is granted to anyone who can | ||
| # reach envoy-internal. flux-web-admin adds create on batch/jobs (arbitrary pod | ||
| # execution), patch/restart on workloads and delete on pods; flux-web-user is the | ||
| # chart's read-only variant. It still carries get/list/watch on *, which includes | ||
| # Secrets -- only real auth or dropping web.httpRoute closes that. | ||
| name: flux-web-user |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
helm template flux-operator \
oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator \
--version 0.57.0 \
--set web.rbac.createRoles=true \
--set web.networkPolicy.create=true \
--set serviceMonitor.create=true |
rg -n -C8 'name: flux-web-user|namespaceSelector: \{\}'
kubectl auth can-i --as=flux --as-group=flux-admin \
get secrets --all-namespacesRepository: Tanguille/cluster
Length of output: 193
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- clusterrolebinding.yaml ---'
cat -n kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml
printf '%s\n' '--- helmrelease.yaml ---'
cat -n kubernetes/apps/flux-system/flux-operator/app/helmrelease.yaml
printf '%s\n' '--- related flux-operator manifests ---'
find kubernetes/apps/flux-system/flux-operator -maxdepth 3 -type f -print | sort
printf '%s\n' '--- flux web references ---'
rg -n -C4 'flux-web-user|flux-web-admin|web\.rbac|web\.networkPolicy|httpRoute|envoy-internal|flux-operator' kubernetes/apps kubernetes/flux-system 2>/dev/null || trueRepository: Tanguille/cluster
Length of output: 50373
🌐 Web query:
Flux Operator Helm chart 0.57.0 flux-web-user networkPolicy namespaceSelector web RBAC anonymous UI
💡 Result:
In Flux Operator Helm chart 0.57.0, the web interface configuration and security features interact as follows: Web UI and RBAC The Flux Web UI uses Kubernetes RBAC for user authorization [1]. By default, the operator includes the flux-web-user ClusterRole, which provides read-only access to Flux resources and workloads [1][2]. For environments requiring custom, least-privilege access, administrators can bind more restrictive roles to users or groups [2][3]. Anonymous Authentication The UI supports an Anonymous authentication mode [4]. When enabled via the Web Config API (web.config), all users share a fixed identity (username and groups) defined in the configuration [1][4]. Because this identity is shared by all users, this mode is intended only for secure environments with trusted users [4]. Network Policy and Namespace Selection - NetworkPolicy: The Helm chart includes a web.networkPolicy object that defaults to create: true to secure access to the web interface [5][6][7]. - Namespace Filtering: The UI uses the web server's service account to internally list all namespaces and then applies a SelfSubjectAccessReview for the current user to filter the list [8]. Users only see namespaces in the UI for which they have get permissions on the ResourceSet custom resource [1][2][8]. This design prevents the need for cluster-wide namespace list permissions [8]. Helm Chart Configuration (web.serverOnly) When deploying the Web UI as a standalone service (web.serverOnly: true), it acts as a dedicated Helm release separate from the Flux Operator [5][6]. In this mode, if web.userActions.access is set to FineGrained, the web server performs actions using its own service account privileges rather than impersonating the user [6][7]. References: [5][6][7] Official Helm chart documentation for configuration values (web.enabled, web.networkPolicy, etc.). [1][4][8] Flux Operator Web UI documentation on User Management, Web Config API, and RBAC least privilege. [2] Documentation on the flux-web-user ClusterRole. [3] GitHub issue #760 discussing minimal RBAC for the Web UI.
Citations:
- 1: https://fluxoperator.dev/docs/web-ui/user-management/
- 2: https://fluxoperator.dev/docs/web-ui/user-management.md
- 3: [web-ui] Minimal RBAC permissions required to resolve "Limited Access" namespace warning in Web UI controlplaneio-fluxcd/flux-operator#760
- 4: https://fluxoperator.dev/docs/web-ui/web-config-api/
- 5: https://github.com/controlplaneio-fluxcd/charts/blob/main/charts/flux-operator/README.md
- 6: https://artifacthub.io/packages/helm/flux-operator/flux-operator
- 7: https://fluxoperator.dev/docs/charts/flux-operator
- 8: https://fluxoperator.dev/docs/web-ui/web-least-privilege-rbac/
🏁 Script executed (no clone):
#!/usr/bin/env bash
set -euo pipefail
api='https://api.github.com/repos/controlplaneio-fluxcd/charts'
for ref in main v0.57.0 0.57.0; do
printf '%s\n' "--- ref: ${ref} ---"
curl -fsSL "${api}/git/trees/${ref}?recursive=1" |
jq -r '.tree[]?.path' |
rg 'flux-operator/.+(role|network|values|template)|charts/flux-operator' |
head -80 || true
done
printf '%s\n' '--- documentation references ---'
for url in \
'https://raw.githubusercontent.com/controlplaneio-fluxcd/charts/main/charts/flux-operator/templates/web/clusterrole.yaml' \
'https://raw.githubusercontent.com/controlplaneio-fluxcd/charts/main/charts/flux-operator/templates/web/networkpolicy.yaml' \
'https://raw.githubusercontent.com/controlplaneio-fluxcd/charts/main/charts/flux-operator/values.yaml' \
'https://raw.githubusercontent.com/controlplaneio-fluxcd/flux-operator/main/docs/web-ui/user-management.md'; do
printf '%s\n' "### ${url}"
curl -fsSL "$url" | rg -n -C5 'flux-web-user|networkPolicy|namespaceSelector|Secrets|Anonymous|authentication' || true
done
Length of output: 4199
🏁 Script executed (no clone):
#!/usr/bin/env bash
set -euo pipefail
base='https://raw.githubusercontent.com/controlplaneio-fluxcd/charts/main/charts/flux-operator'
for file in \
templates/web-clusterrole.yaml \
templates/web-standard-roles.yaml \
templates/networkpolicy.yaml \
values.yaml \
README.md; do
printf '%s\n' "### ${file}"
curl -fsSL "${base}/${file}" |
rg -n -C8 'flux-web-user|networkPolicy|namespaceSelector|secrets|Secrets|web\.rbac|web\.config|Anonymous' || true
done
printf '%s\n' '--- Flux Operator documentation tree ---'
curl -fsSL 'https://api.github.com/repos/controlplaneio-fluxcd/flux-operator/git/trees/main?recursive=1' |
jq -r '.tree[]?.path' |
rg 'web.*(management|rbac|config)|user-management|least-privilege' || true
Length of output: 9506
🏁 Script executed (no clone):
#!/usr/bin/env bash
set -euo pipefail
base='https://raw.githubusercontent.com/controlplaneio-fluxcd/charts/main/charts/flux-operator'
for file in templates/web-standard-roles.yaml templates/web-clusterrole.yaml templates/networkpolicy.yaml; do
printf '%s\n' "### ${file}"
curl -fsSL "${base}/${file}" | sed -n '1,240p'
done
for file in docs/web/web-user-management.md docs/web/web-least-privilege-rbac.md docs/web/web-config-api.md; do
printf '%s\n' "### ${file}"
curl -fsSL "https://raw.githubusercontent.com/controlplaneio-fluxcd/flux-operator/main/${file}" |
rg -n -C5 'Anonymous|flux-web-user|Secret|least.?privilege|group|authentication' || true
done
Length of output: 30810
🏁 Script executed (no clone):
#!/usr/bin/env bash
set -euo pipefail
url='https://raw.githubusercontent.com/controlplaneio-fluxcd/flux-operator/main/docs/web/web-user-management.md'
printf '%s\n' '--- predefined role and sensitive-data behavior ---'
curl -fsSL "$url" | sed -n '165,225p'
curl -fsSL "$url" | sed -n '270,300p'
printf '%s\n' '--- least-privilege sensitive-data references ---'
curl -fsSL 'https://raw.githubusercontent.com/controlplaneio-fluxcd/flux-operator/main/docs/web/web-least-privilege-rbac.md' |
rg -n -C8 'Secret|ConfigMap|sensitive|redact|filter|sanitize' || true
printf '%s\n' '--- OCI chart tags ---'
curl -fsSL 'https://ghcr.io/v2/controlplaneio-fluxcd/charts/flux-operator/tags/list' |
jq -r '.tags[]?' | rg '^0\.57\.0$|^v?0\.57' || true
Length of output: 11945
🏁 Script executed (no clone):
#!/usr/bin/env bash
set -euo pipefail
repo='controlplaneio-fluxcd/charts/flux-operator'
token="$(
curl -fsSL \
"https://ghcr.io/token?service=ghcr.io&scope=repository:${repo}:pull" |
jq -r .token
)"
test -n "$token" && test "$token" != null
manifest="$(
curl -fsSL \
-H "Authorization: Bearer ${token}" \
-H 'Accept: application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json' \
"https://ghcr.io/v2/${repo}/manifests/0.57.0"
)"
printf '%s\n' '--- chart manifest ---'
printf '%s' "$manifest" | jq '{schemaVersion, mediaType, config, layers}'
digest="$(printf '%s' "$manifest" | jq -r '.layers[] | select(.mediaType | test("helm.chart.content|gzip")) | .digest' | head -1)"
test -n "$digest" && test "$digest" != null
curl -fsSL \
-H "Authorization: Bearer ${token}" \
"https://ghcr.io/v2/${repo}/blobs/${digest}" \
-o /tmp/flux-operator-0.57.0.tgz
python3 - <<'PY'
import tarfile
from pathlib import Path
archive = Path('/tmp/flux-operator-0.57.0.tgz')
with tarfile.open(archive, 'r:gz') as tf:
names = tf.getnames()
print('--- matching chart files ---')
for name in names:
if any(x in name for x in ('web-standard-roles.yaml', 'networkpolicy.yaml', 'values.yaml')):
print(name)
print('--- relevant templates and values ---')
for name in names:
if name.endswith((
'templates/web-standard-roles.yaml',
'templates/networkpolicy.yaml',
'values.yaml',
)):
print(f'### {name}')
text = tf.extractfile(name).read().decode()
for i, line in enumerate(text.splitlines(), 1):
if any(term.lower() in line.lower() for term in (
'flux-web-user', 'namespaceSelector', 'networkPolicy',
'resources:', 'verbs:', 'createRoles',
)):
print(f'{i}:{line}')
PY
Length of output: 176
Restrict anonymous Flux web access.
Every visitor receives the flux-admin identity, which is bound to flux-web-user with cluster-wide get, list, and watch permissions. The NetworkPolicy also permits pods from every namespace to reach port 9080. The UI does not return Secret or ConfigMap data, but this still exposes all other UI-readable resources to unauthenticated callers.
Use OAuth2/OIDC, or bind the anonymous identity to a custom least-privilege role. Restrict port 9080 to the Envoy source and port 8080 to the monitoring source. Apply these changes through the repository manifests and Flux reconciliation.
📍 Affects 2 files
kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml#L13-L18(this comment)kubernetes/apps/flux-system/flux-operator/app/helmrelease.yaml#L30-L34
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml` around
lines 13 - 18, Restrict anonymous Flux web access by configuring OAuth2/OIDC or
binding the anonymous identity to a custom least-privilege role instead of
flux-web-user; update
kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml lines
13-18 accordingly. In
kubernetes/apps/flux-system/flux-operator/app/helmrelease.yaml lines 30-34,
restrict port 9080 to Envoy sources and port 8080 to monitoring sources, then
apply both changes through the repository manifests and Flux reconciliation.
Source: Coding guidelines
| # execution), patch/restart on workloads and delete on pods; flux-web-user is the | ||
| # chart's read-only variant. It still carries get/list/watch on *, which includes | ||
| # Secrets -- only real auth or dropping web.httpRoute closes that. | ||
| name: flux-web-user |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
kubectl get clusterrolebinding flux-admin \
-o jsonpath='{.roleRef.name}{"\n"}'
kubectl apply --server-side --dry-run=server \
-f kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yamlRepository: Tanguille/cluster
Length of output: 200
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- target manifest ---'
cat -n kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml
printf '%s\n' '--- related Flux operator manifests ---'
rg -n -C 5 'flux-admin|flux-web-admin|flux-web-user|roleRef|ClusterRoleBinding|anonymous|flux-admin' \
kubernetes/apps/flux-system/flux-operator kubernetes 2>/dev/null || true
printf '%s\n' '--- target history summary ---'
git log --oneline -8 -- kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yamlRepository: Tanguille/cluster
Length of output: 13120
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- target manifest history and parent state ---'
git show --format=fuller --stat 65e51b09 -- kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml
git show 65e51b09^:kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml 2>/dev/null | cat -n || true
git show 65e51b09:kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml | cat -n
printf '%s\n' '--- Flux Kustomization and app files ---'
fd -t f . kubernetes/apps/flux-system/flux-operator | sort
rg -n -C 6 'kind: Kustomization|prune:|clusterrolebinding.yaml|flux-operator' \
kubernetes/apps/flux-system/flux-operator
printf '%s\n' '--- all repository references to the binding and roles ---'
rg -n -C 4 'flux-admin|flux-web-admin|flux-web-user' . --glob '!*.lock' || true
printf '%s\n' '--- recent commits affecting the Flux RBAC transition ---'
git log --oneline --all -- kubernetes/apps/flux-system/flux-operatorRepository: Tanguille/cluster
Length of output: 11311
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- working-tree change under review ---'
git status --short
git diff -- kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml \
kubernetes/apps/flux-system/flux-operator/app/helmrelease.yaml \
kubernetes/apps/flux-system/flux-operator/ks.yaml
printf '%s\n' '--- exact prior binding introduction/change ---'
git log --oneline -S'flux-web-admin' -- kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml
git log --oneline -S'flux-web-user' -- kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml
git show --format=fuller --stat 1981a310 -- kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml kubernetes/apps/flux-system/flux-operator/app/helmrelease.yaml
git show 1981a310 -- kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml kubernetes/apps/flux-system/flux-operator/app/helmrelease.yaml
printf '%s\n' '--- Flux Kustomization controller settings and prune documentation references ---'
rg -n -C 3 'server-side|force|prune|inventory|delet(e|ion)' \
kubernetes flux-system .github 2>/dev/null | head -250 || trueRepository: Tanguille/cluster
Length of output: 18963
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- current revision and target-file commits ---'
git rev-parse HEAD
git log --format='%h %ad %s' --date=short -12 -- kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml
git log --format='%h %ad %s' --date=short -12 -S'flux-web-user' -- kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml
printf '%s\n' '--- commits that contain the current role reference ---'
for c in $(git log --format='%H' -S'flux-web-user' -- kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml | head -5); do
echo "commit $c"
git show --format='%h %s' --no-ext-diff "$c" -- kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml | sed -n '1,100p'
done
printf '%s\n' '--- Flux Kustomization resource and controller version context ---'
cat -n kubernetes/apps/flux-system/flux-operator/ks.yaml
cat -n kubernetes/flux/cluster/ks.yaml | sed -n '1,80p'
rg -n 'fluxcd|kustomize-controller|flux-system' kubernetes/apps/flux-system kubernetes/flux | head -120Repository: Tanguille/cluster
Length of output: 12316
🌐 Web query:
FluxCD Kustomization prune behavior when a resource is renamed, and whether prune deletes the old resource after creating the new resource
💡 Result:
When you rename a resource managed by FluxCD—specifically if you mean renaming the Kustomization object itself—the behavior depends on your prune configuration [1][2]. If spec.prune is set to true, renaming the Kustomization object will cause Flux to treat the original Kustomization as deleted, which triggers the garbage collection (pruning) of all resources previously managed by that Kustomization [1][3][2]. Consequently, those resources will be deleted from the cluster before the new Kustomization (with the new name) attempts to create them [2]. To rename a Flux Kustomization without triggering this deletion, you must perform the following steps [1][2]: 1. Disable garbage collection by setting spec.prune: false in your Kustomization manifest [1]. 2. Commit and push this change to your repository, then wait for Flux to reconcile and acknowledge the change (you can verify this using flux export kustomization) [1]. 3. Rename the Kustomization object in your manifest, then commit and push the change [1]. 4. Flux will delete the old Kustomization object but will not delete the associated workloads because pruning was disabled [1]. 5. Once the new Kustomization is active, you can re-enable pruning by setting spec.prune: true [1]. Flux will then adopt ownership of the existing resources [1]. If you are referring to renaming a specific resource within the manifests being synced by a Kustomization (rather than the Kustomization object itself), Flux will perceive this as a deletion of the old resource and an addition of a new one [3]. Because the old resource is missing from the source revision, Flux will prune it immediately if pruning is enabled [3]. There is no native behavior to ensure the new resource is created before the old one is deleted; the deletion of orphaned resources happens as part of the reconciliation process [3]. To move resources safely between different Kustomizations (or rename them in a way that requires avoiding deletion), the best practice is to temporarily disable pruning while the migration occurs [1].
Citations:
- 1: https://fluxcd.io/flux/faq/
- 2: Changing Kustomization name will deleted all objects fluxcd/flux2#3316
- 3: https://fluxcd.io/flux/components/kustomize/kustomizations/
Recreate the binding during the role transition. The flux-operator Kustomization has prune: true, but changing roleRef.name on the existing flux-admin ClusterRoleBinding is rejected because roleRef is immutable. Use a new binding name, then let Flux prune the old binding through a controlled migration.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kubernetes/apps/flux-system/flux-operator/app/clusterrolebinding.yaml` at
line 18, Rename the ClusterRoleBinding identified by `name: flux-web-user` to a
new binding name so the role transition creates a replacement resource instead
of modifying the immutable `roleRef`. Preserve the existing subject and role
references, allowing the `flux-operator` Kustomization’s `prune: true` behavior
to remove the old `flux-admin` binding during migration.
Source: Coding guidelines
| expr: |- | ||
| time() - kopiur_policy_last_backup_success_timestamp_seconds > 6 * 60 * 60 | ||
| for: 30m |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- target file ---'
cat -n kubernetes/apps/kopiur-system/kopiur/app/prometheusrule.yaml
printf '%s\n' '--- Kopiur-related files ---'
git ls-files | rg -i 'kopiur|snapshot.?policy|prometheus.?rule|servicemonitor|vmrule|prometheus'
printf '%s\n' '--- Kopiur references ---'
rg -n -i 'kopiur_policy_last_backup_success_timestamp_seconds|kopiur_repository|kopiur.*leader|SnapshotPolicy|prometheusRule|ruleSelector|vmrule|prometheusrule' kubernetes --glob '*.yaml' --glob '*.yml' || trueRepository: Tanguille/cluster
Length of output: 14744
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- target file ---'
cat -n kubernetes/apps/kopiur-system/kopiur/app/prometheusrule.yaml
printf '%s\n' '--- Kopiur-related files ---'
git ls-files | rg -i 'kopiur|snapshot.?policy|prometheus.?rule|servicemonitor|vmrule|prometheus'
printf '%s\n' '--- Kopiur references ---'
rg -n -i 'kopiur_policy_last_backup_success_timestamp_seconds|kopiur_repository|kopiur.*leader|SnapshotPolicy|prometheusRule|ruleSelector|vmrule|prometheusrule' kubernetes --glob '*.yaml' --glob '*.yml' || trueRepository: Tanguille/cluster
Length of output: 14744
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- SnapshotPolicy component ---'
cat -n kubernetes/components/kopiur/backup/snapshotpolicy.yaml
cat -n kubernetes/components/kopiur/backup/kustomization.yaml
cat -n kubernetes/components/kopiur/kustomization.yaml
printf '%s\n' '--- Kopiur application configuration ---'
cat -n kubernetes/apps/kopiur-system/kopiur/app/helmrelease.yaml
cat -n kubernetes/apps/kopiur-system/kopiur/app/kustomization.yaml
cat -n kubernetes/apps/kopiur-system/kopiur/ks.yaml
printf '%s\n' '--- monitoring selectors and PrometheusRule patterns ---'
rg -n -i 'ruleSelector|ruleNamespaceSelector|prometheus:|release:|monitoring.coreos.com/v1|kind: PrometheusRule' kubernetes --glob '*.yaml' --glob '*.yml' | head -250
printf '%s\n' '--- all SnapshotPolicy declarations ---'
rg -n -C 12 'kind: SnapshotPolicy' kubernetes --glob '*.yaml' --glob '*.yml' || trueRepository: Tanguille/cluster
Length of output: 14538
🏁 Script executed:
#!/bin/bash
set -eu
rg -n -i 'kopiur|snapshotpolicy|prometheusrule|ruleSelector|ruleNamespaceSelector' . --glob '*.yaml' --glob '*.yml' --glob '*.md' --glob '*.json' || trueRepository: Tanguille/LLMKube
Length of output: 11859
🌐 Web query:
Kopiur kopiur_policy_last_backup_success_timestamp_seconds SnapshotPolicy metrics
💡 Result:
The metric kopiur_policy_last_backup_success_timestamp_seconds is a component of the Kopiur observability suite, which monitors the backup operations of the Kopiur Kubernetes operator [1]. Kopiur is a Kopia-native operator that manages backups by separating backup recipes, schedules, and invocations into Kubernetes Custom Resource Definitions (CRDs) such as SnapshotPolicy, SnapshotSchedule, and Snapshot [2][3]. Key details regarding this metric and its context include: 1. Function: The metric kopiur_policy_last_backup_success_timestamp_seconds records the Unix timestamp of the last successful backup operation associated with a specific SnapshotPolicy [1]. It allows administrators to track the recency of backups for various PVCs and workloads managed by Kopiur [1][4]. 2. Observability: Metrics are exposed by the Kopiur controller via its /metrics endpoint [1]. Users can enable the metrics.serviceMonitor in the Kopiur Helm chart to scrape these metrics using Prometheus [1]. 3. Configuration and Alerting: The Kopiur Helm chart includes configuration options for these metrics, such as metrics.prometheusRule.backupStaleAfterSeconds, which defines the duration in seconds after which a SnapshotPolicy's last success timestamp is considered "stale" for alerting purposes [1]. 4. Relationship to Kopia: While Kopiur relies on Kopia for the underlying backup storage and movement [2][3], the kopiur_ prefixed metrics are specific to the Kopiur operator's management of its CRDs [1]. Kopia itself also implements native snapshot metrics (often under the kopia_ namespace, such as kopia_last_snapshot_age_seconds) [5][6], but kopiur_policy_last_backup_success_timestamp_seconds is specifically scoped to the operation of the Kopiur SnapshotPolicy resources [1].
Citations:
- 1: https://github.com/home-operations/kopiur/blob/main/deploy/helm/kopiur/README.md
- 2: https://kopiur.home-operations.com/
- 3: https://github.com/home-operations/kopiur
- 4: https://github.com/mitchross/talos-argocd-proxmox/blob/fedcc608/docs/storage-architecture.md
- 5: feat(server): snapshot metrics kopia/kopia#4100
- 6: [Feature Request] Prometheus metrics from kopia server kopia/kopia#609
🌐 Web query:
Kopiur GitHub kopiur_policy_last_backup_success_timestamp_seconds
💡 Result:
The term kopiur_policy_last_backup_success_timestamp_seconds is a Prometheus metric exported by Kopiur [1], a Kopia-native Kubernetes backup operator [2][3]. This metric tracks the Unix timestamp (in seconds) of the last successful backup operation associated with a Kopiur SnapshotPolicy [1]. It is primarily used within Prometheus alerting rules to monitor backup freshness. For example, it is often compared against a threshold (such as backupStaleAfterSeconds, which defaults to 172,800 seconds or 48 hours) to determine if a backup policy is considered stale [1]. Kopiur itself is an open-source project under the Home Operations organization, designed to manage Kopia repositories and snapshots as first-class Kubernetes custom resources (CRDs) [2][3][4].
Citations:
- 1: https://github.com/home-operations/kopiur/blob/main/deploy/helm/kopiur/README.md
- 2: https://github.com/home-operations/kopiur
- 3: https://kopiur.home-operations.com/
- 4: https://github.com/home-operations
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- Kopiur component consumers ---'
rg -n -C 8 'kopiur/backup|components:|SnapshotSchedule|BACKUP_SNAPSHOTCLASS|KOPIUR_PUID' kubernetes/apps kubernetes/components --glob '*.yaml' --glob '*.yml' | head -400
printf '%s\n' '--- Victoria Metrics and kube-state-metrics configuration ---'
cat -n kubernetes/apps/observability/victoria-metrics/app/helmrelease.yaml | sed -n '1,380p'
fd -i 'helmrelease.yaml' kubernetes/apps/observability/kube-state-metrics kubernetes/apps/observability 2>/dev/null | xargs -r -n1 sh -c 'echo "--- $0"; cat -n "$0"'
rg -n -i 'customResourceState|customresources|kube_customresource|kube-state-metrics|ruleSelector|ruleNamespaceSelector|Prometheus' kubernetes/apps/observability kubernetes/config --glob '*.yaml' --glob '*.yml' || trueRepository: Tanguille/cluster
Length of output: 50376
🏁 Script executed (no clone):
#!/bin/bash
set -eu
base='https://raw.githubusercontent.com/home-operations/kopiur/main'
for path in deploy/helm/kopiur/README.md deploy/helm/kopiur/values.yaml deploy/helm/kopiur/templates/prometheusrule.yaml; do
echo "--- $path"
curl -fsSL "$base/$path" | rg -n -C 8 'policy_last_backup|leader_is_leader|repo_snapshot_count|PrometheusRule|stale|serviceMonitor' || true
done
Length of output: 10915
🏁 Script executed (no clone):
#!/bin/bash
set -eu
python3 - <<'PY'
# Read-only semantic model of the relevant PromQL behavior.
# Each vector is represented by its label set and value.
from math import isclose
def absent(vector):
return [({}, 1)] if not vector else []
def freshness(vector, now, threshold):
return [(labels, now - value) for labels, value in vector
if now - value > threshold]
now = 1000
healthy = [
({"namespace": "app", "policy": "one"}, 900),
({"namespace": "app", "policy": "two"}, 900),
]
one_missing = [
({"namespace": "app", "policy": "one"}, 900),
]
print("healthy freshness:", freshness(healthy, now, 360))
print("one-policy-missing freshness:", freshness(one_missing, now, 360))
print("absent(healthy):", absent(healthy))
print("absent(one-policy-missing):", absent(one_missing))
print("absent(all-policies-missing):", absent([]))
PY
Length of output: 271
🏁 Script executed (no clone):
#!/bin/bash
set -eu
repo='home-operations/kopiur'
api="https://api.github.com/repos/$repo/git/trees/main?recursive=1"
curl -fsSL "$api" |
jq -r '.tree[].path' |
rg 'observability|metric|prometheus|helm/.+yaml$' |
head -120
Length of output: 174
🏁 Script executed (no clone):
#!/bin/bash
set -eu
for path in \
docs/dev/observability.md \
deploy/helm/kopiur/templates/monitoring/prometheusrule.yaml \
deploy/helm/kopiur/templates/prometheus-rule.yaml \
deploy/helm/kopiur/templates/servicemonitor.yaml; do
echo "--- $path"
curl -fsSL "https://raw.githubusercontent.com/home-operations/kopiur/main/$path" |
rg -n -C 10 'policy_last_backup|leader_is_leader|repo_snapshot_count|SnapshotPolicy|labels:|absent|stale' || true
done
Length of output: 18617
🏁 Script executed (no clone):
#!/bin/bash
set -eu
python3 - <<'PY'
# Minimal read-only vector model for the exact failure mode.
# A missing policy has no sample; a healthy remaining policy keeps the metric present.
metric = [
({"namespace": "media", "policy": "jellyfin"}, 900),
({"namespace": "media", "policy": "sonarr"}, 900),
]
after_one_policy_disappears = [
series for series in metric if series[0]["policy"] != "sonarr"
]
def global_absent(vector):
return "alert" if len(vector) == 0 else "no alert"
def stale(vector, now=1000, threshold=360):
return [labels for labels, timestamp in vector if now - timestamp > threshold]
print("freshness result after sonarr disappears:", stale(after_one_policy_disappears))
print("global absent result after sonarr disappears:", global_absent(after_one_policy_disappears))
print("global absent result after all policies disappear:", global_absent([]))
PY
Length of output: 282
Add per-policy missing-series detection. absent(kopiur_policy_last_backup_success_timestamp_seconds) stays false when any other policy still reports. Add an independent SnapshotPolicy inventory metric and join it with the per-policy metric to alert on missing or stale policies. Keep the global absence rule as a fleet-wide fallback.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kubernetes/apps/kopiur-system/kopiur/app/prometheusrule.yaml` around lines 18
- 20, Update the Prometheus rules around the existing stale-backup expression to
add an independent SnapshotPolicy inventory metric, then join that inventory
with kopiur_policy_last_backup_success_timestamp_seconds so each policy alerts
when its series is missing or older than six hours for 30 minutes. Retain the
existing global absent rule as the fleet-wide fallback.
1 participant
Backups ran hourly for 18d with
LAST-VERIFIEDempty on all 18 policies, and a cron stayed dark 40h. Both signals already existed on disk and in VictoriaMetrics; only the rules were missing.Changes
ai/hermes/app/prometheusrule.yamlkopiur-system/kopiur/app/prometheusrule.yamlcomponents/kopiur/backup/snapshotpolicy.yamlquickverificationflux-system/flux-operator/app/*.gitignoreage.key*Evidence
last_over_time(...[30m])on every hermes rule is load-bearing, not style. The series are pushed, not scraped, and VM's 5m staleness makes a bare selector empty between pushes:Group is
kopiur.freshness.rulesand the freshness alert isKopiurBackupOverduebecause the chart already ships akopiur.rulesgroup containingKopiurBackupStaleon the same metric at the same severity — a shared name would emit identical alert fingerprints.flux-web-admingrantedcreateonbatch/jobs,patch/restarton workloads anddeleteon pods to anyone reachingenvoy-internal, underauthentication.type: Anonymous. Swapped for the chart's read-onlyflux-web-user. Anonymous read of Secrets remains until the UI gets real auth or the route is dropped.*.keyand/age.keyboth missage.key.bak-x25519-only, which decrypts the tracked sops files in this public repo.Validation
kustomize buildon all changed dirskubectl apply --dry-run=serveron both PrometheusRules and an envsubst'd SnapshotPolicyOn merge
HermesCronJobFailedfires immediately for three jobs currently atlast_status_ok == 0(daily-trending-functional-3d-prints,homeassistant-improvement-daily,life-os-morning-briefing). That is the rule working. All 18 kopiur policies are under 1h old, so the backup alerts are silent.Deliberately not included
KopiurBackupOverdueat 24h to stop double-notification withKopiurBackupCritical— a threshold change, and a 24h-dead backup arguably deserves the noise.KopiurBackupStaleis now dominated by the 6h rule and adds a third notification at 48h.Summary by CodeRabbit
New Features
Chores