feat(hermes): alert on cron job health

[Tanguille]

The scheduler already records last_status, last_error, last_delivery_error and next_run_at per job in cron/jobs.json. Nothing read it, which is how a cron stayed dark for 40h. A script on the hermes PVC pushes that state to VictoriaMetrics every 10m; these are the rules over it.

Five alerts

HermesCronJobFailed, HermesCronDeliveryFailed (delivery is a separate failure from execution — the job can succeed and the post still vanish), HermesCronOverdue, HermesCronExporterStale, HermesCronExporterAbsent.

The last_over_time wrapper is load-bearing

The series are pushed, not scraped, and VM's 5m staleness makes a bare instant selector empty between pushes. Verified live:

hermes_cron_last_status_ok                       -> 0 series
last_over_time(hermes_cron_last_status_ok[30m])  -> 14 series

Without it absent() flaps and the failure rules go blind half the time. HermesCronExporterAbsent deliberately has no for: — absent() over a 2h window cannot flap, and any delay reopens a gap where Stale has resolved but Absent has not yet fired, reporting "recovered" mid-outage.

Validation

kustomize build, kubectl apply --dry-run=server, and every expression run against live VictoriaMetrics.

On merge

Fires immediately for jobs currently at last_status_ok == 0. That is the rule working, not a regression.

[coderabbitai]

Warning

Review limit reached

@Tanguille, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 40 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 46811104-bab8-42db-88cf-2692bc32f54b

📥 Commits

Reviewing files that changed from the base of the PR and between 6ac1a8b and 32bf8ab.

📒 Files selected for processing (2)

• kubernetes/apps/ai/hermes/app/kustomization.yaml
• kubernetes/apps/ai/hermes/app/prometheusrule.yaml

---

_{Comment @coderabbitai help to get the list of available commands.}

[tanguille-cluster]

@@ (root level) @@
# monitoring.coreos.com/v1/PrometheusRule/ai/hermes-rules
! + one document added:
+ apiVersion: monitoring.coreos.com/v1
+ kind: PrometheusRule
+ metadata:
+   name: hermes-rules
+   namespace: ai
+   labels:
+     kustomize.toolkit.fluxcd.io/name: hermes
+     kustomize.toolkit.fluxcd.io/namespace: ai
+ spec:
+   groups:
+   - name: hermes.rules
+     rules:
+     - alert: HermesCronJobFailed
+       annotations:
+         summary: "Hermes cron {{ $labels.name }} last run failed — it will keep failing on schedule until the cause is fixed"
+       expr: "last_over_time(hermes_cron_last_status_ok[30m]) == 0"
+       for: 15m
+       labels:
+         severity: warning
+     - alert: HermesCronDeliveryFailed
+       annotations:
+         summary: "Hermes cron {{ $labels.name }} ran but could not deliver its output — the result is lost, not late"
+       expr: "last_over_time(hermes_cron_delivery_failed[30m]) == 1"
+       for: 15m
+       labels:
+         severity: warning
+     - alert: HermesCronOverdue
+       annotations:
+         summary: "Hermes cron {{ $labels.name }} is {{ $value | humanizeDuration }} past its scheduled run — the scheduler is not firing it"
+       expr: "last_over_time(hermes_cron_overdue_seconds[30m]) > 3600"
+       for: 15m
+       labels:
+         severity: warning
+     - alert: HermesCronExporterStale
+       annotations:
+         summary: "Hermes cron health metrics are {{ $value | humanizeDuration }} stale — cron alerting is blind, check the hermes pod"
+       expr: "time() - last_over_time(hermes_cron_export_timestamp[2h]) > 1800"
+       for: 5m
+       labels:
+         severity: warning
+     - alert: HermesCronExporterAbsent
+       annotations:
+         summary: "No Hermes cron health metrics at all — the exporter never ran or was removed"
+       expr: absent(last_over_time(hermes_cron_export_timestamp[2h]))
+       labels:
+         severity: warning

[deepsource-io]

DeepSource Code Review

We reviewed changes in 6ac1a8b...32bf8ab on this pull request. Below is the summary for the review, and you can see the individual issues we found as inline review comments.

See full review on DeepSource ↗

PR Report Card

Overall Grade

Security   Reliability   Complexity   Hygiene

Code Review Summary

Analyzer	Status	Updated (UTC)	Details
JavaScript		Aug 2, 2026 3:02p.m.	Review ↗
Shell		Aug 2, 2026 3:02p.m.	Review ↗

---

Important

AI Review is run only on demand for your team. We're only showing results of static analysis review right now. To trigger AI Review, comment @deepsourcebot review on this thread.

[github-actions]

AI Automated Review

Analysis engine: qwen-3.6-fast@http://litellm.ai.svc.cluster.local/v1 (openai)

Recommendation: Approve

This PR adds well-documented PrometheusRule alerts for the Hermes cron job health exporter. The implementation follows repository conventions for naming, structure, and PromQL patterns used elsewhere in the cluster.

Change-by-Change Findings

1. kubernetes/apps/ai/hermes/app/kustomization.yaml: Correctly adds prometheusrule.yaml to the resources list. This is consistent with other apps in the repo (e.g., toolhive, cert-manager, coredns) that include their own PrometheusRules.
2. kubernetes/apps/ai/hermes/app/prometheusrule.yaml:
   • Uses apiVersion: monitoring.coreos.com/v1, which is standard for this cluster's VictoriaMetrics/Prometheus operator setup.
   • The use of last_over_time() is explicitly justified in comments and the PR body to handle push-based metrics staleness, preventing false positives from VictoriaMetrics' 5m default staleness threshold. This pattern is internally consistent with the exporter design described.
   • Alert thresholds (for: 15m, for: 5m) and expressions are reasonable for cron job health monitoring.
   • No resource quota or limit changes are introduced by these rules; they are purely declarative alerting logic.

Standards Compliance

• Follows Conventional Commit title format (feat(hermes): ...).
• Adheres to repository structure (kubernetes/apps/...).
• Consistent with existing PrometheusRule implementations in the repo (e.g., toolhive/config/prometheusrule.yaml, cert-manager/app/prometheusrule.yaml).
• Includes schema reference comment (# yaml-language-server: $schema=...).

Must-Check Items

• Validate manifest against target cluster version: The monitoring.coreos.com/v1 API is supported by the cluster's Prometheus/VictoriaMetrics operator. The syntax is valid YAML and uses standard PromQL functions available in VictoriaMetrics.
• Check for resource quota / limit changes: This PR only adds a PrometheusRule resource. It does not modify any Deployment, StatefulSet, or HelmRelease values that would affect CPU/memory requests or limits. No quota impact.

Unknowns or Needs Verification

None. The change is self-contained and relies on an existing exporter (scripts/cron_health_export.py on the PVC) whose existence is confirmed by the PR description and code comments.