feat(observability): alert when TrueNAS SSD_Pool free space runs low

[Tanguille]

SSD_Pool had no free-space alerting. It has already hit 0.00 GiB once, unnoticed.

Why it matters

control-1's disk is a sparse zvol on this pool:

SSD_Pool/vm/TALOS.block
  volsize          501G
  refreservation   none     <-- thin provisioned
  referenced       416G

With no refreservation, exhausting the pool fails guest writes and stops the VM. It does not merely fill a share.

Every other rule in this file is scoped kubernetes_node!="", which excludes the external TrueNAS target, so the pool backing the cluster's only VM was never covered.

It already happened

2026-07-28, at 1m scrape resolution:

first sample under 20 GiB	21:10 (16.57 GiB)
minimum reached	0.00 GiB
last sample under 20 GiB	22:26
sustained duration	76 minutes, 77 samples

Cause was back-to-back image bumps (#4227 at 20:01, #4234 at 22:17) writing a second 64G HiCache generation alongside the live one. for: 15m would have fired at 21:25, about an hour before it bottomed out.

Worth noting: this window overlaps the unexplained 07-28/29 performance drop, which was previously attributed to a FORK_REF bump. A pool at zero free space is a much better candidate.

The rule

expr: node_filesystem_avail_bytes{job="node-exporter", fstype="zfs", mountpoint="/mnt/SSD_Pool"} < 21474836480
for: 15m
severity: critical

No exporter needed, the metric is already scraped from 192.168.0.27:9100.

Verification

check	result
selector matches	exactly 1 series
fires now (58.9 GiB free)	no
fires across the 07-28 event	yes, all 77 samples
existing TrueNAS silence conflicts	no, scoped to alertname: NodeMemoryHighUtilization
renders + server dry-run	pass

Caveat on the threshold

20 GiB is roughly half a day of runway, not more. Daily snapshot deltas on this volume have been 43.4 G (07-30) and 143 G (07-29), against a 2-day retention. Fine as a "wake up now" line, but it will not give days of notice. A second, higher warning tier could be added later if that proves too tight.

[coderabbitai]

Warning

Review limit reached

@Tanguille, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 41 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 1d37cb11-f662-456e-a318-6ab00a6d2833

📥 Commits

Reviewing files that changed from the base of the PR and between 3b6bfb7 and a32baf3.

📒 Files selected for processing (1)

• kubernetes/apps/observability/exporters/node-exporter/app/prometheusrule.yaml

---

_{Comment @coderabbitai help to get the list of available commands.}

[tanguille-cluster]

@@ spec.groups.node-exporter.rules.rules @@
# monitoring.coreos.com/v1/PrometheusRule/observability/node-exporter-rules
! + one list entry added:
+ - alert: TrueNASSSDPoolSpaceLow
+   annotations:
+     summary: "TrueNAS SSD_Pool has under 20GiB free — control-1's thin-provisioned disk fails writes if this pool fills"
+   expr: "node_filesystem_avail_bytes{job=\"node-exporter\", fstype=\"zfs\", mountpoint=\"/mnt/SSD_Pool\"} < 21474836480"
+   for: 15m
+   labels:
+     severity: critical

[github-actions]

AI Automated Review

Analysis engine: qwen-3.6-fast@http://litellm.ai.svc.cluster.local/v1 (openai)

Recommendation

Approve. This is a well-justified, narrowly scoped addition of a critical alert for an infrastructure blind spot that has already caused production impact. The implementation follows repository conventions and the PR body provides thorough verification evidence.

Change-by-Change Findings

kubernetes/apps/observability/exporters/node-exporter/app/prometheusrule.yaml

• Alert name TrueNASSSDPoolSpaceLow: Clear, descriptive naming consistent with existing alerts in the file (e.g., NodeMemoryAvailableLow, NodeGTTMemoryHigh).
• Expression: Uses standard node-exporter metric node_filesystem_avail_bytes with appropriate label selectors (job="node-exporter", fstype="zfs", mountpoint="/mnt/SSD_Pool"). The threshold 21474836480 (20 GiB) is explicitly justified by historical data from the 2026-07-28 incident.
• Duration for: 15m: Consistent with other alerts in this file (NodeGTTMemoryHigh, NodeDRMCollectorFailed) and other PrometheusRule files in the repo. The comment explains why 15m would have provided ~1h advance warning before hitting zero.
• Severity critical: Appropriate given that pool exhaustion stops the VM hosting control-1. Matches the severity level used for NodeMemoryAvailableLow which also threatens cluster stability.
• Annotations: Summary clearly communicates both the condition and the consequence (thin-provisioned disk write failures). Follows the multi-line summary pattern used throughout the file.
• Comments: Detailed inline comments explain the architectural context (sparse zvol, no refreservation), the gap in existing coverage (kubernetes_node!="" excludes external targets), and the historical incident timeline. This level of documentation matches the style of other rules in this file.

Standards Compliance

• Conventional Commit title: ✅ feat(observability): alert when TrueNAS SSD_Pool free space runs low — follows the feat(scope): description pattern from AGENTS.md.
• Small and focused change: ✅ Single file, single alert addition, no unrelated modifications.
• YAML structure: ✅ Correctly appended to the existing rules list under spec.groups[0]. No syntax issues visible in the diff.
• No secrets committed: ✅ No credentials or sensitive data introduced.

Unknowns or Needs Verification

• Metric availability: The PR states the metric is "already scraped from 192.168.0.27:9100" but I cannot independently verify the scrape configuration for the external TrueNAS target. The PR author's verification table confirms "selector matches: exactly 1 series" and "renders + server dry-run: pass," which provides confidence. If this were a higher-risk change, I would request confirmation of the ServiceMonitor/PodMonitor configuration exposing this target.
• Silence conflicts: The PR notes "existing TrueNAS silence conflicts: no, scoped to alertname: NodeMemoryHighUtilization" — this is reasonable but relies on the author's knowledge of existing Alertmanager configurations not visible in this diff.