fix: resolve firing and recurring cluster alerts - #4195
Merged
Conversation
Both ha-mcp containers peak at ~255Mi against a 256Mi limit; the primary server was OOMKilled 13 times in the last 30 days.
…y rule etcd-defrag reclaimed 302MB to 130MB per member on its last run but only runs weekly; fragmentation crosses the 50% alert threshold within ~4 days, so etcdDatabaseHighFragmentationRatio fires for roughly half of every week. A full three-member pass costs ~95s. vmsingle, vmagent, vlogs and the log collector all requested less memory than their observed peaks, leaving the entire metrics and logging path burstable and first in line for eviction on a node that is already at 97% of allocatable. vmsingle's 4Gi limit was within 80Mi of its 14d peak. RecordingRulesNoData flags recording rules that are empty by design: count:up0 is count(up == 0), which yields no samples precisely when every scrape target is healthy.
The ha VirtualMCPServer was OOMKilled twice at its 1Gi limit and the observability one at the operator's 512Mi default; both swing by an order of magnitude with session load. karakeep's mcp container peaked at 84% of its 100Mi limit.
Both sit at 100m and burst well past it: grafana throttles on 13% of CFS periods rendering dashboards, and speedtest-exporter on 74%, which caps the throughput it exists to measure. CPU is compressible, so the request still does the isolation work.
|
Warning Review limit reached
Next review available in: 2 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe changes adjust Kubernetes workload resources, make etcd-defrag run daily, update Gatus gateway monitoring, add Victoria Metrics rule configuration, and modify Xmrig sensor, thermal scaling, and alerting behavior. ChangesAI workload configuration
Platform and observability configuration
Xmrig thermal safeguards
Estimated code review effort: 3 (Moderate) | ~20 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
@@ spec.schedule @@
# batch/v1/CronJob/kube-system/etcd-defrag
! ± value change
- 0 3 * * 0
+ 0 3 * * *
@@ spec.template.spec.initContainers.gatus-sidecar.args @@
# apps/v1/Deployment/observability/gatus
! + one list entry added:
+ - "--gateway-name=envoy-external"
@@ spec.template.spec.containers.speedtest-exporter.resources.limits.cpu @@
# apps/v1/Deployment/observability/speedtest-exporter
! ± value change
- 100m
+ 1
@@ spec.template.spec.containers.vlogs.resources.requests.memory @@
# apps/v1/StatefulSet/observability/victoria-logs
! ± value change
- 512Mi
+ 1Gi
@@ spec.template.spec.containers.vlagent.resources.limits.memory @@
# apps/v1/DaemonSet/observability/victoria-logs-collector
! ± value change
- 256Mi
+ 512Mi
@@ spec.template.spec.containers.vlagent.resources.requests.memory @@
# apps/v1/DaemonSet/observability/victoria-logs-collector
! ± value change
- 128Mi
+ 256Mi
@@ data.config.yaml @@
# v1/ConfigMap/observability/victoria-metrics-sync-job-config
! ± value change in multiline text (one insert, no deletions)
common:
clusterLabel: "cluster"
multicluster: false
dashboards:
[93 lines unchanged)]
labels: {}
rules:
etcdMemberCommunicationSlow:
enabled: false
+ recordingRulesNoData:
+ enabled: false
groups:
kubernetes-apps:
jobNamespaces:
kube-state-metrics: .*
[34 lines unchanged)]
- enabled: true
url: https://raw.githubusercontent.com/VictoriaMetrics/VictoriaMetrics/master/deployment/docker/rules/alerts-single-node.yml
- enabled: false
url: https://raw.githubusercontent.com/VictoriaMetrics/VictoriaTraces/master/deployment/docker/rules/alerts-health.yml
@@ spec.resources.requests.memory @@
# operator.victoriametrics.com/v1beta1/VMAgent/observability/victoria-metrics
! ± value change
- 256Mi
+ 512Mi
@@ spec.resources.limits.memory @@
# operator.victoriametrics.com/v1beta1/VMSingle/observability/victoria-metrics
! ± value change
- 4Gi
+ 6Gi
@@ spec.resources.requests.memory @@
# operator.victoriametrics.com/v1beta1/VMSingle/observability/victoria-metrics
! ± value change
- 2Gi
+ 3Gi
|
@@ spec @@
# toolhive.stacklok.dev/v1beta1/VirtualMCPServer/ai/observability
! + one map entry added:
+ podTemplateSpec:
+ spec:
+ containers:
+ - name: vmcp
+ resources:
+ limits:
+ cpu: 500m
+ memory: 1Gi
+ requests:
+ cpu: 100m
+ memory: 256Mi
@@ spec.podTemplateSpec.spec.containers.vmcp.resources.limits.memory @@
# toolhive.stacklok.dev/v1beta1/VirtualMCPServer/ai/ha
! ± value change
- 1Gi
+ 2Gi
@@ spec.podTemplateSpec.spec.containers.mcp.resources.limits.memory @@
# toolhive.stacklok.dev/v1beta1/MCPServer/ai/karakeep
! ± value change
- 100Mi
+ 256Mi
@@ spec.podTemplateSpec.spec.containers.mcp.resources.limits.memory @@
# toolhive.stacklok.dev/v1beta1/MCPServer/ai/karakeep-opt
! ± value change
- 100Mi
+ 256Mi
@@ spec.podTemplateSpec.spec.containers.mcp.resources.limits.memory @@
# toolhive.stacklok.dev/v1beta1/MCPServer/ai/homeassistant
! ± value change
- 256Mi
+ 512Mi
@@ spec.podTemplateSpec.spec.containers.mcp.resources.limits.memory @@
# toolhive.stacklok.dev/v1beta1/MCPServer/ai/homeassistant-opt
! ± value change
- 256Mi
+ 512Mi
@@ spec.values.controllers.etcd-defrag.cronjob.schedule @@
# helm.toolkit.fluxcd.io/v2/HelmRelease/kube-system/etcd-defrag
! ± value change
- 0 3 * * 0
+ 0 3 * * *
@@ spec.values.sidecar.gatewayNames @@
# helm.toolkit.fluxcd.io/v2/HelmRelease/observability/gatus
! + one list entry added:
+ - envoy-external
@@ spec.deployment.spec.template.spec.containers.grafana.resources.limits.cpu @@
# grafana.integreatly.org/v1beta1/Grafana/observability/grafana
! ± value change
- 100m
+ 1
@@ spec.values.controllers.speedtest-exporter.containers.speedtest-exporter.resources.limits.cpu @@
# helm.toolkit.fluxcd.io/v2/HelmRelease/observability/speedtest-exporter
! ± value change
- 100m
+ 1
@@ spec.values.server.resources.requests.memory @@
# helm.toolkit.fluxcd.io/v2/HelmRelease/observability/victoria-logs
! ± value change
- 512Mi
+ 1Gi
@@ spec.values.resources.limits.memory @@
# helm.toolkit.fluxcd.io/v2/HelmRelease/observability/victoria-logs-collector
! ± value change
- 256Mi
+ 512Mi
@@ spec.values.resources.requests.memory @@
# helm.toolkit.fluxcd.io/v2/HelmRelease/observability/victoria-logs-collector
! ± value change
- 128Mi
+ 256Mi
@@ spec.values.defaultRules.rules @@
# helm.toolkit.fluxcd.io/v2/HelmRelease/observability/victoria-metrics
! + one map entry added:
+ recordingRulesNoData:
+ create: false
@@ spec.values.vmagent.spec.resources.requests.memory @@
# helm.toolkit.fluxcd.io/v2/HelmRelease/observability/victoria-metrics
! ± value change
- 256Mi
+ 512Mi
@@ spec.values.vmsingle.spec.resources.limits.memory @@
# helm.toolkit.fluxcd.io/v2/HelmRelease/observability/victoria-metrics
! ± value change
- 4Gi
+ 6Gi
@@ spec.values.vmsingle.spec.resources.requests.memory @@
# helm.toolkit.fluxcd.io/v2/HelmRelease/observability/victoria-metrics
! ± value change
- 2Gi
+ 3Gi
@@ spec.advanced.horizontalPodAutoscalerConfig.behavior.scaleDown.policies.0.periodSeconds @@
# keda.sh/v1alpha1/ScaledObject/web3/xmrig
! ± value change
- 60
+ 15
@@ spec.advanced.horizontalPodAutoscalerConfig.behavior.scaleDown.policies.0.type @@
# keda.sh/v1alpha1/ScaledObject/web3/xmrig
! ± value change
- Pods
+ Percent
@@ spec.advanced.horizontalPodAutoscalerConfig.behavior.scaleDown.policies.0.value @@
# keda.sh/v1alpha1/ScaledObject/web3/xmrig
! ± value change
- 1
+ 100
@@ spec.advanced.horizontalPodAutoscalerConfig.behavior.scaleDown.stabilizationWindowSeconds @@
# keda.sh/v1alpha1/ScaledObject/web3/xmrig
! ± value change
- 180
+ 0
@@ spec.groups.xmrig-guard.rules.rules @@
# monitoring.coreos.com/v1/PrometheusRule/web3/xmrig-guard-rules
! + one list entry added:
+ - alert: XmrigGuardLatchedUnsafe
+ annotations:
+ summary: "{{ $labels.node }} has been gated unsafe for 6h while its NVMe stayed within the 60C recovery band, so the guard is latched rather than thermally tripped and mining is disabled with no signal"
+ expr: |
+ max_over_time(xmrig_guard_safe{node=~"control-[123]"}[6h]) == 0
+ and on (node)
+ max_over_time(xmrig_guard_nvme_temp_max_celsius{node=~"control-[123]"}[2h]) <= 60
+ for: 30m
+ labels:
+ severity: warning
@@ data.controller.py @@
# v1/ConfigMap/web3/xmrig-guard
! ± value change in multiline text (three inserts, two deletions)
"""Small, dependency-free XMRig safety signal controller.
The controller deliberately treats telemetry as untrusted input. A complete
set of fresh samples is required before a node can become safe. Policy is
[ten lines unchanged)]
from dataclasses import dataclass
from datetime import datetime, timezone
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+ # temp1 is the Composite sensor, which is what the drives' 70C rating specifies and
+ # what smartctl reports. temp2-temp4 are internal die sensors that run ~9C hotter and
+ # carry no comparable rating, so including them gated a Composite threshold against
+ # the wrong reading. Fewer series also means fewer chances for a single missing sample
+ # to fail the identity check and latch a node closed.
SENSORS = {
"control-1": (),
- "control-2": (("nvme_nvme0", "temp1"), ("nvme_nvme0", "temp2"), ("nvme_nvme0", "temp3"), ("nvme_nvme1", "temp1"), ("nvme_nvme1", "temp2"), ("nvme_nvme1", "temp3"), ("nvme_nvme1", "temp4")),
- "control-3": (("nvme_nvme0", "temp1"), ("nvme_nvme0", "temp2"), ("nvme_nvme0", "temp3"), ("nvme_nvme0", "temp4"), ("nvme_nvme1", "temp1"), ("nvme_nvme1", "temp2"), ("nvme_nvme1", "temp3"), ("nvme_nvme1", "temp4")),
+ "control-2": (("nvme_nvme0", "temp1"), ("nvme_nvme1", "temp1")),
+ "control-3": (("nvme_nvme0", "temp1"), ("nvme_nvme1", "temp1")),
}
ENDPOINT = "http://vmauth-victoria-metrics.observability.svc.cluster.local:8427"
EVALUATION_INTERVAL_SECONDS = 60
SOURCE_SAMPLE_MAX_AGE_SECONDS = 120
[212 lines unchanged)]
class GuardController:
def __init__(self, telemetry, clock=time.monotonic, wall_clock=lambda: datetime.now(timezone.utc)):
self.telemetry = telemetry
self.clock, self.wall_clock = clock, wall_clock
- self.policies = {"control-2": DwellPolicy(60, 70, 600, 120, MAX_SOURCE_GAP_SECONDS), "control-3": DwellPolicy(60, 70, 600, 120, MAX_SOURCE_GAP_SECONDS)}
+ # Trip 64C / recover 60C on Composite, against a 70C drive rating. Mining raises
+ # Composite at up to 1.1C/min, so the 6C band is ~5.5 minutes wide. Worst-case
+ # response is ~4.25: up to one evaluation interval to sample the crossing, 120s
+ # trip dwell, 60s of KEDA polling, then the HPA drop (scaledobject.yaml sheds all
+ # replicas at once for exactly this reason). That leaves the peak near 69C, and is
+ # conservative because it assumes full heat output until the last miner exits.
+ # Idle Composite never exceeded 62C over 7d on either node, so the trip does not
+ # false-fire, and it sits at or below 60C for 100%/90% of the time, so recovery is
+ # reachable rather than the permanent latch the old 60C/70C pair produced.
+ self.policies = {"control-2": DwellPolicy(60, 64, 600, 120, MAX_SOURCE_GAP_SECONDS), "control-3": DwellPolicy(60, 64, 600, 120, MAX_SOURCE_GAP_SECONDS)}
self.cpu_policy = DwellPolicy(50, 70, 600, 120, MAX_SOURCE_GAP_SECONDS)
self.ready = False
self.metrics = {
"evaluations": 0, "query_errors": {node: 0 for node in SENSORS},
[108 lines unchanged)]
if __name__ == "__main__":
main()
|
Tanguille
marked this pull request as ready for review
July 27, 2026 12:53
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
kubernetes/apps/observability/victoria-metrics/app/helmrelease.yaml (1)
333-337: 🩺 Stability & Availability | 🔵 TrivialVerify that the
vmsinglerequest covers steady-state usage.The comment reports steady-state usage up to
3.4Gi, but the request is only3Gi. Under node pressure, this can make a normally operating Burstable pod more eviction-prone. Confirm that3Giis an intentional sizing target; otherwise raise it to the observed steady-state/p95 level.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kubernetes/apps/observability/victoria-metrics/app/helmrelease.yaml` around lines 333 - 337, Update the vmsingle resource request under the memory resources to cover the reported steady-state usage of up to 3.4Gi, unless 3Gi is explicitly confirmed as the intentional sizing target; align the request with the observed steady-state or p95 level while preserving the existing memory limit.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kubernetes/apps/ai/litellm/instance/proxy.yaml`:
- Around line 66-73: Update the retry_policy under routerSettings to use a retry
policy key supported by LiteLLM v1.93.0 for connection failures surfaced as
InternalServerError. Remove InternalServerErrorRetries and configure the
applicable supported policy while preserving zero retries for timeouts.
In `@kubernetes/apps/ai/toolhive/config/virtualmcpservers.yaml`:
- Around line 30-41: Move the vmcp resource override from the VirtualMCPServer
named observability to the intended flux resource, preserving the existing CPU
and memory requests/limits under flux.spec.podTemplateSpec. If observability is
actually the target, leave it there and add the equivalent override to flux
instead.
In `@kubernetes/apps/kube-system/etcd-defrag/app/helmrelease.yaml`:
- Around line 35-38: Update the comment above the CronJob schedule in the etcd
defragmentation configuration to describe the configured daily 03:00 cadence
instead of a weekly run, and revise the fragmentation and runtime rationale so
it accurately reflects daily execution. Leave the timeZone and schedule values
unchanged.
---
Nitpick comments:
In `@kubernetes/apps/observability/victoria-metrics/app/helmrelease.yaml`:
- Around line 333-337: Update the vmsingle resource request under the memory
resources to cover the reported steady-state usage of up to 3.4Gi, unless 3Gi is
explicitly confirmed as the intentional sizing target; align the request with
the observed steady-state or p95 level while preserving the existing memory
limit.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: a4bf4728-4a6f-4169-a636-d4d4454764f5
📒 Files selected for processing (12)
kubernetes/apps/ai/litellm/instance/proxy.yamlkubernetes/apps/ai/toolhive/config/homeassistant.yamlkubernetes/apps/ai/toolhive/config/karakeep.yamlkubernetes/apps/ai/toolhive/config/virtualmcpservers.yamlkubernetes/apps/kube-system/etcd-defrag/app/helmrelease.yamlkubernetes/apps/observability/exporters/speedtest-exporter/app/helmrelease.yamlkubernetes/apps/observability/gatus/app/helmrelease.yamlkubernetes/apps/observability/grafana/instance/grafana.yamlkubernetes/apps/observability/victoria-logs/app/helmrelease.yamlkubernetes/apps/observability/victoria-logs/collector/helmrelease.yamlkubernetes/apps/observability/victoria-metrics/app/helmrelease.yamlkubernetes/apps/web3/xmrig-guard/app/prometheusrule.yaml
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
Tanguille/LLMKube(auto-detected)
…d xmrig guard #4086 added the isolated external probe and dropped envoy-external from the sidecar's gatewayNames in the same change. Since 2026-07-20 22:00 that has left all eight public hostnames undiscovered, and GatusEndpointDown filters on group=~"external|services", so the only externally-alerting endpoint was the probe itself. The probe is a genuine round trip through Cloudflare's edge and back down the tunnel, but it terminates on the dedicated probe gateway, not the envoy-external one that serves users. The xmrig guard recovers only at or below 60C, which neither mining node reaches, so a node that trips once stays gated off. Thresholds are left alone on purpose: on every sustained-mining day control-3 peaked at 89-95C against 66-75C when idle, which is the range that previously forced a cold power cycle. The gate being shut is the correct outcome. What was missing is a signal for the case where it is shut without cause, so the new rule pairs the gate state with the temperature actually sitting in the recovery band.
… band The guard read max() across every NVMe hwmon sensor, but only temp1 is Composite, which is the reading the drives' 70C rating specifies and the one smartctl reports. temp2-temp4 are internal die sensors running ~9C hotter with no comparable rating, so a Composite-spec threshold was being compared against the wrong number and every measurement was ~9C pessimistic. On Composite the drives are well inside their rating at idle: median 57.9C and 58.9C, and 0.0% of the last 7d above 70C on either node. Mining is what breaks the rating, peaking at 79.8C and 82.8C, so the gate still has to shut during mining. Trip moves 70C -> 65C and recovery stays at 60C. Idle Composite never reached 65C over 7d, so the trip does not false-fire, and it sits at or below 60C for 100% (control-2) and 90% (control-3) of the time, so recovery is reachable and the permanent latch is gone. Mining raises Composite at up to 1.1C/min, giving ~4.5 minutes across the 65-70C band; the 180s scale-down stabilization did not fit inside that with the 120s dwell and 60s poll, so it drops to 0. Restricting to Composite also shrinks the audited set from 7 and 8 series to 2, which is what failed today when one missing sample tripped the identity check and latched control-2 closed.
Tanguille
force-pushed
the
fix/alert-remediation
branch
from
July 27, 2026 13:08
ec49390 to
5fcebb1
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
kubernetes/apps/web3/xmrig-guard/app/resources/test_controller.py (1)
163-166: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winAssert the exact audited sensor pairs.
These assertions verify only the count and
temp1label. Assert the exactnvme_nvme0/temp1andnvme_nvme1/temp1pairs to catch drive-identity regressions.Suggested assertion
+ expected = { + ("nvme_nvme0", "temp1"), + ("nvme_nvme1", "temp1"), + } - self.assertEqual(len(controller.SENSORS["control-2"]), 2) - self.assertEqual(len(controller.SENSORS["control-3"]), 2) - self.assertTrue(all(sensor == "temp1" for node in ("control-2", "control-3") for _, sensor in controller.SENSORS[node])) + for node in ("control-2", "control-3"): + self.assertEqual(set(controller.SENSORS[node]), expected)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kubernetes/apps/web3/xmrig-guard/app/resources/test_controller.py` around lines 163 - 166, Update the sensor assertions in the controller test to verify the exact audited pairs for both control-2 and control-3: nvme_nvme0/temp1 and nvme_nvme1/temp1. Replace or supplement the count/label-only checks while preserving the existing exclusion of die sensors.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kubernetes/apps/web3/monero/xmrig/scaledobject.yaml`:
- Around line 27-31: The configured three-replica scale-down path can exceed the
70C safety margin before miners are removed. In
kubernetes/apps/web3/monero/xmrig/scaledobject.yaml lines 27-31, adjust the HPA
scale-down behavior by increasing responsiveness, raising the thermal trip
threshold, or relaxing the scale-down cap; in
kubernetes/apps/web3/xmrig-guard/app/resources/controller.py lines 248-253,
update the 65C/60C timing justification to accurately reflect the resulting HPA
response budget.
---
Nitpick comments:
In `@kubernetes/apps/web3/xmrig-guard/app/resources/test_controller.py`:
- Around line 163-166: Update the sensor assertions in the controller test to
verify the exact audited pairs for both control-2 and control-3:
nvme_nvme0/temp1 and nvme_nvme1/temp1. Replace or supplement the
count/label-only checks while preserving the existing exclusion of die sensors.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 4680840a-c09c-4b68-b13f-89ad2461fa82
📒 Files selected for processing (7)
kubernetes/apps/kube-system/etcd-defrag/app/helmrelease.yamlkubernetes/apps/observability/gatus/app/helmrelease.yamlkubernetes/apps/observability/victoria-metrics/app/helmrelease.yamlkubernetes/apps/web3/monero/xmrig/scaledobject.yamlkubernetes/apps/web3/xmrig-guard/app/prometheusrule.yamlkubernetes/apps/web3/xmrig-guard/app/resources/controller.pykubernetes/apps/web3/xmrig-guard/app/resources/test_controller.py
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
Tanguille/LLMKube(auto-detected)
🚧 Files skipped from review as they are similar to previous changes (3)
- kubernetes/apps/kube-system/etcd-defrag/app/helmrelease.yaml
- kubernetes/apps/web3/xmrig-guard/app/prometheusrule.yaml
- kubernetes/apps/observability/gatus/app/helmrelease.yaml
AI Automated Review (incremental)Analysis engine: qwen-3.6-fast@http://litellm.ai.svc.cluster.local/v1 (openai) Recommendation: Approve This incremental commit ( Change-by-Change Findings
Standards Compliance
Unknowns or Needs VerificationNone. The changes are self-contained and supported by inline comments referencing observed cluster telemetry. |
The scale-down policy capped removal at one pod per 60s, so draining three replicas took a further 180s that the previous timing note did not count. Full worst case was 60s to sample the crossing + 120s trip dwell + 60s KEDA polling + 180s drain = 7 minutes, which at the observed 1.1C/min peak put the drives at 72.7C, past their 70C rating. Removing the per-pod cap brings the response to ~4.25 minutes. Pairing that with a 64C trip gives a 6C band worth ~5.5 minutes and lands the peak near 68.7C. Idle Composite never exceeded 62C over 7d on either node, so the lower trip does not false-fire and still leaves 4C of hysteresis above the 60C recovery point. Gradual drain only ever made sense for tracking a falling solar curve, never for a thermal trip, and mining is free to interrupt.
This was referenced Jul 27, 2026
This was referenced Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves the alerts that were firing, plus the ones that fire on a regular cycle. Every resource value here is set from a measured 14d peak, not a guess.
etcd fragmentation
The
etcd-defragCronJob works exactly as designed: its last run reclaimed 302MB down to 130MB on all three members in 95s, transferred leadership cleanly, and found no alarms. The problem is cadence. Fragmentation crosses the alert's 50% threshold around day 4, soetcdDatabaseHighFragmentationRatiofires for roughly half of every week while nothing is actually wrong (206MB in a 2GB quota). Now runs daily.RecordingRulesNoData
Disabled via
defaultRules.rules.recordingRulesNoData.create: false, matching the existingetcdMemberCommunicationSlowentry. All three rules it flags are empty by design:count:up0iscount(up == 0), which yields no samples precisely when every scrape target is healthy, as do the orphan-pod variants ofnamespace_workload_pod:kube_pod_owner:relabel.Undersized memory
The entire metrics and logging path requested less than it used, leaving it burstable and first in line for eviction on a node already at 97% of allocatable. vmsingle's limit was within 80Mi of its 14d peak.
CPU throttling
grafana and speedtest-exporter both sat at a 100m limit and burst well past it: 13% and 74% of CFS periods throttled, against p95 usage of 34m and 8m. Raised both to 1 CPU. Requests are unchanged, so isolation is unaffected since CPU is compressible. The speedtest limit was capping the very throughput that exporter exists to measure.
Capacity note
This adds roughly 2.4Gi of requests against 5.97Gi of headroom (control-1 1.82Gi, control-2 3.13Gi, control-3 1.02Gi). That is deliberate. There is no meaningful over-requesting left to reclaim: the remaining slack sits in qwen3-embedding's GTT headroom, which cgroup accounting cannot see, in the
osd_memory_target-driven ceph OSDs, and in the runner requests that fixed the earlier OOM cascade. A scheduler that believes vmsingle needs 2Gi when it uses 3.4Gi is the more expensive problem.KubeMemoryOvercommitwill get louder as a result, and it is telling the truth. control-2 and control-3 have 21.7Gi allocatable each against control-1's 53Gi, and that asymmetry is the real constraint. The descheduler'sLowNodeUtilizationwould be a no-op since no node is underutilized.vmsingle currently sits on control-3 with 1.02Gi free, so its +1Gi is a tight fit. It carries
homelab-critical(500000,PreemptLowerPriority), so the worst case is preemption rather than a Pending pod.Done outside this branch
21054436eon main moved llmkube'sbindAddressto the InferenceService spec, clearing threeKustomizationReconciliationFailurealerts.llmkube-models,litellmandmeminihad been blocked since 07-24 and are Ready again.Failedkopiur Snapshot records, 129 of them from the 07-20 CRD outage. Backups are current and all 18 policies reportkopiur_snapshotpolicy_last_backup_success.Deliberately not included
GatusEndpointDown: 75 of 84 samples are a single series, "Gatus external probe". External path, not app outages.TooManyLogs: the VM stack's own log volume.NodeGTTMemoryHigh: the known unmonitored iGPU GTT leak, worth its own pass.Gatus external coverage
PR #4086 added the isolated external probe and removed
envoy-externalfrom the sidecar'sgatewayNamesin the same change. The external endpoint count drops from 7 to 1 at exactly 07-20 22:00 and never recovers. SinceGatusEndpointDownfilters ongroup=~"external|services", the only externally-alerting endpoint for the past week has been the probe itself, leaving these eight public hostnames unmonitored: nextcloud, share, flux-webhook, jellyfin, seerr/jellyseerr, wizarr, homeassistant, kromgo.The probe is well built and stays: it resolves via 1.1.1.1 to dodge split-horizon DNS, CNAMEs to the cloudflared tunnel, and so makes a genuine round trip out through Cloudflare's edge and back down the tunnel. What it does not do is traverse
envoy-external, the gateway that actually serves users; it terminates on the dedicated probe gateway. The two checks are complementary, so re-adding the gateway restores per-app coverage without touching the probe. Also drops a dead&portYAML anchor the language server was flagging.xmrig guard
The guard read
max()across every NVMe hwmon sensor, but onlytemp1is Composite, which is the reading the drives' 70C rating specifies and the one smartctl reports.temp2-temp4are internal die sensors running about 9C hotter with no comparable rating. Confirmed against smartctl on both nodes: Composite matches to within a degree (46.85 vs 47, 55.85 vs 56), while "Sensor 1" is the one that was driving the gate. Every reading the policy acted on was therefore roughly 9C pessimistic.Recomputed on Composite, the drives are well inside their rating at rest and it is mining that breaks it:
Three coupled changes follow from that:
SENSORSdrops totemp1per drive. This also shrinks the audited set from 7 and 8 series to 2, which is what failed on 07-27 when a single missing sample tripped the identity check and latched control-2 shut withincomplete or changed NVMe identity set.min(xmrig_guard_safe)across all three nodes, that alone pinned mining at zero regardless of solar surplus.XmrigGuardLatchedUnsafecovers the remaining gap. The existing rules fire when the gate is bypassed or the signal is missing, never when it is stuck shut, which is how a week of disabled mining went unnoticed. It pairs the gate state with the temperature actually sitting in the recovery band, so it fires for a stuck latch and stays quiet when the drives are genuinely too hot. Verified against live data: silent at the 60C bound, and matching control-3 when relaxed to 70C, which confirms theand on (node)join is wired correctly.19/19 controller tests pass, including an added assertion that only Composite sensors are audited.