feat(web3): drop hugepages, fold the xmrig guard into monero - #4385
Merged
Conversation
Contributor
 |
|
Overall Grade |
Security Reliability Complexity Hygiene |
Code Review Summary
| Analyzer | Status | Updated (UTC) | Details |
|---|---|---|---|
| JavaScript |  |
Aug 8, 2026 1:13p.m. | Review ↗ |
| Shell | |
Aug 8, 2026 1:13p.m. | Review ↗ |
Important
AI Review is run only on demand for your team. We're only showing results of static analysis review right now. To trigger AI Review, comment @deepsourcebot review on this thread.
|
Warning Review limit reached
Next review available in: 51 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (14)
📝 WalkthroughWalkthroughThe change relocates and renames the XMRig guard, removes hugepage reservations, adds per-node freshness budgets, lowers NVMe thermal thresholds and recovery dwell, updates monitoring rules and tests, and revises operational documentation. ChangesHugepage removal and memory settings
Guard relocation and policy integration
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant GuardController
participant VictoriaMetrics
participant NodePolicy
GuardController->>VictoriaMetrics: Query CPU and NVMe telemetry
VictoriaMetrics-->>GuardController: Return samples
GuardController->>NodePolicy: Check freshness and thermal limits
NodePolicy-->>GuardController: Return dwell or panic decision
GuardController-->>GuardController: Write guard status
Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1⚔️ Resolve merge conflicts 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/database/postgres-cnpg-performance-check.md`:
- Line 107: Update the memory summary in the documentation sentence to match the
values defined by the cluster configuration: shared_buffers 1GB, work_mem 16MB,
maintenance_work_mem 512MB, a 3Gi per-instance limit, and 150 connections.
Recalculate the stated peak work_mem usage accordingly and preserve the
surrounding cache-hit guidance.
In `@docs/xmrig-solar-mining.md`:
- Line 156: Update the operational documentation sentence to spell the product
name as “cAdvisor” instead of “cadvisor,” without changing the surrounding
freshness-budget details.
In `@kubernetes/apps/web3/monero/guard/resources/controller.py`:
- Line 370: Rename the log_message method’s format parameter to message_format,
preserving the method override signature behavior and positional *args handling.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: b443efd7-57f1-4dbd-b3b6-b87c540e4b45
📒 Files selected for processing (14)
docs/database/postgres-cnpg-performance-check.mddocs/xmrig-solar-mining.mdkubernetes/apps/database/cloudnative-pg/cluster/cluster.yamlkubernetes/apps/web3/kustomization.yamlkubernetes/apps/web3/monero/guard/helmrelease.yamlkubernetes/apps/web3/monero/guard/kustomization.yamlkubernetes/apps/web3/monero/guard/prometheusrule.yamlkubernetes/apps/web3/monero/guard/resources/controller.pykubernetes/apps/web3/monero/guard/resources/test_controller.pykubernetes/apps/web3/monero/guard/servicemonitor.yamlkubernetes/apps/web3/monero/ks.yamlkubernetes/apps/web3/monero/xmrig/resourceset.yamlkubernetes/apps/web3/xmrig-guard/ks.yamltalos/patches/global/machine-sysctls.yaml
💤 Files with no reviewable changes (3)
- talos/patches/global/machine-sysctls.yaml
- kubernetes/apps/web3/xmrig-guard/ks.yaml
- kubernetes/apps/web3/kustomization.yaml
…tion Jellyfin clients request dozens of unique /HomeScreen/CachedImage and /Items/<id>/Images URIs per page load, tripping http-crawl-non_statics (47 events in 3s, all HTTP 200) and banning the client for 24h. Scoped to 2xx/3xx so scanners probing jellyfin still get counted.
vm.nr_hugepages withheld 2368Mi from every node's allocatable memory even at 0 replicas, while control-2/3 sat at 99% booked with a 150Mi exporter Pending. Drop the sysctl and move xmrig to regular pages. xmrig requests 2Gi against the ~2336Mi RandomX needs: post-reclaim headroom is 2389Mi/2373Mi, and the true figure parks the miner Pending forever behind preemptionPolicy: Never. /tmp gains a sizeLimit since the dataset now shares the container's memory limit rather than living in hugetlb. Move xmrig-guard to monero/guard and rename its resources. Metrics stay xmrig_guard_* : the KEDA triggers, five alerts, the dashboard and 180d of series all reference them. Tuning, replayed over 180d at the native 20s cadence: - recovery dwell 300->180s, +0.8pp/+1.1pp over 30d, no change above 65C - panic 68->67C, zero measured cost, caps the hottest safe sample at 66.8C - recover 62 / trip 65 / trip_dwell 60 held against a 612-candidate grid Nothing is loosened: real bursts peak at 69.8C against a 70C rating, and the rise is p99 1.35C/min, not the 1.1 the design was sized on. control-1's flapping was telemetry, not policy. cadvisor scrapes at 60s where the other six sources scrape at 20s and query_cpu dates the observation by min() across them, so it self-invalidated 465 times in 10.26d against 45 and 49 on the NVMe nodes, each latching a 600s recovery dwell. Its budget is now 180s and carried on the policy, so the freshness check, the anti-replay gap and the dwell reset cannot drift apart; the VM step follows or the samples would never be returned.
… spelling The doc recommended reducing shared_buffers to 1GB and this branch does it, so the 'current tuning' paragraph and the recommendation below it both went stale. Current figures now match cluster.yaml; the 24h and 7d observations are left as measured and labelled as taken under the previous 2GB / 12MB config. Also renames the log_message 'format' parameter, which shadowed the builtin (ruff A002).
Talos reserves nothing by default; every byte below was this repo's choice. On the 29GB nodes the reservations withheld 7.35Gi of 29.8Gi capacity, and both sat at 99-100% booked with a 150Mi exporter Pending while 12GB of real RAM sat idle. evictionHard 7% -> 1Gi: a percentage made the 64GB node reserve 4.39Gi against the small nodes' 2.04Gi, so the node with the most headroom set aside the most. Flat is still 10x the k8s default of 100Mi, against 8.7-10GB of reclaimable page cache per node. systemReserved 1Gi -> 512Mi: measured /system + /init peak is 391MB on control-3 and 203MB on control-2. kubeReserved stays at 2Gi and is now commented as a deliberate under- reservation: measured /podruntime is 2.89GB steady-state and peaked at 14.1GB. Raising it to the truth costs allocatable on nodes that have none. With the hugepages removal already in the stack: node alloc now alloc after gain booked now -> after control-1 54335Mi 60690Mi 6355Mi 83% -> 74% control-2 22271Mi 26213Mi 3942Mi 100% -> 85% control-3 22271Mi 26213Mi 3942Mi 100% -> 85% Applies on next reboot.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@talos/patches/global/machine-kubelet.yaml`:
- Around line 21-23: Update the memory reservation rationale near
kubeReserved.memory and systemReserved.memory to document complete observed
usage, including /podruntime, /system, /init, kernel memory, safety headroom,
and any TAP/Kubernetes reservation coverage; if the measurements are not
adequately covered, increase the affected reservation values instead of
accepting unexplained under-reservation. Apply this guidance at
talos/patches/global/machine-kubelet.yaml lines 21-23 and 31-32.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: d2913af8-d786-475b-97b6-42461faf4e5f
📒 Files selected for processing (1)
talos/patches/global/machine-kubelet.yaml
Comment on lines
+21
to
+23
| # Left at 2Gi despite being under-reserved: measured /podruntime is 2.89GB steady-state on | ||
| # control-2 and peaked at 14.1GB on control-3. Raising it to the truth costs allocatable on | ||
| # nodes that have none to spare, so the under-reservation is accepted deliberately. |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Repository files of interest:"
git ls-files | rg '(^|/)learned-workspace\.md$|talos/patches/global/machine-kubelet\.ya?ml$|task.*talos' || true
echo
echo "Relevant Talos patches:"
if [ -f talos/patches/global/machine-kubelet.yaml ]; then
cat -n talos/patches/global/machine-kubelet.yaml
fi
echo
echo "Learned workspace facts excerpt if present:"
if [ -f ../.agents/learned-workspace.md ]; then
rg -n -C 4 -i 'learned|talos|kubelet|reserved|memory|podruntime|system|init|control-[23]|kubeReserved|systemReserved' ../.agents/learned-workspace.md || true
else
echo "../.agents/learned-workspace.md not found"
fi
echo
echo "All occurrences of reserved memory config/comments:"
rg -n -C 2 'kubeReserved|systemReserved|podruntime|/podruntime|MemoryReserved|SystemReserved|memoryReserved|systemReserved|512Mi|2Gi' . || trueRepository: Tanguille/cluster
Length of output: 29378
🌐 Web query:
Talos Linux kubeReserved systemReserved kubernetes documentation
💡 Result:
In Talos Linux, you can configure kubeReserved and systemReserved by overriding the Kubelet configuration via the Talos machine configuration [1][2]. There are two primary methods to apply these settings, depending on whether you prefer to use Kubelet configuration files or command-line flags. Method 1: Using extraConfig The recommended approach for modern Kubelet configurations is using the machine.kubelet.extraConfig field in your Talos machine configuration [1]. This allows you to specify the KubeletConfiguration parameters directly as YAML [1][3]. Example: machine: kubelet: extraConfig: systemReserved: cpu: "500m" memory: "1Gi" ephemeral-storage: "2Gi" kubeReserved: cpu: "200m" memory: "512Mi" ephemeral-storage: "1Gi" enforceNodeAllocatable: - pods - system-reserved - kube-reserved Method 2: Using extraArgs Alternatively, you can pass these settings as command-line flags to the Kubelet using the machine.kubelet.extraArgs field [2][4]. This is useful if you need to maintain older configuration patterns or require specific flag-based enforcement [2]. Example: machine: kubelet: extraArgs: system-reserved: "cpu=500m,memory=512Mi" kube-reserved: "cpu=500m,memory=512Mi" enforce-node-allocatable: "pods,system-reserved,kube-reserved" Key Considerations 1. Purpose: systemReserved is intended to reserve resources for OS-level processes (like Talos services and containerd), while kubeReserved is for Kubernetes-specific components (like the kubelet and kube-proxy) [1][5][3]. 2. Enforcement: To ensure these limits are actually enforced at the OS level (via cgroups), you must include the respective categories in the enforceNodeAllocatable list [1][5]. 3. Tuning: These values should be based on the actual observed resource usage of your system processes on your specific nodes [1][3]. Setting them too low can lead to node instability, while setting them too high unnecessarily reduces the allocatable capacity available for your workloads [1][2][3]. 4. Cgroup Drivers: Talos defaults to using the systemd cgroup driver; ensure any cgroup paths you reference (such as system-reserved-cgroup) align with the systemd hierarchy if you choose to configure them explicitly [4].
Citations:
- 1: https://oneuptime.com/blog/post/2026-03-03-set-kubelet-extra-config-in-talos-linux/view
- 2: https://oneuptime.com/blog/post/2026-03-03-configure-kubelet-extra-args-in-talos-linux/view
- 3: https://oneuptime.com/blog/post/2026-03-03-tune-kubelet-performance-on-talos-linux/view
- 4: https://oneuptime.com/blog/post/2026-03-03-configure-cgroup-v2-settings-on-talos-linux/view
- 5: https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/
Reconcile the accepted under-reservation with complete memory measurements.
The reservation rationale is present, but it still leaves kubeReserved.memory below the reported /podruntime steady-state and peak usage, and systemReserved.memory only references /system + /init peaks without kernel memory or safety headroom. Document full observed usage and any TAP/Kubernetes reservation coverage before accepting under-reservation, or increase the affected reservation values.
📍 Affects 1 file
talos/patches/global/machine-kubelet.yaml#L21-L23(this comment)talos/patches/global/machine-kubelet.yaml#L31-L32
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@talos/patches/global/machine-kubelet.yaml` around lines 21 - 23, Update the
memory reservation rationale near kubeReserved.memory and systemReserved.memory
to document complete observed usage, including /podruntime, /system, /init,
kernel memory, safety headroom, and any TAP/Kubernetes reservation coverage; if
the measurements are not adequately covered, increase the affected reservation
values instead of accepting unexplained under-reservation. Apply this guidance
at talos/patches/global/machine-kubelet.yaml lines 21-23 and 31-32.
Tanguille
force-pushed
the
feat/web3-guard-hugepages
branch
from
August 8, 2026 13:13
1165e4f to
4549326
Compare
@@ data @@
# v1/ConfigMap/security/crowdsec-parsers-s02-enrich
! + one map entry added:
+ jellyfin-crawl-whitelist.yaml: |
+ name: hydaz/jellyfin-crawl-whitelist
+ description: "Jellyfin clients fan out to dozens of unique /HomeScreen/CachedImage URIs per page load, which reads as http-crawl-non_statics"
+ filter: "evt.Meta.service == 'http' && evt.Meta.log_type == 'http_access-log' && evt.Meta.target_fqdn == 'jellyfin...PLACEHOLDER_SECRET_DOMAIN..'"
+ whitelist:
+ reason: "successful jellyfin api traffic"
+ expression:
+ - "evt.Meta.http_status startsWith '2' or evt.Meta.http_status startsWith '3'"
@@ spec.template.spec.containers.crowdsec-agent.volumeMounts @@
# apps/v1/DaemonSet/security/crowdsec-agent
! + one list entry added:
+ - name: crowdsec-parsers-s02-enrich-jellyfin-crawl-whitelist
+ mountPath: /etc/crowdsec/parsers/s02-enrich/jellyfin-crawl-whitelist.yaml
+ subPath: jellyfin-crawl-whitelist.yaml
@@ spec.template.spec.volumes @@
# apps/v1/DaemonSet/security/crowdsec-agent
! + one list entry added:
+ - name: crowdsec-parsers-s02-enrich-jellyfin-crawl-whitelist
+ configMap:
+ name: crowdsec-parsers-s02-enrich
+ items:
+ - key: jellyfin-crawl-whitelist.yaml
+ path: jellyfin-crawl-whitelist.yaml
@@ spec.template.spec.containers.app.args @@
# apps/v1/Deployment/web3/xmrig-control-1
! - one list entry removed:
- - "--huge-pages"
! + one list entry added:
+ - "--no-huge-pages"
@@ spec.template.spec.containers.app.resources.limits @@
# apps/v1/Deployment/web3/xmrig-control-1
! - one map entry removed:
- hugepages-2Mi: 2368Mi
@@ spec.template.spec.containers.app.resources.limits.memory @@
# apps/v1/Deployment/web3/xmrig-control-1
! ± value change
- 128Mi
+ 3Gi
@@ spec.template.spec.containers.app.resources.requests @@
# apps/v1/Deployment/web3/xmrig-control-1
! - one map entry removed:
- hugepages-2Mi: 2368Mi
@@ spec.template.spec.containers.app.resources.requests.memory @@
# apps/v1/Deployment/web3/xmrig-control-1
! ± value change
- 32Mi
+ 2Gi
@@ spec.template.spec.containers.app.securityContext.capabilities @@
# apps/v1/Deployment/web3/xmrig-control-1
! - one map entry removed:
- add:
- - IPC_LOCK
@@ spec.template.spec.containers.app.volumeMounts @@
# apps/v1/Deployment/web3/xmrig-control-1
! - one list entry removed:
- - name: hugepages
- mountPath: /dev/hugepages
@@ spec.template.spec.volumes @@
# apps/v1/Deployment/web3/xmrig-control-1
! - one list entry removed:
- - name: hugepages
- emptyDir:
- medium: HugePages
@@ spec.template.spec.volumes.tmp.emptyDir @@
# apps/v1/Deployment/web3/xmrig-control-1
! + one map entry added:
+ sizeLimit: 64Mi
@@ spec.template.spec.containers.app.args @@
# apps/v1/Deployment/web3/xmrig-control-2
! - one list entry removed:
- - "--huge-pages"
! + one list entry added:
+ - "--no-huge-pages"
@@ spec.template.spec.containers.app.resources.limits @@
# apps/v1/Deployment/web3/xmrig-control-2
! - one map entry removed:
- hugepages-2Mi: 2368Mi
@@ spec.template.spec.containers.app.resources.limits.memory @@
# apps/v1/Deployment/web3/xmrig-control-2
! ± value change
- 128Mi
+ 3Gi
@@ spec.template.spec.containers.app.resources.requests @@
# apps/v1/Deployment/web3/xmrig-control-2
! - one map entry removed:
- hugepages-2Mi: 2368Mi
@@ spec.template.spec.containers.app.resources.requests.memory @@
# apps/v1/Deployment/web3/xmrig-control-2
! ± value change
- 32Mi
+ 2Gi
@@ spec.template.spec.containers.app.securityContext.capabilities @@
# apps/v1/Deployment/web3/xmrig-control-2
! - one map entry removed:
- add:
- - IPC_LOCK
@@ spec.template.spec.containers.app.volumeMounts @@
# apps/v1/Deployment/web3/xmrig-control-2
! - one list entry removed:
- - name: hugepages
- mountPath: /dev/hugepages
@@ spec.template.spec.volumes @@
# apps/v1/Deployment/web3/xmrig-control-2
! - one list entry removed:
- - name: hugepages
- emptyDir:
- medium: HugePages
@@ spec.template.spec.volumes.tmp.emptyDir @@
# apps/v1/Deployment/web3/xmrig-control-2
! + one map entry added:
+ sizeLimit: 64Mi
@@ spec.template.spec.containers.app.args @@
# apps/v1/Deployment/web3/xmrig-control-3
! - one list entry removed:
- - "--huge-pages"
! + one list entry added:
+ - "--no-huge-pages"
@@ spec.template.spec.containers.app.resources.limits @@
# apps/v1/Deployment/web3/xmrig-control-3
! - one map entry removed:
- hugepages-2Mi: 2368Mi
@@ spec.template.spec.containers.app.resources.limits.memory @@
# apps/v1/Deployment/web3/xmrig-control-3
! ± value change
- 128Mi
+ 3Gi
@@ spec.template.spec.containers.app.resources.requests @@
# apps/v1/Deployment/web3/xmrig-control-3
! - one map entry removed:
- hugepages-2Mi: 2368Mi
@@ spec.template.spec.containers.app.resources.requests.memory @@
# apps/v1/Deployment/web3/xmrig-control-3
! ± value change
- 32Mi
+ 2Gi
@@ spec.template.spec.containers.app.securityContext.capabilities @@
# apps/v1/Deployment/web3/xmrig-control-3
! - one map entry removed:
- add:
- - IPC_LOCK
@@ spec.template.spec.containers.app.volumeMounts @@
# apps/v1/Deployment/web3/xmrig-control-3
! - one list entry removed:
- - name: hugepages
- mountPath: /dev/hugepages
@@ spec.template.spec.volumes @@
# apps/v1/Deployment/web3/xmrig-control-3
! - one list entry removed:
- - name: hugepages
- emptyDir:
- medium: HugePages
@@ spec.template.spec.volumes.tmp.emptyDir @@
# apps/v1/Deployment/web3/xmrig-control-3
! + one map entry added:
+ sizeLimit: 64Mi
@@ (root level) @@
# v1/ServiceAccount/web3/xmrig-guard
! - one document removed:
- apiVersion: v1
- kind: ServiceAccount
- metadata:
- name: xmrig-guard
- namespace: web3
- labels:
- app.kubernetes.io/instance: xmrig-guard
- app.kubernetes.io/managed-by: Helm
- app.kubernetes.io/name: xmrig-guard
- helm.toolkit.fluxcd.io/name: xmrig-guard
- helm.toolkit.fluxcd.io/namespace: web3
@@ (root level) @@
# v1/Service/web3/xmrig-guard
! - one document removed:
- apiVersion: v1
- kind: Service
- metadata:
- name: xmrig-guard
- namespace: web3
- labels:
- app.kubernetes.io/instance: xmrig-guard
- app.kubernetes.io/managed-by: Helm
- app.kubernetes.io/name: xmrig-guard
- app.kubernetes.io/service: xmrig-guard
- helm.toolkit.fluxcd.io/name: xmrig-guard
- helm.toolkit.fluxcd.io/namespace: web3
- spec:
- type: ClusterIP
- selector:
- app.kubernetes.io/controller: xmrig-guard
- app.kubernetes.io/instance: xmrig-guard
- app.kubernetes.io/name: xmrig-guard
- ports:
- - name: http
- port: 8080
- protocol: TCP
- targetPort: 8080
@@ (root level) @@
# apps/v1/Deployment/web3/xmrig-guard
! - one document removed:
- apiVersion: apps/v1
- kind: Deployment
- metadata:
- name: xmrig-guard
- namespace: web3
- annotations:
- reloader.stakater.com/auto: "true"
- labels:
- app.kubernetes.io/controller: xmrig-guard
- app.kubernetes.io/instance: xmrig-guard
- app.kubernetes.io/managed-by: Helm
- app.kubernetes.io/name: xmrig-guard
- helm.toolkit.fluxcd.io/name: xmrig-guard
- helm.toolkit.fluxcd.io/namespace: web3
- spec:
- replicas: 1
- revisionHistoryLimit: 3
- selector:
- matchLabels:
- app.kubernetes.io/controller: xmrig-guard
- app.kubernetes.io/instance: xmrig-guard
- app.kubernetes.io/name: xmrig-guard
- strategy:
- type: Recreate
- template:
- metadata:
- labels:
- app.kubernetes.io/controller: xmrig-guard
- app.kubernetes.io/instance: xmrig-guard
- app.kubernetes.io/name: xmrig-guard
- spec:
- affinity:
- nodeAffinity:
- preferredDuringSchedulingIgnoredDuringExecution:
- - preference:
- matchExpressions:
- - key: kubernetes.io/hostname
- operator: In
- values:
- - control-1
- weight: 100
- automountServiceAccountToken: false
- containers:
- - name: app
- image: "python:3.14-slim@sha256:a7fb1e634c4a578f9e0bd6327f11a3cde11b7a9395f48e24360c0988bcc5c2bc"
- command:
- - python3
- - /config/controller.py
- livenessProbe:
- httpGet:
- path: /healthz
- port: http
- ports:
- - name: http
- containerPort: 8080
- readinessProbe:
- httpGet:
- path: /readyz
- port: http
- resources:
- limits:
- cpu: 100m
- memory: 128Mi
- requests:
- cpu: 10m
- memory: 32Mi
- volumeMounts:
- - name: config
- mountPath: /config
- readOnly: true
- - name: tmp
- mountPath: /tmp
- securityContext:
- allowPrivilegeEscalation: false
- capabilities:
- drop:
- - ALL
- readOnlyRootFilesystem: true
- dnsPolicy: ClusterFirst
- enableServiceLinks: false
- hostIPC: false
- hostNetwork: false
- hostPID: false
- securityContext:
- fsGroup: 65532
- fsGroupChangePolicy: OnRootMismatch
- runAsGroup: 65532
- runAsNonRoot: true
- runAsUser: 65532
- seccompProfile:
- type: RuntimeDefault
- serviceAccountName: xmrig-guard
- terminationGracePeriodSeconds: 15
- volumes:
- - name: config
- configMap:
- name: xmrig-guard
- - name: tmp
- emptyDir: {}
@@ (root level) @@
# v1/ServiceAccount/web3/guard
! + one document added:
+ apiVersion: v1
+ kind: ServiceAccount
+ metadata:
+ name: guard
+ namespace: web3
+ labels:
+ app.kubernetes.io/instance: guard
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/name: guard
+ helm.toolkit.fluxcd.io/name: guard
+ helm.toolkit.fluxcd.io/namespace: web3
@@ (root level) @@
# v1/Service/web3/guard
! + one document added:
+ apiVersion: v1
+ kind: Service
+ metadata:
+ name: guard
+ namespace: web3
+ labels:
+ app.kubernetes.io/instance: guard
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/name: guard
+ app.kubernetes.io/service: guard
+ helm.toolkit.fluxcd.io/name: guard
+ helm.toolkit.fluxcd.io/namespace: web3
+ spec:
+ type: ClusterIP
+ selector:
+ app.kubernetes.io/controller: guard
+ app.kubernetes.io/instance: guard
+ app.kubernetes.io/name: guard
+ ports:
+ - name: http
+ port: 8080
+ protocol: TCP
+ targetPort: 8080
@@ (root level) @@
# apps/v1/Deployment/web3/guard
! + one document added:
+ apiVersion: apps/v1
+ kind: Deployment
+ metadata:
+ name: guard
+ namespace: web3
+ annotations:
+ reloader.stakater.com/auto: "true"
+ labels:
+ app.kubernetes.io/controller: guard
+ app.kubernetes.io/instance: guard
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/name: guard
+ helm.toolkit.fluxcd.io/name: guard
+ helm.toolkit.fluxcd.io/namespace: web3
+ spec:
+ replicas: 1
+ revisionHistoryLimit: 3
+ selector:
+ matchLabels:
+ app.kubernetes.io/controller: guard
+ app.kubernetes.io/instance: guard
+ app.kubernetes.io/name: guard
+ strategy:
+ type: Recreate
+ template:
+ metadata:
+ labels:
+ app.kubernetes.io/controller: guard
+ app.kubernetes.io/instance: guard
+ app.kubernetes.io/name: guard
+ spec:
+ affinity:
+ nodeAffinity:
+ preferredDuringSchedulingIgnoredDuringExecution:
+ - preference:
+ matchExpressions:
+ - key: kubernetes.io/hostname
+ operator: In
+ values:
+ - control-1
+ weight: 100
+ automountServiceAccountToken: false
+ containers:
+ - name: app
+ image: "python:3.14-slim@sha256:a7fb1e634c4a578f9e0bd6327f11a3cde11b7a9395f48e24360c0988bcc5c2bc"
+ command:
+ - python3
+ - /config/controller.py
+ livenessProbe:
+ httpGet:
+ path: /healthz
+ port: http
+ ports:
+ - name: http
+ containerPort: 8080
+ readinessProbe:
+ httpGet:
+ path: /readyz
+ port: http
+ resources:
+ limits:
+ cpu: 100m
+ memory: 128Mi
+ requests:
+ cpu: 10m
+ memory: 32Mi
+ volumeMounts:
+ - name: config
+ mountPath: /config
+ readOnly: true
+ - name: tmp
+ mountPath: /tmp
+ securityContext:
+ allowPrivilegeEscalation: false
+ capabilities:
+ drop:
+ - ALL
+ readOnlyRootFilesystem: true
+ dnsPolicy: ClusterFirst
+ enableServiceLinks: false
+ hostIPC: false
+ hostNetwork: false
+ hostPID: false
+ securityContext:
+ fsGroup: 65532
+ fsGroupChangePolicy: OnRootMismatch
+ runAsGroup: 65532
+ runAsNonRoot: true
+ runAsUser: 65532
+ seccompProfile:
+ type: RuntimeDefault
+ serviceAccountName: guard
+ terminationGracePeriodSeconds: 15
+ volumes:
+ - name: config
+ configMap:
+ name: guard
+ - name: tmp
+ emptyDir: {}
|
@@ spec.values.config.parsers.s02-enrich @@
# helm.toolkit.fluxcd.io/v2/HelmRelease/security/crowdsec
! + one map entry added:
+ jellyfin-crawl-whitelist.yaml: |
+ name: hydaz/jellyfin-crawl-whitelist
+ description: "Jellyfin clients fan out to dozens of unique /HomeScreen/CachedImage URIs per page load, which reads as http-crawl-non_statics"
+ filter: "evt.Meta.service == 'http' && evt.Meta.log_type == 'http_access-log' && evt.Meta.target_fqdn == 'jellyfin...PLACEHOLDER_SECRET_DOMAIN..'"
+ whitelist:
+ reason: "successful jellyfin api traffic"
+ expression:
+ - "evt.Meta.http_status startsWith '2' or evt.Meta.http_status startsWith '3'"
@@ spec.resources.helm.toolkit.fluxcd.io/v2/HelmRelease/web3/xmrig-<< inputs.node >>.spec.values.controllers.xmrig-<< inputs.node >>.containers.app.args @@
# fluxcd.controlplane.io/v1/ResourceSet/web3/xmrig
! - one list entry removed:
- - "--huge-pages"
! + one list entry added:
+ - "--no-huge-pages"
@@ spec.resources.helm.toolkit.fluxcd.io/v2/HelmRelease/web3/xmrig-<< inputs.node >>.spec.values.controllers.xmrig-<< inputs.node >>.containers.app.resources.limits @@
# fluxcd.controlplane.io/v1/ResourceSet/web3/xmrig
! - one map entry removed:
- hugepages-2Mi: 2368Mi
@@ spec.resources.helm.toolkit.fluxcd.io/v2/HelmRelease/web3/xmrig-<< inputs.node >>.spec.values.controllers.xmrig-<< inputs.node >>.containers.app.resources.limits.memory @@
# fluxcd.controlplane.io/v1/ResourceSet/web3/xmrig
! ± value change
- 128Mi
+ 3Gi
@@ spec.resources.helm.toolkit.fluxcd.io/v2/HelmRelease/web3/xmrig-<< inputs.node >>.spec.values.controllers.xmrig-<< inputs.node >>.containers.app.resources.requests @@
# fluxcd.controlplane.io/v1/ResourceSet/web3/xmrig
! - one map entry removed:
- hugepages-2Mi: 2368Mi
@@ spec.resources.helm.toolkit.fluxcd.io/v2/HelmRelease/web3/xmrig-<< inputs.node >>.spec.values.controllers.xmrig-<< inputs.node >>.containers.app.resources.requests.memory @@
# fluxcd.controlplane.io/v1/ResourceSet/web3/xmrig
! ± value change
- 32Mi
+ 2Gi
@@ spec.resources.helm.toolkit.fluxcd.io/v2/HelmRelease/web3/xmrig-<< inputs.node >>.spec.values.controllers.xmrig-<< inputs.node >>.containers.app.securityContext.capabilities @@
# fluxcd.controlplane.io/v1/ResourceSet/web3/xmrig
! - one map entry removed:
- add:
- - IPC_LOCK
@@ spec.resources.helm.toolkit.fluxcd.io/v2/HelmRelease/web3/xmrig-<< inputs.node >>.spec.values.persistence @@
# fluxcd.controlplane.io/v1/ResourceSet/web3/xmrig
! - one map entry removed:
- hugepages:
- type: emptyDir
- medium: HugePages
- globalMounts:
- - path: /dev/hugepages
@@ spec.resources.helm.toolkit.fluxcd.io/v2/HelmRelease/web3/xmrig-<< inputs.node >>.spec.values.persistence.tmp @@
# fluxcd.controlplane.io/v1/ResourceSet/web3/xmrig
! + one map entry added:
+ sizeLimit: 64Mi
@@ (root level) @@
# kustomize.toolkit.fluxcd.io/v1/Kustomization/web3/xmrig-guard
! - one document removed:
- apiVersion: kustomize.toolkit.fluxcd.io/v1
- kind: Kustomization
- metadata:
- name: xmrig-guard
- namespace: web3
- labels:
- kustomize.toolkit.fluxcd.io/name: cluster-apps
- kustomize.toolkit.fluxcd.io/namespace: flux-system
- spec:
- decryption:
- provider: sops
- secretRef:
- name: sops-age
- deletionPolicy: WaitForTermination
- interval: 30m
- patches:
- - patch: |
- apiVersion: helm.toolkit.fluxcd.io/v2
- kind: HelmRelease
- metadata:
- name: _
- spec:
- install:
- crds: CreateReplace
- rollback:
- cleanupOnFail: true
- upgrade:
- cleanupOnFail: true
- crds: CreateReplace
- strategy:
- name: RemediateOnFailure
- remediation:
- remediateLastFailure: true
- retries: 2
- target:
- kind: HelmRelease
- group: helm.toolkit.fluxcd.io
- path: ./kubernetes/apps/web3/xmrig-guard/app
- postBuild:
- substituteFrom:
- - name: cluster-settings
- kind: ConfigMap
- optional: false
- - name: cluster-secrets
- kind: Secret
- optional: false
- prune: true
- retryInterval: 2m
- sourceRef:
- name: flux-system
- kind: GitRepository
- namespace: flux-system
- targetNamespace: web3
- timeout: 5m
- wait: true
@@ (root level) @@
# helm.toolkit.fluxcd.io/v2/HelmRelease/web3/xmrig-guard
! - one document removed:
- apiVersion: helm.toolkit.fluxcd.io/v2
- kind: HelmRelease
- metadata:
- name: xmrig-guard
- namespace: web3
- labels:
- kustomize.toolkit.fluxcd.io/name: xmrig-guard
- kustomize.toolkit.fluxcd.io/namespace: web3
- spec:
- chartRef:
- name: app-template
- kind: OCIRepository
- install:
- crds: CreateReplace
- interval: 1h
- rollback:
- cleanupOnFail: true
- upgrade:
- cleanupOnFail: true
- crds: CreateReplace
- remediation:
- remediateLastFailure: true
- retries: 2
- strategy:
- name: RemediateOnFailure
- values:
- controllers:
- xmrig-guard:
- annotations:
- reloader.stakater.com/auto: "true"
- containers:
- app:
- resources:
- limits:
- cpu: 100m
- memory: 128Mi
- requests:
- cpu: 10m
- memory: 32Mi
- command:
- - python3
- - /config/controller.py
- image:
- repository: python
- tag: "3.14-slim@sha256:a7fb1e634c4a578f9e0bd6327f11a3cde11b7a9395f48e24360c0988bcc5c2bc"
- ports:
- - name: http
- containerPort: 8080
- securityContext:
- allowPrivilegeEscalation: false
- capabilities:
- drop:
- - ALL
- readOnlyRootFilesystem: true
- probes:
- liveness:
- spec:
- httpGet:
- path: /healthz
- port: http
- custom: true
- enabled: true
- readiness:
- spec:
- httpGet:
- path: /readyz
- port: http
- custom: true
- enabled: true
- defaultPodOptions:
- affinity:
- nodeAffinity:
- preferredDuringSchedulingIgnoredDuringExecution:
- - preference:
- matchExpressions:
- - key: kubernetes.io/hostname
- operator: In
- values:
- - control-1
- weight: 100
- automountServiceAccountToken: false
- securityContext:
- fsGroup: 65532
- fsGroupChangePolicy: OnRootMismatch
- runAsGroup: 65532
- runAsNonRoot: true
- runAsUser: 65532
- seccompProfile:
- type: RuntimeDefault
- terminationGracePeriodSeconds: 15
- persistence:
- config:
- name: xmrig-guard
- type: configMap
- globalMounts:
- - path: /config
- readOnly: true
- tmp:
- type: emptyDir
- globalMounts:
- - path: /tmp
- service:
- app:
- controller: xmrig-guard
- ports:
- http:
- port: 8080
@@ (root level) @@
# monitoring.coreos.com/v1/PrometheusRule/web3/xmrig-guard-rules
! - one document removed:
- apiVersion: monitoring.coreos.com/v1
- kind: PrometheusRule
- metadata:
- name: xmrig-guard-rules
- namespace: web3
- labels:
- kustomize.toolkit.fluxcd.io/name: xmrig-guard
- kustomize.toolkit.fluxcd.io/namespace: web3
- spec:
- groups:
- - name: xmrig-guard.rules
- rules:
- - alert: XmrigGuardEnforcementBypassed
- annotations:
- summary: "{{ $labels.deployment }} is running while its node's thermal gate should be closed — the KEDA trip is not being enforced (operator down, HPA wedged, or manual scale)"
- expr: |
- kube_deployment_status_replicas{namespace="web3", deployment=~"xmrig-control-.+"} > 0
- unless on (deployment)
- label_replace(
- (xmrig_guard_safe == 1)
- and on (node) (count by (node) (xmrig_guard_safe) == 1)
- and on (node) (timestamp(xmrig_guard_safe) >= time() - 120),
- "deployment", "xmrig-$1", "node", "(.*)"
- )
- for: 10m
- labels:
- severity: critical
- - alert: XmrigGuardAbsent
- annotations:
- summary: "Guard safety signal missing or wrong cardinality — the KEDA gate is closed and mining is silently disabled (fails closed)"
- expr: |
- count(xmrig_guard_safe{node=~"control-[123]"}) != 3
- or absent(xmrig_guard_safe{node=~"control-[123]"})
- for: 15m
- labels:
- severity: warning
- - alert: XmrigGuardLatchedUnsafe
- annotations:
- summary: "{{ $labels.node }} has been gated unsafe for 6h while its NVMe stayed within the 62C recovery band, so the guard is latched rather than thermally tripped and mining is disabled with no signal"
- expr: |
- max_over_time(xmrig_guard_safe[6h]) == 0
- and on (node)
- max_over_time(xmrig_guard_nvme_temp_max_celsius[2h]) <= 62
- for: 30m
- labels:
- severity: warning
- - alert: XmrigGuardThermalPanic
- annotations:
- summary: "{{ $labels.node }} is above the 68C panic trip with a miner still running — the zero-dwell fast path did not shed it and the drive is near its 70C rating"
- expr: |
- max by (node) (xmrig_guard_nvme_temp_max_celsius) > 68
- and on (node)
- label_replace(
- kube_deployment_status_replicas{namespace="web3", deployment=~"xmrig-control-.+"} > 0,
- "node", "$1", "deployment", "xmrig-(.*)"
- )
- for: 2m
- labels:
- severity: critical
- - alert: XmrigGuardEvaluationErrors
- annotations:
- summary: "Guard evaluations are failing repeatedly — affected nodes fail closed and can never re-earn safe=1 until queries recover"
- expr: "sum(rate(xmrig_guard_query_errors_total[10m])) > 0.01"
- for: 15m
- labels:
- severity: warning
@@ (root level) @@
# monitoring.coreos.com/v1/ServiceMonitor/web3/xmrig-guard
! - one document removed:
- apiVersion: monitoring.coreos.com/v1
- kind: ServiceMonitor
- metadata:
- name: xmrig-guard
- namespace: web3
- labels:
- kustomize.toolkit.fluxcd.io/name: xmrig-guard
- kustomize.toolkit.fluxcd.io/namespace: web3
- spec:
- endpoints:
- - port: http
- selector:
- matchLabels:
- app.kubernetes.io/instance: xmrig-guard
- app.kubernetes.io/name: xmrig-guard
@@ (root level) @@
# v1/ConfigMap/web3/xmrig-guard
! - one document removed:
- apiVersion: v1
- kind: ConfigMap
- metadata:
- name: xmrig-guard
- namespace: web3
- annotations:
- kustomize.toolkit.fluxcd.io/substitute: disabled
- labels:
- kustomize.toolkit.fluxcd.io/name: xmrig-guard
- kustomize.toolkit.fluxcd.io/namespace: web3
- data:
- controller.py: |
- """Small, dependency-free XMRig safety signal controller.
-
- The controller deliberately treats telemetry as untrusted input. A complete
- set of fresh samples is required before a node can become safe. Policy is
- code: changing thresholds, sensors, or timing requires a reviewed diff here.
- """
- import json
- import logging
- import math
- import threading
- import time
- import urllib.error
- import urllib.parse
- import urllib.request
- from dataclasses import dataclass
- from datetime import datetime, timezone
- from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
-
- # temp1 is the Composite sensor, which is what the drives' 70C rating specifies and
- # what smartctl reports. temp2-temp4 are internal die sensors that run ~9C hotter and
- # carry no comparable rating, so including them gated a Composite threshold against
- # the wrong reading. Fewer series also means fewer chances for a single missing sample
- # to fail the identity check and latch a node closed.
- SENSORS = {
- # empty tuple = no NVMe visible to this node (control-1 is a VM); it is gated on CPU
- # headroom instead, which is what every `if sensors` branch below keys off.
- "control-1": (),
- "control-2": (("nvme_nvme0", "temp1"), ("nvme_nvme1", "temp1")),
- "control-3": (("nvme_nvme0", "temp1"), ("nvme_nvme1", "temp1")),
- }
- # Miner slot priority, best first. control-1 has no NVMe to cook and was safe 96% of the
- # last 7d against 60%/30% for the bare-metal nodes. Each ScaledObject counts how many safe
- # nodes outrank it and subtracts one miner's draw per rank, so this order decides who gets
- # scarce watts. It lives here rather than in the manifests because it is a property of the
- # node telemetry, and a future headroom-derived ranking replaces this tuple alone.
- PRIORITY = ("control-1", "control-2", "control-3")
- ENDPOINT = "http://vmauth-victoria-metrics.observability.svc.cluster.local:8427"
- EVALUATION_INTERVAL_SECONDS = 30
- SOURCE_SAMPLE_MAX_AGE_SECONDS = 120
- MAX_SOURCE_GAP_SECONDS = 120
- # 7 serial queries per CPU evaluation: at 10s a hung VictoriaMetrics blocked 70s against a 30s interval
- HTTP_TIMEOUT_SECONDS = 5
-
-
- def _dt(value):
- value = float(value)
- if not math.isfinite(value):
- raise ValueError("non-finite timestamp")
- return datetime.fromtimestamp(value, timezone.utc)
-
-
- def _fresh(timestamp, evaluation, max_age):
- """Return whether a source timestamp is not future-dated or too old."""
- age = (evaluation - timestamp).total_seconds()
- return 0 <= age <= max_age
-
-
- class DwellPolicy:
- def __init__(self, recovery_limit, trip_limit, recovery_dwell, trip_dwell, max_gap_seconds=120, panic_limit=None):
- self.recovery_limit, self.trip_limit = recovery_limit, trip_limit
- self.recovery_dwell, self.trip_dwell = recovery_dwell, trip_dwell
- self.panic_limit = panic_limit
- self.max_gap = float(max_gap_seconds)
- self.safe = False
- self._last_source = None
- self._pending = None
- self._since = None
-
- def invalidate(self):
- self.safe = False
- self._last_source = self._pending = self._since = None
-
- def observe(self, value, source_time, monotonic_now):
- if not isinstance(value, (int, float)) or not math.isfinite(value) or not isinstance(source_time, datetime):
- self.invalidate()
- return False
- source_seconds = source_time.timestamp()
- if self._last_source is not None:
- gap = source_seconds - self._last_source
- if gap <= 0:
- return self.safe # duplicate/out-of-order samples cannot advance dwell
- if gap > self.max_gap:
- self._pending = self._since = None
- self._last_source = source_seconds
- # third, dwell-free state: 2C from the rating, confirming the reading over a 60s dwell
- # would spend most of the margin. Clearing _pending forces a full recovery dwell after.
- if self.panic_limit is not None and value >= self.panic_limit:
- self.safe = False
- self._pending = self._since = None
- return False
- kind = "recover" if value <= self.recovery_limit else "trip" if value >= self.trip_limit else None
- if kind is None:
- self._pending = self._since = None
- return self.safe
- if (kind == "recover") == self.safe:
- self._pending = self._since = None
- return self.safe
- if kind != self._pending:
- self._pending, self._since = kind, monotonic_now
- else:
- dwell = self.recovery_dwell if kind == "recover" else self.trip_dwell
- if monotonic_now - self._since >= dwell:
- self.safe = kind == "recover"
- self._pending = self._since = None
- return self.safe
-
-
- @dataclass(frozen=True)
- class Source:
- value: float
- timestamp: datetime
-
-
- @dataclass(frozen=True)
- class CPUObservation:
- host: Source
- xmrig: Source | None
- presence: Source
-
-
- def cpu_value(observation):
- if not isinstance(observation.host, Source):
- raise ValueError("host source is required")
- xmrig = observation.xmrig.value if observation.xmrig else 0.0
- return max(0.0, min(100.0, observation.host.value - xmrig))
-
-
- class VictoriaMetricsClient:
- def __init__(self, endpoint, transport=None, timeout=HTTP_TIMEOUT_SECONDS, step_seconds=120):
- self.endpoint = endpoint.rstrip("/")
- self.transport = transport or _HTTPTransport(timeout)
- self.step = f"{int(step_seconds)}s"
-
- def _query(self, expression, evaluation):
- # explicit step: default 5m step makes timestamp(a or b) snap to 5-min boundaries,
- # which made 60% of freshness checks fail; step=max_age also bounds VM lookbehind
- params = {"query": expression, "time": evaluation.astimezone(timezone.utc).isoformat().replace("+00:00", "Z"), "step": self.step}
- try:
- payload = self.transport.get(self.endpoint + "/api/v1/query", params)
- except urllib.error.HTTPError as exc:
- # a rejected query is undiagnosable without its text: 40h of bare 422s went unseen
- raise ValueError(f"VictoriaMetrics rejected ({exc.code}): {expression}") from exc
- if payload.get("status") != "success" or payload.get("data", {}).get("resultType") != "vector" or not isinstance(payload["data"].get("result"), list):
- raise ValueError(f"invalid VictoriaMetrics response for: {expression}")
- return payload["data"]["result"]
-
- @staticmethod
- def _sources(rows, identity=None, raw_timestamp=False):
- out = {}
- for row in rows:
- metric = row.get("metric", {})
- key = identity(metric) if identity else tuple(sorted(metric.items()))
- if key in out or not isinstance(row.get("value"), list) or len(row["value"]) != 2:
- raise ValueError("malformed or duplicate telemetry")
- try:
- value = float(row["value"][1])
- timestamp = _dt(row["value"][1] if raw_timestamp else row["value"][0])
- except (TypeError, ValueError, OverflowError) as exc:
- raise ValueError("malformed telemetry value") from exc
- if not math.isfinite(value):
- raise ValueError("non-finite telemetry value")
- out[key] = Source(value, timestamp)
- return out
-
- def query_nvme(self, node, sensors, evaluation):
- parts = [f'node_hwmon_temp_celsius{{kubernetes_node="{node}",chip="{chip}",sensor="{sensor}"}}' for chip, sensor in sensors]
- expression = " or ".join(parts)
- rows = self._query(expression, evaluation)
- # timestamp() over an or-expression becomes a step-aligned subquery in VictoriaMetrics
- # (fake boundary stamps broke 60% of freshness checks); single selectors return raw stamps
- timestamps = [row for part in parts for row in self._query("timestamp(" + part + ")", evaluation)]
- key = lambda m: (m.get("chip"), m.get("sensor"))
- if any(row.get("metric", {}).get("kubernetes_node") != node for row in rows + timestamps):
- raise ValueError("NVMe node identity changed")
- found = self._sources(rows, key)
- stamped = self._sources(timestamps, key, raw_timestamp=True)
- if set(found) != set(sensors):
- raise ValueError("incomplete or changed NVMe identity set")
- if set(stamped) != set(sensors):
- raise ValueError("incomplete or changed NVMe timestamp identity set")
- result = []
- for item in sensors:
- if not -40 <= found[item].value <= 150:
- raise ValueError("implausible NVMe temperature")
- result.append(Source(found[item].value, stamped[item].timestamp))
- return result
-
- def query_cpu(self, node, evaluation, window="5m"):
- host_raw = f'node_cpu_seconds_total{{kubernetes_node="{node}",mode!="idle"}}'
- idle_raw = f'node_cpu_seconds_total{{kubernetes_node="{node}",mode="idle"}}'
- cadvisor_raw = f'container_cpu_usage_seconds_total{{node="{node}",namespace="web3",container!="",container!="POD"}}'
- host_query = f'sum(rate({host_raw}[{window}])) / count(count({idle_raw}) by (cpu)) * 100'
- ksm_raw = 'kube_pod_info{namespace="web3"}'
- label_selector = 'namespace="web3",label_app_kubernetes_io_component="thermal-guarded"'
- node_pods_raw = f'kube_pod_info{{namespace="web3",node="{node}"}}'
- xmrig_presence_raw = f'kube_pod_labels{{{label_selector}}} * on(namespace,pod) group_left(node) {node_pods_raw}'
- xmrig_presence_query = f'count({xmrig_presence_raw}) or vector(0)'
- xmrig_query = f'100 * (sum((sum by (namespace,pod) (rate({cadvisor_raw}[{window}])) * on(namespace,pod) group_left(node) ({xmrig_presence_raw})) or vector(0)) / count(count({idle_raw}) by (cpu)))'
- labels_raw = f'kube_pod_labels{{{label_selector}}}'
- def one(query):
- values = self._sources(self._query(query, evaluation))
- if len(values) != 1:
- raise ValueError(f"CPU source must be one scalar: {query}")
- return next(iter(values.values()))
- def oldest(*selectors):
- out = []
- for selector in selectors:
- rows = self._query("timestamp(" + selector + ")", evaluation)
- if not rows:
- raise ValueError(f"missing raw timestamps: {selector}")
- out.extend(self._sources(rows, raw_timestamp=True).values())
- return min(item.timestamp for item in out)
- presence = one(xmrig_presence_query)
- if presence.value < 0:
- raise ValueError("invalid labelled XMRig presence source")
- if presence.value > 0:
- # the membership stamps date both the presence count and the subtraction, so
- # they are fetched once and reused rather than queried twice per evaluation
- membership = oldest(labels_raw, node_pods_raw)
- presence = Source(presence.value, membership)
- # cadvisor joins the same membership selectors, so cadvisor freshness is
- # verified exactly when its data enters the subtraction
- xmrig = Source(one(xmrig_query).value, min(membership, oldest(cadvisor_raw)))
- else:
- # no labelled miner on this node: anchor presence freshness to pod-info stamps
- presence = Source(0, oldest(ksm_raw))
- xmrig = None
- host = Source(one(host_query).value, oldest(host_raw))
- return CPUObservation(host, xmrig, presence)
-
-
- class _HTTPTransport:
- def __init__(self, timeout=HTTP_TIMEOUT_SECONDS):
- self.timeout = timeout
-
- def get(self, url, params):
- request = urllib.request.Request(url + "?" + urllib.parse.urlencode(params))
- with urllib.request.urlopen(request, timeout=self.timeout) as response:
- return json.load(response)
-
-
- class GuardController:
- def __init__(self, telemetry, clock=time.monotonic, wall_clock=lambda: datetime.now(timezone.utc)):
- self.telemetry = telemetry
- self.clock, self.wall_clock = clock, wall_clock
- # Trip 65C / recover 62C on Composite, against a 70C drive rating. Mining raises
- # Composite at up to 1.1C/min. Trip-to-drain is 135s (30s evaluation + 60s dwell + 30s
- # KEDA poll + 15s drain), so 2.5C of rise to a ~67.5C peak; test_trip_to_drain_budget
- # pins it. The 68C panic trip is dwell-free and peaks near 69.4C.
- # The drives idle in the 60-64C band on warm afternoons, so recovery sits at 62C:
- # replaying 7d puts control-2/3 at 55.0%/38.8% safe against 44.7%/31.5% at 60C/600s.
- # control-1 is keyed the same way on CPU headroom rather than temperature; SENSORS
- # decides which source feeds which node, so the policy dict needs no special case. No
- # panic limit there: a busy CPU has no equivalent of a drive's absolute rating.
- self.policies = {
- "control-1": DwellPolicy(50, 70, 600, 120, MAX_SOURCE_GAP_SECONDS),
- "control-2": DwellPolicy(62, 65, 300, 60, MAX_SOURCE_GAP_SECONDS, panic_limit=68),
- "control-3": DwellPolicy(62, 65, 300, 60, MAX_SOURCE_GAP_SECONDS, panic_limit=68),
- }
- self.ready = False
- self.metrics = {
- "evaluations": 0, "query_errors": {node: 0 for node in SENSORS},
- "safe": {node: 0 for node in SENSORS},
- "nvme_temp_max": {node: 0.0 for node in SENSORS if SENSORS[node]},
- "source_age_seconds": {node: 0.0 for node in SENSORS},
- "cpu_non_xmrig": {node: 0.0 for node in SENSORS if not SENSORS[node]},
- "rank": {node: PRIORITY.index(node) for node in SENSORS},
- }
- self._last_source_stamps = {node: {} for node in SENSORS}
-
- def _new_source_set(self, node, sources):
- # Keyed by source identity, not position: a miner starting or stopping adds or removes
- # the xmrig source, and comparing by position made that look like tampering. It failed
- # control-1 closed 19 times in 12h, draining the miner, which changed the set back.
- # Only keys in both sets are comparable; a genuinely new key is exempt for one cycle
- # rather than the whole evaluation going unchecked.
- stamps = {key: source.timestamp.timestamp() for key, source in sources.items()}
- previous = self._last_source_stamps[node]
- shared = stamps.keys() & previous.keys()
- if any(stamps[key] - previous[key] > MAX_SOURCE_GAP_SECONDS for key in shared):
- raise ValueError("source gap exceeded maximum")
- if shared and any(stamps[key] <= previous[key] for key in shared):
- return False
- self._last_source_stamps[node] = stamps
- return True
-
- def evaluate(self, evaluation=None):
- evaluation = evaluation or self.wall_clock()
- now = self.clock()
- for node, sensors in SENSORS.items():
- try:
- # no sensors means a node with no visible NVMe (control-1, a VM): it is gated
- # on CPU headroom instead. The dwell policy and metrics are keyed identically.
- if sensors:
- samples = dict(zip(sensors, self.telemetry.query_nvme(node, sensors, evaluation)))
- # trip on the hottest drive, date it by the newest sample it was read from
- value = max(item.value for item in samples.values())
- stamp = max(item.timestamp for item in samples.values())
- self.metrics["nvme_temp_max"][node] = value
- else:
- obs = self.telemetry.query_cpu(node, evaluation)
- samples = {"host": obs.host, "presence": obs.presence}
- if obs.xmrig:
- samples["xmrig"] = obs.xmrig
- value = cpu_value(obs)
- stamp = min(item.timestamp for item in samples.values())
- self.metrics["cpu_non_xmrig"][node] = value
- if not all(_fresh(item.timestamp, evaluation, SOURCE_SAMPLE_MAX_AGE_SECONDS) for item in samples.values()):
- raise ValueError("stale or future source")
- policy = self.policies[node]
- safe = policy.observe(value, stamp, now) if self._new_source_set(node, samples) else policy.safe
- self.metrics["source_age_seconds"][node] = max(0.0, evaluation.timestamp() - min(item.timestamp for item in samples.values()).timestamp())
- self.metrics["safe"][node] = int(safe)
- except Exception as exc:
- # one line per failure, no traceback: the query text travels in the exception
- logging.error(f"evaluation failed for {node}: {exc!r}")
- self.metrics["query_errors"][node] += 1
- self.policies[node].invalidate()
- self._last_source_stamps[node] = {}
- self.metrics["safe"][node] = 0
- self.metrics["source_age_seconds"][node] = float("nan")
- if sensors:
- self.metrics["nvme_temp_max"][node] = float("nan")
- else:
- self.metrics["cpu_non_xmrig"][node] = float("nan")
- self.metrics["evaluations"] += 1
- self.ready = True
- return dict(self.metrics["safe"])
-
-
- def render_metrics(controller):
- m = controller.metrics
- lines = [f'xmrig_guard_evaluations_total {m["evaluations"]}']
- for metric, values in (
- ("safe", m["safe"]),
- ("query_errors_total", m["query_errors"]),
- ("source_age_seconds", m["source_age_seconds"]),
- ("nvme_temp_max_celsius", m["nvme_temp_max"]),
- ("cpu_non_xmrig_percent", m["cpu_non_xmrig"]),
- ("rank", m["rank"]),
- ):
- metric_name = "xmrig_guard_" + metric
- lines.extend(f'{metric_name}{{node="{node}"}} {value}' for node, value in values.items())
- return "\n".join(lines) + "\n"
-
-
- class _StatusHandler(BaseHTTPRequestHandler):
- controller = None # assigned before the server starts
- def do_GET(self):
- if self.path == "/healthz":
- self._send(200, "ok\n", "text/plain")
- elif self.path == "/readyz":
- self._send(200 if self.controller.ready else 503, "ready\n" if self.controller.ready else "not ready\n", "text/plain")
- elif self.path == "/metrics":
- body = render_metrics(self.controller)
- self._send(200, body, "text/plain; version=0.0.4")
- else:
- self._send(404, "not found\n", "text/plain")
- def _send(self, status, body, content_type):
- data = body.encode()
- self.send_response(status); self.send_header("Content-Type", content_type); self.send_header("Content-Length", str(len(data))); self.end_headers(); self.wfile.write(data)
- def log_message(self, format, *args):
- return
-
-
- def main():
- controller = GuardController(VictoriaMetricsClient(ENDPOINT, step_seconds=SOURCE_SAMPLE_MAX_AGE_SECONDS))
- _StatusHandler.controller = controller
- server = ThreadingHTTPServer(("0.0.0.0", 8080), _StatusHandler)
- threading.Thread(target=server.serve_forever, daemon=True).start()
- while True:
- # sleep to a deadline, not a flat interval: sleeping after the work made the true
- # period drift by the evaluation duration, stretching it against a fixed freshness budget
- deadline = time.monotonic() + EVALUATION_INTERVAL_SECONDS
- controller.evaluate()
- time.sleep(max(0, deadline - time.monotonic()))
-
-
- if __name__ == "__main__":
- main()
@@ (root level) @@
# kustomize.toolkit.fluxcd.io/v1/Kustomization/web3/guard
! + one document added:
+ apiVersion: kustomize.toolkit.fluxcd.io/v1
+ kind: Kustomization
+ metadata:
+ name: guard
+ namespace: web3
+ labels:
+ kustomize.toolkit.fluxcd.io/name: cluster-apps
+ kustomize.toolkit.fluxcd.io/namespace: flux-system
+ spec:
+ decryption:
+ provider: sops
+ secretRef:
+ name: sops-age
+ deletionPolicy: WaitForTermination
+ interval: 30m
+ patches:
+ - patch: |
+ apiVersion: helm.toolkit.fluxcd.io/v2
+ kind: HelmRelease
+ metadata:
+ name: _
+ spec:
+ install:
+ crds: CreateReplace
+ rollback:
+ cleanupOnFail: true
+ upgrade:
+ cleanupOnFail: true
+ crds: CreateReplace
+ strategy:
+ name: RemediateOnFailure
+ remediation:
+ remediateLastFailure: true
+ retries: 2
+ target:
+ kind: HelmRelease
+ group: helm.toolkit.fluxcd.io
+ path: ./kubernetes/apps/web3/monero/guard
+ postBuild:
+ substituteFrom:
+ - name: cluster-settings
+ kind: ConfigMap
+ optional: false
+ - name: cluster-secrets
+ kind: Secret
+ optional: false
+ prune: true
+ retryInterval: 2m
+ sourceRef:
+ name: flux-system
+ kind: GitRepository
+ namespace: flux-system
+ targetNamespace: web3
+ timeout: 5m
+ wait: true
@@ (root level) @@
# helm.toolkit.fluxcd.io/v2/HelmRelease/web3/guard
! + one document added:
+ apiVersion: helm.toolkit.fluxcd.io/v2
+ kind: HelmRelease
+ metadata:
+ name: guard
+ namespace: web3
+ labels:
+ kustomize.toolkit.fluxcd.io/name: guard
+ kustomize.toolkit.fluxcd.io/namespace: web3
+ spec:
+ chartRef:
+ name: app-template
+ kind: OCIRepository
+ install:
+ crds: CreateReplace
+ interval: 1h
+ rollback:
+ cleanupOnFail: true
+ upgrade:
+ cleanupOnFail: true
+ crds: CreateReplace
+ remediation:
+ remediateLastFailure: true
+ retries: 2
+ strategy:
+ name: RemediateOnFailure
+ values:
+ controllers:
+ guard:
+ annotations:
+ reloader.stakater.com/auto: "true"
+ containers:
+ app:
+ resources:
+ limits:
+ cpu: 100m
+ memory: 128Mi
+ requests:
+ cpu: 10m
+ memory: 32Mi
+ command:
+ - python3
+ - /config/controller.py
+ image:
+ repository: python
+ tag: "3.14-slim@sha256:a7fb1e634c4a578f9e0bd6327f11a3cde11b7a9395f48e24360c0988bcc5c2bc"
+ ports:
+ - name: http
+ containerPort: 8080
+ securityContext:
+ allowPrivilegeEscalation: false
+ capabilities:
+ drop:
+ - ALL
+ readOnlyRootFilesystem: true
+ probes:
+ liveness:
+ spec:
+ httpGet:
+ path: /healthz
+ port: http
+ custom: true
+ enabled: true
+ readiness:
+ spec:
+ httpGet:
+ path: /readyz
+ port: http
+ custom: true
+ enabled: true
+ defaultPodOptions:
+ affinity:
+ nodeAffinity:
+ preferredDuringSchedulingIgnoredDuringExecution:
+ - preference:
+ matchExpressions:
+ - key: kubernetes.io/hostname
+ operator: In
+ values:
+ - control-1
+ weight: 100
+ automountServiceAccountToken: false
+ securityContext:
+ fsGroup: 65532
+ fsGroupChangePolicy: OnRootMismatch
+ runAsGroup: 65532
+ runAsNonRoot: true
+ runAsUser: 65532
+ seccompProfile:
+ type: RuntimeDefault
+ terminationGracePeriodSeconds: 15
+ persistence:
+ config:
+ name: guard
+ type: configMap
+ globalMounts:
+ - path: /config
+ readOnly: true
+ tmp:
+ type: emptyDir
+ globalMounts:
+ - path: /tmp
+ service:
+ app:
+ controller: guard
+ ports:
+ http:
+ port: 8080
@@ (root level) @@
# monitoring.coreos.com/v1/PrometheusRule/web3/guard-rules
! + one document added:
+ apiVersion: monitoring.coreos.com/v1
+ kind: PrometheusRule
+ metadata:
+ name: guard-rules
+ namespace: web3
+ labels:
+ kustomize.toolkit.fluxcd.io/name: guard
+ kustomize.toolkit.fluxcd.io/namespace: web3
+ spec:
+ groups:
+ - name: guard.rules
+ rules:
+ - alert: XmrigGuardEnforcementBypassed
+ annotations:
+ summary: "{{ $labels.deployment }} is running while its node's thermal gate should be closed — the KEDA trip is not being enforced (operator down, HPA wedged, or manual scale)"
+ expr: |
+ kube_deployment_status_replicas{namespace="web3", deployment=~"xmrig-control-.+"} > 0
+ unless on (deployment)
+ label_replace(
+ (xmrig_guard_safe == 1)
+ and on (node) (count by (node) (xmrig_guard_safe) == 1)
+ and on (node) (timestamp(xmrig_guard_safe) >= time() - 120),
+ "deployment", "xmrig-$1", "node", "(.*)"
+ )
+ for: 10m
+ labels:
+ severity: critical
+ - alert: XmrigGuardAbsent
+ annotations:
+ summary: "Guard safety signal missing or wrong cardinality — the KEDA gate is closed and mining is silently disabled (fails closed)"
+ expr: |
+ count(xmrig_guard_safe{node=~"control-[123]"}) != 3
+ or absent(xmrig_guard_safe{node=~"control-[123]"})
+ for: 15m
+ labels:
+ severity: warning
+ - alert: XmrigGuardLatchedUnsafe
+ annotations:
+ summary: "{{ $labels.node }} has been gated unsafe for 6h while its NVMe stayed within the 62C recovery band, so the guard is latched rather than thermally tripped and mining is disabled with no signal"
+ expr: |
+ max_over_time(xmrig_guard_safe[6h]) == 0
+ and on (node)
+ max_over_time(xmrig_guard_nvme_temp_max_celsius[2h]) <= 62
+ for: 30m
+ labels:
+ severity: warning
+ - alert: XmrigGuardThermalPanic
+ annotations:
+ summary: "{{ $labels.node }} is above the 67C panic trip with a miner still running — the zero-dwell fast path did not shed it and the drive is near its 70C rating"
+ expr: |
+ max by (node) (xmrig_guard_nvme_temp_max_celsius) > 67
+ and on (node)
+ label_replace(
+ kube_deployment_status_replicas{namespace="web3", deployment=~"xmrig-control-.+"} > 0,
+ "node", "$1", "deployment", "xmrig-(.*)"
+ )
+ for: 2m
+ labels:
+ severity: critical
+ - alert: XmrigGuardEvaluationErrors
+ annotations:
+ summary: "Guard evaluations are failing repeatedly — affected nodes fail closed and can never re-earn safe=1 until queries recover"
+ expr: "sum(rate(xmrig_guard_query_errors_total[10m])) > 0.01"
+ for: 15m
+ labels:
+ severity: warning
@@ (root level) @@
# monitoring.coreos.com/v1/ServiceMonitor/web3/guard
! + one document added:
+ apiVersion: monitoring.coreos.com/v1
+ kind: ServiceMonitor
+ metadata:
+ name: guard
+ namespace: web3
+ labels:
+ kustomize.toolkit.fluxcd.io/name: guard
+ kustomize.toolkit.fluxcd.io/namespace: web3
+ spec:
+ endpoints:
+ - port: http
+ selector:
+ matchLabels:
+ app.kubernetes.io/instance: guard
+ app.kubernetes.io/name: guard
@@ (root level) @@
# v1/ConfigMap/web3/guard
! + one document added:
+ apiVersion: v1
+ kind: ConfigMap
+ metadata:
+ name: guard
+ namespace: web3
+ annotations:
+ kustomize.toolkit.fluxcd.io/substitute: disabled
+ labels:
+ kustomize.toolkit.fluxcd.io/name: guard
+ kustomize.toolkit.fluxcd.io/namespace: web3
+ data:
+ controller.py: |
+ """Small, dependency-free XMRig safety signal controller.
+
+ The controller deliberately treats telemetry as untrusted input. A complete
+ set of fresh samples is required before a node can become safe. Policy is
+ code: changing thresholds, sensors, or timing requires a reviewed diff here.
+ """
+ import json
+ import logging
+ import math
+ import threading
+ import time
+ import urllib.error
+ import urllib.parse
+ import urllib.request
+ from dataclasses import dataclass
+ from datetime import datetime, timezone
+ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+ # temp1 is the Composite sensor, which is what the drives' 70C rating specifies and
+ # what smartctl reports. temp2-temp4 are internal die sensors that run ~9C hotter and
+ # carry no comparable rating, so including them gated a Composite threshold against
+ # the wrong reading. Fewer series also means fewer chances for a single missing sample
+ # to fail the identity check and latch a node closed.
+ SENSORS = {
+ # empty tuple = no NVMe visible to this node (control-1 is a VM); it is gated on CPU
+ # headroom instead, which is what every `if sensors` branch below keys off.
+ "control-1": (),
+ "control-2": (("nvme_nvme0", "temp1"), ("nvme_nvme1", "temp1")),
+ "control-3": (("nvme_nvme0", "temp1"), ("nvme_nvme1", "temp1")),
+ }
+ # Miner slot priority, best first. control-1 has no NVMe to cook and was safe 96% of the
+ # last 7d against 60%/30% for the bare-metal nodes. Each ScaledObject counts how many safe
+ # nodes outrank it and subtracts one miner's draw per rank, so this order decides who gets
+ # scarce watts. It lives here rather than in the manifests because it is a property of the
+ # node telemetry, and a future headroom-derived ranking replaces this tuple alone.
+ PRIORITY = ("control-1", "control-2", "control-3")
+ ENDPOINT = "http://vmauth-victoria-metrics.observability.svc.cluster.local:8427"
+ EVALUATION_INTERVAL_SECONDS = 30
+ SOURCE_SAMPLE_MAX_AGE_SECONDS = 120
+ # The CPU path joins cadvisor, which scrapes at 60s where the other six sources scrape at 20s,
+ # and query_cpu dates the observation by min() across all of them. At 120s control-1 ran an age
+ # p99 of 102s against that ceiling and self-invalidated 465 times in 10.26d (45 and 49 on the
+ # NVMe nodes), each latching a 600s recovery dwell that drained the miner that caused it.
+ CPU_SAMPLE_MAX_AGE_SECONDS = 180
+ # 7 serial queries per CPU evaluation: at 10s a hung VictoriaMetrics blocked 70s against a 30s interval
+ HTTP_TIMEOUT_SECONDS = 5
+
+
+ def _dt(value):
+ value = float(value)
+ if not math.isfinite(value):
+ raise ValueError("non-finite timestamp")
+ return datetime.fromtimestamp(value, timezone.utc)
+
+
+ def _fresh(timestamp, evaluation, max_age):
+ """Return whether a source timestamp is not future-dated or too old."""
+ age = (evaluation - timestamp).total_seconds()
+ return 0 <= age <= max_age
+
+
+ class DwellPolicy:
+ def __init__(self, recovery_limit, trip_limit, recovery_dwell, trip_dwell, max_gap_seconds=SOURCE_SAMPLE_MAX_AGE_SECONDS, panic_limit=None):
+ self.recovery_limit, self.trip_limit = recovery_limit, trip_limit
+ self.recovery_dwell, self.trip_dwell = recovery_dwell, trip_dwell
+ self.panic_limit = panic_limit
+ self.max_gap = float(max_gap_seconds)
+ self.safe = False
+ self._last_source = None
+ self._pending = None
+ self._since = None
+
+ def invalidate(self):
+ self.safe = False
+ self._last_source = self._pending = self._since = None
+
+ def observe(self, value, source_time, monotonic_now):
+ if not isinstance(value, (int, float)) or not math.isfinite(value) or not isinstance(source_time, datetime):
+ self.invalidate()
+ return False
+ source_seconds = source_time.timestamp()
+ if self._last_source is not None:
+ gap = source_seconds - self._last_source
+ if gap <= 0:
+ return self.safe # duplicate/out-of-order samples cannot advance dwell
+ if gap > self.max_gap:
+ self._pending = self._since = None
+ self._last_source = source_seconds
+ # third, dwell-free state: 2C from the rating, confirming the reading over a 60s dwell
+ # would spend most of the margin. Clearing _pending forces a full recovery dwell after.
+ if self.panic_limit is not None and value >= self.panic_limit:
+ self.safe = False
+ self._pending = self._since = None
+ return False
+ kind = "recover" if value <= self.recovery_limit else "trip" if value >= self.trip_limit else None
+ if kind is None:
+ self._pending = self._since = None
+ return self.safe
+ if (kind == "recover") == self.safe:
+ self._pending = self._since = None
+ return self.safe
+ if kind != self._pending:
+ self._pending, self._since = kind, monotonic_now
+ else:
+ dwell = self.recovery_dwell if kind == "recover" else self.trip_dwell
+ if monotonic_now - self._since >= dwell:
+ self.safe = kind == "recover"
+ self._pending = self._since = None
+ return self.safe
+
+
+ @dataclass(frozen=True)
+ class Source:
+ value: float
+ timestamp: datetime
+
+
+ @dataclass(frozen=True)
+ class CPUObservation:
+ host: Source
+ xmrig: Source | None
+ presence: Source
+
+
+ def cpu_value(observation):
+ if not isinstance(observation.host, Source):
+ raise ValueError("host source is required")
+ xmrig = observation.xmrig.value if observation.xmrig else 0.0
+ return max(0.0, min(100.0, observation.host.value - xmrig))
+
+
+ class VictoriaMetricsClient:
+ def __init__(self, endpoint, transport=None, timeout=HTTP_TIMEOUT_SECONDS, step_seconds=120):
+ self.endpoint = endpoint.rstrip("/")
+ self.transport = transport or _HTTPTransport(timeout)
+ self.step = f"{int(step_seconds)}s"
+
+ def _query(self, expression, evaluation):
+ # explicit step: default 5m step makes timestamp(a or b) snap to 5-min boundaries,
+ # which made 60% of freshness checks fail; step=max_age also bounds VM lookbehind
+ params = {"query": expression, "time": evaluation.astimezone(timezone.utc).isoformat().replace("+00:00", "Z"), "step": self.step}
+ try:
+ payload = self.transport.get(self.endpoint + "/api/v1/query", params)
+ except urllib.error.HTTPError as exc:
+ # a rejected query is undiagnosable without its text: 40h of bare 422s went unseen
+ raise ValueError(f"VictoriaMetrics rejected ({exc.code}): {expression}") from exc
+ if payload.get("status") != "success" or payload.get("data", {}).get("resultType") != "vector" or not isinstance(payload["data"].get("result"), list):
+ raise ValueError(f"invalid VictoriaMetrics response for: {expression}")
+ return payload["data"]["result"]
+
+ @staticmethod
+ def _sources(rows, identity=None, raw_timestamp=False):
+ out = {}
+ for row in rows:
+ metric = row.get("metric", {})
+ key = identity(metric) if identity else tuple(sorted(metric.items()))
+ if key in out or not isinstance(row.get("value"), list) or len(row["value"]) != 2:
+ raise ValueError("malformed or duplicate telemetry")
+ try:
+ value = float(row["value"][1])
+ timestamp = _dt(row["value"][1] if raw_timestamp else row["value"][0])
+ except (TypeError, ValueError, OverflowError) as exc:
+ raise ValueError("malformed telemetry value") from exc
+ if not math.isfinite(value):
+ raise ValueError("non-finite telemetry value")
+ out[key] = Source(value, timestamp)
+ return out
+
+ def query_nvme(self, node, sensors, evaluation):
+ parts = [f'node_hwmon_temp_celsius{{kubernetes_node="{node}",chip="{chip}",sensor="{sensor}"}}' for chip, sensor in sensors]
+ expression = " or ".join(parts)
+ rows = self._query(expression, evaluation)
+ # timestamp() over an or-expression becomes a step-aligned subquery in VictoriaMetrics
+ # (fake boundary stamps broke 60% of freshness checks); single selectors return raw stamps
+ timestamps = [row for part in parts for row in self._query("timestamp(" + part + ")", evaluation)]
+ key = lambda m: (m.get("chip"), m.get("sensor"))
+ if any(row.get("metric", {}).get("kubernetes_node") != node for row in rows + timestamps):
+ raise ValueError("NVMe node identity changed")
+ found = self._sources(rows, key)
+ stamped = self._sources(timestamps, key, raw_timestamp=True)
+ if set(found) != set(sensors):
+ raise ValueError("incomplete or changed NVMe identity set")
+ if set(stamped) != set(sensors):
+ raise ValueError("incomplete or changed NVMe timestamp identity set")
+ result = []
+ for item in sensors:
+ if not -40 <= found[item].value <= 150:
+ raise ValueError("implausible NVMe temperature")
+ result.append(Source(found[item].value, stamped[item].timestamp))
+ return result
+
+ def query_cpu(self, node, evaluation, window="5m"):
+ host_raw = f'node_cpu_seconds_total{{kubernetes_node="{node}",mode!="idle"}}'
+ idle_raw = f'node_cpu_seconds_total{{kubernetes_node="{node}",mode="idle"}}'
+ cadvisor_raw = f'container_cpu_usage_seconds_total{{node="{node}",namespace="web3",container!="",container!="POD"}}'
+ host_query = f'sum(rate({host_raw}[{window}])) / count(count({idle_raw}) by (cpu)) * 100'
+ ksm_raw = 'kube_pod_info{namespace="web3"}'
+ label_selector = 'namespace="web3",label_app_kubernetes_io_component="thermal-guarded"'
+ node_pods_raw = f'kube_pod_info{{namespace="web3",node="{node}"}}'
+ xmrig_presence_raw = f'kube_pod_labels{{{label_selector}}} * on(namespace,pod) group_left(node) {node_pods_raw}'
+ xmrig_presence_query = f'count({xmrig_presence_raw}) or vector(0)'
+ xmrig_query = f'100 * (sum((sum by (namespace,pod) (rate({cadvisor_raw}[{window}])) * on(namespace,pod) group_left(node) ({xmrig_presence_raw})) or vector(0)) / count(count({idle_raw}) by (cpu)))'
+ labels_raw = f'kube_pod_labels{{{label_selector}}}'
+ def one(query):
+ values = self._sources(self._query(query, evaluation))
+ if len(values) != 1:
+ raise ValueError(f"CPU source must be one scalar: {query}")
+ return next(iter(values.values()))
+ def oldest(*selectors):
+ out = []
+ for selector in selectors:
+ rows = self._query("timestamp(" + selector + ")", evaluation)
+ if not rows:
+ raise ValueError(f"missing raw timestamps: {selector}")
+ out.extend(self._sources(rows, raw_timestamp=True).values())
+ return min(item.timestamp for item in out)
+ presence = one(xmrig_presence_query)
+ if presence.value < 0:
+ raise ValueError("invalid labelled XMRig presence source")
+ if presence.value > 0:
+ # the membership stamps date both the presence count and the subtraction, so
+ # they are fetched once and reused rather than queried twice per evaluation
+ membership = oldest(labels_raw, node_pods_raw)
+ presence = Source(presence.value, membership)
+ # cadvisor joins the same membership selectors, so cadvisor freshness is
+ # verified exactly when its data enters the subtraction
+ xmrig = Source(one(xmrig_query).value, min(membership, oldest(cadvisor_raw)))
+ else:
+ # no labelled miner on this node: anchor presence freshness to pod-info stamps
+ presence = Source(0, oldest(ksm_raw))
+ xmrig = None
+ host = Source(one(host_query).value, oldest(host_raw))
+ return CPUObservation(host, xmrig, presence)
+
+
+ class _HTTPTransport:
+ def __init__(self, timeout=HTTP_TIMEOUT_SECONDS):
+ self.timeout = timeout
+
+ def get(self, url, params):
+ request = urllib.request.Request(url + "?" + urllib.parse.urlencode(params))
+ with urllib.request.urlopen(request, timeout=self.timeout) as response:
+ return json.load(response)
+
+
+ class GuardController:
+ def __init__(self, telemetry, clock=time.monotonic, wall_clock=lambda: datetime.now(timezone.utc)):
+ self.telemetry = telemetry
+ self.clock, self.wall_clock = clock, wall_clock
+ # Trip 65C / recover 62C on Composite, against a 70C drive rating. Over 1151 measured
+ # miner starts the rise is p50 0.88 / p90 1.13 / p99 1.35 C/min. Trip-to-drain is 135s
+ # (30s evaluation + 60s dwell + 30s KEDA poll + 15s drain) = 3.0C at p99, and
+ # test_trip_to_drain_budget pins it. Real bursts under these parameters peaked at p50 67.8
+ # / p90 68.8 / max 69.8C, so the margin to the rating is 0.2C and nothing may be loosened.
+ # max_gap is each node's slowest scrape: it dates the freshness check, the anti-replay gap
+ # and the dwell reset, which must agree or a gap inside one budget resets dwell under
+ # another. control-1 has no NVMe to cook and no panic limit; a busy CPU carries no
+ # equivalent of a drive's absolute rating.
+ self.policies = {
+ "control-1": DwellPolicy(50, 70, 600, 120, CPU_SAMPLE_MAX_AGE_SECONDS),
+ "control-2": DwellPolicy(62, 65, 180, 60, panic_limit=67),
+ "control-3": DwellPolicy(62, 65, 180, 60, panic_limit=67),
+ }
+ self.ready = False
+ self.metrics = {
+ "evaluations": 0, "query_errors": {node: 0 for node in SENSORS},
+ "safe": {node: 0 for node in SENSORS},
+ "nvme_temp_max": {node: 0.0 for node in SENSORS if SENSORS[node]},
+ "source_age_seconds": {node: 0.0 for node in SENSORS},
+ "cpu_non_xmrig": {node: 0.0 for node in SENSORS if not SENSORS[node]},
+ "rank": {node: PRIORITY.index(node) for node in SENSORS},
+ }
+ self._last_source_stamps = {node: {} for node in SENSORS}
+
+ def _new_source_set(self, node, sources):
+ # Keyed by source identity, not position: a miner starting or stopping adds or removes
+ # the xmrig source, and comparing by position made that look like tampering. It failed
+ # control-1 closed 19 times in 12h, draining the miner, which changed the set back.
+ # Only keys in both sets are comparable; a genuinely new key is exempt for one cycle
+ # rather than the whole evaluation going unchecked.
+ stamps = {key: source.timestamp.timestamp() for key, source in sources.items()}
+ previous = self._last_source_stamps[node]
+ shared = stamps.keys() & previous.keys()
+ if any(stamps[key] - previous[key] > self.policies[node].max_gap for key in shared):
+ raise ValueError("source gap exceeded maximum")
+ if shared and any(stamps[key] <= previous[key] for key in shared):
+ return False
+ self._last_source_stamps[node] = stamps
+ return True
+
+ def evaluate(self, evaluation=None):
+ evaluation = evaluation or self.wall_clock()
+ now = self.clock()
+ for node, sensors in SENSORS.items():
+ try:
+ # no sensors means a node with no visible NVMe (control-1, a VM): it is gated
+ # on CPU headroom instead. The dwell policy and metrics are keyed identically.
+ if sensors:
+ samples = dict(zip(sensors, self.telemetry.query_nvme(node, sensors, evaluation)))
+ # trip on the hottest drive, date it by the newest sample it was read from
+ value = max(item.value for item in samples.values())
+ stamp = max(item.timestamp for item in samples.values())
+ self.metrics["nvme_temp_max"][node] = value
+ else:
+ obs = self.telemetry.query_cpu(node, evaluation)
+ samples = {"host": obs.host, "presence": obs.presence}
+ if obs.xmrig:
+ samples["xmrig"] = obs.xmrig
+ value = cpu_value(obs)
+ stamp = min(item.timestamp for item in samples.values())
+ self.metrics["cpu_non_xmrig"][node] = value
+ policy = self.policies[node]
+ if not samples or not all(_fresh(item.timestamp, evaluation, policy.max_gap) for item in samples.values()):
+ raise ValueError("stale or future source")
+ safe = policy.observe(value, stamp, now) if self._new_source_set(node, samples) else policy.safe
+ self.metrics["source_age_seconds"][node] = max(0.0, evaluation.timestamp() - min(item.timestamp for item in samples.values()).timestamp())
+ self.metrics["safe"][node] = int(safe)
+ except Exception as exc:
+ # one line per failure, no traceback: the query text travels in the exception
+ logging.error(f"evaluation failed for {node}: {exc!r}")
+ self.metrics["query_errors"][node] += 1
+ self.policies[node].invalidate()
+ self._last_source_stamps[node] = {}
+ self.metrics["safe"][node] = 0
+ self.metrics["source_age_seconds"][node] = float("nan")
+ if sensors:
+ self.metrics["nvme_temp_max"][node] = float("nan")
+ else:
+ self.metrics["cpu_non_xmrig"][node] = float("nan")
+ self.metrics["evaluations"] += 1
+ self.ready = True
+ return dict(self.metrics["safe"])
+
+
+ def render_metrics(controller):
+ m = controller.metrics
+ lines = [f'xmrig_guard_evaluations_total {m["evaluations"]}']
+ This message was truncated. Download full message |
AI Automated ReviewAnalysis engine: omniroute@http://litellm.ai.svc.cluster.local/v1 (openai) Recommendation: Approve. The PR is internally consistent, well-tested, and addresses the must_check items appropriately for a thermal-guard tuning change. Must Check Verification
Findings
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based on
main. Independent of the #4378/#4379 stack: touches only talos patches,web3/, the CNPG cluster and two docs.Hugepages
vm.nr_hugepages: 1184withheld 2368Mi from every node's allocatable memory even at 0 replicas. xmrig has been 0/0 on all three nodes for 6d12h under the thermal guard;HugePages_Free == HugePages_Totaland cluster-widehugepages-2Mirequests were 0%.Unblocks
rook-ceph-exporter-control-2(150Mi, Pending 2h).Applies on next reboot. Nothing was applied to the cluster.
Coupled edits required by the removal:
--huge-pages->--no-huge-pages, bothhugepages-2Miblocks, the HugePages emptyDir, andIPC_LOCK(xmrig takes it only to raiseRLIMIT_MEMLOCKforMAP_HUGETLB).xmrig memory 32Mi/128Mi -> 2Gi/3Gi. Requested under the ~2336Mi RandomX needs: at the true figure the miner never schedules and
preemptionPolicy: Nevermeans it waits rather than preempts./tmpgainssizeLimit: 64Mibecause the dataset now shares the container limit instead of living in hugetlb.Net reclaim is 2368Mi idle, ~320Mi while mining. RandomX loses hashrate without hugepages; accepted.
Guard move
web3/xmrig-guard/app/->web3/monero/guard/, standalone ks.yaml folded intomonero/ks.yaml. Resources renamed toguard.Metrics stay
xmrig_guard_*. The KEDA triggers, 5 alert rules, the dashboard and 180d of series reference them.No
dependsOnadded: KEDA'sfallback: {failureThreshold: 1, replicas: 0}plus thecount == bool 1andtimestamp >= time() - 120gates mean an absent guard yields 0 replicas, so ordering cannot produce mining without a gate.Tuning
Replayed 180d of
node_hwmon_temp_celsiusat the native 20s cadence over a 612-candidate grid. Only 2026-07-27 14:02 onward reflects the deployed policy; before that the guard read all sensors rather thantemp1(~9C hotter), and 07-17..21 it emitted without enforcing.Nothing loosened. Measured bursts peak at p50 67.8 / p90 68.8 / max 69.8C against a 70C rating. Rise is p50 0.88 / p90 1.13 / p99 1.35 C/min, not the 1.1 the design was sized on; the budget test now uses 1.35 and pins the panic path, which governs on a fast ramp (68.69C at 75s) since the trip path models 68.04C above the panic limit.
Guard is working: zero samples >=70C while
safe=1. Before it existed, ungated mining put control-3 over its rating 51% of the time it mined, peaking at 86C.The prior
55.0%/38.8% vs 44.7%/31.5%figures do not reproduce on either sensor set. Direction holds, magnitudes were stale; restated in the doc.control-1 freshness
Flapping was telemetry, not policy: 465 query errors in 10.26d against 45 and 49 on the NVMe nodes,
source_age_secondsp99 102s against a 120s ceiling. cadvisor scrapes at 60s where the other six sources scrape at 20s andquery_cpudates the observation bymin()across them. Self-reinforcing at 1.7 errors/h idle vs 4.8/h with a miner, because the cadvisor join only engages when one is present. 85 of 118 mining bursts died under 5 minutes.Budget is now 180s for the CPU path, carried on the policy so the freshness check, anti-replay gap and dwell reset read one number and cannot drift. VM
stepfollows, or the samples would never be returned. Its thresholds replay at 93.8% safe against 75.8% observed.Follow-ups, not in this PR
evictionHard: 7%->1GiandsystemReserved: 1Gi->512Mi: another +1.54Gi/node.kubeReservedmust NOT be cut, measured/podruntimeis already above its 2Gi.nvme1Micron 7450 480GB runs ~10C hotter than the 980 PRO beside it and sets the gate on both nodes. Sun-hour idle baseline is 63-64C against a workload adding 8.5C in 7 min, so 68-72% of surplus goes unmined. Cooling it beats any threshold change.Verification
26/26 tests pass;
kustomize buildclean on web3, monero/guard, monero/xmrig, cnpg cluster.Summary by CodeRabbit
New Features
Improvements
Documentation