feat(xmrig): gate each miner on its own node's thermal verdict - #4268
Merged
Conversation
The KEDA trigger gated all mining on min(xmrig_guard_safe) across all three nodes, so control-3's NVMe stopped control-1 too. Over the last 7d that spent 8.7 of 137 desired replica-hours (6%) against 47.5h of solar export, with the gate open only 25% of the time while control-1 alone was safe 96%. Split the single Deployment into one per node via a flux-operator ResourceSet: each miner is pinned to its node with a nodeSelector and scaled by a ScaledObject reading only that node's guard verdict. Activation is staggered 25/75/125W so the three triggers, which all read the same global export figure, add miners in sequence at ~50W each instead of all firing at 50W. Stagger order follows measured thermal availability (96%/60%/30%), not power efficiency. A single Deployment could not do this: the HPA sheds an arbitrary pod, so nothing guaranteed the pod on the node that just tripped was the one to die. Guard thresholds are unchanged at trip 64C / recover 60C. Modelled at 73 replica-hours on last week's data against 8.7 delivered.
|
Warning Review limit reached
Next review available in: 20 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (6)
📝 WalkthroughWalkthroughXmrig is migrated from fleet-wide thermal gating to three node-pinned workloads generated by a Flux ResourceSet. Each workload receives independent KEDA scaling from power and guard metrics, while the bypass alert evaluates matching deployments individually. ChangesPer-node xmrig gating
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Prometheus
participant KEDA
participant NodeDeployment
participant XmrigService
Prometheus->>KEDA: Evaluate node power and guard metrics
KEDA->>NodeDeployment: Set node deployment replicas to 0 or 1
NodeDeployment->>XmrigService: Serve the xmrig HTTP API on port 42000
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
@@ (root level) @@
# v1/ServiceAccount/web3/xmrig
! - one document removed:
- apiVersion: v1
- kind: ServiceAccount
- metadata:
- name: xmrig
- namespace: web3
- labels:
- app.kubernetes.io/instance: xmrig
- app.kubernetes.io/managed-by: Helm
- app.kubernetes.io/name: xmrig
- helm.toolkit.fluxcd.io/name: xmrig
- helm.toolkit.fluxcd.io/namespace: web3
@@ (root level) @@
# v1/Service/web3/xmrig
! - one document removed:
- apiVersion: v1
- kind: Service
- metadata:
- name: xmrig
- namespace: web3
- labels:
- app.kubernetes.io/instance: xmrig
- app.kubernetes.io/managed-by: Helm
- app.kubernetes.io/name: xmrig
- app.kubernetes.io/service: xmrig
- helm.toolkit.fluxcd.io/name: xmrig
- helm.toolkit.fluxcd.io/namespace: web3
- spec:
- type: ClusterIP
- selector:
- app.kubernetes.io/controller: xmrig
- app.kubernetes.io/instance: xmrig
- app.kubernetes.io/name: xmrig
- ports:
- - name: http-api
- port: 42000
- protocol: TCP
- targetPort: 42000
@@ (root level) @@
# apps/v1/Deployment/web3/xmrig
! - one document removed:
- apiVersion: apps/v1
- kind: Deployment
- metadata:
- name: xmrig
- namespace: web3
- labels:
- app.kubernetes.io/controller: xmrig
- app.kubernetes.io/instance: xmrig
- app.kubernetes.io/managed-by: Helm
- app.kubernetes.io/name: xmrig
- helm.toolkit.fluxcd.io/name: xmrig
- helm.toolkit.fluxcd.io/namespace: web3
- spec:
- replicas: 1
- revisionHistoryLimit: 3
- selector:
- matchLabels:
- app.kubernetes.io/controller: xmrig
- app.kubernetes.io/instance: xmrig
- app.kubernetes.io/name: xmrig
- strategy:
- type: Recreate
- template:
- metadata:
- labels:
- app.kubernetes.io/component: thermal-guarded
- app.kubernetes.io/controller: xmrig
- app.kubernetes.io/instance: xmrig
- app.kubernetes.io/name: xmrig
- spec:
- affinity:
- nodeAffinity:
- preferredDuringSchedulingIgnoredDuringExecution:
- - preference:
- matchExpressions:
- - key: kubernetes.io/hostname
- operator: In
- values:
- - control-2
- - control-3
- weight: 100
- podAntiAffinity:
- requiredDuringSchedulingIgnoredDuringExecution:
- - labelSelector:
- matchExpressions:
- - key: app.kubernetes.io/name
- operator: In
- values:
- - xmrig
- topologyKey: kubernetes.io/hostname
- automountServiceAccountToken: false
- containers:
- - name: app
- image: "djerfy/xmrig:6.26.0@sha256:bd72a33a41d67a74956fd494ab7ee0d79efdf616db8f0829fb5186eec91152d3"
- args:
- - "-o"
- - "p2pool.web3.svc.cluster.local:3333"
- - "-u"
- - x+5000
- - "-k"
- - "--coin=monero"
- - "--cpu-max-threads-hint=50"
- - "--huge-pages"
- - "--http-port=42000"
- - "--http-host=0.0.0.0"
- command:
- - /usr/bin/xmrig
- resources:
- limits:
- cpu: 6
- hugepages-2Mi: 2368Mi
- memory: 128Mi
- requests:
- cpu: 1
- hugepages-2Mi: 2368Mi
- memory: 32Mi
- volumeMounts:
- - name: hugepages
- mountPath: /dev/hugepages
- - name: tmp
- mountPath: /tmp
- securityContext:
- allowPrivilegeEscalation: false
- capabilities:
- add:
- - IPC_LOCK
- drop:
- - ALL
- readOnlyRootFilesystem: true
- dnsPolicy: ClusterFirst
- enableServiceLinks: false
- hostIPC: false
- hostNetwork: false
- hostPID: false
- priorityClassName: low-priority-mining
- securityContext:
- fsGroup: 1000
- fsGroupChangePolicy: OnRootMismatch
- runAsGroup: 1000
- runAsNonRoot: true
- runAsUser: 1000
- serviceAccountName: xmrig
- terminationGracePeriodSeconds: 15
- volumes:
- - name: hugepages
- emptyDir:
- medium: HugePages
- - name: tmp
- emptyDir:
- medium: Memory
@@ (root level) @@
# v1/ServiceAccount/web3/xmrig-control-1
! + one document added:
+ apiVersion: v1
+ kind: ServiceAccount
+ metadata:
+ name: xmrig-control-1
+ namespace: web3
+ labels:
+ app.kubernetes.io/instance: xmrig-control-1
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/name: xmrig-control-1
+ helm.toolkit.fluxcd.io/name: xmrig-control-1
+ helm.toolkit.fluxcd.io/namespace: web3
@@ (root level) @@
# apps/v1/Deployment/web3/xmrig-control-1
! + one document added:
+ apiVersion: apps/v1
+ kind: Deployment
+ metadata:
+ name: xmrig-control-1
+ namespace: web3
+ labels:
+ app.kubernetes.io/controller: xmrig-control-1
+ app.kubernetes.io/instance: xmrig-control-1
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/name: xmrig-control-1
+ helm.toolkit.fluxcd.io/name: xmrig-control-1
+ helm.toolkit.fluxcd.io/namespace: web3
+ spec:
+ replicas: 1
+ revisionHistoryLimit: 3
+ selector:
+ matchLabels:
+ app.kubernetes.io/controller: xmrig-control-1
+ app.kubernetes.io/instance: xmrig-control-1
+ app.kubernetes.io/name: xmrig-control-1
+ strategy:
+ type: Recreate
+ template:
+ metadata:
+ labels:
+ app.kubernetes.io/component: thermal-guarded
+ app.kubernetes.io/controller: xmrig-control-1
+ app.kubernetes.io/instance: xmrig-control-1
+ app.kubernetes.io/name: xmrig-control-1
+ spec:
+ automountServiceAccountToken: false
+ containers:
+ - name: app
+ image: "djerfy/xmrig:6.26.0@sha256:bd72a33a41d67a74956fd494ab7ee0d79efdf616db8f0829fb5186eec91152d3"
+ args:
+ - "-o"
+ - "p2pool.web3.svc.cluster.local:3333"
+ - "-u"
+ - x+5000
+ - "-k"
+ - "--coin=monero"
+ - "--cpu-max-threads-hint=50"
+ - "--huge-pages"
+ - "--http-port=42000"
+ - "--http-host=0.0.0.0"
+ command:
+ - /usr/bin/xmrig
+ resources:
+ limits:
+ cpu: 6
+ hugepages-2Mi: 2368Mi
+ memory: 128Mi
+ requests:
+ cpu: 1
+ hugepages-2Mi: 2368Mi
+ memory: 32Mi
+ volumeMounts:
+ - name: hugepages
+ mountPath: /dev/hugepages
+ - name: tmp
+ mountPath: /tmp
+ securityContext:
+ allowPrivilegeEscalation: false
+ capabilities:
+ add:
+ - IPC_LOCK
+ drop:
+ - ALL
+ readOnlyRootFilesystem: true
+ dnsPolicy: ClusterFirst
+ enableServiceLinks: false
+ hostIPC: false
+ hostNetwork: false
+ hostPID: false
+ nodeSelector:
+ kubernetes.io/hostname: control-1
+ priorityClassName: low-priority-mining
+ securityContext:
+ fsGroup: 1000
+ fsGroupChangePolicy: OnRootMismatch
+ runAsGroup: 1000
+ runAsNonRoot: true
+ runAsUser: 1000
+ serviceAccountName: xmrig-control-1
+ terminationGracePeriodSeconds: 15
+ volumes:
+ - name: hugepages
+ emptyDir:
+ medium: HugePages
+ - name: tmp
+ emptyDir:
+ medium: Memory
@@ (root level) @@
# v1/ServiceAccount/web3/xmrig-control-2
! + one document added:
+ apiVersion: v1
+ kind: ServiceAccount
+ metadata:
+ name: xmrig-control-2
+ namespace: web3
+ labels:
+ app.kubernetes.io/instance: xmrig-control-2
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/name: xmrig-control-2
+ helm.toolkit.fluxcd.io/name: xmrig-control-2
+ helm.toolkit.fluxcd.io/namespace: web3
@@ (root level) @@
# apps/v1/Deployment/web3/xmrig-control-2
! + one document added:
+ apiVersion: apps/v1
+ kind: Deployment
+ metadata:
+ name: xmrig-control-2
+ namespace: web3
+ labels:
+ app.kubernetes.io/controller: xmrig-control-2
+ app.kubernetes.io/instance: xmrig-control-2
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/name: xmrig-control-2
+ helm.toolkit.fluxcd.io/name: xmrig-control-2
+ helm.toolkit.fluxcd.io/namespace: web3
+ spec:
+ replicas: 1
+ revisionHistoryLimit: 3
+ selector:
+ matchLabels:
+ app.kubernetes.io/controller: xmrig-control-2
+ app.kubernetes.io/instance: xmrig-control-2
+ app.kubernetes.io/name: xmrig-control-2
+ strategy:
+ type: Recreate
+ template:
+ metadata:
+ labels:
+ app.kubernetes.io/component: thermal-guarded
+ app.kubernetes.io/controller: xmrig-control-2
+ app.kubernetes.io/instance: xmrig-control-2
+ app.kubernetes.io/name: xmrig-control-2
+ spec:
+ automountServiceAccountToken: false
+ containers:
+ - name: app
+ image: "djerfy/xmrig:6.26.0@sha256:bd72a33a41d67a74956fd494ab7ee0d79efdf616db8f0829fb5186eec91152d3"
+ args:
+ - "-o"
+ - "p2pool.web3.svc.cluster.local:3333"
+ - "-u"
+ - x+5000
+ - "-k"
+ - "--coin=monero"
+ - "--cpu-max-threads-hint=50"
+ - "--huge-pages"
+ - "--http-port=42000"
+ - "--http-host=0.0.0.0"
+ command:
+ - /usr/bin/xmrig
+ resources:
+ limits:
+ cpu: 6
+ hugepages-2Mi: 2368Mi
+ memory: 128Mi
+ requests:
+ cpu: 1
+ hugepages-2Mi: 2368Mi
+ memory: 32Mi
+ volumeMounts:
+ - name: hugepages
+ mountPath: /dev/hugepages
+ - name: tmp
+ mountPath: /tmp
+ securityContext:
+ allowPrivilegeEscalation: false
+ capabilities:
+ add:
+ - IPC_LOCK
+ drop:
+ - ALL
+ readOnlyRootFilesystem: true
+ dnsPolicy: ClusterFirst
+ enableServiceLinks: false
+ hostIPC: false
+ hostNetwork: false
+ hostPID: false
+ nodeSelector:
+ kubernetes.io/hostname: control-2
+ priorityClassName: low-priority-mining
+ securityContext:
+ fsGroup: 1000
+ fsGroupChangePolicy: OnRootMismatch
+ runAsGroup: 1000
+ runAsNonRoot: true
+ runAsUser: 1000
+ serviceAccountName: xmrig-control-2
+ terminationGracePeriodSeconds: 15
+ volumes:
+ - name: hugepages
+ emptyDir:
+ medium: HugePages
+ - name: tmp
+ emptyDir:
+ medium: Memory
@@ (root level) @@
# v1/ServiceAccount/web3/xmrig-control-3
! + one document added:
+ apiVersion: v1
+ kind: ServiceAccount
+ metadata:
+ name: xmrig-control-3
+ namespace: web3
+ labels:
+ app.kubernetes.io/instance: xmrig-control-3
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/name: xmrig-control-3
+ helm.toolkit.fluxcd.io/name: xmrig-control-3
+ helm.toolkit.fluxcd.io/namespace: web3
@@ (root level) @@
# apps/v1/Deployment/web3/xmrig-control-3
! + one document added:
+ apiVersion: apps/v1
+ kind: Deployment
+ metadata:
+ name: xmrig-control-3
+ namespace: web3
+ labels:
+ app.kubernetes.io/controller: xmrig-control-3
+ app.kubernetes.io/instance: xmrig-control-3
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/name: xmrig-control-3
+ helm.toolkit.fluxcd.io/name: xmrig-control-3
+ helm.toolkit.fluxcd.io/namespace: web3
+ spec:
+ replicas: 1
+ revisionHistoryLimit: 3
+ selector:
+ matchLabels:
+ app.kubernetes.io/controller: xmrig-control-3
+ app.kubernetes.io/instance: xmrig-control-3
+ app.kubernetes.io/name: xmrig-control-3
+ strategy:
+ type: Recreate
+ template:
+ metadata:
+ labels:
+ app.kubernetes.io/component: thermal-guarded
+ app.kubernetes.io/controller: xmrig-control-3
+ app.kubernetes.io/instance: xmrig-control-3
+ app.kubernetes.io/name: xmrig-control-3
+ spec:
+ automountServiceAccountToken: false
+ containers:
+ - name: app
+ image: "djerfy/xmrig:6.26.0@sha256:bd72a33a41d67a74956fd494ab7ee0d79efdf616db8f0829fb5186eec91152d3"
+ args:
+ - "-o"
+ - "p2pool.web3.svc.cluster.local:3333"
+ - "-u"
+ - x+5000
+ - "-k"
+ - "--coin=monero"
+ - "--cpu-max-threads-hint=50"
+ - "--huge-pages"
+ - "--http-port=42000"
+ - "--http-host=0.0.0.0"
+ command:
+ - /usr/bin/xmrig
+ resources:
+ limits:
+ cpu: 6
+ hugepages-2Mi: 2368Mi
+ memory: 128Mi
+ requests:
+ cpu: 1
+ hugepages-2Mi: 2368Mi
+ memory: 32Mi
+ volumeMounts:
+ - name: hugepages
+ mountPath: /dev/hugepages
+ - name: tmp
+ mountPath: /tmp
+ securityContext:
+ allowPrivilegeEscalation: false
+ capabilities:
+ add:
+ - IPC_LOCK
+ drop:
+ - ALL
+ readOnlyRootFilesystem: true
+ dnsPolicy: ClusterFirst
+ enableServiceLinks: false
+ hostIPC: false
+ hostNetwork: false
+ hostPID: false
+ nodeSelector:
+ kubernetes.io/hostname: control-3
+ priorityClassName: low-priority-mining
+ securityContext:
+ fsGroup: 1000
+ fsGroupChangePolicy: OnRootMismatch
+ runAsGroup: 1000
+ runAsNonRoot: true
+ runAsUser: 1000
+ serviceAccountName: xmrig-control-3
+ terminationGracePeriodSeconds: 15
+ volumes:
+ - name: hugepages
+ emptyDir:
+ medium: HugePages
+ - name: tmp
+ emptyDir:
+ medium: Memory
|
@@ spec.groups.xmrig-guard.rules.rules.XmrigGuardEnforcementBypassed.annotations.summary @@
# monitoring.coreos.com/v1/PrometheusRule/web3/xmrig-guard-rules
! ± value change
- Miners are running while the thermal gate should be closed — the KEDA trip is not being enforced (operator down, HPA wedged, or manual scale)
+ {{ $labels.deployment }} is running while its node's thermal gate should be closed — the KEDA trip is not being enforced (operator down, HPA wedged, or manual scale)
@@ spec.groups.xmrig-guard.rules.rules.XmrigGuardEnforcementBypassed.expr @@
# monitoring.coreos.com/v1/PrometheusRule/web3/xmrig-guard-rules
! ± value change in multiline text (one insert, one deletion)
- (
- min(xmrig_guard_safe{node=~"control-[123]"}) == 0
- or count(xmrig_guard_safe{node=~"control-[123]"}) != 3
- or absent(xmrig_guard_safe{node=~"control-[123]"})
- )
- and on ()
- kube_deployment_status_replicas{namespace="web3", deployment="xmrig"} > 0
+ kube_deployment_status_replicas{namespace="web3", deployment=~"xmrig-control-.+"} > 0
+ unless on (deployment)
+ label_replace(
+ (xmrig_guard_safe == 1)
+ and on (node) (count by (node) (xmrig_guard_safe) == 1)
+ and on (node) (timestamp(xmrig_guard_safe) >= time() - 120),
+ "deployment", "xmrig-$1", "node", "(.*)"
+ )
@@ spec.groups.xmrig-guard.rules.rules.XmrigGuardLatchedUnsafe.expr @@
# monitoring.coreos.com/v1/PrometheusRule/web3/xmrig-guard-rules
! ± value change in multiline text (two inserts, two deletions)
- max_over_time(xmrig_guard_safe{node=~"control-[123]"}[6h]) == 0
+ max_over_time(xmrig_guard_safe[6h]) == 0
and on (node)
- max_over_time(xmrig_guard_nvme_temp_max_celsius{node=~"control-[123]"}[2h]) <= 60
+ max_over_time(xmrig_guard_nvme_temp_max_celsius[2h]) <= 60
@@ data.controller.py @@
# v1/ConfigMap/web3/xmrig-guard
! ± value change in multiline text (24 inserts, 22 deletions)
"""Small, dependency-free XMRig safety signal controller.
The controller deliberately treats telemetry as untrusted input. A complete
set of fresh samples is required before a node can become safe. Policy is
[16 lines unchanged)]
# carry no comparable rating, so including them gated a Composite threshold against
# the wrong reading. Fewer series also means fewer chances for a single missing sample
# to fail the identity check and latch a node closed.
SENSORS = {
+ # empty tuple = no NVMe visible to this node (control-1 is a VM); it is gated on CPU
+ # headroom instead, which is what every `if sensors` branch below keys off.
"control-1": (),
"control-2": (("nvme_nvme0", "temp1"), ("nvme_nvme1", "temp1")),
"control-3": (("nvme_nvme0", "temp1"), ("nvme_nvme1", "temp1")),
}
+ # Miner slot priority, best first. control-1 has no NVMe to cook and was safe 96% of the
+ # last 7d against 60%/30% for the bare-metal nodes. Each ScaledObject counts how many safe
+ # nodes outrank it and subtracts one miner's draw per rank, so this order decides who gets
+ # scarce watts. It lives here rather than in the manifests because it is a property of the
+ # node telemetry, and a future headroom-derived ranking replaces this tuple alone.
+ PRIORITY = ("control-1", "control-2", "control-3")
ENDPOINT = "http://vmauth-victoria-metrics.observability.svc.cluster.local:8427"
EVALUATION_INTERVAL_SECONDS = 60
SOURCE_SAMPLE_MAX_AGE_SECONDS = 120
MAX_SOURCE_GAP_SECONDS = 120
[76 lines unchanged)]
return max(0.0, min(100.0, observation.host.value - xmrig))
class VictoriaMetricsClient:
- def __init__(self, endpoint, transport=None, timeout=10, step_seconds=120):
+ def __init__(self, endpoint, transport=None, timeout=HTTP_TIMEOUT_SECONDS, step_seconds=120):
self.endpoint = endpoint.rstrip("/")
self.transport = transport or _HTTPTransport(timeout)
self.step = f"{int(step_seconds)}s"
[61 lines unchanged)]
node_pods_raw = f'kube_pod_info{{namespace="web3",node="{node}"}}'
xmrig_presence_raw = f'kube_pod_labels{{{label_selector}}} * on(namespace,pod) group_left(node) {node_pods_raw}'
xmrig_presence_query = f'count({xmrig_presence_raw}) or vector(0)'
xmrig_query = f'100 * (sum((sum by (namespace,pod) (rate({cadvisor_raw}[{window}])) * on(namespace,pod) group_left(node) ({xmrig_presence_raw})) or vector(0)) / count(count({idle_raw}) by (cpu)))'
- def one(query, optional=False, raw_timestamp=False):
- rows = self._query(query, evaluation)
- if not rows and optional:
- return None
- values = self._sources(rows, raw_timestamp=raw_timestamp)
+ labels_raw = f'kube_pod_labels{{{label_selector}}}'
+ def one(query):
+ values = self._sources(self._query(query, evaluation))
if len(values) != 1:
- raise ValueError("CPU source must be one scalar")
+ raise ValueError(f"CPU source must be one scalar: {query}")
return next(iter(values.values()))
- def source(query, raw_selector, optional=False):
- value = one(query, optional)
- if value is None:
- if optional:
- return None
- raise ValueError("incomplete CPU source")
- selectors = (raw_selector,) if isinstance(raw_selector, str) else raw_selector
- stamps = []
+ def oldest(*selectors):
+ out = []
for selector in selectors:
rows = self._query("timestamp(" + selector + ")", evaluation)
if not rows:
- raise ValueError("missing raw CPU timestamps")
- stamps.extend(self._sources(rows, raw_timestamp=True).values())
- return Source(value.value, min(item.timestamp for item in stamps))
- presence_rows = self._query(xmrig_presence_query, evaluation)
- presence_values = self._sources(presence_rows)
- if len(presence_values) != 1 or next(iter(presence_values.values())).value < 0:
+ raise ValueError(f"missing raw timestamps: {selector}")
+ out.extend(self._sources(rows, raw_timestamp=True).values())
+ return min(item.timestamp for item in out)
+ presence = one(xmrig_presence_query)
+ if presence.value < 0:
raise ValueError("invalid labelled XMRig presence source")
- presence = next(iter(presence_values.values()))
if presence.value > 0:
- presence_timestamps = []
- for raw_selector in (f'kube_pod_labels{{{label_selector}}}', node_pods_raw):
- rows = self._query("timestamp(" + raw_selector + ")", evaluation)
- if not rows:
- raise ValueError("missing raw XMRig membership timestamps")
- presence_timestamps.extend(self._sources(rows, raw_timestamp=True).values())
- presence = Source(presence.value, min(item.timestamp for item in presence_timestamps))
- # xmrig's raw selectors include cadvisor_raw, so cadvisor freshness is
+ # the membership stamps date both the presence count and the subtraction, so
+ # they are fetched once and reused rather than queried twice per evaluation
+ membership = oldest(labels_raw, node_pods_raw)
+ presence = Source(presence.value, membership)
+ # cadvisor joins the same membership selectors, so cadvisor freshness is
# verified exactly when its data enters the subtraction
- xmrig = source(
- xmrig_query,
- (cadvisor_raw, f'kube_pod_labels{{{label_selector}}}', node_pods_raw),
- )
+ xmrig = Source(one(xmrig_query).value, min(membership, oldest(cadvisor_raw)))
else:
# no labelled miner on this node: anchor presence freshness to pod-info stamps
- rows = self._query("timestamp(" + ksm_raw + ")", evaluation)
- if not rows:
- raise ValueError("missing raw pod-info timestamps")
- presence = Source(0, min(item.timestamp for item in self._sources(rows, raw_timestamp=True).values()))
+ presence = Source(0, oldest(ksm_raw))
xmrig = None
- return CPUObservation(source(host_query, host_raw), xmrig, presence)
+ host = Source(one(host_query).value, oldest(host_raw))
+ return CPUObservation(host, xmrig, presence)
class _HTTPTransport:
- def __init__(self, timeout=10, headers=None, context=None):
- self.timeout, self.headers, self.context = timeout, headers or {}, context
+ def __init__(self, timeout=HTTP_TIMEOUT_SECONDS):
+ self.timeout = timeout
def get(self, url, params):
- request = urllib.request.Request(url + "?" + urllib.parse.urlencode(params), headers=self.headers)
- with urllib.request.urlopen(request, timeout=self.timeout, context=self.context) as response:
+ request = urllib.request.Request(url + "?" + urllib.parse.urlencode(params))
+ with urllib.request.urlopen(request, timeout=self.timeout) as response:
return json.load(response)
class GuardController:
[eight lines unchanged)]
# conservative because it assumes full heat output until the last miner exits.
# Idle Composite never exceeded 62C over 7d on either node, so the trip does not
# false-fire, and it sits at or below 60C for 100%/90% of the time, so recovery is
# reachable rather than the permanent latch the old 60C/70C pair produced.
- self.policies = {"control-2": DwellPolicy(60, 64, 600, 120, MAX_SOURCE_GAP_SECONDS), "control-3": DwellPolicy(60, 64, 600, 120, MAX_SOURCE_GAP_SECONDS)}
- self.cpu_policy = DwellPolicy(50, 70, 600, 120, MAX_SOURCE_GAP_SECONDS)
+ # control-1 is keyed the same way on CPU headroom rather than temperature; SENSORS
+ # decides which source feeds which node, so the policy dict needs no special case.
+ self.policies = {
+ "control-1": DwellPolicy(50, 70, 600, 120, MAX_SOURCE_GAP_SECONDS),
+ "control-2": DwellPolicy(60, 64, 600, 120, MAX_SOURCE_GAP_SECONDS),
+ "control-3": DwellPolicy(60, 64, 600, 120, MAX_SOURCE_GAP_SECONDS),
+ }
self.ready = False
self.metrics = {
"evaluations": 0, "query_errors": {node: 0 for node in SENSORS},
"safe": {node: 0 for node in SENSORS},
- "nvme_temp_max": {node: 0.0 for node in SENSORS if node != "control-1"},
+ "nvme_temp_max": {node: 0.0 for node in SENSORS if SENSORS[node]},
"source_age_seconds": {node: 0.0 for node in SENSORS},
- "cpu_non_xmrig": 0.0,
+ "cpu_non_xmrig": {node: 0.0 for node in SENSORS if not SENSORS[node]},
+ "rank": {node: PRIORITY.index(node) for node in SENSORS},
}
self._last_source_stamps = {node: () for node in SENSORS}
def _new_source_set(self, node, sources):
stamps = tuple(source.timestamp.timestamp() for source in sources)
previous = self._last_source_stamps[node]
- if previous and len(previous) != len(stamps):
- raise ValueError("CPU source membership changed")
- if previous and len(previous) == len(stamps) and any(current - old > MAX_SOURCE_GAP_SECONDS for current, old in zip(stamps, previous)):
- raise ValueError("source gap exceeded maximum")
- if previous and len(previous) == len(stamps) and any(current <= old for current, old in zip(stamps, previous)):
- return False
+ if previous:
+ if len(previous) != len(stamps):
+ raise ValueError("source membership changed")
+ if any(current - old > MAX_SOURCE_GAP_SECONDS for current, old in zip(stamps, previous)):
+ raise ValueError("source gap exceeded maximum")
+ if any(current <= old for current, old in zip(stamps, previous)):
+ return False
self._last_source_stamps[node] = stamps
return True
def evaluate(self, evaluation=None):
evaluation = evaluation or self.wall_clock()
now = self.clock()
for node, sensors in SENSORS.items():
try:
- if node == "control-1":
+ # no sensors means a node with no visible NVMe (control-1, a VM): it is gated
+ # on CPU headroom instead. The dwell policy and metrics are keyed identically.
+ if sensors:
+ samples = self.telemetry.query_nvme(node, sensors, evaluation)
+ # trip on the hottest drive, date it by the newest sample it was read from
+ value, stamp = max(item.value for item in samples), max(item.timestamp for item in samples)
+ self.metrics["nvme_temp_max"][node] = value
+ else:
obs = self.telemetry.query_cpu(node, evaluation)
- sources = (obs.host, obs.presence) + ((obs.xmrig,) if obs.xmrig else ())
- if not all(_fresh(source.timestamp, evaluation, SOURCE_SAMPLE_MAX_AGE_SECONDS) for source in sources):
- raise ValueError("stale or future CPU source")
- if self._new_source_set(node, sources):
- safe = self.cpu_policy.observe(cpu_value(obs), min(source.timestamp for source in sources), now)
- else:
- safe = self.cpu_policy.safe
- self.metrics["cpu_non_xmrig"] = cpu_value(obs)
- self.metrics["source_age_seconds"][node] = max(0.0, evaluation.timestamp() - min(source.timestamp for source in sources).timestamp())
- else:
- samples = self.telemetry.query_nvme(node, sensors, evaluation)
- if not samples or not all(_fresh(sample.timestamp, evaluation, SOURCE_SAMPLE_MAX_AGE_SECONDS) for sample in samples):
- raise ValueError("stale or future NVMe source")
- if self._new_source_set(node, samples):
- safe = self.policies[node].observe(max(sample.value for sample in samples), max(sample.timestamp for sample in samples), now)
- else:
- safe = self.policies[node].safe
- self.metrics["nvme_temp_max"][node] = max(sample.value for sample in samples)
- self.metrics["source_age_seconds"][node] = max(0.0, evaluation.timestamp() - min(sample.timestamp for sample in samples).timestamp())
+ samples = (obs.host, obs.presence) + ((obs.xmrig,) if obs.xmrig else ())
+ value, stamp = cpu_value(obs), min(item.timestamp for item in samples)
+ self.metrics["cpu_non_xmrig"][node] = value
+ if not samples or not all(_fresh(item.timestamp, evaluation, SOURCE_SAMPLE_MAX_AGE_SECONDS) for item in samples):
+ raise ValueError("stale or future source")
+ policy = self.policies[node]
+ safe = policy.observe(value, stamp, now) if self._new_source_set(node, samples) else policy.safe
+ self.metrics["source_age_seconds"][node] = max(0.0, evaluation.timestamp() - min(item.timestamp for item in samples).timestamp())
self.metrics["safe"][node] = int(safe)
except Exception as exc:
# one line per failure, no traceback: the query text travels in the exception
logging.error(f"evaluation failed for {node}: {exc!r}")
self.metrics["query_errors"][node] += 1
- policy = self.cpu_policy if node == "control-1" else self.policies[node]
- policy.invalidate()
+ self.policies[node].invalidate()
self._last_source_stamps[node] = ()
self.metrics["safe"][node] = 0
self.metrics["source_age_seconds"][node] = float("nan")
- if node == "control-1":
- self.metrics["cpu_non_xmrig"] = float("nan")
+ if sensors:
+ self.metrics["nvme_temp_max"][node] = float("nan")
else:
- self.metrics["nvme_temp_max"][node] = float("nan")
+ self.metrics["cpu_non_xmrig"][node] = float("nan")
self.metrics["evaluations"] += 1
self.ready = True
return dict(self.metrics["safe"])
def render_metrics(controller):
m = controller.metrics
- lines = [
- f'xmrig_guard_evaluations_total {m["evaluations"]}',
- f'xmrig_guard_cpu_non_xmrig_percent {m["cpu_non_xmrig"]}',
- ]
+ lines = [f'xmrig_guard_evaluations_total {m["evaluations"]}']
for metric, values in (
("safe", m["safe"]),
("query_errors_total", m["query_errors"]),
("source_age_seconds", m["source_age_seconds"]),
("nvme_temp_max_celsius", m["nvme_temp_max"]),
+ ("cpu_non_xmrig_percent", m["cpu_non_xmrig"]),
+ ("rank", m["rank"]),
):
metric_name = "xmrig_guard_" + metric
lines.extend(f'{metric_name}{{node="{node}"}} {value}' for node, value in values.items())
return "\n".join(lines) + "\n"
[29 lines unchanged)]
if __name__ == "__main__":
main()
@@ (root level) @@
# helm.toolkit.fluxcd.io/v2/HelmRelease/web3/xmrig
! - one document removed:
- apiVersion: helm.toolkit.fluxcd.io/v2
- kind: HelmRelease
- metadata:
- name: xmrig
- namespace: web3
- labels:
- kustomize.toolkit.fluxcd.io/name: xmrig
- kustomize.toolkit.fluxcd.io/namespace: web3
- spec:
- chartRef:
- name: app-template
- kind: OCIRepository
- install:
- crds: CreateReplace
- interval: 1h
- rollback:
- cleanupOnFail: true
- upgrade:
- cleanupOnFail: true
- crds: CreateReplace
- remediation:
- remediateLastFailure: true
- retries: 2
- strategy:
- name: RemediateOnFailure
- values:
- controllers:
- xmrig:
- containers:
- app:
- resources:
- limits:
- cpu: 6
- hugepages-2Mi: 2368Mi
- memory: 128Mi
- requests:
- cpu: 1
- hugepages-2Mi: 2368Mi
- memory: 32Mi
- args:
- - "-o"
- - "p2pool.web3.svc.cluster.local:3333"
- - "-u"
- - x+5000
- - "-k"
- - "--coin=monero"
- - "--cpu-max-threads-hint=50"
- - "--huge-pages"
- - "--http-port=42000"
- - "--http-host=0.0.0.0"
- command:
- - /usr/bin/xmrig
- image:
- repository: djerfy/xmrig
- tag: "6.26.0@sha256:bd72a33a41d67a74956fd494ab7ee0d79efdf616db8f0829fb5186eec91152d3"
- securityContext:
- allowPrivilegeEscalation: false
- capabilities:
- add:
- - IPC_LOCK
- drop:
- - ALL
- readOnlyRootFilesystem: true
- pod:
- labels:
- app.kubernetes.io/component: thermal-guarded
- defaultPodOptions:
- affinity:
- nodeAffinity:
- preferredDuringSchedulingIgnoredDuringExecution:
- - preference:
- matchExpressions:
- - key: kubernetes.io/hostname
- operator: In
- values:
- - control-2
- - control-3
- weight: 100
- podAntiAffinity:
- requiredDuringSchedulingIgnoredDuringExecution:
- - labelSelector:
- matchExpressions:
- - key: app.kubernetes.io/name
- operator: In
- values:
- - xmrig
- topologyKey: kubernetes.io/hostname
- priorityClassName: low-priority-mining
- securityContext:
- fsGroup: 1000
- fsGroupChangePolicy: OnRootMismatch
- runAsGroup: 1000
- runAsNonRoot: true
- runAsUser: 1000
- terminationGracePeriodSeconds: 15
- persistence:
- hugepages:
- type: emptyDir
- medium: HugePages
- globalMounts:
- - path: /dev/hugepages
- tmp:
- type: emptyDir
- medium: Memory
- globalMounts:
- - path: /tmp
- service:
- xmrig:
- ports:
- http-api:
- port: 42000
@@ (root level) @@
# keda.sh/v1alpha1/ScaledObject/web3/xmrig
! - one document removed:
- apiVersion: keda.sh/v1alpha1
- kind: ScaledObject
- metadata:
- name: xmrig
- namespace: web3
- labels:
- app.kubernetes.io/component: autoscaler
- app.kubernetes.io/name: xmrig
- kustomize.toolkit.fluxcd.io/name: xmrig
- kustomize.toolkit.fluxcd.io/namespace: web3
- spec:
- advanced:
- horizontalPodAutoscalerConfig:
- behavior:
- scaleDown:
- policies:
- - type: Percent
- periodSeconds: 15
- value: 100
- stabilizationWindowSeconds: 0
- scaleUp:
- policies:
- - type: Percent
- periodSeconds: 60
- value: 100
- stabilizationWindowSeconds: 60
- cooldownPeriod: 0
- fallback:
- failureThreshold: 1
- replicas: 0
- maxReplicaCount: 3
- minReplicaCount: 0
- pollingInterval: 60
- scaleTargetRef:
- apiVersion: apps/v1
- kind: Deployment
- name: xmrig
- triggers:
- - type: prometheus
- metadata:
- activationThreshold: "25"
- ignoreNullValues: "false"
- query: |
- max(
-
- clamp_max(
- clamp_min(-avg_over_time(hass_sensor_power_w{entity="sensor.p1_meter_power",job="homeassistant"}[15m]), 0),
- 150
- )
- * on()
- (
- min(xmrig_guard_safe{node=~"control-[123]"})
- * (count(xmrig_guard_safe{node=~"control-[123]"}) == bool 3)
- * (min(timestamp(xmrig_guard_safe{node=~"control-[123]"})) >= bool (time() - 120))
- )
- ) or vector(0)
- serverAddress: "http://vmsingle-victoria-metrics.observability.svc.cluster.local:8428"
- threshold: "50"
@@ (root level) @@
# fluxcd.controlplane.io/v1/ResourceSet/web3/xmrig
! + one document added:
+ apiVersion: fluxcd.controlplane.io/v1
+ kind: ResourceSet
+ metadata:
+ name: xmrig
+ namespace: web3
+ labels:
+ kustomize.toolkit.fluxcd.io/name: xmrig
+ kustomize.toolkit.fluxcd.io/namespace: web3
+ spec:
+ resources:
+ - apiVersion: helm.toolkit.fluxcd.io/v2
+ kind: HelmRelease
+ metadata:
+ name: "xmrig-<< inputs.node >>"
+ namespace: web3
+ spec:
+ chartRef:
+ name: app-template
+ kind: OCIRepository
+ interval: 1h
+ values:
+ controllers:
+ xmrig-<< inputs.node >>:
+ containers:
+ app:
+ resources:
+ limits:
+ cpu: 6
+ hugepages-2Mi: 2368Mi
+ memory: 128Mi
+ requests:
+ cpu: 1
+ hugepages-2Mi: 2368Mi
+ memory: 32Mi
+ args:
+ - "-o"
+ - "p2pool.web3.svc.cluster.local:3333"
+ - "-u"
+ - x+5000
+ - "-k"
+ - "--coin=monero"
+ - "--cpu-max-threads-hint=50"
+ - "--huge-pages"
+ - "--http-port=42000"
+ - "--http-host=0.0.0.0"
+ command:
+ - /usr/bin/xmrig
+ image:
+ repository: djerfy/xmrig
+ tag: "6.26.0@sha256:bd72a33a41d67a74956fd494ab7ee0d79efdf616db8f0829fb5186eec91152d3"
+ securityContext:
+ allowPrivilegeEscalation: false
+ capabilities:
+ add:
+ - IPC_LOCK
+ drop:
+ - ALL
+ readOnlyRootFilesystem: true
+ pod:
+ labels:
+ app.kubernetes.io/component: thermal-guarded
+ defaultPodOptions:
+ nodeSelector:
+ kubernetes.io/hostname: "<< inputs.node >>"
+ priorityClassName: low-priority-mining
+ securityContext:
+ fsGroup: 1000
+ fsGroupChangePolicy: OnRootMismatch
+ runAsGroup: 1000
+ runAsNonRoot: true
+ runAsUser: 1000
+ terminationGracePeriodSeconds: 15
+ persistence:
+ hugepages:
+ type: emptyDir
+ medium: HugePages
+ globalMounts:
+ - path: /dev/hugepages
+ tmp:
+ type: emptyDir
+ medium: Memory
+ globalMounts:
+ - path: /tmp
+ - apiVersion: keda.sh/v1alpha1
+ kind: ScaledObject
+ metadata:
+ name: "xmrig-<< inputs.node >>"
+ namespace: web3
+ labels:
+ app.kubernetes.io/component: autoscaler
+ app.kubernetes.io/name: "xmrig-<< inputs.node >>"
+ spec:
+ advanced:
+ horizontalPodAutoscalerConfig:
+ behavior:
+ scaleDown:
+ policies:
+ - type: Percent
+ periodSeconds: 15
+ value: 100
+ stabilizationWindowSeconds: 0
+ scaleUp:
+ policies:
+ - type: Percent
+ periodSeconds: 60
+ value: 100
+ stabilizationWindowSeconds: 60
+ cooldownPeriod: 0
+ fallback:
+ failureThreshold: 1
+ replicas: 0
+ maxReplicaCount: 1
+ minReplicaCount: 0
+ pollingInterval: 60
+ scaleTargetRef:
+ apiVersion: apps/v1
+ kind: Deployment
+ name: "xmrig-<< inputs.node >>"
+ triggers:
+ - type: prometheus
+ metadata:
+ activationThreshold: "25"
+ ignoreNullValues: "false"
+ query: |
+ (
+
+ clamp_min(
+ clamp_max(
+ clamp_min(-avg_over_time(hass_sensor_power_w{entity="sensor.p1_meter_power",job="homeassistant"}[15m]), 0),
+ 150
+ )
+ - 50 * (
+ count(
+ (xmrig_guard_safe == 1)
+ and on(node) (xmrig_guard_rank < scalar(xmrig_guard_rank{node="<< inputs.node >>"}))
+ ) or vector(0)
+ ),
+ 0
+ )
+ * on()
+ (
+ min(xmrig_guard_safe{node="<< inputs.node >>"})
+ * (count(xmrig_guard_safe{node="<< inputs.node >>"}) == bool 1)
+ * (min(timestamp(xmrig_guard_safe{node="<< inputs.node >>"})) >= bool (time() - 120))
+ )
+ ) or vector(0)
+ serverAddress: "http://vmsingle-victoria-metrics.observability.svc.cluster.local:8428"
+ threshold: "50"
+ inputs:
+ - node: control-1
+ - node: control-2
+ - node: control-3
@@ (root level) @@
# v1/Service/web3/xmrig
! + one document added:
+ apiVersion: v1
+ kind: Service
+ metadata:
+ name: xmrig
+ namespace: web3
+ labels:
+ kustomize.toolkit.fluxcd.io/name: xmrig
+ kustomize.toolkit.fluxcd.io/namespace: web3
+ spec:
+ ports:
+ - name: http-api
+ port: 42000
+ targetPort: 42000
+ selector:
+ app.kubernetes.io/component: thermal-guarded
@@ (root level) @@
# keda.sh/v1alpha1/ScaledObject/web3/xmrig-control-1
! + one document added:
+ apiVersion: keda.sh/v1alpha1
+ kind: ScaledObject
+ metadata:
+ name: xmrig-control-1
+ namespace: web3
+ labels:
+ app.kubernetes.io/component: autoscaler
+ app.kubernetes.io/name: xmrig-control-1
+ resourceset.fluxcd.controlplane.io/name: xmrig
+ resourceset.fluxcd.controlplane.io/namespace: web3
+ spec:
+ advanced:
+ horizontalPodAutoscalerConfig:
+ behavior:
+ scaleDown:
+ policies:
+ - type: Percent
+ periodSeconds: 15
+ value: 100
+ stabilizationWindowSeconds: 0
+ scaleUp:
+ policies:
+ - type: Percent
+ periodSeconds: 60
+ value: 100
+ stabilizationWindowSeconds: 60
+ cooldownPeriod: 0
+ fallback:
+ failureThreshold: 1
+ replicas: 0
+ maxReplicaCount: 1
+ minReplicaCount: 0
+ pollingInterval: 60
+ scaleTargetRef:
+ apiVersion: apps/v1
+ kind: Deployment
+ name: xmrig-control-1
+ triggers:
+ - type: prometheus
+ metadata:
+ activationThreshold: "25"
+ ignoreNullValues: "false"
+ query: |
+ (
+
+ clamp_min(
+ clamp_max(
+ clamp_min(-avg_over_time(hass_sensor_power_w{entity="sensor.p1_meter_power",job="homeassistant"}[15m]), 0),
+ 150
+ )
+ - 50 * (
+ count(
+ (xmrig_guard_safe == 1)
+ and on(node) (xmrig_guard_rank < scalar(xmrig_guard_rank{node="control-1"}))
+ ) or vector(0)
+ ),
+ 0
+ )
+ * on()
+ (
+ min(xmrig_guard_safe{node="control-1"})
+ * (count(xmrig_guard_safe{node="control-1"}) == bool 1)
+ * (min(timestamp(xmrig_guard_safe{node="control-1"})) >= bool (time() - 120))
+ )
+ ) or vector(0)
+ serverAddress: "http://vmsingle-victoria-metrics.observability.svc.cluster.local:8428"
+ threshold: "50"
@@ (root level) @@
# keda.sh/v1alpha1/ScaledObject/web3/xmrig-control-2
! + one document added:
+ apiVersion: keda.sh/v1alpha1
+ kind: ScaledObject
+ metadata:
+ name: xmrig-control-2
+ namespace: web3
+ labels:
+ app.kubernetes.io/component: autoscaler
+ app.kubernetes.io/name: xmrig-control-2
+ resourceset.fluxcd.controlplane.io/name: xmrig
+ resourceset.fluxcd.controlplane.io/namespace: web3
+ spec:
+ advanced:
+ horizontalPodAutoscalerConfig:
+ behavior:
+ scaleDown:
+ policies:
+ - type: Percent
+ periodSeconds: 15
+ value: 100
+ stabilizationWindowSeconds: 0
+ scaleUp:
+ policies:
+ - type: Percent
+ periodSeconds: 60
+ value: 100
+ stabilizationWindowSeconds: 60
+ cooldownPeriod: 0
+ fallback:
+ failureThreshold: 1
+ replicas: 0
+ maxReplicaCount: 1
+ minReplicaCount: 0
+ pollingInterval: 60
+ scaleTargetRef:
+ apiVersion: apps/v1
+ kind: Deployment
+ name: xmrig-control-2
+ triggers:
+ - type: prometheus
+ metadata:
+ activationThreshold: "25"
+ ignoreNullValues: "false"
+ query: |
+ (
+
+ clamp_min(
+ clamp_max(
+ clamp_min(-avg_over_time(hass_sensor_power_w{entity="sensor.p1_meter_power",job="homeassistant"}[15m]), 0),
+ 150
+ )
+ - 50 * (
+ count(
+ (xmrig_guard_safe == 1)
+ and on(node) (xmrig_guard_rank < scalar(xmrig_guard_rank{node="control-2"}))
+ ) or vector(0)
+ ),
+ 0
+ )
+ * on()
+ (
+ min(xmrig_guard_safe{node="control-2"})
+ * (count(xmrig_guard_safe{node="control-2"}) == bool 1)
+ * (min(timestamp(xmrig_guard_safe{node="control-2"})) >= bool (time() - 120))
+ )
+ ) or vector(0)
+ serverAddress: "http://vmsingle-victoria-metrics.observability.svc.cluster.local:8428"
+ threshold: "50"
@@ (root level) @@
# keda.sh/v1alpha1/ScaledObject/web3/xmrig-control-3
! + one document added:
+ apiVersion: keda.sh/v1alpha1
+ kind: ScaledObject
+ metadata:
+ name: xmrig-control-3
+ namespace: web3
+ labels:
+ app.kubernetes.io/component: autoscaler
+ app.kubernetes.io/name: xmrig-control-3
+ resourceset.fluxcd.controlplane.io/name: xmrig
+ resourceset.fluxcd.controlplane.io/namespace: web3
+ spec:
+ advanced:
+ horizontalPodAutoscalerConfig:
+ behavior:
+ scaleDown:
+ policies:
+ - type: Percent
+ periodSeconds: 15
+ value: 100
+ stabilizationWindowSeconds: 0
+ scaleUp:
+ policies:
+ - type: Percent
+ periodSeconds: 60
+ value: 100
+ stabilizationWindowSeconds: 60
+ cooldownPeriod: 0
+ fallback:
+ failureThreshold: 1
+ replicas: 0
+ maxReplicaCount: 1
+ minReplicaCount: 0
+ pollingInterval: 60
+ scaleTargetRef:
+ apiVersion: apps/v1
+ kind: Deployment
+ name: xmrig-control-3
+ triggers:
+ - type: prometheus
+ metadata:
+ activationThreshold: "25"
+ ignoreNullValues: "false"
+ query: |
+ (
+
+ clamp_min(
+ clamp_max(
+ clamp_min(-avg_over_time(hass_sensor_power_w{entity="sensor.p1_meter_power",job="homeassistant"}[15m]), 0),
+ 150
+ )
+ - 50 * (
+ count(
+ (xmrig_guard_safe == 1)
+ and on(node) (xmrig_guard_rank < scalar(xmrig_guard_rank{node="control-3"}))
+ ) or vector(0)
+ ),
+ 0
+ )
+ * on()
+ (
+ min(xmrig_guard_safe{node="control-3"})
+ * (count(xmrig_guard_safe{node="control-3"}) == bool 1)
+ * (min(timestamp(xmrig_guard_safe{node="control-3"})) >= bool (time() - 120))
+ )
+ ) or vector(0)
+ serverAddress: "http://vmsingle-victoria-metrics.observability.svc.cluster.local:8428"
+ threshold: "50"
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kubernetes/apps/web3/monero/xmrig/resourceset.yaml`:
- Around line 46-50: Update the endpoint arguments in the resource set
configuration to use the repository’s `${SECRET_DOMAIN}` parameter instead of
hard-coded service domains, including both the p2pool and
vmsingle-victoria-metrics references. If `${SECRET_DOMAIN}` cannot support
in-cluster resolution, clarify the endpoint policy rather than adding a
manifest-specific exception.
In `@kubernetes/apps/web3/xmrig-guard/app/prometheusrule.yaml`:
- Around line 18-20: Update the PromQL expression in the alert predicate to
mirror KEDA’s complete gate: before the label_replace around xmrig_guard_safe,
retain only nodes with exactly one safe series and a sample fresher than 120
seconds. Then apply label_replace to this filtered result so stale or duplicate
safe=1 series cannot suppress the bypass alert.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 376d10d5-34f0-443d-8f38-884be7a65484
📒 Files selected for processing (7)
docs/xmrig-per-node-gating-plan.mdkubernetes/apps/web3/monero/xmrig/helmrelease.yamlkubernetes/apps/web3/monero/xmrig/kustomization.yamlkubernetes/apps/web3/monero/xmrig/resourceset.yamlkubernetes/apps/web3/monero/xmrig/scaledobject.yamlkubernetes/apps/web3/monero/xmrig/service.yamlkubernetes/apps/web3/xmrig-guard/app/prometheusrule.yaml
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
Tanguille/LLMKube(auto-detected)Tanguille/toolhive(auto-detected)Tanguille/2x-R9700-RDNA4-GFX1201-sglang-inference(auto-detected)
💤 Files with no reviewable changes (2)
- kubernetes/apps/web3/monero/xmrig/scaledobject.yaml
- kubernetes/apps/web3/monero/xmrig/helmrelease.yaml
…in the bypass alert A duplicate or stale safe=1 closes KEDA's gate but was still subtracted by the alert, so a wedged HPA leaving miners running during exactly those conditions stayed silent. Apply the same count==1 and 120s freshness filters before the label_replace.
…double-fetching stamps SENSORS already says which nodes have NVMe, so control-1's CPU gate no longer needs a node-name branch: its DwellPolicy joins the same dict and evaluate() tests `if not sensors`. That removes the special case from the value path and from the error handler, where a missed branch would have left the policy un-invalidated on failure. query_cpu fetched the kube_pod_labels and kube_pod_info timestamps twice per evaluation, once to date the presence count and again inside the subtraction: 9 round trips where 7 suffice. A test pins the count. Also narrows XmrigGuardLatchedUnsafe to control-2/3. It joins on an NVMe metric control-1 never emits, so the wider regex claimed coverage the query could not deliver.
The previous commit claimed SENSORS decides which source feeds which node, then left `node != "control-1"` in the nvme_temp_max comprehension and kept cpu_non_xmrig as a bare float. A second CPU-gated node would have silently overwritten that value each evaluation and the error path would have NaN'd a metric belonging to a different node. Both are now keyed like the rest, so xmrig_guard_cpu_non_xmrig_percent gains a node label (nothing in the repo selects on it). Also unpicks a nested ternary that encoded one bit three ways, returns the oldest stamp from the timestamp helper instead of making all four callers reduce it, drops the node matchers from XmrigGuardLatchedUnsafe since only NVMe-gated nodes emit the metric the join needs, and stops tracking __pycache__ committed by a test run.
…vation steps The 25/75/125W activation thresholds encoded "n miners need n*50W" as three literals that had to stay consistent with each other and with threshold: 50, and they reserved watts for a node whether or not it could actually mine: with control-1 CPU-saturated, control-2 still waited for 75W before starting. Each trigger now subtracts 50W per *safe* node that outranks it and takes the remainder, with a uniform 25W activation. Ranking moves to the guard as xmrig_guard_rank, driven by a PRIORITY tuple, so re-prioritising is a guard change rather than a manifest edit and a headroom-derived ranking later replaces that tuple alone. Worth 0.6 replica-hours a month against the static stagger; export here is bimodal so the ranking rarely binds. Taken for deleting the magic numbers.
AI Automated ReviewAnalysis engine: qwen-3.6-fast@http://litellm.ai.svc.cluster.local/v1 (openai) Recommendation: Approve with minor verification notes. This PR replaces a single global Required Checks
Change-by-Change Findings
Standards Compliance
Unknowns or Needs Verification
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| - node: control-3 | ||
| resources: | ||
| - apiVersion: helm.toolkit.fluxcd.io/v2 | ||
| kind: HelmRelease |
There was a problem hiding this comment.
Info (question): Confirm that the flux-operator version installed in the cluster supports the fluxcd.controlplane.io/v1 ResourceSet API.
Automated finding from AI PR review.
| fsGroup: 1000 | ||
| fsGroupChangePolicy: OnRootMismatch | ||
| - apiVersion: keda.sh/v1alpha1 | ||
| kind: ScaledObject |
There was a problem hiding this comment.
Info: Verify that all three control nodes have sufficient hugepages-2Mi available (2368Mi each) to support concurrent mining if thermal conditions allow.
Automated finding from AI PR review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One hot NVMe on control-3 stopped mining on control-1 too, because the KEDA trigger gated everything on
min(xmrig_guard_safe)across all three nodes.Measured, last 7d (2026-07-23 to 07-30)
Capture was 103-107% before the guard went enforcing (~07-14), so the solar half is fine. The drives now idle in the 60-64C band year round (5-week weekly averages 59.9 / 61.3 / 64.3 / 63.7 / 62.4 C), on top of both thresholds.
Change
One Deployment per node via a flux-operator
ResourceSet, each pinned with anodeSelectorand scaled by a ScaledObject reading only that node's guard verdict. Activation staggered 25/75/125W at ~50W per miner; without the stagger 50W of export lights all three. Stagger order follows measured thermal availability, which inverts the old "prefer c2/c3, more power efficient" preference.A single Deployment cannot express this: the HPA sheds an arbitrary pod, so nothing guaranteed the pod on the node that just tripped was the one to die. Guard-written node labels or taints were rejected (RBAC on a pod deliberately running with
automountServiceAccountToken: false, plus NoExecute eviction semantics), as were descheduler and priority/preemption.Guard thresholds unchanged at trip 64C / recover 60C. Modelled at 73 replica-hours against 8.7 delivered.
Verification
kubectl kustomize, both dirskubectl apply --dry-run=serverhelm templateapp-template 5.0.1xmrig-control-N, nodeSelector applied--dry-run=extractresourceset.yamlis a packageFile withdjerfy/xmrigat the pinned digest, detected by both helm-values and kubernetes, so no config change neededKnown and accepted
xmrigService is currently Helm-owned; when Flux prunes the old HelmRelease, helm-controller deletes it and the Kustomization recreates it within one reconcile. Brief gap on a dashboard panel that already tolerates the miner being down, with miners at 0 anyway.maxReplicaCount: 1the HPAbehaviorblock should be inert, since both transitions are KEDA activation. Kept rather than deleted on a safety path: if that reasoning is wrong the HPA default is a 300s scaleDown stabilization, which alone blows the thermal budget. A measured trip settles it.The pod label
app.kubernetes.io/component: thermal-guardedis preserved on all three Deployments (verified in the rendered chart output), so the guard's control-1 CPU accounting, which selects miner pods by that label, keeps working.Static stagger vs dynamic allocation
The 25/75/125W thresholds are static, so a miner's slot is not reallocated when a higher-ranked node is unsafe. Measured cost of that against perfect allocation: 0.02 replica-hours over 7d, 0.6 over 30d. Export at this site is bimodal (44.6h of the 47.5h above 25W last week were at or above the 150W cap), so the partial-export window where the stagger binds is 4-12% of solar time in every 30-day period on record back to January. Nothing stalls on control-1 at full export: control-2 clears 75W and control-3 clears 125W regardless of control-1's verdict. Left static; a dynamic form is only worth it if the export curve flattens into the 25-125W band, e.g. after adding a battery or EV charging.
Steps 2 (response-time levers) and 3 (trip 65C, recovery 62C/300s) are written up with measurement gates in
docs/xmrig-per-node-gating-plan.mdand are not in this PR.Summary by CodeRabbit
New Features
Documentation