diff --git a/docs/xmrig-per-node-gating-plan.md b/docs/xmrig-per-node-gating-plan.md deleted file mode 100644 index 57214677d0..0000000000 --- a/docs/xmrig-per-node-gating-plan.md +++ /dev/null @@ -1,164 +0,0 @@ -# xmrig per-node thermal gating - -**Branch**: `feat/xmrig-per-node-gating` -**Worktree**: `.claude/worktrees/xmrig-per-node-gating` - -## Problem - -The KEDA trigger gates all mining on `min(xmrig_guard_safe)` across all three nodes, so one -hot NVMe stops the whole fleet. Measured over 7d (2026-07-23 → 07-30): - -| metric | value | -|---|---| -| solar available (>=25W export) | 47.5h | -| desired replica-hours | 137 | -| delivered replica-hours | 8.7 (6%) | -| gate open | 25% of wall time | -| per-node safe fraction | control-1 96%, control-2 60%, control-3 30% | -| solar hours lost per node | c1 3.8h, c2 29.1h, c3 40.0h | -| NVMe above the 64C trip during solar hours | c2 28.9h, c3 35.3h | - -Before the guard went enforcing (~2026-07-14) capture was 103-107%, so the solar half of the -system is fine. The drives now idle in the 60-64C band all year (5-week weekly averages 59.9, -61.3, 64.3, 63.7, 62.4 C), i.e. on top of both thresholds, which is why the gate is a coin flip. - -Modelled replica-hours for last week's actual solar and guard data (hysteresis-free upper -bound; the same model reproduces today's design at 8.6 vs 8.7 actual): - -| design | replica-h/7d | -|---|---| -| today | 8.7 | -| per-node split, trip 64C | 73 | -| + trip 65C | 82 | -| + trip 66C | 90 | - -The split is the 8x. Threshold tuning is worth roughly 5h per node per degree on top. - -## Why one Deployment cannot do this - -The HPA sheds an arbitrary pod, so nothing guarantees the pod on the node that just tripped is -the one that dies. Making placement follow the guard needs the guard to write node labels or -taints: RBAC on a pod that deliberately runs with `automountServiceAccountToken: false`, plus -`NoExecute` semantics on a control-plane node, plus new guard code. Per-node autonomy in -Kubernetes is spelled "one workload per node". Rejected alternatives: descheduler (new -component, unbounded eviction latency), `IgnoredDuringExecution` affinity on a guard-written -label (does not evict the pod on the node that just went hot, so it is unsound on the safety -path), priority/preemption (nothing to preempt with). - -Three copies are expressed as a flux-operator `ResourceSet` (already installed on the cluster, -first use in this repo) rather than three files or three kustomize overlays: - -| approach | lines | delta vs 170 today | -|---|---|---| -| naive 3x copy | ~510 | +340 | -| kustomize overlays (base + 3) | ~250 | +80 | -| ResourceSet | ~180 | ~+10 | - -A kustomize *component* is the wrong tool: it injects resources into a kustomization, it does -not instantiate N copies of a base. - -## Step 1 — the split, thresholds unchanged - -Delivers the 8x on its own with no change to drive safety. Guard thresholds stay at trip 64C / -recover 60C. - -- `kubernetes/apps/web3/monero/xmrig/resourceset.yaml` replaces `helmrelease.yaml` and - `scaledobject.yaml`. Inputs are the three nodes plus a staggered activation threshold. -- Each trigger subtracts 50W per *safe* node that outranks it and takes what is left, with a - uniform 25W activation. Without that subtraction, three ScaledObjects reading one global - export figure would all start at 50W. Unlike a fixed activation threshold it does not - reserve watts for a node that is unavailable: if control-1 is CPU-saturated, control-2 - takes the first slot rather than waiting for 75W. -- Ranking is the guard's `xmrig_guard_rank`, ordered by measured thermal availability - (c1 96% → c3 30%), which inverts the old "prefer control-2/control-3, more power efficient" - preference. It lives in the guard's `PRIORITY` tuple, so re-prioritising needs no manifest - edit, and a future headroom-derived ranking replaces that tuple alone. -- Measured worth of dynamic over the static 25/75/125 stagger it replaced: 0.02 replica-hours - over 7d, 0.6 over 30d. The reason it is small is that export here is bimodal (44.6h of the - 47.5h above 25W last week sat at the 150W cap), so the ranking rarely binds. It was adopted - for removing three hardcoded numbers that had to stay consistent with `threshold: 50`, not - for the throughput. -- Each HelmRelease pins its Deployment to one node with a `nodeSelector`. - `podAntiAffinity` goes away: one replica per node is now structural. -- The gate query per node collapses from the `min() x count()==3 x freshness` cardinality dance - to the same shape against a single series. -- `kubernetes/apps/web3/monero/xmrig/service.yaml` keeps one stable `xmrig` Service across the - three Deployments, selecting the existing `app.kubernetes.io/component: thermal-guarded` pod - label, so the dashboard's `http://xmrig.web3.svc.cluster.local:42000` keeps working. -- `XmrigGuardEnforcementBypassed` becomes per-node, joining guard verdict to the matching - Deployment by node. Its old global `min()` form would false-fire constantly once one node - can mine while another is unsafe. - -Known accepted behaviours: - -- If the guard's rank series is missing for a node, that node's `scalar()` yields NaN, every - rank comparison is false, and it behaves as top priority. That fails open on *allocation* - only (worst case one extra miner, 50W over-drawn); the safety gate is a separate factor in - the same query and still fails closed. -- The dashboard polls one Service and gets whichever pod answers, so it under-reports total - hashrate when more than one miner runs. Pre-existing, unchanged by this step. -- With `maxReplicaCount: 1` the HPA `behavior` block is probably inert (0->1 and 1->0 are KEDA - activation transitions, not HPA scaling). It is kept unchanged rather than deleted on a safety - path; revisit once step 1 measurement confirms real drain latency. - -**Measure for ~5 days**: delivered replica-hours vs the 73h model; per-node safe fractions; -no `XmrigGuardEnforcementBypassed` false-fires; force a single-node trip and confirm only that -node drains, inside the timing budget. - -**Renovate coverage: verified, no config change needed.** A local `--dry-run=extract` lists -`resourceset.yaml` as a packageFile carrying `djerfy/xmrig` at the pinned digest, detected -twice by `helm-values` and `kubernetes` exactly as the `.renovaterc.json5` comment describes -for plain HelmReleases. - -## Step 2 — response-time levers - -No threshold change. Buys back the time that step 3 spends. - -- guard `EVALUATION_INTERVAL_SECONDS` 60 -> 30 (node-exporter scrapes at 20s, verified, so this - is real time and not just polling a stale sample) -- trip dwell 120s -> 60s -- KEDA `pollingInterval` 60 -> 30 on all three ScaledObjects -- new zero-dwell panic trip at 68C in `DwellPolicy.observe()`, plus a critical alert for - `nvme_temp > 68 and replicas > 0` (the panic line crossed with miners still running means the - fast path failed) -- `HTTP_TIMEOUT_SECONDS` 10 -> 5, and a deadline-based sleep. One evaluation now issues 13 - queries (7 for control-1, 3 per NVMe node), so a hung VictoriaMetrics blocks up to 130s of - serial wait against a 120s freshness budget: the guard fails closed on every node plus a - stale scrape, later and more expensively than a faster timeout would. The loop also sleeps - a flat 60s *after* the work, so the true period drifts by the evaluation duration; - `time.sleep(max(0, deadline - time.monotonic()))` fixes both. Two lines, no new concepts. - -**Measure**: observed trip-to-drain latency on a real or forced trip, target < 2.5 min. - -## Step 3 — thresholds - -- trip 64C -> 65C. Chain at 65C: scrape + 30s eval + 60s dwell + 30s poll + drain ~= 3.5 min = - 3.85C at the observed 1.1C/min, peak ~68.9C against the 70C rating: the same ~1.1C margin the - 64C design has today. 66C is defensible on the same math but leaves ~0.9C; take it only if - step 2's measured latency beats the budget. -- recovery 60C -> 62C, recovery dwell 600s -> 300s. Recovers the ~5h/week currently spent - latched closed while the drive is already cool, and the ~11h/week stuck in the 60-64C dead - band. Note the code comment claiming "idle Composite never exceeded 62C over 7d" is stale: - the drives now idle above 64C on hot afternoons. This does not unlock summer afternoons, and - no threshold can - it unlocks mornings and winter. -- Anti-flap at trip 65 / recover 62: the miner must climb 3C at <=1.1C/min (>=2.7 min) plus the - 60s dwell, then cool back to 62C plus 300s dwell. Full cycle >= 12-15 min. That is duty - cycling sized by the drive's thermal mass, not flapping. -- `XmrigGuardLatchedUnsafe` hardcodes 60C twice; update to 62. -- Rewrite the timing-budget comments in `scaledobject.yaml` and `controller.py`. Every number in - them changes and they are the design record. - -**Measure for a week**: weekly max Composite stays < 70C; cycle period >= 10 min; latched-cool -hours ~= 0; transition count; control-3 hugepage allocation health under repeated 2368Mi -alloc/free on a node that runs chronically near 95% memory; no NPD XFS/NVMe conditions. - -## Process Instructions - -- After completing each step, update the plan with the current status. -- Pause for user confirmation before proceeding to next step. -- Suggest the prompt for continuing to the next step. -- After the last step, make a final documentation pass. Once the contents of the plan have been - consolidated into existing documentation, the plan file can be removed. If there is no - relevant existing documentation, the plan should be reworked into a reference document. - -**Important**: Every prompt should verify the branch and worktree before doing any work. diff --git a/docs/xmrig-solar-mining.md b/docs/xmrig-solar-mining.md new file mode 100644 index 0000000000..b372d77271 --- /dev/null +++ b/docs/xmrig-solar-mining.md @@ -0,0 +1,140 @@ +# xmrig solar-gated mining + +Monero mining that runs only on exported solar power, and only on nodes whose hardware has +headroom. One miner per control-plane node, each gated independently. + +## Shape + +`kubernetes/apps/web3/monero/xmrig/resourceset.yaml` is a flux-operator `ResourceSet` +templated over three node inputs. Each input produces one HelmRelease (a Deployment pinned +with `nodeSelector`) and one ScaledObject. + +One Deployment with three replicas cannot express this: the HPA sheds an arbitrary pod, so +nothing guarantees the pod on the node that just tripped is the one that dies. Per-node +autonomy in Kubernetes is spelled "one workload per node". Rejected alternatives: guard-written +node labels or taints (needs RBAC on a pod that deliberately runs with +`automountServiceAccountToken: false`, plus `NoExecute` semantics on a control-plane node), +descheduler (new component, unbounded eviction latency), priority/preemption (nothing to +preempt with). + +`service.yaml` keeps one stable `xmrig` Service across the three Deployments, selecting +`app.kubernetes.io/component: thermal-guarded`, so the dashboard's +`http://xmrig.web3.svc.cluster.local:42000` resolves regardless of which node is mining. + +## The guard + +`kubernetes/apps/web3/xmrig-guard/app/resources/controller.py` polls VictoriaMetrics every 30s +and exports `xmrig_guard_safe{node}` plus `xmrig_guard_rank{node}`. It treats telemetry as +untrusted: a complete set of fresh samples is required before a node can be safe, and every +failure path fails closed. Policy is code, so changing a threshold requires a reviewed diff. + +`SENSORS` decides which source gates which node. control-1 is a VM with no visible NVMe, so it +is gated on CPU headroom; the bare-metal mini PCs are gated on NVMe Composite temperature. + +| node | source | recover | trip | recovery dwell | trip dwell | panic | +|---|---|---|---|---|---|---| +| control-1 | non-xmrig CPU % | 50 | 70 | 600s | 120s | — | +| control-2 | NVMe Composite C | 62 | 65 | 300s | 60s | 68C | +| control-3 | NVMe Composite C | 62 | 65 | 300s | 60s | 68C | + +Only `temp1` (Composite) is read. `temp2`-`temp4` are internal die sensors running ~9C hotter +with no comparable rating, so including them gated a Composite threshold against the wrong value. + +The CPU gate has no panic limit: a busy CPU carries no equivalent of a drive's absolute rating. + +### Timing budget + +The 65C trip is sized against the 70C drive rating and a measured ~1.1C/min rise under load: + +| leg | seconds | +|---|---| +| evaluation interval (sample the crossing) | 30 | +| trip dwell | 60 | +| KEDA `pollingInterval` | 30 | +| drain (`terminationGracePeriodSeconds`) | 15 | +| **total** | **135** | + +135s is 2.5C of rise, peaking near 67.5C, leaving 2.5C of margin. The 68C panic trip skips the +dwell entirely (75s, ~69.4C) and bounds the case where the ramp beats the dwell. + +Changing any leg invalidates the trip threshold. They move together. + +## Allocation + +Each ScaledObject reads its own node's verdict and takes only the export left after every +*safe* node that outranks it has taken its 50W. Without that subtraction, three ScaledObjects +reading one global export figure would all activate at 50W. + +Ranking is `xmrig_guard_rank`, derived from the `PRIORITY` tuple in the controller, so +re-prioritising needs no manifest edit. Order follows measured availability: over 7d, +control-1 97.8%, control-2 44.7%, control-3 31.5% (pre-step-3 numbers). + +Unlike a fixed activation threshold, the subtraction skips an unavailable node instead of +reserving watts for it: if control-1 is CPU-saturated, control-2 takes the first slot rather +than waiting for 75W. Measured worth of dynamic over the static 25/75/125 stagger it replaced +is small (0.6 replica-hours over 30d) because export here is bimodal; it was adopted for +removing three hardcoded numbers that had to stay consistent with `threshold: 50`. + +The gate itself is a separate factor in the same query: cardinality must be exactly 1 and the +sample must be under 120s old, so a duplicate or stale `safe=1` closes the gate. + +## Measured + +Per-node gating, first 24h after rollout (2026-07-31): + +| metric | before | after | +|---|---|---| +| capture vs solar hours ≥25W | 6% | 51% | +| peak concurrent miners | 1 | 3 | + +Replaying 7d of real telemetry through `DwellPolicy` (`recover 60/trip 64/600s/120s` → +`62/65/300s/60s` + 68C panic): 65.4h → 84.2h of delivered replica-hours, +29%. control-2 44.7% +→ 62.3% safe, control-3 31.5% → 40.4%. + +That replay is an upper bound: it applies a more permissive policy to temperatures recorded +under the restrictive one, and more mining means more heat. It matched actuals closely on the +*current* policy (control-2 20.0% modelled vs 20.4% actual, control-3 8.5% vs 8.5%), which +validates the harness but not the counterfactual. + +Threshold tuning is worth roughly 5h per node per degree. The per-node split was the 8x; the +thresholds are the next ~30%. + +## Alerts + +| alert | fires when | +|---|---| +| `XmrigGuardEnforcementBypassed` | a miner runs while its node's gate should be shut (KEDA down, HPA wedged, manual scale). Mirrors KEDA's cardinality and freshness gate exactly, so a stale `safe=1` cannot suppress it. | +| `XmrigGuardThermalPanic` | a drive is above 68C with a miner still running: the zero-dwell fast path failed | +| `XmrigGuardAbsent` | guard signal missing or wrong cardinality (gate closed, mining silently off) | +| `XmrigGuardLatchedUnsafe` | gated unsafe 6h while the drive stayed within the 62C recovery band, i.e. latched rather than hot | +| `XmrigGuardEvaluationErrors` | >0.01/s evaluation failures; affected nodes cannot re-earn `safe=1` | + +## Known and accepted + +- **A full node forfeits its slot.** The hard `nodeSelector` means a miner cannot shop for + another node. control-2 and control-3 have 21.7Gi allocatable and sit near 99% memory + requests, which cost 1.69 replica-hours in the first 24h. `preemptionPolicy: Never` on + `low-priority-mining` means the miner waits rather than evicting real work, which is correct. +- **The miner needs all of a node's hugepages.** 2368Mi requested against 2368Mi allocatable, + so exactly one miner fits per node. Any other hugepages consumer on a control node blocks a + miner outright; the CNPG cluster carries an explicit cap for this reason. +- **The dashboard under-reports hashrate when more than one miner runs**, because it polls one + Service and gets whichever pod answers. +- **A missing rank series fails open on allocation only.** `scalar()` yields NaN, every rank + comparison is false, and the node behaves as top priority: worst case one extra miner, 50W + over-drawn. The safety gate is a separate factor and still fails closed. +- **Summer afternoons stay shut.** The drives idle in the 60-64C band on hot days, on top of + both thresholds. No threshold unlocks that; these thresholds unlock mornings and winter. + +## Operating notes + +- The guard's CPU path issues 7 serial queries against a 5s timeout. control-1 is the only node + on that path, so evaluation errors concentrate there. +- A miner starting or stopping changes control-1's CPU sample count (host+presence, plus xmrig + only while one runs). That is expected and re-baselines the source set; it must not be treated + as tampering, which previously made mining on control-1 trip control-1. +- `xmrig_guard_safe` has one series per node. Range queries spanning a guard or + kube-state-metrics restart return two partial series; use `max by (node)` in a subquery, and + remember gaps are not zeros (`or vector(0)` where absence means zero). +- kube-state-metrics scrapes at 20s, not 60s. `sum_over_time(...)/60` overcounts replica-hours + 3x; prefer `avg_over_time(...) * hours`. diff --git a/kubernetes/apps/web3/monero/xmrig/resourceset.yaml b/kubernetes/apps/web3/monero/xmrig/resourceset.yaml index 6d4036650c..f4b98ed7f0 100644 --- a/kubernetes/apps/web3/monero/xmrig/resourceset.yaml +++ b/kubernetes/apps/web3/monero/xmrig/resourceset.yaml @@ -107,7 +107,9 @@ spec: minReplicaCount: 0 maxReplicaCount: 1 cooldownPeriod: 0 # Guard dwell already debounces trips and recovery - pollingInterval: 60 + # 30s to match the guard's evaluation interval: this is the last serial leg of the + # 135s trip-to-drain chain the 65C trip is sized against. + pollingInterval: 30 fallback: # A VictoriaMetrics/scaler error is unsafe: do not keep miners running on stale state. failureThreshold: 1 diff --git a/kubernetes/apps/web3/xmrig-guard/app/prometheusrule.yaml b/kubernetes/apps/web3/xmrig-guard/app/prometheusrule.yaml index bc9789a2e3..63570a2573 100644 --- a/kubernetes/apps/web3/xmrig-guard/app/prometheusrule.yaml +++ b/kubernetes/apps/web3/xmrig-guard/app/prometheusrule.yaml @@ -46,7 +46,7 @@ spec: severity: warning - alert: XmrigGuardLatchedUnsafe # A node that trips or hits an evaluation error only re-earns safe=1 after - # 600s at or below 60C. Paired here with the temperature actually being in + # 300s at or below 62C. Paired here with the temperature actually being in # that band, so this fires for a stuck latch (the 2026-07-27 control-2 # sensor-identity blip) and stays quiet when the gate is shut because the # drives are genuinely too hot, which is the expected steady state. @@ -55,15 +55,35 @@ spec: expr: |- max_over_time(xmrig_guard_safe[6h]) == 0 and on (node) - max_over_time(xmrig_guard_nvme_temp_max_celsius[2h]) <= 60 + max_over_time(xmrig_guard_nvme_temp_max_celsius[2h]) <= 62 for: 30m annotations: summary: >- {{ $labels.node }} has been gated unsafe for 6h while its NVMe stayed - within the 60C recovery band, so the guard is latched rather than + within the 62C recovery band, so the guard is latched rather than thermally tripped and mining is disabled with no signal labels: severity: warning + - alert: XmrigGuardThermalPanic + # The 68C panic trip skips the dwell, so a miner still running above it means the + # fast path itself failed (guard wedged, KEDA down, HPA stuck). 70C is the rating, + # and the drive climbs ~1.1C/min under load, so this is roughly 1.8 minutes of margin. + # Mirrors XmrigGuardEnforcementBypassed's deployment<->node join with the regex + # reversed; the "xmrig-" convention lives in both and must change together. + expr: |- + max by (node) (xmrig_guard_nvme_temp_max_celsius) > 68 + and on (node) + label_replace( + kube_deployment_status_replicas{namespace="web3", deployment=~"xmrig-control-.+"} > 0, + "node", "$1", "deployment", "xmrig-(.*)" + ) + for: 2m + annotations: + summary: >- + {{ $labels.node }} is above the 68C panic trip with a miner still running — + the zero-dwell fast path did not shed it and the drive is near its 70C rating + labels: + severity: critical - alert: XmrigGuardEvaluationErrors # 0.01/s ~= 36 errors/h; the 2026-07 telemetry breakage ran 47-100/h expr: |- diff --git a/kubernetes/apps/web3/xmrig-guard/app/resources/controller.py b/kubernetes/apps/web3/xmrig-guard/app/resources/controller.py index e6c31176dc..851b417644 100644 --- a/kubernetes/apps/web3/xmrig-guard/app/resources/controller.py +++ b/kubernetes/apps/web3/xmrig-guard/app/resources/controller.py @@ -35,10 +35,11 @@ # node telemetry, and a future headroom-derived ranking replaces this tuple alone. PRIORITY = ("control-1", "control-2", "control-3") ENDPOINT = "http://vmauth-victoria-metrics.observability.svc.cluster.local:8427" -EVALUATION_INTERVAL_SECONDS = 60 +EVALUATION_INTERVAL_SECONDS = 30 SOURCE_SAMPLE_MAX_AGE_SECONDS = 120 MAX_SOURCE_GAP_SECONDS = 120 -HTTP_TIMEOUT_SECONDS = 10 +# 7 serial queries per CPU evaluation: at 10s a hung VictoriaMetrics blocked 70s against a 30s interval +HTTP_TIMEOUT_SECONDS = 5 def _dt(value): @@ -55,9 +56,10 @@ def _fresh(timestamp, evaluation, max_age): class DwellPolicy: - def __init__(self, recovery_limit, trip_limit, recovery_dwell, trip_dwell, max_gap_seconds=120): + def __init__(self, recovery_limit, trip_limit, recovery_dwell, trip_dwell, max_gap_seconds=120, panic_limit=None): self.recovery_limit, self.trip_limit = recovery_limit, trip_limit self.recovery_dwell, self.trip_dwell = recovery_dwell, trip_dwell + self.panic_limit = panic_limit self.max_gap = float(max_gap_seconds) self.safe = False self._last_source = None @@ -80,6 +82,12 @@ def observe(self, value, source_time, monotonic_now): if gap > self.max_gap: self._pending = self._since = None self._last_source = source_seconds + # third, dwell-free state: 2C from the rating, confirming the reading over a 60s dwell + # would spend most of the margin. Clearing _pending forces a full recovery dwell after. + if self.panic_limit is not None and value >= self.panic_limit: + self.safe = False + self._pending = self._since = None + return False kind = "recover" if value <= self.recovery_limit else "trip" if value >= self.trip_limit else None if kind is None: self._pending = self._since = None @@ -235,21 +243,19 @@ class GuardController: def __init__(self, telemetry, clock=time.monotonic, wall_clock=lambda: datetime.now(timezone.utc)): self.telemetry = telemetry self.clock, self.wall_clock = clock, wall_clock - # Trip 64C / recover 60C on Composite, against a 70C drive rating. Mining raises - # Composite at up to 1.1C/min, so the 6C band is ~5.5 minutes wide. Worst-case - # response is ~4.25: up to one evaluation interval to sample the crossing, 120s - # trip dwell, 60s of KEDA polling, then the HPA drop (scaledobject.yaml sheds all - # replicas at once for exactly this reason). That leaves the peak near 69C, and is - # conservative because it assumes full heat output until the last miner exits. - # Idle Composite never exceeded 62C over 7d on either node, so the trip does not - # false-fire, and it sits at or below 60C for 100%/90% of the time, so recovery is - # reachable rather than the permanent latch the old 60C/70C pair produced. + # Trip 65C / recover 62C on Composite, against a 70C drive rating. Mining raises + # Composite at up to 1.1C/min. Trip-to-drain is 135s (30s evaluation + 60s dwell + 30s + # KEDA poll + 15s drain), so 2.5C of rise to a ~67.5C peak; test_trip_to_drain_budget + # pins it. The 68C panic trip is dwell-free and peaks near 69.4C. + # The drives idle in the 60-64C band on warm afternoons, so recovery sits at 62C: + # replaying 7d puts control-2/3 at 55.0%/38.8% safe against 44.7%/31.5% at 60C/600s. # control-1 is keyed the same way on CPU headroom rather than temperature; SENSORS - # decides which source feeds which node, so the policy dict needs no special case. + # decides which source feeds which node, so the policy dict needs no special case. No + # panic limit there: a busy CPU has no equivalent of a drive's absolute rating. self.policies = { "control-1": DwellPolicy(50, 70, 600, 120, MAX_SOURCE_GAP_SECONDS), - "control-2": DwellPolicy(60, 64, 600, 120, MAX_SOURCE_GAP_SECONDS), - "control-3": DwellPolicy(60, 64, 600, 120, MAX_SOURCE_GAP_SECONDS), + "control-2": DwellPolicy(62, 65, 300, 60, MAX_SOURCE_GAP_SECONDS, panic_limit=68), + "control-3": DwellPolicy(62, 65, 300, 60, MAX_SOURCE_GAP_SECONDS, panic_limit=68), } self.ready = False self.metrics = { @@ -260,18 +266,21 @@ def __init__(self, telemetry, clock=time.monotonic, wall_clock=lambda: datetime. "cpu_non_xmrig": {node: 0.0 for node in SENSORS if not SENSORS[node]}, "rank": {node: PRIORITY.index(node) for node in SENSORS}, } - self._last_source_stamps = {node: () for node in SENSORS} + self._last_source_stamps = {node: {} for node in SENSORS} def _new_source_set(self, node, sources): - stamps = tuple(source.timestamp.timestamp() for source in sources) + # Keyed by source identity, not position: a miner starting or stopping adds or removes + # the xmrig source, and comparing by position made that look like tampering. It failed + # control-1 closed 19 times in 12h, draining the miner, which changed the set back. + # Only keys in both sets are comparable; a genuinely new key is exempt for one cycle + # rather than the whole evaluation going unchecked. + stamps = {key: source.timestamp.timestamp() for key, source in sources.items()} previous = self._last_source_stamps[node] - if previous: - if len(previous) != len(stamps): - raise ValueError("source membership changed") - if any(current - old > MAX_SOURCE_GAP_SECONDS for current, old in zip(stamps, previous)): - raise ValueError("source gap exceeded maximum") - if any(current <= old for current, old in zip(stamps, previous)): - return False + shared = stamps.keys() & previous.keys() + if any(stamps[key] - previous[key] > MAX_SOURCE_GAP_SECONDS for key in shared): + raise ValueError("source gap exceeded maximum") + if shared and any(stamps[key] <= previous[key] for key in shared): + return False self._last_source_stamps[node] = stamps return True @@ -283,27 +292,31 @@ def evaluate(self, evaluation=None): # no sensors means a node with no visible NVMe (control-1, a VM): it is gated # on CPU headroom instead. The dwell policy and metrics are keyed identically. if sensors: - samples = self.telemetry.query_nvme(node, sensors, evaluation) + samples = dict(zip(sensors, self.telemetry.query_nvme(node, sensors, evaluation))) # trip on the hottest drive, date it by the newest sample it was read from - value, stamp = max(item.value for item in samples), max(item.timestamp for item in samples) + value = max(item.value for item in samples.values()) + stamp = max(item.timestamp for item in samples.values()) self.metrics["nvme_temp_max"][node] = value else: obs = self.telemetry.query_cpu(node, evaluation) - samples = (obs.host, obs.presence) + ((obs.xmrig,) if obs.xmrig else ()) - value, stamp = cpu_value(obs), min(item.timestamp for item in samples) + samples = {"host": obs.host, "presence": obs.presence} + if obs.xmrig: + samples["xmrig"] = obs.xmrig + value = cpu_value(obs) + stamp = min(item.timestamp for item in samples.values()) self.metrics["cpu_non_xmrig"][node] = value - if not samples or not all(_fresh(item.timestamp, evaluation, SOURCE_SAMPLE_MAX_AGE_SECONDS) for item in samples): + if not samples or not all(_fresh(item.timestamp, evaluation, SOURCE_SAMPLE_MAX_AGE_SECONDS) for item in samples.values()): raise ValueError("stale or future source") policy = self.policies[node] safe = policy.observe(value, stamp, now) if self._new_source_set(node, samples) else policy.safe - self.metrics["source_age_seconds"][node] = max(0.0, evaluation.timestamp() - min(item.timestamp for item in samples).timestamp()) + self.metrics["source_age_seconds"][node] = max(0.0, evaluation.timestamp() - min(item.timestamp for item in samples.values()).timestamp()) self.metrics["safe"][node] = int(safe) except Exception as exc: # one line per failure, no traceback: the query text travels in the exception logging.error(f"evaluation failed for {node}: {exc!r}") self.metrics["query_errors"][node] += 1 self.policies[node].invalidate() - self._last_source_stamps[node] = () + self._last_source_stamps[node] = {} self.metrics["safe"][node] = 0 self.metrics["source_age_seconds"][node] = float("nan") if sensors: @@ -356,8 +369,11 @@ def main(): server = ThreadingHTTPServer(("0.0.0.0", 8080), _StatusHandler) threading.Thread(target=server.serve_forever, daemon=True).start() while True: + # sleep to a deadline, not a flat interval: sleeping after the work made the true + # period drift by the evaluation duration, stretching it against a fixed freshness budget + deadline = time.monotonic() + EVALUATION_INTERVAL_SECONDS controller.evaluate() - time.sleep(EVALUATION_INTERVAL_SECONDS) + time.sleep(max(0, deadline - time.monotonic())) if __name__ == "__main__": diff --git a/kubernetes/apps/web3/xmrig-guard/app/resources/test_controller.py b/kubernetes/apps/web3/xmrig-guard/app/resources/test_controller.py index c2b8baef1e..a77867463c 100644 --- a/kubernetes/apps/web3/xmrig-guard/app/resources/test_controller.py +++ b/kubernetes/apps/web3/xmrig-guard/app/resources/test_controller.py @@ -1,5 +1,6 @@ import math import os +import re import sys import unittest import urllib.error @@ -10,6 +11,17 @@ import controller UTC = timezone.utc +# The other two legs of the trip-to-drain budget live in the miner's manifest, so they are read +# from it rather than copied: a manifest-only change must fail the budget test, not pass it. +RESOURCESET = os.path.join(os.path.dirname(__file__), "..", "..", "..", "monero", "xmrig", "resourceset.yaml") + + +def manifest_seconds(field): + with open(RESOURCESET) as handle: + found = re.findall(rf"^\s*{field}:\s*(\d+)\s*(?:#.*)?$", handle.read(), re.MULTILINE) + if len(found) != 1: + raise AssertionError(f"expected exactly one {field} in resourceset.yaml, found {len(found)}") + return int(found[0]) class PolicyTests(unittest.TestCase): @@ -267,13 +279,95 @@ def cpu(node, evaluation): self.assertIn("xmrig_guard_source_age_seconds", text) self.assertIn("xmrig_guard_query_errors_total", text) - def test_cpu_membership_change_is_fail_closed(self): + def test_miner_arriving_exempts_only_the_new_source(self): + # a miner starting adds the xmrig source, which must not fail the node closed guard = controller.GuardController(Mock()) stamp = datetime(2026, 1, 1, tzinfo=UTC) - sources = [controller.Source(1, stamp)] * 3 - self.assertTrue(guard._new_source_set("control-1", sources)) + base = {"host": controller.Source(1, stamp), "presence": controller.Source(1, stamp)} + self.assertTrue(guard._new_source_set("control-1", base)) + later = {key: controller.Source(1, stamp + timedelta(seconds=30)) for key in base} + self.assertTrue(guard._new_source_set("control-1", later | {"xmrig": controller.Source(1, stamp)})) + # the shared keys are still checked: unchanged stamps mean no advancement + self.assertFalse(guard._new_source_set("control-1", later | {"xmrig": controller.Source(1, stamp)})) + + def test_miner_starting_and_stopping_never_faults_control1(self): + # the regression this replaces: the xmrig source appearing and disappearing changed the + # sample count, read as tampering, and drained the miner it was caused by + telemetry = Mock() + base = datetime(2026, 1, 1, tzinfo=UTC) + mining = [False] + + def cpu(_node, evaluation): + source = controller.Source(30, evaluation) + return controller.CPUObservation(source, controller.Source(5, evaluation) if mining[0] else None, source) + + telemetry.query_cpu.side_effect = cpu + telemetry.query_nvme.side_effect = lambda _n, sensors, evaluation: [controller.Source(40, evaluation)] * len(sensors) + guard = controller.GuardController(telemetry, clock=lambda: 0) + for step, running in enumerate([False, True, True, False, True]): + mining[0] = running + guard.evaluate(base + timedelta(seconds=30 * step)) + self.assertEqual(guard.metrics["query_errors"]["control-1"], 0) + + def test_nvme_source_keys_survive_reordering(self): + # keys are the sensor identities themselves, so iteration order must not matter + guard = controller.GuardController(Mock()) + first, second = controller.SENSORS["control-2"] + base = datetime(2026, 1, 1, tzinfo=UTC) + previous = { + first: controller.Source(40, base), + second: controller.Source(41, base + timedelta(seconds=30)), + } + self.assertTrue(guard._new_source_set("control-2", previous)) + # advance each sensor's own timestamp, then reverse the iteration order + reordered = { + second: controller.Source(41, base + timedelta(seconds=60)), + first: controller.Source(40, base + timedelta(seconds=30)), + } + self.assertTrue(guard._new_source_set("control-2", reordered)) + # and the anti-replay check still binds across the reorder + self.assertFalse(guard._new_source_set("control-2", reordered)) + + def test_shared_key_gap_fails_closed_even_when_a_source_is_added(self): + guard = controller.GuardController(Mock()) + stamp = datetime(2026, 1, 1, tzinfo=UTC) + base = {"host": controller.Source(1, stamp), "presence": controller.Source(1, stamp)} + self.assertTrue(guard._new_source_set("control-1", base)) + far = stamp + timedelta(seconds=controller.MAX_SOURCE_GAP_SECONDS + 1) + stale = {key: controller.Source(1, far) for key in base} with self.assertRaises(ValueError): - guard._new_source_set("control-1", sources + [controller.Source(1, stamp)]) + guard._new_source_set("control-1", stale | {"xmrig": controller.Source(1, far)}) + + def test_panic_limit_trips_without_dwell_and_needs_full_recovery(self): + p = controller.DwellPolicy(62, 65, 300, 60, 120, panic_limit=68) + source = datetime(2026, 1, 1, tzinfo=UTC) + p.safe = True + self.assertFalse(p.observe(68, source, 0.0)) + self.assertFalse(p.safe) + # recovery is not instant afterwards: the full 300s dwell must elapse below 62C + self.assertFalse(p.observe(60, source + timedelta(seconds=1), 1.0)) + self.assertFalse(p.observe(60, source + timedelta(seconds=2), 299.0)) + self.assertTrue(p.observe(60, source + timedelta(seconds=3), 302.0)) + + def test_trip_to_drain_budget_fits_the_thermal_margin(self): + # The 65C trip is sized on a 135s chain against a 70C rating at ~1.1C/min. Two legs are + # policy here, two are read from the miner's manifest, so either side drifting fails this. + keda_poll = manifest_seconds("pollingInterval") + drain = manifest_seconds("terminationGracePeriodSeconds") + rise_c_per_min, rating = 1.1, 70 + policies = controller.GuardController(Mock()).policies + for node in ("control-2", "control-3"): + policy = policies[node] + budget = controller.EVALUATION_INTERVAL_SECONDS + policy.trip_dwell + keda_poll + drain + peak = policy.trip_limit + rise_c_per_min * budget / 60 + self.assertLessEqual(budget, 135, node) + self.assertLess(peak, rating, f"{node} peaks at {peak}C against a {rating}C rating") + + def test_nvme_policies_carry_the_panic_limit_and_control1_does_not(self): + guard = controller.GuardController(Mock()) + self.assertEqual(guard.policies["control-2"].panic_limit, 68) + self.assertEqual(guard.policies["control-3"].panic_limit, 68) + self.assertIsNone(guard.policies["control-1"].panic_limit) def test_failure_invalidates_values(self): telemetry = Mock()