feat(container)!: Update immich monorepo (v2.7.5 ➔ v3.1.0) - #3618
Merged
Conversation
Contributor
 |
|
Overall Grade |
Security Reliability Complexity Hygiene |
Code Review Summary
| Analyzer | Status | Updated (UTC) | Details |
|---|---|---|---|
| JavaScript |  |
Aug 8, 2026 1:19p.m. | Review ↗ |
| Shell | |
Aug 8, 2026 1:19p.m. | Review ↗ |
Important
AI Review is run only on demand for your team. We're only showing results of static analysis review right now. To trigger AI Review, comment @deepsourcebot review on this thread.
@@ spec.template.spec.containers.app.args @@
# apps/v1/Deployment/web3/xmrig-control-1
! - one list entry removed:
- - "--no-huge-pages"
! + one list entry added:
+ - "--huge-pages"
@@ spec.template.spec.containers.app.resources.limits @@
# apps/v1/Deployment/web3/xmrig-control-1
! + one map entry added:
+ hugepages-2Mi: 2368Mi
@@ spec.template.spec.containers.app.resources.limits.memory @@
# apps/v1/Deployment/web3/xmrig-control-1
! ± value change
- 3Gi
+ 128Mi
@@ spec.template.spec.containers.app.resources.requests @@
# apps/v1/Deployment/web3/xmrig-control-1
! + one map entry added:
+ hugepages-2Mi: 2368Mi
@@ spec.template.spec.containers.app.resources.requests.memory @@
# apps/v1/Deployment/web3/xmrig-control-1
! ± value change
- 2Gi
+ 32Mi
@@ spec.template.spec.containers.app.securityContext.capabilities @@
# apps/v1/Deployment/web3/xmrig-control-1
! + one map entry added:
+ add:
+ - IPC_LOCK
@@ spec.template.spec.containers.app.volumeMounts @@
# apps/v1/Deployment/web3/xmrig-control-1
! + one list entry added:
+ - name: hugepages
+ mountPath: /dev/hugepages
@@ spec.template.spec.volumes @@
# apps/v1/Deployment/web3/xmrig-control-1
! + one list entry added:
+ - name: hugepages
+ emptyDir:
+ medium: HugePages
@@ spec.template.spec.volumes.tmp.emptyDir @@
# apps/v1/Deployment/web3/xmrig-control-1
! - one map entry removed:
- sizeLimit: 64Mi
@@ spec.template.spec.containers.app.args @@
# apps/v1/Deployment/web3/xmrig-control-2
! - one list entry removed:
- - "--no-huge-pages"
! + one list entry added:
+ - "--huge-pages"
@@ spec.template.spec.containers.app.resources.limits @@
# apps/v1/Deployment/web3/xmrig-control-2
! + one map entry added:
+ hugepages-2Mi: 2368Mi
@@ spec.template.spec.containers.app.resources.limits.memory @@
# apps/v1/Deployment/web3/xmrig-control-2
! ± value change
- 3Gi
+ 128Mi
@@ spec.template.spec.containers.app.resources.requests @@
# apps/v1/Deployment/web3/xmrig-control-2
! + one map entry added:
+ hugepages-2Mi: 2368Mi
@@ spec.template.spec.containers.app.resources.requests.memory @@
# apps/v1/Deployment/web3/xmrig-control-2
! ± value change
- 2Gi
+ 32Mi
@@ spec.template.spec.containers.app.securityContext.capabilities @@
# apps/v1/Deployment/web3/xmrig-control-2
! + one map entry added:
+ add:
+ - IPC_LOCK
@@ spec.template.spec.containers.app.volumeMounts @@
# apps/v1/Deployment/web3/xmrig-control-2
! + one list entry added:
+ - name: hugepages
+ mountPath: /dev/hugepages
@@ spec.template.spec.volumes @@
# apps/v1/Deployment/web3/xmrig-control-2
! + one list entry added:
+ - name: hugepages
+ emptyDir:
+ medium: HugePages
@@ spec.template.spec.volumes.tmp.emptyDir @@
# apps/v1/Deployment/web3/xmrig-control-2
! - one map entry removed:
- sizeLimit: 64Mi
@@ spec.template.spec.containers.app.args @@
# apps/v1/Deployment/web3/xmrig-control-3
! - one list entry removed:
- - "--no-huge-pages"
! + one list entry added:
+ - "--huge-pages"
@@ spec.template.spec.containers.app.resources.limits @@
# apps/v1/Deployment/web3/xmrig-control-3
! + one map entry added:
+ hugepages-2Mi: 2368Mi
@@ spec.template.spec.containers.app.resources.limits.memory @@
# apps/v1/Deployment/web3/xmrig-control-3
! ± value change
- 3Gi
+ 128Mi
@@ spec.template.spec.containers.app.resources.requests @@
# apps/v1/Deployment/web3/xmrig-control-3
! + one map entry added:
+ hugepages-2Mi: 2368Mi
@@ spec.template.spec.containers.app.resources.requests.memory @@
# apps/v1/Deployment/web3/xmrig-control-3
! ± value change
- 2Gi
+ 32Mi
@@ spec.template.spec.containers.app.securityContext.capabilities @@
# apps/v1/Deployment/web3/xmrig-control-3
! + one map entry added:
+ add:
+ - IPC_LOCK
@@ spec.template.spec.containers.app.volumeMounts @@
# apps/v1/Deployment/web3/xmrig-control-3
! + one list entry added:
+ - name: hugepages
+ mountPath: /dev/hugepages
@@ spec.template.spec.volumes @@
# apps/v1/Deployment/web3/xmrig-control-3
! + one list entry added:
+ - name: hugepages
+ emptyDir:
+ medium: HugePages
@@ spec.template.spec.volumes.tmp.emptyDir @@
# apps/v1/Deployment/web3/xmrig-control-3
! - one map entry removed:
- sizeLimit: 64Mi
@@ (root level) @@
# v1/ServiceAccount/web3/guard
! - one document removed:
- apiVersion: v1
- kind: ServiceAccount
- metadata:
- name: guard
- namespace: web3
- labels:
- app.kubernetes.io/instance: guard
- app.kubernetes.io/managed-by: Helm
- app.kubernetes.io/name: guard
- helm.toolkit.fluxcd.io/name: guard
- helm.toolkit.fluxcd.io/namespace: web3
@@ (root level) @@
# v1/Service/web3/guard
! - one document removed:
- apiVersion: v1
- kind: Service
- metadata:
- name: guard
- namespace: web3
- labels:
- app.kubernetes.io/instance: guard
- app.kubernetes.io/managed-by: Helm
- app.kubernetes.io/name: guard
- app.kubernetes.io/service: guard
- helm.toolkit.fluxcd.io/name: guard
- helm.toolkit.fluxcd.io/namespace: web3
- spec:
- type: ClusterIP
- selector:
- app.kubernetes.io/controller: guard
- app.kubernetes.io/instance: guard
- app.kubernetes.io/name: guard
- ports:
- - name: http
- port: 8080
- protocol: TCP
- targetPort: 8080
@@ (root level) @@
# apps/v1/Deployment/web3/guard
! - one document removed:
- apiVersion: apps/v1
- kind: Deployment
- metadata:
- name: guard
- namespace: web3
- annotations:
- reloader.stakater.com/auto: "true"
- labels:
- app.kubernetes.io/controller: guard
- app.kubernetes.io/instance: guard
- app.kubernetes.io/managed-by: Helm
- app.kubernetes.io/name: guard
- helm.toolkit.fluxcd.io/name: guard
- helm.toolkit.fluxcd.io/namespace: web3
- spec:
- replicas: 1
- revisionHistoryLimit: 3
- selector:
- matchLabels:
- app.kubernetes.io/controller: guard
- app.kubernetes.io/instance: guard
- app.kubernetes.io/name: guard
- strategy:
- type: Recreate
- template:
- metadata:
- labels:
- app.kubernetes.io/controller: guard
- app.kubernetes.io/instance: guard
- app.kubernetes.io/name: guard
- spec:
- affinity:
- nodeAffinity:
- preferredDuringSchedulingIgnoredDuringExecution:
- - preference:
- matchExpressions:
- - key: kubernetes.io/hostname
- operator: In
- values:
- - control-1
- weight: 100
- automountServiceAccountToken: false
- containers:
- - name: app
- image: "python:3.14-slim@sha256:a7fb1e634c4a578f9e0bd6327f11a3cde11b7a9395f48e24360c0988bcc5c2bc"
- command:
- - python3
- - /config/controller.py
- livenessProbe:
- httpGet:
- path: /healthz
- port: http
- ports:
- - name: http
- containerPort: 8080
- readinessProbe:
- httpGet:
- path: /readyz
- port: http
- resources:
- limits:
- cpu: 100m
- memory: 128Mi
- requests:
- cpu: 10m
- memory: 32Mi
- volumeMounts:
- - name: config
- mountPath: /config
- readOnly: true
- - name: tmp
- mountPath: /tmp
- securityContext:
- allowPrivilegeEscalation: false
- capabilities:
- drop:
- - ALL
- readOnlyRootFilesystem: true
- dnsPolicy: ClusterFirst
- enableServiceLinks: false
- hostIPC: false
- hostNetwork: false
- hostPID: false
- securityContext:
- fsGroup: 65532
- fsGroupChangePolicy: OnRootMismatch
- runAsGroup: 65532
- runAsNonRoot: true
- runAsUser: 65532
- seccompProfile:
- type: RuntimeDefault
- serviceAccountName: guard
- terminationGracePeriodSeconds: 15
- volumes:
- - name: config
- configMap:
- name: guard
- - name: tmp
- emptyDir: {}
@@ (root level) @@
# v1/ServiceAccount/web3/xmrig-guard
! + one document added:
+ apiVersion: v1
+ kind: ServiceAccount
+ metadata:
+ name: xmrig-guard
+ namespace: web3
+ labels:
+ app.kubernetes.io/instance: xmrig-guard
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/name: xmrig-guard
+ helm.toolkit.fluxcd.io/name: xmrig-guard
+ helm.toolkit.fluxcd.io/namespace: web3
@@ (root level) @@
# v1/Service/web3/xmrig-guard
! + one document added:
+ apiVersion: v1
+ kind: Service
+ metadata:
+ name: xmrig-guard
+ namespace: web3
+ labels:
+ app.kubernetes.io/instance: xmrig-guard
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/name: xmrig-guard
+ app.kubernetes.io/service: xmrig-guard
+ helm.toolkit.fluxcd.io/name: xmrig-guard
+ helm.toolkit.fluxcd.io/namespace: web3
+ spec:
+ type: ClusterIP
+ selector:
+ app.kubernetes.io/controller: xmrig-guard
+ app.kubernetes.io/instance: xmrig-guard
+ app.kubernetes.io/name: xmrig-guard
+ ports:
+ - name: http
+ port: 8080
+ protocol: TCP
+ targetPort: 8080
@@ (root level) @@
# apps/v1/Deployment/web3/xmrig-guard
! + one document added:
+ apiVersion: apps/v1
+ kind: Deployment
+ metadata:
+ name: xmrig-guard
+ namespace: web3
+ annotations:
+ reloader.stakater.com/auto: "true"
+ labels:
+ app.kubernetes.io/controller: xmrig-guard
+ app.kubernetes.io/instance: xmrig-guard
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/name: xmrig-guard
+ helm.toolkit.fluxcd.io/name: xmrig-guard
+ helm.toolkit.fluxcd.io/namespace: web3
+ spec:
+ replicas: 1
+ revisionHistoryLimit: 3
+ selector:
+ matchLabels:
+ app.kubernetes.io/controller: xmrig-guard
+ app.kubernetes.io/instance: xmrig-guard
+ app.kubernetes.io/name: xmrig-guard
+ strategy:
+ type: Recreate
+ template:
+ metadata:
+ labels:
+ app.kubernetes.io/controller: xmrig-guard
+ app.kubernetes.io/instance: xmrig-guard
+ app.kubernetes.io/name: xmrig-guard
+ spec:
+ affinity:
+ nodeAffinity:
+ preferredDuringSchedulingIgnoredDuringExecution:
+ - preference:
+ matchExpressions:
+ - key: kubernetes.io/hostname
+ operator: In
+ values:
+ - control-1
+ weight: 100
+ automountServiceAccountToken: false
+ containers:
+ - name: app
+ image: "python:3.14-slim@sha256:a7fb1e634c4a578f9e0bd6327f11a3cde11b7a9395f48e24360c0988bcc5c2bc"
+ command:
+ - python3
+ - /config/controller.py
+ livenessProbe:
+ httpGet:
+ path: /healthz
+ port: http
+ ports:
+ - name: http
+ containerPort: 8080
+ readinessProbe:
+ httpGet:
+ path: /readyz
+ port: http
+ resources:
+ limits:
+ cpu: 100m
+ memory: 128Mi
+ requests:
+ cpu: 10m
+ memory: 32Mi
+ volumeMounts:
+ - name: config
+ mountPath: /config
+ readOnly: true
+ - name: tmp
+ mountPath: /tmp
+ securityContext:
+ allowPrivilegeEscalation: false
+ capabilities:
+ drop:
+ - ALL
+ readOnlyRootFilesystem: true
+ dnsPolicy: ClusterFirst
+ enableServiceLinks: false
+ hostIPC: false
+ hostNetwork: false
+ hostPID: false
+ securityContext:
+ fsGroup: 65532
+ fsGroupChangePolicy: OnRootMismatch
+ runAsGroup: 65532
+ runAsNonRoot: true
+ runAsUser: 65532
+ seccompProfile:
+ type: RuntimeDefault
+ serviceAccountName: xmrig-guard
+ terminationGracePeriodSeconds: 15
+ volumes:
+ - name: config
+ configMap:
+ name: xmrig-guard
+ - name: tmp
+ emptyDir: {}
|
AI Automated ReviewAnalysis engine: omniroute@http://litellm.ai.svc.cluster.local/v1 (openai) Recommendation: Approve with minor verification notes. Release notes Required Checks
Sources
Unknowns or Needs Verification
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
renovate
Bot
force-pushed
the
renovate/major-immich-monorepo
branch
4 times, most recently
from
July 2, 2026 21:57
8bbff83 to
5814465
Compare
renovate
Bot
changed the title
renovate
Bot
force-pushed
the
renovate/major-immich-monorepo
branch
from
July 3, 2026 01:00
5814465 to
7dd74c4
Compare
renovate
Bot
changed the title
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
renovate
Bot
force-pushed
the
renovate/major-immich-monorepo
branch
2 times, most recently
from
July 4, 2026 15:07
10d6e4c to
a5a20c1
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
renovate
Bot
force-pushed
the
renovate/major-immich-monorepo
branch
from
July 8, 2026 01:02
a5a20c1 to
eb04961
Compare
renovate
Bot
changed the title
renovate
Bot
force-pushed
the
renovate/major-immich-monorepo
branch
from
July 8, 2026 04:40
eb04961 to
10c8ecd
Compare
renovate
Bot
changed the title
renovate
Bot
force-pushed
the
renovate/major-immich-monorepo
branch
from
July 9, 2026 19:10
10c8ecd to
6adce9c
Compare
renovate
Bot
changed the title
renovate
Bot
force-pushed
the
renovate/major-immich-monorepo
branch
from
July 10, 2026 15:10
6adce9c to
8553f56
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
renovate
Bot
force-pushed
the
renovate/major-immich-monorepo
branch
2 times, most recently
from
July 13, 2026 18:26
f9b6ac8 to
5574519
Compare
renovate
Bot
force-pushed
the
renovate/major-immich-monorepo
branch
3 times, most recently
from
July 15, 2026 16:51
6f47b8e to
8ef8e2d
Compare
renovate
Bot
changed the title
renovate
Bot
force-pushed
the
renovate/major-immich-monorepo
branch
from
July 15, 2026 17:46
8ef8e2d to
12a4cc5
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
renovate
Bot
force-pushed
the
renovate/major-immich-monorepo
branch
3 times, most recently
from
July 21, 2026 01:10
dce9613 to
aeed5cc
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
renovate
Bot
force-pushed
the
renovate/major-immich-monorepo
branch
from
July 24, 2026 16:24
aeed5cc to
90be035
Compare
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughImmich server and machine-learning HelmReleases update their pinned container images from v2.7.5 to v3.1.0 with updated digests. ChangesImmich image update
Estimated code review effort: 1 (Trivial) | ~3 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
renovate
Bot
force-pushed
the
renovate/major-immich-monorepo
branch
2 times, most recently
from
July 29, 2026 17:34
4478c77 to
56899e9
Compare
renovate
Bot
changed the title
renovate
Bot
force-pushed
the
renovate/major-immich-monorepo
branch
from
August 2, 2026 20:00
56899e9 to
f0390ed
Compare
renovate
Bot
changed the title
renovate
Bot
force-pushed
the
renovate/major-immich-monorepo
branch
from
August 2, 2026 20:42
f0390ed to
5e2f295
Compare
renovate
Bot
changed the title
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kubernetes/apps/default/immich/server/helmrelease.yaml`:
- Line 37: Before changing the image tag in the Immich HelmRelease, verify that
the surrounding Flux manifests define the required v3 migration prerequisites:
database migration handling, PostgreSQL extension compatibility, backup and
rollback procedures, postgres-init:v18, and updated referenced configuration
with deprecated variables removed. If any prerequisite is missing, block the
image rollout and commit the necessary manifest changes first.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 5df6a05e-81c6-4b5c-8cdc-5b242ee85dda
📒 Files selected for processing (2)
kubernetes/apps/default/immich/machine-learning/helmrelease.yamlkubernetes/apps/default/immich/server/helmrelease.yaml
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
Tanguille/LLMKube(auto-detected)Tanguille/toolhive(auto-detected)Tanguille/pr-reviewer-action(auto-detected)Tanguille/2x-R9700-RDNA4-GFX1201-sglang-inference(auto-detected)
🚧 Files skipped from review as they are similar to previous changes (1)
- kubernetes/apps/default/immich/machine-learning/helmrelease.yaml
| image: | ||
| repository: ghcr.io/immich-app/immich-server | ||
| tag: v2.7.5@sha256:c15bff75068effb03f4355997d03dc7e0fc58720c2b54ad6f7f10d1bc57efaa5 | ||
| tag: v3.1.0@sha256:b434cb9287eea1471c9974845914d4dd328c9c2d652e446ed4930f99944f0ceb |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Verify Immich v3 migration prerequisites before rollout.
This change upgrades Immich from v2.7.5 to v3.1.0. The PR objective lists breaking changes, including removal of deprecated environment variables and pgvecto.rs support. This manifest only changes the image reference.
Confirm that database migration, PostgreSQL extension compatibility, backup, rollback, postgres-init:v18, and referenced configuration are handled by other Flux manifests before deployment. If they are not, block the rollout until the migration plan is committed.
#!/usr/bin/env bash
set -euo pipefail
rg -n -C 5 \
'pgvecto|pgvector|postgres-init|DB_URL|envFrom|secretRef|configMapRef|migration|backup|rollback' \
kubernetes/apps/default/immich🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kubernetes/apps/default/immich/server/helmrelease.yaml` at line 37, Before
changing the image tag in the Immich HelmRelease, verify that the surrounding
Flux manifests define the required v3 migration prerequisites: database
migration handling, PostgreSQL extension compatibility, backup and rollback
procedures, postgres-init:v18, and updated referenced configuration with
deprecated variables removed. If any prerequisite is missing, block the image
rollout and commit the necessary manifest changes first.
…tion Jellyfin clients request dozens of unique /HomeScreen/CachedImage and /Items/<id>/Images URIs per page load, tripping http-crawl-non_statics (47 events in 3s, all HTTP 200) and banning the client for 24h. Scoped to 2xx/3xx so scanners probing jellyfin still get counted.
@@ spec.resources.helm.toolkit.fluxcd.io/v2/HelmRelease/web3/xmrig-<< inputs.node >>.spec.values.controllers.xmrig-<< inputs.node >>.containers.app.args @@
# fluxcd.controlplane.io/v1/ResourceSet/web3/xmrig
! - one list entry removed:
- - "--no-huge-pages"
! + one list entry added:
+ - "--huge-pages"
@@ spec.resources.helm.toolkit.fluxcd.io/v2/HelmRelease/web3/xmrig-<< inputs.node >>.spec.values.controllers.xmrig-<< inputs.node >>.containers.app.resources.limits @@
# fluxcd.controlplane.io/v1/ResourceSet/web3/xmrig
! + one map entry added:
+ hugepages-2Mi: 2368Mi
@@ spec.resources.helm.toolkit.fluxcd.io/v2/HelmRelease/web3/xmrig-<< inputs.node >>.spec.values.controllers.xmrig-<< inputs.node >>.containers.app.resources.limits.memory @@
# fluxcd.controlplane.io/v1/ResourceSet/web3/xmrig
! ± value change
- 3Gi
+ 128Mi
@@ spec.resources.helm.toolkit.fluxcd.io/v2/HelmRelease/web3/xmrig-<< inputs.node >>.spec.values.controllers.xmrig-<< inputs.node >>.containers.app.resources.requests @@
# fluxcd.controlplane.io/v1/ResourceSet/web3/xmrig
! + one map entry added:
+ hugepages-2Mi: 2368Mi
@@ spec.resources.helm.toolkit.fluxcd.io/v2/HelmRelease/web3/xmrig-<< inputs.node >>.spec.values.controllers.xmrig-<< inputs.node >>.containers.app.resources.requests.memory @@
# fluxcd.controlplane.io/v1/ResourceSet/web3/xmrig
! ± value change
- 2Gi
+ 32Mi
@@ spec.resources.helm.toolkit.fluxcd.io/v2/HelmRelease/web3/xmrig-<< inputs.node >>.spec.values.controllers.xmrig-<< inputs.node >>.containers.app.securityContext.capabilities @@
# fluxcd.controlplane.io/v1/ResourceSet/web3/xmrig
! + one map entry added:
+ add:
+ - IPC_LOCK
@@ spec.resources.helm.toolkit.fluxcd.io/v2/HelmRelease/web3/xmrig-<< inputs.node >>.spec.values.persistence @@
# fluxcd.controlplane.io/v1/ResourceSet/web3/xmrig
! + one map entry added:
+ hugepages:
+ type: emptyDir
+ medium: HugePages
+ globalMounts:
+ - path: /dev/hugepages
@@ spec.resources.helm.toolkit.fluxcd.io/v2/HelmRelease/web3/xmrig-<< inputs.node >>.spec.values.persistence.tmp @@
# fluxcd.controlplane.io/v1/ResourceSet/web3/xmrig
! - one map entry removed:
- sizeLimit: 64Mi
@@ (root level) @@
# kustomize.toolkit.fluxcd.io/v1/Kustomization/web3/guard
! - one document removed:
- apiVersion: kustomize.toolkit.fluxcd.io/v1
- kind: Kustomization
- metadata:
- name: guard
- namespace: web3
- labels:
- kustomize.toolkit.fluxcd.io/name: cluster-apps
- kustomize.toolkit.fluxcd.io/namespace: flux-system
- spec:
- decryption:
- provider: sops
- secretRef:
- name: sops-age
- deletionPolicy: WaitForTermination
- interval: 30m
- patches:
- - patch: |
- apiVersion: helm.toolkit.fluxcd.io/v2
- kind: HelmRelease
- metadata:
- name: _
- spec:
- install:
- crds: CreateReplace
- rollback:
- cleanupOnFail: true
- upgrade:
- cleanupOnFail: true
- crds: CreateReplace
- strategy:
- name: RemediateOnFailure
- remediation:
- remediateLastFailure: true
- retries: 2
- target:
- kind: HelmRelease
- group: helm.toolkit.fluxcd.io
- path: ./kubernetes/apps/web3/monero/guard
- postBuild:
- substituteFrom:
- - name: cluster-settings
- kind: ConfigMap
- optional: false
- - name: cluster-secrets
- kind: Secret
- optional: false
- prune: true
- retryInterval: 2m
- sourceRef:
- name: flux-system
- kind: GitRepository
- namespace: flux-system
- targetNamespace: web3
- timeout: 5m
- wait: true
@@ (root level) @@
# helm.toolkit.fluxcd.io/v2/HelmRelease/web3/guard
! - one document removed:
- apiVersion: helm.toolkit.fluxcd.io/v2
- kind: HelmRelease
- metadata:
- name: guard
- namespace: web3
- labels:
- kustomize.toolkit.fluxcd.io/name: guard
- kustomize.toolkit.fluxcd.io/namespace: web3
- spec:
- chartRef:
- name: app-template
- kind: OCIRepository
- install:
- crds: CreateReplace
- interval: 1h
- rollback:
- cleanupOnFail: true
- upgrade:
- cleanupOnFail: true
- crds: CreateReplace
- remediation:
- remediateLastFailure: true
- retries: 2
- strategy:
- name: RemediateOnFailure
- values:
- controllers:
- guard:
- annotations:
- reloader.stakater.com/auto: "true"
- containers:
- app:
- resources:
- limits:
- cpu: 100m
- memory: 128Mi
- requests:
- cpu: 10m
- memory: 32Mi
- command:
- - python3
- - /config/controller.py
- image:
- repository: python
- tag: "3.14-slim@sha256:a7fb1e634c4a578f9e0bd6327f11a3cde11b7a9395f48e24360c0988bcc5c2bc"
- ports:
- - name: http
- containerPort: 8080
- securityContext:
- allowPrivilegeEscalation: false
- capabilities:
- drop:
- - ALL
- readOnlyRootFilesystem: true
- probes:
- liveness:
- spec:
- httpGet:
- path: /healthz
- port: http
- custom: true
- enabled: true
- readiness:
- spec:
- httpGet:
- path: /readyz
- port: http
- custom: true
- enabled: true
- defaultPodOptions:
- affinity:
- nodeAffinity:
- preferredDuringSchedulingIgnoredDuringExecution:
- - preference:
- matchExpressions:
- - key: kubernetes.io/hostname
- operator: In
- values:
- - control-1
- weight: 100
- automountServiceAccountToken: false
- securityContext:
- fsGroup: 65532
- fsGroupChangePolicy: OnRootMismatch
- runAsGroup: 65532
- runAsNonRoot: true
- runAsUser: 65532
- seccompProfile:
- type: RuntimeDefault
- terminationGracePeriodSeconds: 15
- persistence:
- config:
- name: guard
- type: configMap
- globalMounts:
- - path: /config
- readOnly: true
- tmp:
- type: emptyDir
- globalMounts:
- - path: /tmp
- service:
- app:
- controller: guard
- ports:
- http:
- port: 8080
@@ (root level) @@
# monitoring.coreos.com/v1/PrometheusRule/web3/guard-rules
! - one document removed:
- apiVersion: monitoring.coreos.com/v1
- kind: PrometheusRule
- metadata:
- name: guard-rules
- namespace: web3
- labels:
- kustomize.toolkit.fluxcd.io/name: guard
- kustomize.toolkit.fluxcd.io/namespace: web3
- spec:
- groups:
- - name: guard.rules
- rules:
- - alert: XmrigGuardEnforcementBypassed
- annotations:
- summary: "{{ $labels.deployment }} is running while its node's thermal gate should be closed — the KEDA trip is not being enforced (operator down, HPA wedged, or manual scale)"
- expr: |
- kube_deployment_status_replicas{namespace="web3", deployment=~"xmrig-control-.+"} > 0
- unless on (deployment)
- label_replace(
- (xmrig_guard_safe == 1)
- and on (node) (count by (node) (xmrig_guard_safe) == 1)
- and on (node) (timestamp(xmrig_guard_safe) >= time() - 120),
- "deployment", "xmrig-$1", "node", "(.*)"
- )
- for: 10m
- labels:
- severity: critical
- - alert: XmrigGuardAbsent
- annotations:
- summary: "Guard safety signal missing or wrong cardinality — the KEDA gate is closed and mining is silently disabled (fails closed)"
- expr: |
- count(xmrig_guard_safe{node=~"control-[123]"}) != 3
- or absent(xmrig_guard_safe{node=~"control-[123]"})
- for: 15m
- labels:
- severity: warning
- - alert: XmrigGuardLatchedUnsafe
- annotations:
- summary: "{{ $labels.node }} has been gated unsafe for 6h while its NVMe stayed within the 62C recovery band, so the guard is latched rather than thermally tripped and mining is disabled with no signal"
- expr: |
- max_over_time(xmrig_guard_safe[6h]) == 0
- and on (node)
- max_over_time(xmrig_guard_nvme_temp_max_celsius[2h]) <= 62
- for: 30m
- labels:
- severity: warning
- - alert: XmrigGuardThermalPanic
- annotations:
- summary: "{{ $labels.node }} is above the 67C panic trip with a miner still running — the zero-dwell fast path did not shed it and the drive is near its 70C rating"
- expr: |
- max by (node) (xmrig_guard_nvme_temp_max_celsius) > 67
- and on (node)
- label_replace(
- kube_deployment_status_replicas{namespace="web3", deployment=~"xmrig-control-.+"} > 0,
- "node", "$1", "deployment", "xmrig-(.*)"
- )
- for: 2m
- labels:
- severity: critical
- - alert: XmrigGuardEvaluationErrors
- annotations:
- summary: "Guard evaluations are failing repeatedly — affected nodes fail closed and can never re-earn safe=1 until queries recover"
- expr: "sum(rate(xmrig_guard_query_errors_total[10m])) > 0.01"
- for: 15m
- labels:
- severity: warning
@@ (root level) @@
# monitoring.coreos.com/v1/ServiceMonitor/web3/guard
! - one document removed:
- apiVersion: monitoring.coreos.com/v1
- kind: ServiceMonitor
- metadata:
- name: guard
- namespace: web3
- labels:
- kustomize.toolkit.fluxcd.io/name: guard
- kustomize.toolkit.fluxcd.io/namespace: web3
- spec:
- endpoints:
- - port: http
- selector:
- matchLabels:
- app.kubernetes.io/instance: guard
- app.kubernetes.io/name: guard
@@ (root level) @@
# v1/ConfigMap/web3/guard
! - one document removed:
- apiVersion: v1
- kind: ConfigMap
- metadata:
- name: guard
- namespace: web3
- annotations:
- kustomize.toolkit.fluxcd.io/substitute: disabled
- labels:
- kustomize.toolkit.fluxcd.io/name: guard
- kustomize.toolkit.fluxcd.io/namespace: web3
- data:
- controller.py: |
- """Small, dependency-free XMRig safety signal controller.
-
- The controller deliberately treats telemetry as untrusted input. A complete
- set of fresh samples is required before a node can become safe. Policy is
- code: changing thresholds, sensors, or timing requires a reviewed diff here.
- """
- import json
- import logging
- import math
- import threading
- import time
- import urllib.error
- import urllib.parse
- import urllib.request
- from dataclasses import dataclass
- from datetime import datetime, timezone
- from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
-
- # temp1 is the Composite sensor, which is what the drives' 70C rating specifies and
- # what smartctl reports. temp2-temp4 are internal die sensors that run ~9C hotter and
- # carry no comparable rating, so including them gated a Composite threshold against
- # the wrong reading. Fewer series also means fewer chances for a single missing sample
- # to fail the identity check and latch a node closed.
- SENSORS = {
- # empty tuple = no NVMe visible to this node (control-1 is a VM); it is gated on CPU
- # headroom instead, which is what every `if sensors` branch below keys off.
- "control-1": (),
- "control-2": (("nvme_nvme0", "temp1"), ("nvme_nvme1", "temp1")),
- "control-3": (("nvme_nvme0", "temp1"), ("nvme_nvme1", "temp1")),
- }
- # Miner slot priority, best first. control-1 has no NVMe to cook and was safe 96% of the
- # last 7d against 60%/30% for the bare-metal nodes. Each ScaledObject counts how many safe
- # nodes outrank it and subtracts one miner's draw per rank, so this order decides who gets
- # scarce watts. It lives here rather than in the manifests because it is a property of the
- # node telemetry, and a future headroom-derived ranking replaces this tuple alone.
- PRIORITY = ("control-1", "control-2", "control-3")
- ENDPOINT = "http://vmauth-victoria-metrics.observability.svc.cluster.local:8427"
- EVALUATION_INTERVAL_SECONDS = 30
- SOURCE_SAMPLE_MAX_AGE_SECONDS = 120
- # The CPU path joins cadvisor, which scrapes at 60s where the other six sources scrape at 20s,
- # and query_cpu dates the observation by min() across all of them. At 120s control-1 ran an age
- # p99 of 102s against that ceiling and self-invalidated 465 times in 10.26d (45 and 49 on the
- # NVMe nodes), each latching a 600s recovery dwell that drained the miner that caused it.
- CPU_SAMPLE_MAX_AGE_SECONDS = 180
- # 7 serial queries per CPU evaluation: at 10s a hung VictoriaMetrics blocked 70s against a 30s interval
- HTTP_TIMEOUT_SECONDS = 5
-
-
- def _dt(value):
- value = float(value)
- if not math.isfinite(value):
- raise ValueError("non-finite timestamp")
- return datetime.fromtimestamp(value, timezone.utc)
-
-
- def _fresh(timestamp, evaluation, max_age):
- """Return whether a source timestamp is not future-dated or too old."""
- age = (evaluation - timestamp).total_seconds()
- return 0 <= age <= max_age
-
-
- class DwellPolicy:
- def __init__(self, recovery_limit, trip_limit, recovery_dwell, trip_dwell, max_gap_seconds=SOURCE_SAMPLE_MAX_AGE_SECONDS, panic_limit=None):
- self.recovery_limit, self.trip_limit = recovery_limit, trip_limit
- self.recovery_dwell, self.trip_dwell = recovery_dwell, trip_dwell
- self.panic_limit = panic_limit
- self.max_gap = float(max_gap_seconds)
- self.safe = False
- self._last_source = None
- self._pending = None
- self._since = None
-
- def invalidate(self):
- self.safe = False
- self._last_source = self._pending = self._since = None
-
- def observe(self, value, source_time, monotonic_now):
- if not isinstance(value, (int, float)) or not math.isfinite(value) or not isinstance(source_time, datetime):
- self.invalidate()
- return False
- source_seconds = source_time.timestamp()
- if self._last_source is not None:
- gap = source_seconds - self._last_source
- if gap <= 0:
- return self.safe # duplicate/out-of-order samples cannot advance dwell
- if gap > self.max_gap:
- self._pending = self._since = None
- self._last_source = source_seconds
- # third, dwell-free state: 2C from the rating, confirming the reading over a 60s dwell
- # would spend most of the margin. Clearing _pending forces a full recovery dwell after.
- if self.panic_limit is not None and value >= self.panic_limit:
- self.safe = False
- self._pending = self._since = None
- return False
- kind = "recover" if value <= self.recovery_limit else "trip" if value >= self.trip_limit else None
- if kind is None:
- self._pending = self._since = None
- return self.safe
- if (kind == "recover") == self.safe:
- self._pending = self._since = None
- return self.safe
- if kind != self._pending:
- self._pending, self._since = kind, monotonic_now
- else:
- dwell = self.recovery_dwell if kind == "recover" else self.trip_dwell
- if monotonic_now - self._since >= dwell:
- self.safe = kind == "recover"
- self._pending = self._since = None
- return self.safe
-
-
- @dataclass(frozen=True)
- class Source:
- value: float
- timestamp: datetime
-
-
- @dataclass(frozen=True)
- class CPUObservation:
- host: Source
- xmrig: Source | None
- presence: Source
-
-
- def cpu_value(observation):
- if not isinstance(observation.host, Source):
- raise ValueError("host source is required")
- xmrig = observation.xmrig.value if observation.xmrig else 0.0
- return max(0.0, min(100.0, observation.host.value - xmrig))
-
-
- class VictoriaMetricsClient:
- def __init__(self, endpoint, transport=None, timeout=HTTP_TIMEOUT_SECONDS, step_seconds=120):
- self.endpoint = endpoint.rstrip("/")
- self.transport = transport or _HTTPTransport(timeout)
- self.step = f"{int(step_seconds)}s"
-
- def _query(self, expression, evaluation):
- # explicit step: default 5m step makes timestamp(a or b) snap to 5-min boundaries,
- # which made 60% of freshness checks fail; step=max_age also bounds VM lookbehind
- params = {"query": expression, "time": evaluation.astimezone(timezone.utc).isoformat().replace("+00:00", "Z"), "step": self.step}
- try:
- payload = self.transport.get(self.endpoint + "/api/v1/query", params)
- except urllib.error.HTTPError as exc:
- # a rejected query is undiagnosable without its text: 40h of bare 422s went unseen
- raise ValueError(f"VictoriaMetrics rejected ({exc.code}): {expression}") from exc
- if payload.get("status") != "success" or payload.get("data", {}).get("resultType") != "vector" or not isinstance(payload["data"].get("result"), list):
- raise ValueError(f"invalid VictoriaMetrics response for: {expression}")
- return payload["data"]["result"]
-
- @staticmethod
- def _sources(rows, identity=None, raw_timestamp=False):
- out = {}
- for row in rows:
- metric = row.get("metric", {})
- key = identity(metric) if identity else tuple(sorted(metric.items()))
- if key in out or not isinstance(row.get("value"), list) or len(row["value"]) != 2:
- raise ValueError("malformed or duplicate telemetry")
- try:
- value = float(row["value"][1])
- timestamp = _dt(row["value"][1] if raw_timestamp else row["value"][0])
- except (TypeError, ValueError, OverflowError) as exc:
- raise ValueError("malformed telemetry value") from exc
- if not math.isfinite(value):
- raise ValueError("non-finite telemetry value")
- out[key] = Source(value, timestamp)
- return out
-
- def query_nvme(self, node, sensors, evaluation):
- parts = [f'node_hwmon_temp_celsius{{kubernetes_node="{node}",chip="{chip}",sensor="{sensor}"}}' for chip, sensor in sensors]
- expression = " or ".join(parts)
- rows = self._query(expression, evaluation)
- # timestamp() over an or-expression becomes a step-aligned subquery in VictoriaMetrics
- # (fake boundary stamps broke 60% of freshness checks); single selectors return raw stamps
- timestamps = [row for part in parts for row in self._query("timestamp(" + part + ")", evaluation)]
- key = lambda m: (m.get("chip"), m.get("sensor"))
- if any(row.get("metric", {}).get("kubernetes_node") != node for row in rows + timestamps):
- raise ValueError("NVMe node identity changed")
- found = self._sources(rows, key)
- stamped = self._sources(timestamps, key, raw_timestamp=True)
- if set(found) != set(sensors):
- raise ValueError("incomplete or changed NVMe identity set")
- if set(stamped) != set(sensors):
- raise ValueError("incomplete or changed NVMe timestamp identity set")
- result = []
- for item in sensors:
- if not -40 <= found[item].value <= 150:
- raise ValueError("implausible NVMe temperature")
- result.append(Source(found[item].value, stamped[item].timestamp))
- return result
-
- def query_cpu(self, node, evaluation, window="5m"):
- host_raw = f'node_cpu_seconds_total{{kubernetes_node="{node}",mode!="idle"}}'
- idle_raw = f'node_cpu_seconds_total{{kubernetes_node="{node}",mode="idle"}}'
- cadvisor_raw = f'container_cpu_usage_seconds_total{{node="{node}",namespace="web3",container!="",container!="POD"}}'
- host_query = f'sum(rate({host_raw}[{window}])) / count(count({idle_raw}) by (cpu)) * 100'
- ksm_raw = 'kube_pod_info{namespace="web3"}'
- label_selector = 'namespace="web3",label_app_kubernetes_io_component="thermal-guarded"'
- node_pods_raw = f'kube_pod_info{{namespace="web3",node="{node}"}}'
- xmrig_presence_raw = f'kube_pod_labels{{{label_selector}}} * on(namespace,pod) group_left(node) {node_pods_raw}'
- xmrig_presence_query = f'count({xmrig_presence_raw}) or vector(0)'
- xmrig_query = f'100 * (sum((sum by (namespace,pod) (rate({cadvisor_raw}[{window}])) * on(namespace,pod) group_left(node) ({xmrig_presence_raw})) or vector(0)) / count(count({idle_raw}) by (cpu)))'
- labels_raw = f'kube_pod_labels{{{label_selector}}}'
- def one(query):
- values = self._sources(self._query(query, evaluation))
- if len(values) != 1:
- raise ValueError(f"CPU source must be one scalar: {query}")
- return next(iter(values.values()))
- def oldest(*selectors):
- out = []
- for selector in selectors:
- rows = self._query("timestamp(" + selector + ")", evaluation)
- if not rows:
- raise ValueError(f"missing raw timestamps: {selector}")
- out.extend(self._sources(rows, raw_timestamp=True).values())
- return min(item.timestamp for item in out)
- presence = one(xmrig_presence_query)
- if presence.value < 0:
- raise ValueError("invalid labelled XMRig presence source")
- if presence.value > 0:
- # the membership stamps date both the presence count and the subtraction, so
- # they are fetched once and reused rather than queried twice per evaluation
- membership = oldest(labels_raw, node_pods_raw)
- presence = Source(presence.value, membership)
- # cadvisor joins the same membership selectors, so cadvisor freshness is
- # verified exactly when its data enters the subtraction
- xmrig = Source(one(xmrig_query).value, min(membership, oldest(cadvisor_raw)))
- else:
- # no labelled miner on this node: anchor presence freshness to pod-info stamps
- presence = Source(0, oldest(ksm_raw))
- xmrig = None
- host = Source(one(host_query).value, oldest(host_raw))
- return CPUObservation(host, xmrig, presence)
-
-
- class _HTTPTransport:
- def __init__(self, timeout=HTTP_TIMEOUT_SECONDS):
- self.timeout = timeout
-
- def get(self, url, params):
- request = urllib.request.Request(url + "?" + urllib.parse.urlencode(params))
- with urllib.request.urlopen(request, timeout=self.timeout) as response:
- return json.load(response)
-
-
- class GuardController:
- def __init__(self, telemetry, clock=time.monotonic, wall_clock=lambda: datetime.now(timezone.utc)):
- self.telemetry = telemetry
- self.clock, self.wall_clock = clock, wall_clock
- # Trip 65C / recover 62C on Composite, against a 70C drive rating. Over 1151 measured
- # miner starts the rise is p50 0.88 / p90 1.13 / p99 1.35 C/min. Trip-to-drain is 135s
- # (30s evaluation + 60s dwell + 30s KEDA poll + 15s drain) = 3.0C at p99, and
- # test_trip_to_drain_budget pins it. Real bursts under these parameters peaked at p50 67.8
- # / p90 68.8 / max 69.8C, so the margin to the rating is 0.2C and nothing may be loosened.
- # max_gap is each node's slowest scrape: it dates the freshness check, the anti-replay gap
- # and the dwell reset, which must agree or a gap inside one budget resets dwell under
- # another. control-1 has no NVMe to cook and no panic limit; a busy CPU carries no
- # equivalent of a drive's absolute rating.
- self.policies = {
- "control-1": DwellPolicy(50, 70, 600, 120, CPU_SAMPLE_MAX_AGE_SECONDS),
- "control-2": DwellPolicy(62, 65, 180, 60, panic_limit=67),
- "control-3": DwellPolicy(62, 65, 180, 60, panic_limit=67),
- }
- self.ready = False
- self.metrics = {
- "evaluations": 0, "query_errors": {node: 0 for node in SENSORS},
- "safe": {node: 0 for node in SENSORS},
- "nvme_temp_max": {node: 0.0 for node in SENSORS if SENSORS[node]},
- "source_age_seconds": {node: 0.0 for node in SENSORS},
- "cpu_non_xmrig": {node: 0.0 for node in SENSORS if not SENSORS[node]},
- "rank": {node: PRIORITY.index(node) for node in SENSORS},
- }
- self._last_source_stamps = {node: {} for node in SENSORS}
-
- def _new_source_set(self, node, sources):
- # Keyed by source identity, not position: a miner starting or stopping adds or removes
- # the xmrig source, and comparing by position made that look like tampering. It failed
- # control-1 closed 19 times in 12h, draining the miner, which changed the set back.
- # Only keys in both sets are comparable; a genuinely new key is exempt for one cycle
- # rather than the whole evaluation going unchecked.
- stamps = {key: source.timestamp.timestamp() for key, source in sources.items()}
- previous = self._last_source_stamps[node]
- shared = stamps.keys() & previous.keys()
- if any(stamps[key] - previous[key] > self.policies[node].max_gap for key in shared):
- raise ValueError("source gap exceeded maximum")
- if shared and any(stamps[key] <= previous[key] for key in shared):
- return False
- self._last_source_stamps[node] = stamps
- return True
-
- def evaluate(self, evaluation=None):
- evaluation = evaluation or self.wall_clock()
- now = self.clock()
- for node, sensors in SENSORS.items():
- try:
- # no sensors means a node with no visible NVMe (control-1, a VM): it is gated
- # on CPU headroom instead. The dwell policy and metrics are keyed identically.
- if sensors:
- samples = dict(zip(sensors, self.telemetry.query_nvme(node, sensors, evaluation)))
- # trip on the hottest drive, date it by the newest sample it was read from
- value = max(item.value for item in samples.values())
- stamp = max(item.timestamp for item in samples.values())
- self.metrics["nvme_temp_max"][node] = value
- else:
- obs = self.telemetry.query_cpu(node, evaluation)
- samples = {"host": obs.host, "presence": obs.presence}
- if obs.xmrig:
- samples["xmrig"] = obs.xmrig
- value = cpu_value(obs)
- stamp = min(item.timestamp for item in samples.values())
- self.metrics["cpu_non_xmrig"][node] = value
- policy = self.policies[node]
- if not samples or not all(_fresh(item.timestamp, evaluation, policy.max_gap) for item in samples.values()):
- raise ValueError("stale or future source")
- safe = policy.observe(value, stamp, now) if self._new_source_set(node, samples) else policy.safe
- self.metrics["source_age_seconds"][node] = max(0.0, evaluation.timestamp() - min(item.timestamp for item in samples.values()).timestamp())
- self.metrics["safe"][node] = int(safe)
- except Exception as exc:
- # one line per failure, no traceback: the query text travels in the exception
- logging.error(f"evaluation failed for {node}: {exc!r}")
- self.metrics["query_errors"][node] += 1
- self.policies[node].invalidate()
- self._last_source_stamps[node] = {}
- self.metrics["safe"][node] = 0
- self.metrics["source_age_seconds"][node] = float("nan")
- if sensors:
- self.metrics["nvme_temp_max"][node] = float("nan")
- else:
- self.metrics["cpu_non_xmrig"][node] = float("nan")
- self.metrics["evaluations"] += 1
- self.ready = True
- return dict(self.metrics["safe"])
-
-
- def render_metrics(controller):
- m = controller.metrics
- lines = [f'xmrig_guard_evaluations_total {m["evaluations"]}']
- for metric, values in (
- ("safe", m["safe"]),
- ("query_errors_total", m["query_errors"]),
- ("source_age_seconds", m["source_age_seconds"]),
- ("nvme_temp_max_celsius", m["nvme_temp_max"]),
- ("cpu_non_xmrig_percent", m["cpu_non_xmrig"]),
- ("rank", m["rank"]),
- ):
- metric_name = "xmrig_guard_" + metric
- lines.extend(f'{metric_name}{{node="{node}"}} {value}' for node, value in values.items())
- return "\n".join(lines) + "\n"
-
-
- class _StatusHandler(BaseHTTPRequestHandler):
- controller = None # assigned before the server starts
- def do_GET(self):
- if self.path == "/healthz":
- self._send(200, "ok\n", "text/plain")
- elif self.path == "/readyz":
- self._send(200 if self.controller.ready else 503, "ready\n" if self.controller.ready else "not ready\n", "text/plain")
- elif self.path == "/metrics":
- body = render_metrics(self.controller)
- self._send(200, body, "text/plain; version=0.0.4")
- else:
- self._send(404, "not found\n", "text/plain")
- def _send(self, status, body, content_type):
- data = body.encode()
- self.send_response(status)
- self.send_header("Content-Type", content_type)
- self.send_header("Content-Length", str(len(data)))
- self.end_headers()
- self.wfile.write(data)
- def log_message(self, message_format, *args):
- return
-
-
- def main():
- # step bounds VM's lookbehind, so it must cover the loosest budget or the CPU path could never
- # return the 121-180s samples its own policy accepts. Per-node max_gap still rejects them for NVMe.
- controller = GuardController(VictoriaMetricsClient(ENDPOINT, step_seconds=CPU_SAMPLE_MAX_AGE_SECONDS))
- _StatusHandler.controller = controller
- server = ThreadingHTTPServer(("0.0.0.0", 8080), _StatusHandler)
- threading.Thread(target=server.serve_forever, daemon=True).start()
- while True:
- # sleep to a deadline, not a flat interval: sleeping after the work made the true
- # period drift by the evaluation duration, stretching it against a fixed freshness budget
- deadline = time.monotonic() + EVALUATION_INTERVAL_SECONDS
- controller.evaluate()
- time.sleep(max(0, deadline - time.monotonic()))
-
-
- if __name__ == "__main__":
- main()
@@ (root level) @@
# kustomize.toolkit.fluxcd.io/v1/Kustomization/web3/xmrig-guard
! + one document added:
+ apiVersion: kustomize.toolkit.fluxcd.io/v1
+ kind: Kustomization
+ metadata:
+ name: xmrig-guard
+ namespace: web3
+ labels:
+ kustomize.toolkit.fluxcd.io/name: cluster-apps
+ kustomize.toolkit.fluxcd.io/namespace: flux-system
+ spec:
+ decryption:
+ provider: sops
+ secretRef:
+ name: sops-age
+ deletionPolicy: WaitForTermination
+ interval: 30m
+ patches:
+ - patch: |
+ apiVersion: helm.toolkit.fluxcd.io/v2
+ kind: HelmRelease
+ metadata:
+ name: _
+ spec:
+ install:
+ crds: CreateReplace
+ rollback:
+ cleanupOnFail: true
+ upgrade:
+ cleanupOnFail: true
+ crds: CreateReplace
+ strategy:
+ name: RemediateOnFailure
+ remediation:
+ remediateLastFailure: true
+ retries: 2
+ target:
+ kind: HelmRelease
+ group: helm.toolkit.fluxcd.io
+ path: ./kubernetes/apps/web3/xmrig-guard/app
+ postBuild:
+ substituteFrom:
+ - name: cluster-settings
+ kind: ConfigMap
+ optional: false
+ - name: cluster-secrets
+ kind: Secret
+ optional: false
+ prune: true
+ retryInterval: 2m
+ sourceRef:
+ name: flux-system
+ kind: GitRepository
+ namespace: flux-system
+ targetNamespace: web3
+ timeout: 5m
+ wait: true
@@ (root level) @@
# helm.toolkit.fluxcd.io/v2/HelmRelease/web3/xmrig-guard
! + one document added:
+ apiVersion: helm.toolkit.fluxcd.io/v2
+ kind: HelmRelease
+ metadata:
+ name: xmrig-guard
+ namespace: web3
+ labels:
+ kustomize.toolkit.fluxcd.io/name: xmrig-guard
+ kustomize.toolkit.fluxcd.io/namespace: web3
+ spec:
+ chartRef:
+ name: app-template
+ kind: OCIRepository
+ install:
+ crds: CreateReplace
+ interval: 1h
+ rollback:
+ cleanupOnFail: true
+ upgrade:
+ cleanupOnFail: true
+ crds: CreateReplace
+ remediation:
+ remediateLastFailure: true
+ retries: 2
+ strategy:
+ name: RemediateOnFailure
+ values:
+ controllers:
+ xmrig-guard:
+ annotations:
+ reloader.stakater.com/auto: "true"
+ containers:
+ app:
+ resources:
+ limits:
+ cpu: 100m
+ memory: 128Mi
+ requests:
+ cpu: 10m
+ memory: 32Mi
+ command:
+ - python3
+ - /config/controller.py
+ image:
+ repository: python
+ tag: "3.14-slim@sha256:a7fb1e634c4a578f9e0bd6327f11a3cde11b7a9395f48e24360c0988bcc5c2bc"
+ ports:
+ - name: http
+ containerPort: 8080
+ securityContext:
+ allowPrivilegeEscalation: false
+ capabilities:
+ drop:
+ - ALL
+ readOnlyRootFilesystem: true
+ probes:
+ liveness:
+ spec:
+ httpGet:
+ path: /healthz
+ port: http
+ custom: true
+ enabled: true
+ readiness:
+ spec:
+ httpGet:
+ path: /readyz
+ port: http
+ custom: true
+ enabled: true
+ defaultPodOptions:
+ affinity:
+ nodeAffinity:
+ preferredDuringSchedulingIgnoredDuringExecution:
+ - preference:
+ matchExpressions:
+ - key: kubernetes.io/hostname
+ operator: In
+ values:
+ - control-1
+ weight: 100
+ automountServiceAccountToken: false
+ securityContext:
+ fsGroup: 65532
+ fsGroupChangePolicy: OnRootMismatch
+ runAsGroup: 65532
+ runAsNonRoot: true
+ runAsUser: 65532
+ seccompProfile:
+ type: RuntimeDefault
+ terminationGracePeriodSeconds: 15
+ persistence:
+ config:
+ name: xmrig-guard
+ type: configMap
+ globalMounts:
+ - path: /config
+ readOnly: true
+ tmp:
+ type: emptyDir
+ globalMounts:
+ - path: /tmp
+ service:
+ app:
+ controller: xmrig-guard
+ ports:
+ http:
+ port: 8080
@@ (root level) @@
# monitoring.coreos.com/v1/PrometheusRule/web3/xmrig-guard-rules
! + one document added:
+ apiVersion: monitoring.coreos.com/v1
+ kind: PrometheusRule
+ metadata:
+ name: xmrig-guard-rules
+ namespace: web3
+ labels:
+ kustomize.toolkit.fluxcd.io/name: xmrig-guard
+ kustomize.toolkit.fluxcd.io/namespace: web3
+ spec:
+ groups:
+ - name: xmrig-guard.rules
+ rules:
+ - alert: XmrigGuardEnforcementBypassed
+ annotations:
+ summary: "{{ $labels.deployment }} is running while its node's thermal gate should be closed — the KEDA trip is not being enforced (operator down, HPA wedged, or manual scale)"
+ expr: |
+ kube_deployment_status_replicas{namespace="web3", deployment=~"xmrig-control-.+"} > 0
+ unless on (deployment)
+ label_replace(
+ (xmrig_guard_safe == 1)
+ and on (node) (count by (node) (xmrig_guard_safe) == 1)
+ and on (node) (timestamp(xmrig_guard_safe) >= time() - 120),
+ "deployment", "xmrig-$1", "node", "(.*)"
+ )
+ for: 10m
+ labels:
+ severity: critical
+ - alert: XmrigGuardAbsent
+ annotations:
+ summary: "Guard safety signal missing or wrong cardinality — the KEDA gate is closed and mining is silently disabled (fails closed)"
+ expr: |
+ count(xmrig_guard_safe{node=~"control-[123]"}) != 3
+ or absent(xmrig_guard_safe{node=~"control-[123]"})
+ for: 15m
+ labels:
+ severity: warning
+ - alert: XmrigGuardLatchedUnsafe
+ annotations:
+ summary: "{{ $labels.node }} has been gated unsafe for 6h while its NVMe stayed within the 62C recovery band, so the guard is latched rather than thermally tripped and mining is disabled with no signal"
+ expr: |
+ max_over_time(xmrig_guard_safe[6h]) == 0
+ and on (node)
+ max_over_time(xmrig_guard_nvme_temp_max_celsius[2h]) <= 62
+ for: 30m
+ labels:
+ severity: warning
+ - alert: XmrigGuardThermalPanic
+ annotations:
+ summary: "{{ $labels.node }} is above the 68C panic trip with a miner still running — the zero-dwell fast path did not shed it and the drive is near its 70C rating"
+ expr: |
+ max by (node) (xmrig_guard_nvme_temp_max_celsius) > 68
+ and on (node)
+ label_replace(
+ kube_deployment_status_replicas{namespace="web3", deployment=~"xmrig-control-.+"} > 0,
+ "node", "$1", "deployment", "xmrig-(.*)"
+ )
+ for: 2m
+ labels:
+ severity: critical
+ - alert: XmrigGuardEvaluationErrors
+ annotations:
+ summary: "Guard evaluations are failing repeatedly — affected nodes fail closed and can never re-earn safe=1 until queries recover"
+ expr: "sum(rate(xmrig_guard_query_errors_total[10m])) > 0.01"
+ for: 15m
+ labels:
+ severity: warning
@@ (root level) @@
# monitoring.coreos.com/v1/ServiceMonitor/web3/xmrig-guard
! + one document added:
+ apiVersion: monitoring.coreos.com/v1
+ kind: ServiceMonitor
+ metadata:
+ name: xmrig-guard
+ namespace: web3
+ labels:
+ kustomize.toolkit.fluxcd.io/name: xmrig-guard
+ kustomize.toolkit.fluxcd.io/namespace: web3
+ spec:
+ endpoints:
+ - port: http
+ selector:
+ matchLabels:
+ app.kubernetes.io/instance: xmrig-guard
+ app.kubernetes.io/name: xmrig-guard
@@ (root level) @@
# v1/ConfigMap/web3/xmrig-guard
! + one document added:
+ apiVersion: v1
+ kind: ConfigMap
+ metadata:
+ name: xmrig-guard
+ namespace: web3
+ annotations:
+ kustomize.toolkit.fluxcd.io/substitute: disabled
+ labels:
+ kustomize.toolkit.fluxcd.io/name: xmrig-guard
+ kustomize.toolkit.fluxcd.io/namespace: web3
+ data:
+ controller.py: |
+ """Small, dependency-free XMRig safety signal controller.
+
+ The controller deliberately treats telemetry as untrusted input. A complete
+ set of fresh samples is required before a node can become safe. Policy is
+ code: changing thresholds, sensors, or timing requires a reviewed diff here.
+ """
+ import json
+ import logging
+ import math
+ import threading
+ import time
+ import urllib.error
+ import urllib.parse
+ import urllib.request
+ from dataclasses import dataclass
+ from datetime import datetime, timezone
+ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+ # temp1 is the Composite sensor, which is what the drives' 70C rating specifies and
+ # what smartctl reports. temp2-temp4 are internal die sensors that run ~9C hotter and
+ # carry no comparable rating, so including them gated a Composite threshold against
+ # the wrong reading. Fewer series also means fewer chances for a single missing sample
+ # to fail the identity check and latch a node closed.
+ SENSORS = {
+ # empty tuple = no NVMe visible to this node (control-1 is a VM); it is gated on CPU
+ # headroom instead, which is what every `if sensors` branch below keys off.
+ "control-1": (),
+ "control-2": (("nvme_nvme0", "temp1"), ("nvme_nvme1", "temp1")),
+ "control-3": (("nvme_nvme0", "temp1"), ("nvme_nvme1", "temp1")),
+ }
+ # Miner slot priority, best first. control-1 has no NVMe to cook and was safe 96% of the
+ # last 7d against 60%/30% for the bare-metal nodes. Each ScaledObject counts how many safe
+ # nodes outrank it and subtracts one miner's draw per rank, so this order decides who gets
+ # scarce watts. It lives here rather than in the manifests because it is a property of the
+ # node telemetry, and a future headroom-derived ranking replaces this tuple alone.
+ PRIORITY = ("control-1", "control-2", "control-3")
+ ENDPOINT = "http://vmauth-victoria-metrics.observability.svc.cluster.local:8427"
+ EVALUATION_INTERVAL_SECONDS = 30
+ SOURCE_SAMPLE_MAX_AGE_SECONDS = 120
+ MAX_SOURCE_GAP_SECONDS = 120
+ # 7 serial queries per CPU evaluation: at 10s a hung VictoriaMetrics blocked 70s against a 30s interval
+ HTTP_TIMEOUT_SECONDS = 5
+
+
+ def _dt(value):
+ value = float(value)
+ if not math.isfinite(value):
+ raise ValueError("non-finite timestamp")
+ return datetime.fromtimestamp(value, timezone.utc)
+
+
+ def _fresh(timestamp, evaluation, max_age):
+ """Return whether a source timestamp is not future-dated or too old."""
+ age = (evaluation - timestamp).total_seconds()
+ return 0 <= age <= max_age
+
+
+ class DwellPolicy:
+ def __init__(self, recovery_limit, trip_limit, recovery_dwell, trip_dwell, max_gap_seconds=120, panic_limit=None):
+ self.recovery_limit, self.trip_limit = recovery_limit, trip_limit
+ self.recovery_dwell, self.trip_dwell = recovery_dwell, trip_dwell
+ self.panic_limit = panic_limit
+ self.max_gap = float(max_gap_seconds)
+ self.safe = False
+ self._last_source = None
+ self._pending = None
+ self._since = None
+
+ def invalidate(self):
+ self.safe = False
+ self._last_source = self._pending = self._since = None
+
+ def observe(self, value, source_time, monotonic_now):
+ if not isinstance(value, (int, float)) or not math.isfinite(value) or not isinstance(source_time, datetime):
+ self.invalidate()
+ return False
+ source_seconds = source_time.timestamp()
+ if self._last_source is not None:
+ gap = source_seconds - self._last_source
+ if gap <= 0:
+ return self.safe # duplicate/out-of-order samples cannot advance dwell
+ if gap > self.max_gap:
+ self._pending = self._since = None
+ self._last_source = source_seconds
+ # third, dwell-free state: 2C from the rating, confirming the reading over a 60s dwell
+ # would spend most of the margin. Clearing _pending forces a full recovery dwell after.
+ if self.panic_limit is not None and value >= self.panic_limit:
+ self.safe = False
+ self._pending = self._since = None
+ return False
+ kind = "recover" if value <= self.recovery_limit else "trip" if value >= self.trip_limit else None
+ if kind is None:
+ self._pending = self._since = None
+ return self.safe
+ if (kind == "recover") == self.safe:
+ self._pending = self._since = None
+ return self.safe
+ if kind != self._pending:
+ self._pending, self._since = kind, monotonic_now
+ else:
+ dwell = self.recovery_dwell if kind == "recover" else self.trip_dwell
+ if monotonic_now - self._since >= dwell:
+ self.safe = kind == "recover"
+ self._pending = self._since = None
+ return self.safe
+
+
+ @dataclass(frozen=True)
+ class Source:
+ value: float
+ timestamp: datetime
+
+
+ @dataclass(frozen=True)
+ class CPUObservation:
+ host: Source
+ xmrig: Source | None
+ presence: Source
+
+
+ def cpu_value(observation):
+ if not isinstance(observation.host, Source):
+ raise ValueError("host source is required")
+ xmrig = observation.xmrig.value if observation.xmrig else 0.0
+ return max(0.0, min(100.0, observation.host.value - xmrig))
+
+
+ class VictoriaMetricsClient:
+ def __init__(self, endpoint, transport=None, timeout=HTTP_TIMEOUT_SECONDS, step_seconds=120):
+ self.endpoint = endpoint.rstrip("/")
+ self.transport = transport or _HTTPTransport(timeout)
+ self.step = f"{int(step_seconds)}s"
+
+ def _query(self, expression, evaluation):
+ # explicit step: default 5m step makes timestamp(a or b) snap to 5-min boundaries,
+ # which made 60% of freshness checks fail; step=max_age also bounds VM lookbehind
+ params = {"query": expression, "time": evaluation.astimezone(timezone.utc).isoformat().replace("+00:00", "Z"), "step": self.step}
+ try:
+ payload = self.transport.get(self.endpoint + "/api/v1/query", params)
+ except urllib.error.HTTPError as exc:
+ # a rejected query is undiagnosable without its text: 40h of bare 422s went unseen
+ raise ValueError(f"VictoriaMetrics rejected ({exc.code}): {expression}") from exc
+ if payload.get("status") != "success" or payload.get("data", {}).get("resultType") != "vector" or not isinstance(payload["data"].get("result"), list):
+ raise ValueError(f"invalid VictoriaMetrics response for: {expression}")
+ return payload["data"]["result"]
+
+ @staticmethod
+ def _sources(rows, identity=None, raw_timestamp=False):
+ out = {}
+ for row in rows:
+ metric = row.get("metric", {})
+ key = identity(metric) if identity else tuple(sorted(metric.items()))
+ if key in out or not isinstance(row.get("value"), list) or len(row["value"]) != 2:
+ raise ValueError("malformed or duplicate telemetry")
+ try:
+ value = float(row["value"][1])
+ timestamp = _dt(row["value"][1] if raw_timestamp else row["value"][0])
+ except (TypeError, ValueError, OverflowError) as exc:
+ raise ValueError("malformed telemetry value") from exc
+ if not math.isfinite(value):
+ raise ValueError("non-finite telemetry value")
+ out[key] = Source(value, timestamp)
+ return out
+
+ def query_nvme(self, node, sensors, evaluation):
+ parts = [f'node_hwmon_temp_celsius{{kubernetes_node="{node}",chip="{chip}",sensor="{sensor}"}}' for chip, sensor in sensors]
+ expression = " or ".join(parts)
+ rows = self._query(expression, evaluation)
+ # timestamp() over an or-expression becomes a step-aligned subquery in VictoriaMetrics
+ # (fake boundary stamps broke 60% of freshness checks); single selectors return raw stamps
+ timestamps = [row for part in parts for row in self._query("timestamp(" + part + ")", evaluation)]
+ key = lambda m: (m.get("chip"), m.get("sensor"))
+ if any(row.get("metric", {}).get("kubernetes_node") != node for row in rows + timestamps):
+ raise ValueError("NVMe node identity changed")
+ found = self._sources(rows, key)
+ stamped = self._sources(timestamps, key, raw_timestamp=True)
+ if set(found) != set(sensors):
+ raise ValueError("incomplete or changed NVMe identity set")
+ if set(stamped) != set(sensors):
+ raise ValueError("incomplete or changed NVMe timestamp identity set")
+ result = []
+ for item in sensors:
+ if not -40 <= found[item].value <= 150:
+ raise ValueError("implausible NVMe temperature")
+ result.append(Source(found[item].value, stamped[item].timestamp))
+ return result
+
+ def query_cpu(self, node, evaluation, window="5m"):
+ host_raw = f'node_cpu_seconds_total{{kubernetes_node="{node}",mode!="idle"}}'
+ idle_raw = f'node_cpu_seconds_total{{kubernetes_node="{node}",mode="idle"}}'
+ cadvisor_raw = f'container_cpu_usage_seconds_total{{node="{node}",namespace="web3",container!="",container!="POD"}}'
+ host_query = f'sum(rate({host_raw}[{window}])) / count(count({idle_raw}) by (cpu)) * 100'
+ ksm_raw = 'kube_pod_info{namespace="web3"}'
+ label_selector = 'namespace="web3",label_app_kubernetes_io_component="thermal-guarded"'
+ node_pods_raw = f'kube_pod_info{{namespace="web3",node="{node}"}}'
+ xmrig_presence_raw = f'kube_pod_labels{{{label_selector}}} * on(namespace,pod) group_left(node) {node_pods_raw}'
+ xmrig_presence_query = f'count({xmrig_presence_raw}) or vector(0)'
+ xmrig_query = f'100 * (sum((sum by (namespace,pod) (rate({cadvisor_raw}[{window}])) * on(namespace,pod) group_left(node) ({xmrig_presence_raw})) or vector(0)) / count(count({idle_raw}) by (cpu)))'
+ labels_raw = f'kube_pod_labels{{{label_selector}}}'
+ def one(query):
+ values = self._sources(self._query(query, evaluation))
+ if len(values) != 1:
+ raise ValueError(f"CPU source must be one scalar: {query}")
+ return next(iter(values.values()))
+ def oldest(*selectors):
+ out = []
+ for selector in selectors:
+ rows = self._query("timestamp(" + selector + ")", evaluation)
+ if not rows:
+ raise ValueError(f"missing raw timestamps: {selector}")
+ out.extend(self._sources(rows, raw_timestamp=True).values())
+ return min(item.timestamp for item in out)
+ presence = one(xmrig_presence_query)
+ if presence.value < 0:
+ raise ValueError("invalid labelled XMRig presence source")
+ if presence.value > 0:
+ # the membership stamps date both the presence count and the subtraction, so
+ # they are fetched once and reused rather than queried twice per evaluation
+ membership = oldest(labels_raw, node_pods_raw)
+ presence = Source(presence.value, membership)
+ # cadvisor joins the same membership selectors, so cadvisor freshness is
+ # verified exactly when its data enters the subtraction
+ xmrig = Source(one(xmrig_query).value, min(membership, oldest(cadvisor_raw)))
+ else:
+ # no labelled miner on this node: anchor presence freshness to pod-info stamps
+ presence = Source(0, oldest(ksm_raw))
+ xmrig = None
+ host = Source(one(host_query).value, oldest(host_raw))
+ return CPUObservation(host, xmrig, presence)
+
+
+ class _HTTPTransport:
+ def __init__(self, timeout=HTTP_TIMEOUT_SECONDS):
+ self.timeout = timeout
+
+ def get(self, url, params):
+ request = urllib.request.Request(url + "?" + urllib.parse.urlencode(params))
+ with urllib.request.urlopen(request, timeout=self.timeout) as response:
+ return json.load(response)
+
+
+ class GuardController:
+ def __init__(self, telemetry, clock=time.monotonic, wall_clock=lambda: datetime.now(timezone.utc)):
+ self.telemetry = telemetry
+ self.clock, self.wall_clock = clock, wall_clock
+ # Trip 65C / recover 62C on Composite, against a 70C drive rating. Mining raises
+ # Composite at up to 1.1C/min. Trip-to-drain is 135s (30s evaluation + 60s dwell + 30s
+ # KEDA poll + 15s drain), so 2.5C of rise to a ~67.5C peak; test_trip_to_drain_budget
+ # pins it. The 68C panic trip is dwell-free and peaks near 69.4C.
+ # The drives idle in the 60-64C band on warm afternoons, so recovery sits at 62C:
+ # replaying 7d puts control-2/3 at 55.0%/38.8% safe against 44.7%/31.5% at 60C/600s.
+ # control-1 is keyed the same way on CPU headroom rather than temperature; SENSORS
+ # decides which source feeds which node, so the policy dict needs no special case. No
+ # panic limit there: a busy CPU has no equivalent of a drive's absolute rating.
+ self.policies = {
+ "control-1": DwellPolicy(50, 70, 600, 120, MAX_SOURCE_GAP_SECONDS),
+ "control-2": DwellPolicy(62, 65, 300, 60, MAX_SOURCE_GAP_SECONDS, panic_limit=68),
+ "control-3": DwellPolicy(62, 65, 300, 60, MAX_SOURCE_GAP_SECONDS, panic_limit=68),
+ }
+ self.ready = False
+ self.metrics = {
+ "evaluations": 0, "query_errors": {node: 0 for node in SENSORS},
+ "safe": {node: 0 for node in SENSORS},
+ "nvme_temp_max": {node: 0.0 for node in SENSORS if SENSORS[node]},
+ "source_age_seconds": {node: 0.0 for node in SENSORS},
+ "cpu_non_xmrig": {node: 0.0 for node in SENSORS if not SENSORS[node]},
+ "rank": {node: PRIORITY.index(node) for node in SENSORS},
+ }
+ self._last_source_stamps = {node: {} for node in SENSORS}
+
+ def _new_source_set(self, node, sources):
+ # Keyed by source identity, not position: a miner starting or stopping adds or removes
+ # the xmrig source, and comparing by position made that look like tampering. It failed
+ # control-1 closed 19 times in 12h, draining the miner, which changed the set back.
+ # Only keys in both sets are comparable; a genuinely new key is exempt for one cycle
+ # rather than the whole evaluation going unchecked.
+ stamps = {key: source.timestamp.timestamp() for key, source in sources.items()}
+ previous = self._last_source_stamps[node]
+ shared = stamps.keys() & previous.keys()
+ if any(stamps[key] - previous[key] > MAX_SOURCE_GAP_SECONDS for key in shared):
+ raise ValueError("source gap exceeded maximum")
+ if shared and any(stamps[key] <= previous[key] for key in shared):
+ return False
+ self._last_source_stamps[node] = stamps
+ return True
+
+ def evaluate(self, evaluation=None):
+ evaluation = evaluation or self.wall_clock()
+ now = self.clock()
+ for node, sensors in SENSORS.items():
+ try:
+ # no sensors means a node with no visible NVMe (control-1, a VM): it is gated
+ # on CPU headroom instead. The dwell policy and metrics are keyed identically.
+ if sensors:
+ samples = dict(zip(sensors, self.telemetry.query_nvme(node, sensors, evaluation)))
+ # trip on the hottest drive, date it by the newest sample it was read from
+ value = max(item.value for item in samples.values())
+ stamp = max(item.timestamp for item in samples.values())
+ self.metrics["nvme_temp_max"][node] = value
+ else:
+ obs = self.telemetry.query_cpu(node, evaluation)
+ samples = {"host": obs.host, "presence": obs.presence}
+ if obs.xmrig:
+ samples["xmrig"] = obs.xmrig
+ value = cpu_value(obs)
+ stamp = min(item.timestamp for item in samples.values())
+ self.metrics["cpu_non_xmrig"][node] = value
+ if not all(_fresh(item.timestamp, evaluation, SOURCE_SAMPLE_MAX_AGE_SECONDS) for item in samples.values()):
+ raise ValueError("stale or future source")
+ policy = self.policies[node]
+ safe = policy.observe(value, stamp, now) if self._new_source_set(node, samples) else policy.safe
+ self.metrics["source_age_seconds"][node] = max(0.0, evaluation.timestamp() - min(item.timestamp for item in samples.values()).timestamp())
+ self.metrics["safe"][node] = int(safe)
+ except Exception as exc:
+ # one line per failure, no traceback: the query text travels in the exception
+ logging.error(f"evaluation failed for {node}: {exc!r}")
+ self.metrics["query_errors"][node] += 1
+ self.policies[node].invalidate()
+ self._last_source_stamps[node] = {}
+ self.metrics["safe"][node] = 0
+ self.metrics["source_age_seconds"][node] = float("nan")
+ if sensors:
+ self.metrics["nvme_temp_max"][node] = float("nan")
+ else:
+ self.metrics["cpu_non_xmrig"][node] = float("nan")
+ self.metrics["evaluations"] += 1
+ self.ready = True
+ return dict(self.metrics["safe"])
+
+
+ def render_metrics(controller):
+ m = controller.metrics
+ lines = [f'xmrig_guard_evaluations_total {m["evaluations"]}']
+ for metric, values in (
+ ("safe", m["safe"]),
+ ("query_errors_total", m["query_errors"]),
+ ("source_age_seconds", m["source_age_seconds"]),
+ ("nvme_temp_max_celsius", m["nvme_temp_max"]),
+ ("cpu_non_xmrig_percent", m["cpu_non_xmrig"]),
+ ("rank", m["rank"]),
+ ):
+ metric_name = "xmrig_guard_" +This message was truncated. Download full message |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains the following updates:
v2.7.5→v3.1.0v2.7.5→v3.1.0Release Notes
immich-app/immich (ghcr.io/immich-app/immich-machine-learning)
v3.1.0Compare Source
Welcome to Immich
v3.1.0!This release includes several quality of life improvements and another round of bug fixes. Keep reading below for the complete highlights.
Highlights
Upload wakelock (web)
Similar to mobile, the web application will now automatically acquire a wakelock to prevent the screen from dimming while uploads are happening. This feature will require an HTTPS connection.
Undo archive (web)
After archiving assets, the success notification now has an “undo” button to undo the action.
Filter assets by server filepath (workflows)
The asset file filter has a new option “Use path”. When set to true, it will filter based on the actual path on the server rather than the original file name.
Filter assets by EXIF metadata (workflows)
A new workflow filter has been added that allows you to filter assets based on their EXIF metadata fields. For now it only supports string comparisons, with numerical comparisons expected to come soon.
Quick navigate to maintenance page (web)
The command palette (
CTRL+Kor/) now includes an item for the maintenance page.Better slideshow button visibility (web)
The slideshow buttons on the web are now wrapped in a container with a backdrop blur, making them easier to view when the current asset is very light.
OIDC role claim sync improvements (OAuth)
The OAuth integration in Immich supports setting an initial
isAdminvalue for new users via a role claim. This same process has been updated to now syncisAdminon subsequent logins. Additionally, the role claim now supports both single values (immich_role: 'admin') in addition to lists of values (immich_role: ["admin", "user"]), making in more flexible.Invalidate sessions on password reset (admin-cli)
When using the admin command line interface to reset a password, there is now a new option to invalidate existing sessions.
Date range for map (mobile)
Similar to web, mobile now also supports filtering the map for assets within a given date range.
What's Changed
🚨 Breaking Changes
🚀 Features
🌟 Enhancements
🐛 Bug fixes
📚 Documentation
🌐 Translations
New Contributors
Full Changelog: immich-app/immich@v3.0.3...v3.1.0
v3.0.3Compare Source
v3.0.3
What's Changed
🚀 Features
🐛 Bug fixes
📚 Documentation
🌐 Translations
New Contributors
Full Changelog: immich-app/immich@v3.0.2...v3.0.3
v3.0.2Compare Source
v3.0.2
This patch release addresses various bugs from v3 release and adds an additional filter to workflows
What's Changed
🌟 Enhancements
🐛 Bug fixes
📚 Documentation
🌐 Translations
New Contributors
Full Changelog: immich-app/immich@v3.0.1...v3.0.2
v3.0.1Compare Source
v3.0.1
What's Changed
🚀 Features
🐛 Bug fixes
🌐 Translations
Full Changelog: immich-app/immich@v3.0.0...v3.0.1
v3.0.0Compare Source
v3.0.0
Welcome to Immich
v3.0.0!After months of hard work from the team and our amazing contributors, we're thrilled to announce the next major version of Immich:
v3.0.0! 🎉Breaking changes
This release includes several breaking changes; read the full migration guide here. It's worth mentioning that many of the breaking changes are updates to API endpoints and affect only third-party tools that integrate with Immich's API. For the vast majority of users, updating works exactly as it always has.
How to update
First, update the
IMMICH_VERSIONin your.envfile tov3:Then run the usual update commands:
docker compose pull && docker compose up -dRelease candidates
If you missed it,
v3.0.0was the first time we used release candidates, also known as prereleases. Release candidates are tested but not yet official releases of Immich, and they allow us to find and fix any outstanding bugs before a final release. If you would like to be notified about release candidates directly through Immich, you can change the release channel from "Stable" to "Release candidate" in theAdmin settings > Version checkoptions (here).New Merch
As part of this release, we're happy to announce we also have some new swag for you!
Check it out now at https://immich.store!
Highlights
Now, let's get right into all the new features in this release:
Mobile non-destructive editing
This is a follow-up to Image Editing on the web, which was released in
v2.5.0. This feature allows you to make non-destructive edits to your photos inside of Immich. Until now, the mobile editor used a completely different system that created new assets instead of editing the photo in place.With this update, we bring a new, easier-to-use editor to mobile devices that has the same features as the web version. You can now edit photos directly in the mobile app, including cropping, rotating, and adjusting your images without ever touching the original file. Similar to the web, edits are non-destructive, so you can revisit or revert them at any time. You can even make edits on mobile and then adjust them on the web later!
Some features from the previous mobile editing implementation have been removed including:
We have plans to bring some of these capabilities back in future releases.
Workflows (preview)
The first preview of Workflows is here! Workflows let you automate actions in your library by chaining triggers, filters, and actions together with a drag-and-drop builder. This is the foundation for many exciting automations to come, and we'd love your feedback as we continue building on it.
You can access the feature from Utilities > Workflows on the web.
From there, you can either create a new blank workflow or browse the premade templates to get a basic understanding of how workflows can be used.
Workflows editor
In the workflows editor, you can switch between the Visual or JSON editor. The visual editor is nice for building out the workflow; the JSON editor is nice for sharing and receiving workflow content from others.
In each workflow, there is a trigger and a sequence of steps.
Sharing a workflow
You can share the workflow you made with others in two ways: text and JSON. Text is nice for sharing on a forum or for show-and-tell content. JSON is nice for others to make an exact copy of your workflow's configuration.
You can copy the text in the workflows summary panel on the lower right of the screen
You can share the JSON content from the copy workflows button in the app bar, switch to the JSON editor, or use the
Show schemabutton in the context menu in the workflows listBackground backup improvements
Background backup on Android is now significantly more reliable. Previously, the background backup on Android was limited to newly taken photos. Now, the app uses a new periodic task scheduler, which allows you to upload your entire library in the background, and it plays nicer with Android's background execution limits, properly cleans up tasks, and warns you when battery optimization and notification settings might interfere with backups.
On iOS, the background refresh task now runs its sync and upload work in parallel, so uploads actually start within the short time window iOS allows.
Recently added page
A new "Recently Added" page on the web and mobile lets you browse your library sorted by when assets were added to Immich, rather than when they were taken. This makes it easier to find what's new when browsing a new batch of imports. You can find the new page in the "Explore" tab on the web and in the "Search" tab on mobile.
Integrity checks
The maintenance page has gotten a new addition: integrity reports! Immich will scan its directories on your file system, and compare it to what it has stored in its database. If there are deviations, they will be surfaced as
You can configure when and how long the job runs each night.
Slideshow (mobile)
The slideshow experience comes to mobile! You can now sit back and let your photos and videos play across the screen, just like on the web.
HLS and Real-Time Video transcoding (preview)
Immich can now transcode videos on-the-fly without needing to generate offline transcodes. This has been a long-requested feature with many benefits:
Please note that this feature is still experimental and can change behavior from version to version. It's currently only implemented in the web app, with the mobile app implementation in progress.
To enable real-time transcoding, go to the video transcoding settings (scroll down). Offline transcoding isn't directly affected by enabling it, so if you'd like to disable offline transcoding, you should also adjust the transcode policy.
Keep in mind that your server needs to be powerful enough to transcode in real-time for this feature to work well. Hardware acceleration is recommended, but not required, when using this feature.
New video player for web
A new custom video player on the web app ensures all your devices share the same controls and layout, matching the Immich design. Some basic functions, like changing the playback rate, are available. This should also fix a lot of the problems on iOS, where the OS's controls are hidden behind the Immich navbar.
Open photos in Immich as a gallery on Android
Immich can now act as a gallery/image viewer app on Android. Tap a photo or video in another app, choose Immich, and it opens directly in the asset viewer with options to share the file or upload it to your library.
This is the first iteration of the feature, and refinements to how Immich recognizes files that are already in your library are on the way
OCR on the mobile app
The asset viewer now has a toggle that highlights recognized text in a photo, and you can select and copy it directly from the image.
Upload assets directly to an album on mobile
You can now upload local photos directly to an album in the mobile app, including from the asset bottom sheet, instead of uploading first and organizing later. A small change that removes a lot of friction from the backup-and-organize flow.
Select image size when sharing on mobile
When sharing photos from the mobile app, you can now choose the image size before sending; it is handy for keeping shared files small for messaging apps while preserving the option to share at full quality when needed.
You can change the default behavior in the
App Settings > PreferencesYou can also pick the option when sharing on-the-fly by long pressing the
SharebuttonTimeline performance Improvements
Browsing months with a large number of assets is now dramatically smoother and prevents the browser tab from locking up when your instance encounters that scenario.
Support Immich
If you find the project helpful, you can support Immich by purchasing a product key at https://buy.immich.app or our merchandise at https://immich.store
What's Changed
🚨 Breaking Changes
🫥 Deprecated Changes
🔒 Security
🚀 Features
Configuration
📅 Schedule: (in timezone Europe/Brussels)
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about these updates again.
This PR was generated by Mend Renovate. View the repository job log.