fix(talos): replace etcd nightly defrag with hourly auto-compaction - #4290
Merged
Conversation
Etcd was growing enough between weekly defrag runs to cross the 50% quota alert by day 4, requiring a daily ahrtr/etcd-defrag CronJob (privileged + hostPID + hostNetwork, mounting Talos etcd PKI). Auto-compaction addresses the root cause: compacting old revisions hourly keeps the DB from growing, so a defrag job is a no-op. Matches the approach in bjw-s-labs/home-ops commit 904940dc8393. Note: compaction does not shrink an already-large DB file; a one-time talosctl etcd defrag per control-plane node is recommended after applying. The kustomize-controller --concurrent reduction from that commit is not included: it targets single-node etcd I/O contention, not defrag.
Contributor
 |
|
Overall Grade |
Security Reliability Complexity Hygiene |
Code Review Summary
| Analyzer | Status | Updated (UTC) | Details |
|---|---|---|---|
| JavaScript |  |
Aug 1, 2026 10:39p.m. | Review ↗ |
| Shell | |
Aug 1, 2026 10:39p.m. | Review ↗ |
Important
AI Review is run only on demand for your team. We're only showing results of static analysis review right now. To trigger AI Review, comment @deepsourcebot review on this thread.
📝 WalkthroughWalkthroughThe PR removes the ChangesEtcd maintenance
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
@@ (root level) @@
# v1/ServiceAccount/kube-system/etcd-defrag
! - one document removed:
- apiVersion: v1
- kind: ServiceAccount
- metadata:
- name: etcd-defrag
- namespace: kube-system
- labels:
- app.kubernetes.io/instance: etcd-defrag
- app.kubernetes.io/managed-by: Helm
- app.kubernetes.io/name: etcd-defrag
- helm.toolkit.fluxcd.io/name: etcd-defrag
- helm.toolkit.fluxcd.io/namespace: kube-system
@@ (root level) @@
# batch/v1/CronJob/kube-system/etcd-defrag
! - one document removed:
- apiVersion: batch/v1
- kind: CronJob
- metadata:
- name: etcd-defrag
- namespace: kube-system
- labels:
- app.kubernetes.io/controller: etcd-defrag
- app.kubernetes.io/instance: etcd-defrag
- app.kubernetes.io/managed-by: Helm
- app.kubernetes.io/name: etcd-defrag
- helm.toolkit.fluxcd.io/name: etcd-defrag
- helm.toolkit.fluxcd.io/namespace: kube-system
- spec:
- concurrencyPolicy: Forbid
- failedJobsHistoryLimit: 1
- jobTemplate:
- spec:
- backoffLimit: 1
- template:
- metadata:
- labels:
- app.kubernetes.io/controller: etcd-defrag
- app.kubernetes.io/instance: etcd-defrag
- app.kubernetes.io/name: etcd-defrag
- spec:
- automountServiceAccountToken: false
- containers:
- - name: app
- image: "ghcr.io/ahrtr/etcd-defrag:v0.42.0@sha256:5138a12eda39067dabcf1d420d66f54f29f1e9297109a6b36574e010b45d58cc"
- args:
- - "--endpoints=https://127.0.0.1:2379"
- - "--cluster"
- - "--cacert=/certs/ca.crt"
- - "--cert=/certs/server.crt"
- - "--key=/certs/server.key"
- - "--defrag-rule=dbQuotaUsage > 0.5 || dbSizeFree > 50*1024*1024"
- - "--move-leader"
- - "--wait-between-defrags=30s"
- - "--etcd-storage-quota-bytes=2147483648"
- - "--auto-disalarm"
- - "--disalarm-threshold=0.8"
- securityContext:
- privileged: true
- resources:
- limits:
- cpu: 100m
- memory: 128Mi
- requests:
- cpu: 25m
- memory: 128Mi
- volumeMounts:
- - name: certs
- mountPath: /certs
- readOnly: true
- - name: tmpfs
- mountPath: /tmp
- subPath: tmp
- dnsPolicy: ClusterFirstWithHostNet
- enableServiceLinks: false
- hostIPC: false
- hostNetwork: true
- hostPID: true
- nodeSelector:
- node-role.kubernetes.io/control-plane:
- restartPolicy: Never
- securityContext:
- runAsGroup: 0
- runAsNonRoot: false
- runAsUser: 0
- serviceAccountName: etcd-defrag
- tolerations:
- - key: node-role.kubernetes.io/control-plane
- effect: NoSchedule
- operator: Exists
- volumes:
- - name: certs
- hostPath:
- path: /system/secrets/etcd
- - name: tmpfs
- emptyDir: {}
- schedule: "0 3 * * *"
- startingDeadlineSeconds: 30
- successfulJobsHistoryLimit: 1
- suspend: false
- timeZone: Europe/Brussels
|
@@ (root level) @@
# kustomize.toolkit.fluxcd.io/v1/Kustomization/kube-system/etcd-defrag
! - one document removed:
- apiVersion: kustomize.toolkit.fluxcd.io/v1
- kind: Kustomization
- metadata:
- name: etcd-defrag
- namespace: kube-system
- labels:
- kustomize.toolkit.fluxcd.io/name: cluster-apps
- kustomize.toolkit.fluxcd.io/namespace: flux-system
- spec:
- decryption:
- provider: sops
- secretRef:
- name: sops-age
- deletionPolicy: WaitForTermination
- interval: 1h
- patches:
- - patch: |
- apiVersion: helm.toolkit.fluxcd.io/v2
- kind: HelmRelease
- metadata:
- name: _
- spec:
- install:
- crds: CreateReplace
- rollback:
- cleanupOnFail: true
- upgrade:
- cleanupOnFail: true
- crds: CreateReplace
- strategy:
- name: RemediateOnFailure
- remediation:
- remediateLastFailure: true
- retries: 2
- target:
- kind: HelmRelease
- group: helm.toolkit.fluxcd.io
- path: ./kubernetes/apps/kube-system/etcd-defrag/app
- postBuild:
- substituteFrom:
- - name: cluster-settings
- kind: ConfigMap
- optional: false
- - name: cluster-secrets
- kind: Secret
- optional: false
- prune: true
- retryInterval: 2m
- sourceRef:
- name: flux-system
- kind: GitRepository
- namespace: flux-system
- targetNamespace: kube-system
- timeout: 5m
- wait: false
@@ (root level) @@
# helm.toolkit.fluxcd.io/v2/HelmRelease/kube-system/etcd-defrag
! - one document removed:
- apiVersion: helm.toolkit.fluxcd.io/v2
- kind: HelmRelease
- metadata:
- name: etcd-defrag
- namespace: kube-system
- labels:
- kustomize.toolkit.fluxcd.io/name: etcd-defrag
- kustomize.toolkit.fluxcd.io/namespace: kube-system
- spec:
- chartRef:
- name: app-template
- kind: OCIRepository
- install:
- crds: CreateReplace
- interval: 30m
- rollback:
- cleanupOnFail: true
- upgrade:
- cleanupOnFail: true
- crds: CreateReplace
- remediation:
- remediateLastFailure: true
- retries: 2
- strategy:
- name: RemediateOnFailure
- values:
- controllers:
- etcd-defrag:
- type: cronjob
- cronjob:
- backoffLimit: 1
- schedule: "0 3 * * *"
- timeZone: Europe/Brussels
- containers:
- app:
- resources:
- limits:
- cpu: 100m
- memory: 128Mi
- requests:
- cpu: 25m
- memory: 128Mi
- args:
- - "--endpoints=https://127.0.0.1:2379"
- - "--cluster"
- - "--cacert=/certs/ca.crt"
- - "--cert=/certs/server.crt"
- - "--key=/certs/server.key"
- - "--defrag-rule=dbQuotaUsage > 0.5 || dbSizeFree > 50*1024*1024"
- - "--move-leader"
- - "--wait-between-defrags=30s"
- - "--etcd-storage-quota-bytes=2147483648"
- - "--auto-disalarm"
- - "--disalarm-threshold=0.8"
- image:
- repository: ghcr.io/ahrtr/etcd-defrag
- tag: "v0.42.0@sha256:5138a12eda39067dabcf1d420d66f54f29f1e9297109a6b36574e010b45d58cc"
- securityContext:
- privileged: true
- defaultPodOptions:
- hostNetwork: true
- hostPID: true
- nodeSelector:
- node-role.kubernetes.io/control-plane:
- securityContext:
- runAsGroup: 0
- runAsNonRoot: false
- runAsUser: 0
- tolerations:
- - key: node-role.kubernetes.io/control-plane
- effect: NoSchedule
- operator: Exists
- persistence:
- certs:
- type: hostPath
- hostPath: /system/secrets/etcd
- globalMounts:
- - path: /certs
- readOnly: true
- tmpfs:
- type: emptyDir
- advancedMounts:
- etcd-defrag:
- app:
- - path: /tmp
- subPath: tmp
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
talos/patches/controller/cluster.yaml (1)
17-21: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winDo not treat compaction as a complete replacement for defragmentation.
These settings limit MVCC revision history. They do not return fragmented backend space to the host filesystem; etcd treats defragmentation as a separate, per-member operation. (etcd.io)
Because this PR removes recurring
etcd-defrag, verify that one-time defragmentation plus compaction keeps each control-plane member below its storage quota. Monitor the version-appropriate physical and in-use database size metrics andNOSPACEalarms. Update the comment if it intends to describe logical history growth only. (etcd.io)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@talos/patches/controller/cluster.yaml` around lines 17 - 21, Update the comment near auto-compaction-mode and auto-compaction-retention to describe only MVCC revision-history compaction, not defragmentation or filesystem-space reclamation. Ensure the change does not imply compaction replaces the removed recurring etcd-defrag operation.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@talos/patches/controller/cluster.yaml`:
- Around line 17-21: Update the comment near auto-compaction-mode and
auto-compaction-retention to describe only MVCC revision-history compaction, not
defragmentation or filesystem-space reclamation. Ensure the change does not
imply compaction replaces the removed recurring etcd-defrag operation.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: d1423e5a-4cda-4508-85d1-14658e916854
📒 Files selected for processing (5)
kubernetes/apps/kube-system/etcd-defrag/app/helmrelease.yamlkubernetes/apps/kube-system/etcd-defrag/app/kustomization.yamlkubernetes/apps/kube-system/etcd-defrag/ks.yamlkubernetes/apps/kube-system/kustomization.yamltalos/patches/controller/cluster.yaml
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
Tanguille/LLMKube(auto-detected)Tanguille/toolhive(auto-detected)Tanguille/pr-reviewer-action(auto-detected)Tanguille/2x-R9700-RDNA4-GFX1201-sglang-inference(auto-detected)
💤 Files with no reviewable changes (4)
- kubernetes/apps/kube-system/kustomization.yaml
- kubernetes/apps/kube-system/etcd-defrag/app/kustomization.yaml
- kubernetes/apps/kube-system/etcd-defrag/app/helmrelease.yaml
- kubernetes/apps/kube-system/etcd-defrag/ks.yaml
AI Automated ReviewAnalysis engine: qwen-3.6-fast@http://litellm.ai.svc.cluster.local/v1 (openai) Recommendation: Approve This PR correctly replaces a privileged, host-networked etcd defragmentation CronJob with native Talos etcd auto-compaction. The change addresses the root cause of database growth rather than just reclaiming space periodically, and significantly reduces the cluster's privilege surface by removing a pod that required Change-by-Change Findings
Standards Compliance
Must Check Items
Unknowns or Needs Verification
No blocker or major issues found. The change is clean, well-justified, and improves cluster security. |
Tanguille
added a commit
that referenced
this pull request
Aug 21, 2026
* fix(etcd): restore the defrag CronJob at a weekly cadence #4290 replaced the daily defrag with hourly auto-compaction and noted a one-time `talosctl etcd defrag` per node was still needed; that never ran. Compaction is doing its job -- in-use held at 137-160MB across the last 20 days -- but it cannot shrink the file, so free space ramped ~8MB/day and the db grew 185MB -> 341MB, tripping etcdDatabaseHighFragmentationRatio (57%) on all three members. Only defrag reclaims that. Weekly rather than the daily cadence it ran at before, since compaction now holds the keyspace flat and the rule only has the fragmentation ramp to clear. Image bumped v0.42.0 -> v0.43.0. * style(etcd-defrag): trim the cadence comment to measured facts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replaces the nightly
etcd-defragCronJob with hourly etcd auto-compaction configured in Talos, matching bjw-s-labs/home-ops@904940dc8393.Why
The previous weekly defrag cadence let fragmentation cross the 50% quota alert by day 4, forcing the daily
ahrtr/etcd-defragCronJob — a privileged + hostPID + hostNetwork pod mounting/system/secrets/etcd.Auto-compaction fixes the root cause: compacting old revisions hourly keeps the DB from growing, so a defrag job is a no-op. It also removes a privileged pod from the cluster.
Changes
talos/patches/controller/cluster.yaml: addauto-compaction-mode: periodic+auto-compaction-retention: 1htocluster.etcd.extraArgskubernetes/apps/kube-system/etcd-defrag/and its entry inkube-system/kustomization.yamlNotes
talosctl etcd defragper control-plane node once.kustomize-controller --concurrentreduction targets single-node etcd I/O contention, not defrag — this cluster's 3-node HA etcd doesn't have that problem.Validation
validate-pr.shexit 0 (warnings are pre-existing repo-wide false positives)talhelper genconfigsucceeds; generated machine config contains the new etcdextraArgsSummary by CodeRabbit
New Features
Changes