fix(ci): stop volume-pruning self-hosted runners (data-loss hazard) - #1868
Conversation
Self-hosted runners co-host LIVE fleet data volumes (esp. SPARK full PMOVES.AI node + ai-lab/Z890). 'docker system prune -af --volumes' on those hosts can destroy fleet data whose service is briefly stopped — and the 'until=48h' filter does NOT protect volumes. Replace with safe image + build-cache reclaim (Known Road: never volume prune on runners; use make volume-reset SERVICE=<name> for intentional volume resets). - runner-maintenance.yml: 4 nightly cleanup jobs (ai-lab/spark/kvm4/kvm2) - integrations-ghcr.yml: 2 in-build free-disk steps (matrix + publish) Reclaim is preserved (docker image prune + docker builder prune); only the volume destruction is removed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Warning Review limit reached
More reviews will be available in 11 minutes and 11 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate. For paid Pro and Pro+ PR reviews, CodeRabbit uses rolling per-developer review limits. Reviews become available again as older review attempts age out of the rolling limit window. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughTwo CI workflow files are updated to remove ChangesDocker Volume Pruning Removal
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 59b51b5073
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Triage — chatgpt-codex-connector[bot] review on #1868 No P0/P1 findings from this review. P2: 1, P3: 0
|
… reclaim Address Codex P2: 'docker image prune -af' only removes images unreferenced by containers, so stopped/exited job containers (and the images they pin) survive — under-reclaiming vs the original. Use 'docker system prune -af' (original behavior MINUS --volumes): reclaims stopped containers + images + networks + build cache, never volumes. Preserves the disk cleanup the maintenance job relied on while keeping fleet data volumes safe. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ther half) (#1869) * docs(ops): fleet daemon.json log rotation + live-restore (disk-full other half) Container logs use the default json-file driver with NO rotation -> unbounded growth in /var/lib/docker fills the root FS on long-lived fleet nodes (SPARK, ai-lab/Z890, KVM4). This is the daemon-level half of the disk-full failures (the runner/build-cache half is PR #1868). live-restore also mitigates the 'containers stuck Created' drift (containers survive docker restart). - deploy/provision/daemon.json: version-controlled baseline (log-opts + live-restore) - DOCKER_DAEMON_HARDENING.md: Log Rotation section, per-node-class sizing, canonical daemon.json + lane-coordinated apply runbook (VPS via Hostinger MCP; Z890 coordinated; dev), verify + rollback Conservative settings only — risky hardening (icc/userland-proxy) and storage-driver pinning excluded/gated (orphan-data footgun documented). Authoring is 4090-lane; APPLY is host-coordinated per the runbook. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(ops): prefer 'systemctl reload docker' (SIGHUP) over restart for rollout Address Codex P2: enabling live-restore for the first time via 'systemctl restart' DOES cycle running containers (the setting isn't active yet), so the 'no downtime' claim was wrong on first rollout. Use 'systemctl reload docker' (SIGHUP) — re-reads daemon.json and applies log-opts + live-restore without stopping containers. Added the reload-vs-restart caveat; updated VPS/Z890/ rollback steps to reload. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
#1876) Full TAC audit across all 34 trees after this session's merges. Routes work by agent_hint owner (Village Rule — no edits to other lanes' trees). Findings: firefly-iii/pmoves-launch-readiness clean; agent-zero-customization (35 fail) / mcp-topology (12) / observability (10) need grep-first reconciliation (likely stale patterns); 9 trees never-run. 4090-lane: cast-gateway 2 fails = real compose-hardening gaps (fold into compose-hardening PR); node-4090-laptop all runtime checks (needs live sitrep, not stale). Proposes new security-posture/ networking nodes for this session's daemon-hardening (#1869) + no-volume-prune (#1868), and re-scoping github-app runner-auth per the §4 PAT-for-GHCR decision. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…aintenance sweep) (#2021) * fix(ci): reclaim leaked BuildKit builder-state volumes in runner maintenance Root cause of kvm4-1 filling to 100% (181 GB): setup-buildx-action creates a fresh docker-container builder per run and self-hosted runners never `buildx rm` it, so each build leaks a `buildx_buildkit_builder-*_state` volume. The nightly runner-maintenance sweep missed them entirely — `docker system prune` skips volumes and `docker builder prune` only clears cache inside builders, never the builder instances or their dedicated state volumes. Add a targeted orphan-volume sweep to all four cleanup jobs (ai-lab/spark/kvm4/ kvm2): remove `buildx_buildkit_builder-*_state` volumes by exact pattern. The pattern can never match a `pmoves_*` data volume, and volumes attached to an in-flight builder fail the rm and are safely skipped — targeted reclaim, NOT the banned `docker volume prune` (#1868). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ci): bounded reusable BuildKit builder across self-hosted workflows (root fix) The runner-maintenance sweep (prev commit) reclaims leaked builder volumes but treats the symptom. Root cause: every self-hosted build called setup-buildx-action with no fixed name, so each run created a fresh docker-container builder whose multi-GB state volume was never reclaimed (40 accumulated -> 181GB -> kvm4-1 100%). Add one canonical composite action `.github/actions/pmoves-buildx` and cascade it across all 7 self-hosted setup-buildx call sites (self-hosted-builds x2, self-hosted-builds-hardened x2, build-images, integrations-ghcr [self-hosted job only], build-nats-workers). The builder is now: - REUSED (name: pmoves-shared -> no per-run builder/volume) - STATE-KEPT (keep-state: true on persistent runners) - SELF-BOUNDED (buildkitd gc: maxUsedSpace/reservedSpace/keepDuration -> BuildKit prunes cache periodically; the state volume can't grow unbounded) ubuntu-latest sites (integrations-ghcr:275, yt-dlp, canary) are ephemeral and left unchanged. deploy/runners/BUILDX_PROVISIONING.md documents the builder + the shared operator/agent access lane (tailscale ssh by hostname) + the :9100 obs the KVMs now export. Docs: docker.com/build/cache/garbage-collection, github.com/docker/setup-buildx-action. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ci): guard orphan-volume sweep against grep no-match abort CodeRabbit Critical + Codex P2 on #2021: grep -E exits 1 when no buildx_buildkit_builder-*_state volumes exist (the steady state after cleanup). Under 'set -euo pipefail' that non-zero pipeline status aborts the step, skipping workdir cleanup, diag cleanup, and the disk report in all 4 jobs. Append '|| true' to the sweep pipeline (matches the existing '|| true' pattern on every other docker command in these jobs). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 participant
Why
Self-hosted runners co-host LIVE fleet data volumes (esp. SPARK — the ARM64 full PMOVES.AI node — and ai-lab/Z890). Several workflows run
docker system prune -af --volumeson those hosts, which can destroy fleet data for any service that's briefly stopped. The--filter "until=48h"does not protect volumes from removal. This is a standing data-loss hazard and a Known-Road violation (never volume-prune co-hosted runners), and a likely contributor to the "containers stuckCreated" env-drift seen on Z890.What
Replace the 6 unsafe prunes with safe image + build-cache reclaim — disk is still reclaimed, but data volumes are never touched:
runner-maintenance.ymlintegrations-ghcr.ymldocker system prune -af --volumes [...]→docker image prune -af [...](the existingdocker builder prune -afreclaim stays).Safety / scope
make volume-reset SERVICE=<name>.--volumesprune remaining).Lane
4090-CLAUDE deploy/CI spine. Read-only to Z890's runner hosts (workflow logic only). Part of the Docker fleet production-readiness audit (
pmoves/docs/handoffs/DOCKER_FLEET_PROD_AUDIT.md); follow-ups: safe pre-flight reclaim+gate, container log rotation (Z890/VPS-coordinated), VPSdaemon.json.🤖 Generated with Claude Code
Summary by CodeRabbit