fix(docker): add json-file log rotation to all tier anchors (root cause of disk-full) - #2420
Conversation
Root cause of recurring disk-full: all 54 containers had NO log rotation
— Docker's json-file driver was writing unbounded logs. The Docker VHDX
grew from 247 GB to 516 GB in ~3 days from accumulated log output.
Added logging block to x-tier-agent-hardened, x-tier-data-hardened, and
x-tier-media-hardened YAML anchors:
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
This caps each container at 30 MB of logs (3 x 10 MB rotated files).
Applied to the tier anchors so every service using <<: *tier-*-hardened
inherits log rotation automatically. Existing containers pick this up
on the next force-recreate.
Also set daemon.json log-opts globally (Docker Desktop node config) as
a defense-in-depth — new containers created outside compose also get
log rotation.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe Compose configuration adds ChangesDocker logging configuration
Estimated code review effort: 1 (Trivial) | ~5 minutes Suggested labels: Suggested reviewers: ✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Docker Hardening ValidationHardening Validation ReportValidated: Wed Aug 5 23:02:12 UTC 2026Services CheckedPMOVES.AI Docker Hardening Validation[INFO] Checking: pmoves/docker-compose.hardened.yml [INFO] Validating: hi-rag-gateway-v2 [INFO] Validating: extract-worker [INFO] Validating: langextract [INFO] Validating: presign [INFO] Validating: render-webhook [INFO] Validating: retrieval-eval [INFO] Validating: pdf-ingest [INFO] Validating: jellyfin-bridge [INFO] Validating: invidious-companion-proxy [INFO] Validating: ffmpeg-whisper [INFO] Validating: media-video [INFO] Validating: media-audio [INFO] Validating: hi-rag-gateway-v2-gpu [INFO] Validating: hi-rag-gateway-gpu [INFO] Validating: deepresearch [INFO] Validating: supaserch [INFO] Validating: publisher-discord [INFO] Validating: mesh-agent [INFO] Validating: nats-echo-req [INFO] Validating: nats-echo-res [INFO] Validating: comfy-watcher [INFO] Validating: grayjay-plugin-host [INFO] Validating: agent-zero [INFO] Validating: p7-room-orchestrator [INFO] Validating: archon [INFO] Validating: channel-monitor [INFO] Validating: pmoves-yt [INFO] Validating: notebook-sync [INFO] Validating: supabase_service_role_key [INFO] Validating: supabase_jwt_secret [INFO] Validating: p7_control_token ====================================== |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 38e83c7f1f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| logging: | ||
| driver: json-file | ||
| options: | ||
| max-size: "10m" | ||
| max-file: "3" |
There was a problem hiding this comment.
On hosts retaining Docker's default unbounded json-file settings, this limit reaches only services that merge exactly tier-agent-hardened or tier-data-hardened; the independent Supabase, API, worker, media, LLM, UI, agent-ro, and agent-rw anchors do not inherit it. Consequently services such as transcribe-backend, ffmpeg-whisper, tensorzero-clickhouse, and all 18 read-only agent services remain unbounded even after force-recreation, so the disk-full root cause persists across most of the stack. Add the logging configuration to every tier variant or merge a shared logging anchor into every service.
Useful? React with 👍 / 👎.
Rebase brought in #2420's json-file log rotation on tier anchors. Regenerate base/core/agents/apps overlays to match. Workers was already correct (compose-split on the pre-rebase branch fixed it). 💘 Generated with Crush
P1 #1 (volume prune banned): removed `docker volume prune -f` — it's banned by damage-control patterns.yaml because fleet hosts co-host data volumes that can be temporarily unreferenced. Replaced with a message directing to `make volume-reset SERVICE=<name>`. P1 #2 (daemon.json destructive fallback): changed bare `except:` to `except FileNotFoundError:` in both scripts. Malformed JSON or permission errors now abort with a clear message instead of silently overwriting daemon config. P2 #3 (missing make target): removed the `make -C pmoves disk-cleanup` reference from the runbook — no such target exists. Changed to direct script invocation: `bash pmoves/scripts/pmoves-disk-cleanup.sh`. P2 #4 (wrong YAML anchor pattern): removed the entire compose-patching section. As of #2420, log rotation is baked into the tier anchors at the source level — the script's regex-based patching was both broken (matching x-tier- not x-env-tier-) and obsolete. 💘 Generated with Crush
…2425) * feat(archon): wire TensorZero routing + Claude auth for ARM64/fleet Archon's TypeScript server exits(1) at boot without Claude auth configured, and its LLM calls default to OpenAI (which fails on fleet nodes routing through TensorZero). Three changes: 1. CLAUDE_USE_GLOBAL_AUTH default false→true — PMOVES fleet uses global auth, not per-session credentials. Parameterized so nodes that need per-session auth can override. 2. OPENAI_API_KEY + OPENAI_API_BASE added to Archon env — routes LLM calls through TensorZero gateway (same pattern as Agent Zero). Dummy key silences the "Missing API Key" warning; TensorZero handles actual auth. 3. CLAUDE_BIN_PATH documented in env.tier-agent.example for ARM64 (SPARK) — the Claude Agent SDK installs at a non-standard path under node_modules on ARM64. Left as ${VAR:-} default in compose (uses PATH on x86_64); uncomment in env.tier-agent on SPARK. Regenerates split overlays (base/core/agents/apps/juicefs pick up #2420's json-file log rotation that propagated through tier anchors). 💘 Generated with Crush * fix(review): address Codex P1+P2 on #2425 — canonical TZ URL + auth default P1: Derive OPENAI_API_BASE from TENSORZERO_BASE_URL (the fleet-canonical remote-endpoint var) instead of hardcoding tensorzero-gateway:3000. Operators on delegated-inference nodes set TENSORZERO_BASE_URL and all services — including Archon — now pick it up. Dropped the redundant ARCHON_OPENAI_API_BASE override. P2: Flip CLAUDE_USE_GLOBAL_AUTH in env.tier-agent.example from false→true to match the compose default. Without this, fresh checkouts that use env.tier-agent.example as an interpolation source still got false, leaving Archon unable to boot. 💘 Generated with Crush --------- Co-authored-by: Agent Zero <agent.zero@pmoves.ai>
…he dev touched ≠ file the runner builds' (#2358) (#2417) * feat(ratchets): validate-dockerfile-paths — durable answer to "file the dev touched ≠ file the runner builds" (#2358) Third pattern from the operator's #2358 meta-callout. Same shape as the validate-composes + validate-tac ratchets (PR #2416): no- network, ~100ms scan, baseline file committed and reviewable in PR diffs, count only goes DOWN over time. Two failure modes: 1. BROKEN_BUILD — a compose `build:` stanza points at a Dockerfile that doesn't exist on disk. (5 real instances fixed in the next commit: agent-zero in vps.override.yml, github-branch-naming + github-crossrepo-sync in docker-compose.yml + docker-compose.workers.yml.) 2. ORPHAN_DOCKERFILE — a Dockerfile in the repo that no compose references AND isn't in the operator-acknowledged baseline. Initial baseline has 26 entries with reasons. Files: - pmoves/tools/validate_dockerfile_paths.py (~280 lines, zero deps beyond pyyaml; same `!override` / `!reset` Compose-tag tolerance as validate_composes) - pmoves/tools/tests/test_validate_dockerfile_paths.py (11 tests, all pass; covers broken-build, baseline-aware orphan detection, sibling-submodule skipping, env-var default substitution, JSON output shape) - pmoves/configs/dockerfiles/_known_orphans.yaml (baseline with 26 entries + reasons; reduce-by-pr pattern) - .github/workflows/validate-dockerfile-paths-ratchet.yml (fires on any Dockerfile*, any docker-compose*.yml, the ratchet script, the ratchet tests, the baseline file, or the workflow itself) - pmoves/Makefile (3 new targets: validate-dockerfile-paths, validate-dockerfile-paths-json, validate-dockerfile-paths-list-orphans) The sibling-submodule / vendor / provisions paths are explicit- skip: the ratchet can't statically check external repos. Out-of- scope paths include anything under pmoves/vendor/, anything with "provisions" in the path, and the common sibling-submodule patterns (PMOVES-Archon, PMOVES.YT, Pmoves-cipher, etc.). The operator keeps those synced via `make submodules`. Counts at initial pass (on this PR): 90 dockerfiles in the scan dirs, 78 referenced by compose, 26 in baseline. Zero findings. 35/35 ratchet tests pass. * fix(compose): 5 real broken build targets the ratchet surfaced The validate-dockerfile-paths ratchet (P1 commit) caught these on its first run. Each one is a compose `build:` stanza that points at a Dockerfile that doesn't exist — the ratchet's exact "merged before the issue was addressed" pattern. 5 fixes (4 files): 1. pmoves/docker-compose.vps.override.yml:43-44 `agent-zero` had `build: { dockerfile: Dockerfile }` with no `context:`. Default context is `.` (pmoves/), so the resolved path was `pmoves/Dockerfile` which doesn't exist. Added `context: .` + `dockerfile: ./services/agent-zero/ Dockerfile` to match the build stanza in pmoves/docker- compose.yml:2911-2913. 2-3. pmoves/docker-compose.yml:5207-5211 (github-branch- naming) + 5247-5251 (github-crossrepo-sync) Both services had `build: { context: ./services/<name>, dockerfile: Dockerfile }` but the service directories don't exist in this repo (the implementation lives in separate `pmoves-github-branch-naming` / `pmoves-github- crossrepo-sync` repos). Dropped the `build:` block; the prebuilt `image: ${...IMAGE:-ghcr.io/...:pmoves-latest}` env var default is the canonical path. Matches the `publisher-discord` (line 3891) + `github-crossrepo-pr` (line 3891 neighborhood) pattern of prebuilt-only. 4-5. Same fix applied to pmoves/docker-compose.workers.yml: 695-699 (github-branch-naming) + 735-739 (github-crossrepo- sync) — these are a hand-curated split of the main file and had the same broken `build:` stanzas. After this commit: validate-dockerfile-paths reports 0 broken builds on the fleet. The next commit (docs) re-adds the missing AGNOTE entries for the prior ratchet PRs (#2415, #2416) plus the release marker for this PR. * docs(agnote): re-add #2415 + #2416 entries lost in rebase + new #2417 release Three entries appended at the bottom of pmoves/docs/AGENTS/AGNOTE4482PHI.t1.md: 1. PR #2415 (RELEASE, 2026-08-05T20:08Z) — env.local stragglers across 3 hand-maintained compose overlays. Lost in the rebase/reset cycle that left main at 0ae6a99; re-adding so the AGNOTE chain reflects the actual merge history. 2. PR #2416 (RELEASE, 2026-08-05T20:39Z) — validate-composes + validate-tac ratchets. Same: lost in the rebase cycle, re-added. 3. PR #2417 (CLAIM, 2026-08-05T22:15Z) — this PR's own validate-dockerfile-paths ratchet. Third ratchet from the #2358 meta-callout, same shape as the prior two, with the 5 broken builds fixed in the prior commit and the 26-entry baseline of intentional orphans. GRAPHITI_MARK comments added for all three so the Graphiti ingest picks them up. No code changes in this commit. * fix(compose): regenerate workers overlay after comment edits The ratchet PR updated inline comments in docker-compose.yml for github-branch-naming / github-crossrepo-sync (explaining why there's no build block) but didn't run `make -C pmoves compose-split` to propagate the changes to docker-compose.workers.yml. Validate Compose Files was correctly reporting drift. Verified idempotent (compose-split produces same output on re-run). 💘 Generated with Crush * fix(compose): regenerate overlays after rebase onto #2420 Rebase brought in #2420's json-file log rotation on tier anchors. Regenerate base/core/agents/apps overlays to match. Workers was already correct (compose-split on the pre-rebase branch fixed it). 💘 Generated with Crush * fix(ratchet): use is_file() not exists() for Dockerfile BROKEN_BUILD check exists() returns True for directories — a compose build.dockerfile that resolves to a directory (e.g. unset ${FOO} collapsing to the context dir) would pass the gate even though Docker can't use it. is_file() ensures only real files satisfy the check. Addresses Codex P2 on #2417. 💘 Generated with Crush --------- Co-authored-by: mavis <mavis@pmoves.ai> Co-authored-by: Agent Zero <agent.zero@pmoves.ai>
* feat(ops): fleet disk cleanup scripts + cross-platform runbook Adds: - pmoves/scripts/pmoves-disk-cleanup.sh — safe Docker cleanup (stopped containers, dangling images, ALL build cache, orphan volumes). No root needed. Reports disk before/after. - pmoves/scripts/pmoves-daemon-log-rotation.sh — adds log rotation to /etc/docker/daemon.json (max-size 10m, max-file 3). Needs root. - pmoves/docs/operations/PMOVES_DISK_CLEANUP.md — cross-platform runbook covering Linux (KVMs, SPARK, B850, Jetsons) AND Windows (Z890, 5090, 4090, Desktop, Missling-Link, Slate) with PowerShell one-liner for Docker Desktop daemon.json. - pmoves/mk/infra.mk — `make -C pmoves disk-cleanup` target Complements the compose tier anchor log rotation from PR #2420. * fix(review): address all 4 Codex threads on #2424 P1 #1 (volume prune banned): removed `docker volume prune -f` — it's banned by damage-control patterns.yaml because fleet hosts co-host data volumes that can be temporarily unreferenced. Replaced with a message directing to `make volume-reset SERVICE=<name>`. P1 #2 (daemon.json destructive fallback): changed bare `except:` to `except FileNotFoundError:` in both scripts. Malformed JSON or permission errors now abort with a clear message instead of silently overwriting daemon config. P2 #3 (missing make target): removed the `make -C pmoves disk-cleanup` reference from the runbook — no such target exists. Changed to direct script invocation: `bash pmoves/scripts/pmoves-disk-cleanup.sh`. P2 #4 (wrong YAML anchor pattern): removed the entire compose-patching section. As of #2420, log rotation is baked into the tier anchors at the source level — the script's regex-based patching was both broken (matching x-tier- not x-env-tier-) and obsolete. 💘 Generated with Crush --------- Co-authored-by: Agent Zero <agent.zero@pmoves.ai>
…n every request (#2587) * fix(firefly): give tmpfs mounts www-data ownership — fixes HTTP 500 on every request Firefly returned HTTP 500 on EVERY request and wrote a 10.7 GB container log, which is what took z890's D: drive to 100% (0 bytes free, repo writes failing). Root cause: docker-compose.external.yml mounts tmpfs over three Laravel runtime dirs (storage/framework/{cache,sessions,views}) for hardening (noexec,nosuid). A tmpfs mount starts EMPTY and ROOT-OWNED on every container start, but Firefly's php-fpm workers run as www-data (uid=33,gid=33) — so they could not create the storage/framework/cache/data tree Laravel's file cache requires: production.ERROR: file_put_contents(.../storage/framework/cache/data/6c/61/<hash>): Failed to open stream: No such file or directory Blackbox Exporter polls the endpoint continuously, so a broken service times a monitoring loop produced unbounded identical stack traces. The disk-fill was the symptom; the 500 was the defect. Fix per Docker's official tmpfs reference, which lists uid/gid/mode as supported options and recommends uid/gid precisely because tmpfs permissions reset across restarts: set uid=33,gid=33 and tighten mode to 0770 (from the effective 775). noexec,nosuid are RETAINED — the hardening intent is preserved, not traded away. Deliberately NOT bundled: tmpfs size= limits. Docker supports them and an unbounded tmpfs can consume RAM, but no measurement of Firefly's steady-state footprint exists and a too-small bound would cause a new outage of the same shape. Tracked as a follow-up in the handoff. Verified after `make -C pmoves up-external`: cache/sessions now www-data:www-data drwxrwx--- (0770) storage/framework/cache/data EXISTS (Laravel created it itself) HTTP 302 (correct unauthenticated redirect), no longer 500 log growth 400 KB/20s -> 12 KB/25s fleet-wide; firefly log now 2.4 MB recreating also applied the #2420 rotation (max-size 10m, max-file 3) Handoff also records the wider finding: #2420's log rotation only applies to containers created after it landed (Docker fixes log config at creation), so 30 of 48 containers on z890 are still unbounded. The fix shipped without a backfill. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(firefly): name the target that exists — up-external, not up-firefly The verification recipe told the reader to run `make -C pmoves up-firefly`. No such target exists, anywhere in pmoves/Makefile or pmoves/mk/*.mk — a grep for `^[a-z0-9-]*firefly[a-z0-9-]*:` returns nothing. The real target is `up-external`, whose own help text reads "Start external integration services (Wger, Firefly, Open Notebook, Jellyfin)". It drives docker-compose.external.yml, which is the file this PR changes, and `up -d` recreates only the services whose definitions moved — so it does exactly what the recipe intended. This is what failed validate-command-anchors-ratchet on this PR: GHOST_TARGET -> up-firefly. The gate was right. Fixing the doc rather than adding a target avoids inventing a bring-up path the fleet does not have. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 participant
Summary
Root cause of the recurring disk-full issue: all 54 containers had zero log rotation. Docker's json-file driver was writing unbounded logs. The Docker VHDX grew from 247 GB to 516 GB in ~3 days from accumulated log output.
The fix:
logging: { driver: json-file, options: { max-size: "10m", max-file: "3" } }tox-tier-agent-hardened,x-tier-data-hardened, andx-tier-media-hardenedYAML anchors<<: *tier-*-hardenedinherits log rotation automaticallydaemon.jsonlog-opts globally as defense-in-depth (applies to containers created outside compose)Existing containers pick this up on the next
docker compose up -d --force-recreate.After merge: the VHDX needs compaction (
Optimize-VHD) to reclaim the ~400 GB of dead slack. That requires Docker Desktop shutdown +wsl --shutdown+ admin PowerShell.10-line additive change. No behavior change for running containers until recreated.
💘 Generated with Crush
Summary by CodeRabbit