deploy(caddy): split error + access logs to rotated files (validates against live CT 112)

[cmeans-claude-dev]

Summary

deploy/caddy/Caddyfile.example previously had a single log { output stdout } stanza that buried request data inside journalctl -u caddy with no separate error vs access split and no rotation. This PR ports the validated production-deployment pattern back into the example so future operators get the same shape.

Log	Path	Rotation	Notes
Error	/var/log/caddy/error.log	roll_size 50MiB / roll_keep 10 / roll_keep_for 2160h (90 d)	Global log default block; level ERROR; JSON. Errors are rare and forensically valuable, so kept longer.
Access	/var/log/caddy/access.log	roll_size 100MiB / roll_keep 14 / roll_keep_for 720h (30 d)	Per-site log block; JSON. Low-traffic badge service — one 100 MiB file may last weeks.

Caddy 2.7+ supports lumberjack rotation natively (roll_size + roll_keep + roll_keep_for). No separate logrotate config required. Same keys, just two destinations.

Validated against the live production deployment

This change was rolled out on the live CT 112 deployment FIRST, validated end-to-end, then ported to the example here.

Live verification:

$ caddy validate --config /tmp/Caddyfile.new --adapter caddyfile
Valid configuration
$ systemctl restart caddy && systemctl is-active caddy
active
$ curl -s -o /dev/null -w '%{http_code}\n' \
    https://pypi-badges.intfar.com/_health.json \
    https://pypi-badges.intfar.com/pypi-winnow-downloads/downloads-30d-non-ci.json \
    https://pypi-badges.intfar.com/does-not-exist/downloads-30d-non-ci.json
200 200 404
$ tail -1 /var/log/caddy/access.log | jq '.status, .request.uri, .duration'
404
"/does-not-exist/downloads-30d-non-ci.json"
0.000110184

error.log stays empty after the test hits — 4xx responses don't go there; only server-level / panic / ACME errors do.

Gotcha documented in the file header

Hit a sharp gotcha during the production rollout: running caddy validate as root (e.g., from a deployment script) pre-creates /var/log/caddy/{error,access}.log as root:root 0600. The caddy daemon (running as user caddy) then can't open them on reload, and the systemd unit gets stuck in reloading state. Recovery: chown caddy:caddy /var/log/caddy/{error,access}.log && systemctl restart caddy.

The Caddyfile.example header comment documents this so future operators don't lose 20 minutes to it.

Test plan

• Validated against live CT 112 deployment (see above)
• caddy validate --config Caddyfile.example --adapter caddyfile (CI's deploy-smoke job in PR ci: add deploy-smoke job for Dockerfile + Compose + Caddyfile examples (#7) #29 covers this once ci: add deploy-smoke job for Dockerfile + Compose + Caddyfile examples (#7) #29 lands)
• Operator confirmation that the rotation knobs match what they want for their traffic profile (the values are conservative defaults; tune as needed)

CHANGELOG

• Added a ## [Unreleased] → ### Added entry describing the split + rotation + gotcha-documentation

Related

• The live production rollout that validated this pattern (CT 112 / Holodeck) was performed via ssh holodeck → pct exec 112 against /etc/caddy/Caddyfile.
• PR ci: add deploy-smoke job for Dockerfile + Compose + Caddyfile examples (#7) #29 (ci/docker-smoke) adds deploy-smoke to CI, which caddy validates this example file on every push — so any future regression here gets caught.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[cmeans]

LGTM

[cmeans]

QA round 1 — QA Failed

The Caddyfile.example change itself is sound — the live-deployment-first pattern is exactly right per feedback_anchor_on_live_deployment.md, and I verified the rotation knobs and structure match CT 112's /etc/caddy/Caddyfile byte-for-byte (50MiB/10/2160h on error, 100MiB/14/720h on access, JSON format on both, level ERROR on the global log default). Live curl tests in PR body reproduced from my side: 200 / 200 / 404 against the three URLs as advertised. CI deploy-smoke validates the new file against caddy:2. So far so good.

The fail is for doc drift in deploy/README.md introduced by this PR.

Substantive finding (blocker): "Native journal logging" claim is now incomplete

deploy/README.md:33:

| **Bare systemd** (Linux host or LXC) | `systemd/`, `caddy/Caddyfile.example` | Smallest moving parts. Predictable. Native journal logging. | Linux-only. Manual user/dir setup. |

That cell describes the Bare systemd path's pros. After this PR, Caddy no longer logs to journal at all — server errors go to /var/log/caddy/error.log, requests go to /var/log/caddy/access.log. Only the collector still uses journal (line 88's journalctl -u pypi-winnow-downloads-collector.service).

So "Native journal logging" describes half the deployment now. An operator picking deployment shape from this table gets a misleading pro: they reasonably expect to journalctl -u caddy to grep request data and find no requests there.

This is exactly the symbol-walking the new lens calls for: changing how Caddy logs → grep deploy/README.md for log / journal → find this row → update it. Per feedback_doc_drift_is_substantive.md, doc drift is substantive, fix in same PR cycle.

Suggested fix: rewrite the cell to either drop the journal-specific claim (e.g., "Native logging integration") or split it ("Collector logs to journal; Caddy logs to rotated files under /var/log/caddy/").

What is correct on this PR

Aspect	State
Live-deployment-first validation	yes — production CT 112 changed first, ported to example after. Pattern matches feedback_anchor_on_live_deployment.md.
Rotation knobs match production	exact: error 50MiB / 10 / 2160h, access 100MiB / 14 / 720h
Structural match	global log default block with level ERROR + per-site log block, both format json, file outputs at /var/log/caddy/{error,access}.log
caddy:2 validates the new file	CI deploy-smoke job SUCCESS on PR head
Live URL behavior matches PR body	200 / 200 / 404 reproduced from my end against pypi-badges.intfar.com
Header gotcha documentation	well-placed: operators copying Caddyfile.example to /etc/caddy/Caddyfile will see the "running validate as root pre-creates the log files as root:root 0600" warning before they can fall into it
CHANGELOG entry	top of ### Added, comprehensive, references the gotcha

Adjacent observation (not a blocker)

deploy/README.md's Bare systemd quickstart (around line 91-93) doesn't mention /var/log/caddy/ ownership or pre-creation. Operators on Debian-family installing caddy from caddyserver.com get the directory created with caddy:caddy ownership by the package's postinst, so the gotcha doesn't bite them. Operators on non-Debian distros (Alpine / Arch / RHEL / Fedora) where the package may not pre-create the directory could hit it. The gotcha is documented in the Caddyfile.example header so an operator reading what they're copying will see it — that's reasonable coverage. Worth thinking about, not worth blocking.

Transitioning label to QA Failed.

[cmeans]

Applying QA Failed — see review above. One substantive finding: deploy/README.md:33 claims 'Native journal logging' as a Bare systemd pro, but this PR moves Caddy logs out of the journal to dedicated rotated files. Symbol-walking the docs for the changed concept would have caught it during drafting. Caddyfile change itself is sound and matches the live CT 112 deployment exactly.

[cmeans-claude-dev]

QA round 1 finding addressed. New commit 5644c59:

• deploy/README.md:33 — Bare systemd row's pros updated from "Native journal logging" to "Collector logs to journal; Caddy logs to rotated files under `/var/log/caddy/`". Symbol-walked the rest of the file: line 34 ("No native log integration" for Docker shape) is still accurate; line 88 (collector journalctl example) is still accurate. Both untouched.
• CHANGELOG bullet for this PR amended to mention the README update so it reflects full PR scope.

Adjacent observation about non-Debian distros needing to pre-create /var/log/caddy/ is acknowledged but not blocking per your call — the Caddyfile.example header gotcha covers operators who'll see the warning before they fall into it.

Re-tagged Ready for QA.

[cmeans]

QA round 2 — clean, signing off

Round-1 doc-drift fix landed as commit 5644c59. The deploy/README.md:33 cell now reads:

Smallest moving parts. Predictable. Collector logs to journal; Caddy logs to rotated files under /var/log/caddy/.

That's accurate and self-explanatory — operators picking deployment shape from the table now get the right mental model for where to grep request data.

Round-1 finding resolution check:

Round-1 ask	Resolution
deploy/README.md:33 "Native journal logging" claim	replaced with split phrasing that names both halves of the deployment correctly
Repo-wide drift recheck	only two journal / log references remain in deploy/README.md: line 33 (now correct) and line 88's journalctl -u pypi-winnow-downloads-collector.service (still accurate — collector still uses journal). Nothing else drifts.
CHANGELOG accuracy	the existing PR bullet was amended to mention the README update so the Unreleased entry reflects the full PR scope, not just the Caddyfile change

No regression on what was already correct in round 1:

• Caddyfile.example itself is unchanged from round 1 (verified by git diff baba6a5...5644c59 -- deploy/caddy/Caddyfile.example — empty). The byte-for-byte match against live CT 112 production still holds (50MiB/10/2160h error, 100MiB/14/720h access, level ERROR global, JSON on both).
• CI on new head: all SUCCESS — including deploy-smoke validating the unchanged Caddyfile.example against caddy:2.
• Live URL behavior on pypi-badges.intfar.com already verified in round 1 (200 / 200 / 404).

No new findings. Transitioning label to Ready for QA Signoff.

[cmeans]

Applying Ready for QA Signoff — see review above. Round-1 doc drift fixed in 5644c59: deploy/README.md:33 now correctly splits the logging story ("Collector logs to journal; Caddy logs to rotated files under /var/log/caddy/"). Caddyfile.example unchanged from round 1, so the byte-for-byte match against live CT 112 production still holds. CI all green on new head.

[cmeans]

LGTM