Test the backup and 404 flows, put the pull in git, and describe every config value once

OPERATIONS.md

@@ -239,7 +239,9 @@ Two properties of the Caddy side are worth knowing before parsing it. Its access
 
 **The off-host copy of the access log exists, and the schedule that maintains it is younger than the copy.** The pull to the backup host is installed as a `systemd` timer running daily at 09:00 UTC, chosen to sit behind both producers on the VPS rather than beside them, and its first copy was made by hand rather than by the timer. Read the unit and its last run on the backup host rather than trusting this paragraph, for the same reason retention is read from the VPS: a claim about a schedule is only worth what the machine says.
 
-**A rename on the VPS does not propagate to that copy, and nothing reports the divergence.** The pull passes no `--delete` for the logs, deliberately, since an append-only record must never be removed by a transfer. So a file **the VPS** renames, merges, or re-compresses after it has been pulled keeps its old name **on the backup host** forever, alongside the new one, and a count that walks that archive by filename double-counts the overlap. This has already happened once, to two archives whose names were a day ahead of their contents. **Read a date from a line's `StartUTC` rather than from the filename that holds it**, and treat a rename on the VPS as something the channel has to carry, because no transfer will.
+**A rename on the VPS does not propagate to that copy, and nothing reports the divergence.** The pull passes no `--delete` for the logs, deliberately, since an append-only record must never be removed by a transfer. So a file **the VPS** renames, merges, or re-compresses after it has been pulled keeps its old name **on the backup host** forever, alongside the new one, and a count that walks that archive by filename double-counts the overlap. This has already happened once, to two archives whose names were a day ahead of their contents. **Read a date from a line's `StartUTC` rather than from the filename that holds it.** The reconciliation itself now travels with the data: the VPS keeps an append-only `RECONCILE.md` **inside the archive directory**, so the pull carries it automatically and a rename does not depend on someone rereading a channel file. It records what a file contained rather than what it was called, and it is counted among the pulled log files, so expect the file count to exceed the number of logs by one.
+
+**A journal with one entry is not evidence of one copy.** The pull can be run directly as well as by its timer, and a direct run writes no service record. Directory mtimes on the backup host are the copy times, where the file mtimes are the VPS's, so those are what to read when establishing when something arrived.
 
 ## Who Owns What
 

TODO.md

@@ -18,7 +18,7 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y
 | Fleet conformance | cataloged in the hub registry, audited, and carrying the current canonical |
 | Deploy pipeline | `deploy-site.yml` is dispatchable and has deployed staging from CI end to end, through a transport retested against the real host |
 | VPS staging | live at `blog.vps.insanegenius.net`, behind the auth gate, serving a pipeline release |
-| VPS production | **M7a done 2026-08-08.** Serving release `20260808-041050` at `blog.insanegenius.net`, answering `200` unauthenticated, verified 9/9 from the host side with the built `baseURL` read from the deployed bytes. DNS for the public name is still on the old platform |
+| VPS production | **M7a done 2026-08-08.** Serving release `20260808-154717` at `blog.insanegenius.net`, answering `200` unauthenticated, deployed from `main` by pipeline with the 1,245-URL contract verified against the live site. `/robots.txt` answers 200 carrying a `.net` sitemap line, and the gallery fix is live. DNS for the public name is still on the old platform |
 | Operations | started, and neither half has completed a **scheduled** run. The off-host log pull is installed, armed for 09:00 UTC daily, and has copied once, started by hand, so the timer itself has never fired and 2026-08-09 is its first scheduled run. The periodic log review has not run at all |
 
 ## Blocked on the maintainer
@@ -33,8 +33,8 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y
   - **`HUGO_BASEURL` on the `production` environment is set to `https://blog.insanegenius.net/`**, done 2026-08-07. It held `https://blog.insanegenius.com/`, the live WordPress address, which is what the workflow both builds with and points the live check at, so a deploy would have baked the old platform's address into every canonical tag, feed link and `sitemap.xml` and then run 1,245 requests at the live site to verify it. **Setting it back to `.com` at M7b is the other half and is not done.**
   - **Production emits `X-Robots-Tag: noindex, nofollow` for the length of the rehearsal**, deliberately, because `.net` serves a public duplicate of a live site and Certificate Transparency publishes the hostname. Where a check asserts `index, follow`, make the expected value a parameter rather than flipping a literal, since it reverts at M7b and a hardcoded literal is one more thing to remember at the wrong moment.
   - **The two questions in §19.3 are answered.** `HUGO_BASEURL` holds the interim `.net` name, per the item above. Exactly one place hardcodes `blog.insanegenius.com`: `baseURL` on line 1 of `hugo.yaml`, which is the production default every environment overrides through `HUGO_BASEURL`. Nothing under `checks/`, `deploy/`, `layouts/`, or `.github/` carries it.
-- **`robots.txt` is decided and built, 2026-08-08, and what remains is that production has not been redeployed since.** The site emits one now, `enableRobotsTXT` is set, and the theme's template derives the `Sitemap:` line from the built `baseURL`, so it names `.net` during the rehearsal and `.com` after the cutover with nothing to remember at M7b. `/robots.txt/` redirects to the real file rather than to the home page, and `check-url-parity.py` gates all of it. The record below is kept because the reasoning is what the next decision about crawl directives will need, and because production still answers 404 until a deploy carries this.
-  - **The first deploy did not fix the 404, and that is what turned this from a gap into a decision.** The VPS agent raised it in §22.10 and both halves were measured rather than assumed: the site emitted no `robots.txt` at all, because `hugo.yaml` set no `enableRobotsTXT`, so the 404 survived the deploy and `X-Robots-Tag` was the only control, while `sitemap.xml` **was** emitted and became fetchable on the interim name at that same deploy. A crawler got a full sitemap and no robots file. `enableRobotsTXT` is now set, so this describes the release production is still serving rather than the current build.
+- **`robots.txt` is decided, built, and deployed, 2026-08-08.** The site emits one, `enableRobotsTXT` is set, and the theme's template derives the `Sitemap:` line from the built `baseURL`, so it names `.net` during the rehearsal and `.com` after the cutover with nothing to remember at M7b. `/robots.txt/` redirects to the real file rather than to the home page, and `check-url-parity.py` gates all of it. Verified from the served bytes on release `20260808-154717`: `/robots.txt` answers 200 advertising `https://blog.insanegenius.net/sitemap.xml`, `/robots.txt/` 301s to it, and `sitemap.xml` carries 312 `.net` URLs and zero `.com`. The record below is kept because the reasoning is what the next decision about crawl directives will need.
+  - **The first deploy did not fix the 404, and that is what turned this from a gap into a decision.** The VPS agent raised it in §22.10 and both halves were measured rather than assumed: the site emitted no `robots.txt` at all, because `hugo.yaml` set no `enableRobotsTXT`, so the 404 survived the deploy and `X-Robots-Tag` was the only control, while `sitemap.xml` **was** emitted and became fetchable on the interim name at that same deploy. A crawler got a full sitemap and no robots file. `enableRobotsTXT` is now set and a deploy has carried it, so this describes the state up to release `20260808-041050` rather than what is served today.
   - **At the cutover this stops being a gap and becomes a loss, which is the half neither side had checked.** The live `.com` blog **serves a `robots.txt` today, carrying a `Sitemap:` line**. Because this site emitted none, M7b would not have been a return to a previous state, it would have been a move from having crawl directives to having none on a site that has had them for years, and the sitemap pointer would have gone with them. The VPS agent measured this from the outside in §23.3, will not put a file in this repository's bundle, and has made it a decision that blocks step 1 of the M7b checklist rather than one discovered after it. The minimum that preserves today's behavior is `User-agent: *`, no `Disallow`, and the sitemap line, since every `Disallow` the old platform serves names a WordPress path this site does not have. **That is what was chosen**, out of three options: preserve today's behavior, write what this site actually wants, or keep emitting nothing and accept the loss. The sitemap URL is derived from the built `baseURL` rather than typed, which is what makes the choice survive the cutover without a second edit.
   - **The log reframes the decision, and it is the `Sitemap:` line that carries it rather than any rule.** Across the interim hostname's first full day, `/robots.txt` was requested nine times and answered 404 every time, five of those from real agents on a hostname with no inbound links. **No crawler fetched `sitemap.xml` or `feed.xml` once**: every request to either came from `curl`, the deploy gate's or the host side's. Crawlers do not guess a sitemap's location, they are told it, and the only thing telling them today is the `robots.txt` the old platform serves, which is the file the cutover deletes. So the question is not whether to have crawl directives, it is whether the sitemap stays advertised at all. Measured on the host side in its §26.4 and recorded here because the decision outlives that channel.
   - **`/robots.txt/`, with a trailing slash, now redirects to the real file** rather than to the home page, in the same change, since the two are only correct together. The fix is in `build-redirects.py` rather than in the generated map, because the map is rewritten from the capture and a hand edit does not survive the next regeneration. `/osd.xml/` stays pointed at the home page: it was the old platform's OpenSearch description and this site emits no such file.
@@ -52,7 +52,9 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y
 
 - **Review the logs for non-200s**, daily for the first week after cutover, then monthly. Real traffic finds what the golden list missed and the crawl that produced the list cannot. Append anything new to `checks/golden-urls.txt` and add a redirect. Read the edge as well as Caddy: a request the proxy refused never reaches the site's log, so a count taken from Caddy alone is a floor, and a staging probe for `/wp-login.php` answered by the auth gate rather than by the site is the shape of what Caddy never sees. The procedure, the three tiers and what each is blind to, and the inward pass that names content nobody has ever requested are in [`OPERATIONS.md`](./OPERATIONS.md) "Log Review".
 - **Pull the log off the VPS, on a schedule**, which is [#53][issue-53]. The access log is deliberately outside the nightly encrypted archives, because those are fourteen full copies with no dedupe and an append-only file would be multiplied by fourteen for no recovery benefit, so the VPS's 400-day window was the **only** copy until this ran. **It is installed**, as `vps-backup-pull.timer` at 09:00 UTC daily with `Persistent=true`, and a first copy exists: 42 archives and 4 log files, pulled 2026-08-08 12:59 UTC. **That run was started by hand, so the timer has never fired**, which is the distinction worth keeping until 2026-08-09 09:00 UTC proves the schedule rather than the script. One copy is a fact; "backed up daily" is still a unit file.
-  - **A rename on the VPS does not reach this copy, and nothing detects that it did not.** The pull deliberately passes no `--delete` for the logs, since that flag exists to mirror the VPS's fourteen-archive window and must never touch an append-only file. So when the host side renamed and merged its two mis-dated archives, the pre-fix name survived here: `access.log-2026-08-08`, 52 lines, every one of them 2026-08-07 traffic and every one already inside the merged `access.log-2026-08-07.gz`, which holds 58. Verified a strict subset with `comm -23` rather than assumed. **A line count over the off-host archive therefore returns 110 lines where 58 exist, half of them filed under a date whose traffic they are not** — which is exactly the defect the host side fixed, surviving on the copy the log review will read once the VPS's window rolls past what it needs. The general form is that any rename, merge, or re-compression of an already-pulled log leaves the old name here permanently, and the only propagation mechanism is a note in the channel. Removing that one file is the maintainer's call, since it is a deletion inside a backup tree.
+  - **A rename on the VPS does not reach this copy, and nothing detects that it did not.** The pull deliberately passes no `--delete` for the logs, since that flag exists to mirror the VPS's fourteen-archive window and must never touch an append-only file. So when the host side renamed and merged its two mis-dated archives, the pre-fix name survived here: `access.log-2026-08-08`, 52 lines, every one of them 2026-08-07 traffic and every one already inside the merged `access.log-2026-08-07.gz`, which holds 58. Verified a strict subset with `comm -23` rather than assumed. **A line count over the off-host archive therefore returned 110 lines where 58 exist, half of them filed under a date whose traffic they are not** — which is exactly the defect the host side fixed, surviving on the copy the log review will read once the VPS's window rolls past what it needs. The general form is that any rename, merge, or re-compression of an already-pulled log leaves the old name here permanently. **That one file is deleted and the archive reads 58**, and the general case now has a mechanism: the host side keeps an append-only `RECONCILE.md` **inside the archive directory**, so the pull carries it alongside the data it explains rather than relying on a note in a channel file nobody rereads. It records what a file contained rather than what it was called, and it will be counted among the pulled log files.
+  - **Read a date from a line's `StartUTC` rather than from the filename holding it.** That is the durable form of the lesson, and it is in [`OPERATIONS.md`](./OPERATIONS.md) "Log Review" as well.
+  - **An off-host copy also predates the timer.** The directory mtimes on the backup host are copy times where the file mtimes are the VPS's, and they show a pull at 2026-08-08 03:31 UTC that the service journal has no record of, because the script was run directly rather than through `systemd`. So a journal with one entry is not evidence of one copy. The whole set was audited both ways afterwards and nothing else had diverged: logs identical, 38 archives shared and identical in size, four newer on the VPS because they postdate the pull, four older retained off-host because the pull passes no `--delete`.
 
 ## Owed to the hub