fix(docker): stage2 API_SERVER_KEY bootstrap no longer depends on .env existing (OOF-285)

[shannonsands]

Problem (OOF-285 — escalated to P1)

A fleet sweep of all started hosted instances on v2026.8.13+ found 144 of 351 (41%) missing API_SERVER_KEY. Without it the loopback gateway api_server (which serves /api/cron/fire on :8642) refuses to start, so every scheduled cron fire on those instances is silently lost — NAS/QStash retries get 503 until the budget exhausts, and nothing surfaces to the customer. Cohort: v2026.8.13 ×74, v2026.8.16 ×50, v2026.8.16.2 ×20. Sampled affected instances have no /opt/data/.env at all.

Root cause chain (each link verified on live instances)

1. .dockerignore excludes .env.example (from an old image-size optimization), so /opt/hermes/.env.example does not exist in shipped images.
2. stage2-hook's first-boot seed seed_one ".env" ".env.example" is a silent no-op when the source is missing → fresh volumes never get a .env.
3. The API_SERVER_KEY generation added in fix(cron): managed-cron fires execute in the gateway process (live adapters + dashboard forwarder) #84339 was gated on [ -f "$HERMES_HOME/.env" ] → never ran on those instances.

The 207 healthy instances got their key by luck: some runtime save_env_value call (Telegram pairing, credential setup, …) created .env organically, and a later reboot's stage2 hook appended the key.

Fixes

• docker/stage2-hook.sh: keygen now creates an owner-only (0600, hermes-owned) .env when missing instead of requiring it to exist. Still append-only w.r.t. operator-provided keys; still refuses symlinked paths; warns loudly if .env can't be created.
• .dockerignore: re-include .env.example (!.env.example after the .env.* exclusion — last match wins) so the first-boot template seed works again.
• Tests: new tests/tools/test_stage2_hook_api_server_keygen.py executes the actual keygen block from the hook in a sandbox: create-when-missing, append-without-clobber, operator-key preservation, symlink refusal, plus a .dockerignore contract test that .env.example survives ignore evaluation.

Validation

• 18/18 stage2/dockerfile contract tests pass, ruff clean, bash -n clean.
• Manual remediation of one production instance (318-6th-ave, the original OOF-285 report) using exactly this keygen procedure restored cron fires end-to-end (verified fire cron_c5e9a881831b_20260818_020000 executed).

Rollout notes

• Existing affected instances self-heal on their next reboot with this image (stage2 runs every container start).
• A one-time fleet remediation sweep for the 144 currently-affected apps is being coordinated separately on OOF-285 (piloting on 2–3 instances first).
• Catch-up for already-lost fires is out of scope here (tracked on OOF-285).

[shannonsands]

Incident timeline: how this composed (blame-free accounting)

For the record — this outage was the composition of three individually-defensible changes, each blind to the others. No single PR was "wrong"; the failure lived in an undocumented dependency between them.

Date	Change	Rationale at the time	Hidden consequence
2026-03-28	dcbdfdbb2b (#3668) — Docker support added	First-boot seed copies /opt/hermes/.env.example → $HERMES_HOME/.env	.env.example became a runtime dependency, but only implicitly — it entered the image via COPY . . with no explicit COPY line, no test, no comment marking it as needed at boot
2026-06-05	eb43a5b5d8 (#6092) — .dockerignore hygiene	Standard .env / .env.* patterns — the correct "never bake secrets into images" reflex	The .env.* glob also matches .env.example, which is a template, not a secret. First (unintentional) breakage of the seed
2026-06-10	a72bb03757 (#38749) — image-size optimization	.env.example explicitly listed under "repo-level dotfiles that are git-only or dev-tooling config" — true for ~99% of repos	Made the exclusion deliberate. The author had no signal that a shell hook reads this file at container boot
Jun 10 → Aug 12	(broken but harmless)	—	Fresh volumes never got a seeded .env, but nothing required one: runtime secrets are written via save_env_value(), which creates the file itself. The seed was pure convenience, so nothing surfaced
2026-08-12	bb597e1c02 (#84339, ships as v2026.8.13) — cron fires move into the gateway's loopback api_server (:8642), which refuses to start without API_SERVER_KEY	Keygen added to stage2-hook, gated on [ -f .env ] — a deliberate "append-only, never create" safety choice	Written assuming the first-boot seed still worked. It hadn't for two months → on volumes with no organically-created .env, the keygen never ran → no api_server → all scheduled cron fires silently lost

Why 41% of the fleet, not 100%

The 207 healthy instances got .env created by accident: some runtime event (Telegram pairing, a credential save) called save_env_value() → file created → a subsequent reboot's stage2 hook saw [ -f .env ] pass and appended the key. Instances whose users never triggered such a write — or never rebooted after one — stayed broken. That's why the affected cohort correlates with customer behavior, not rollout date (missing-key instances split evenly across Jul/Aug creation months).

Root lesson

The only signal that .env.example was load-bearing at runtime was buried inside docker/stage2-hook.sh. Nothing in the Dockerfile, .dockerignore, or CI encoded that dependency, so two reasonable hygiene commits severed it without anyone noticing, and a later feature built on the severed link.

This PR closes both failure links and adds the missing contract: the keygen no longer depends on the seed (creates .env when missing), the seed works again (!.env.example re-include), and a .dockerignore contract test makes the dependency explicit so the next hygiene pass can't silently re-break it.

Fleet impact & remediation tracked on Linear OOF-285 (144/351 started instances on v2026.8.13+ affected; manual remediation sweep in progress).

[shannonsands]

P1 addressed in 31cb0d8 — good catch, and confirmed: hermes_cli/env_loader.py loads the profile .env with override=True, so a generated key would indeed shadow an operator's docker run -e API_SERVER_KEY=... credential and 401 existing clients.

Fix:

• The generation guard now checks the inherited container environment first. API_SERVER_KEY present in env → skip generation entirely (logged: "provided via container environment — skipping generation"). stage2-hook runs as a cont-init.d script and sees the full container env, so -e values are visible to it.
• Conflict case (key in BOTH env and .env): warn explicitly that the .env value wins at runtime (override=True), touch nothing — surfacing the shadowing behavior instead of silently participating in it.

Regression tests added (tests/tools/test_stage2_hook_api_server_keygen.py):

• test_keygen_skips_when_container_env_provides_key — env key set, no .env write, skip logged
• test_keygen_env_key_with_existing_env_file_key_warns_not_clobbers — both sources set → warning, .env byte-identical
• The sandbox runner now explicitly pins/unsets API_SERVER_KEY so all tests are hermetic w.r.t. the host env

Note for the hosted-fleet context (OOF-285): managed instances don't set API_SERVER_KEY via container env (verified during the fleet sweep — the key lives in /opt/data/.env or nowhere), so the remediation path is unaffected; this fix protects self-hosted docker run -e deployments.

On CI: the slice 10/12 failure (test_session_api.py::test_session_chat_stream_treats_pre_existing_poisoned_row_as_no_model) is unrelated to this diff (which touches only docker/stage2-hook.sh, .dockerignore, and its own test file) and passes locally on this branch. The new commit will retrigger the suite.