Skip to content

fix(sandbox): pin podman-static v5.8.4 host bootstrap and harden re-runs - #49

Merged
awtprod merged 1 commit into
mainfrom
fix/podman-static-v58-bootstrap
Aug 22, 2026
Merged

fix(sandbox): pin podman-static v5.8.4 host bootstrap and harden re-runs#49
awtprod merged 1 commit into
mainfrom
fix/podman-static-v58-bootstrap

Conversation

@awtprod

@awtprod awtprod commented Aug 21, 2026

Copy link
Copy Markdown
Owner

Problem

The production sandbox host has been running an uncommitted rewrite of the bootstrap script for days: apt's podman 4.9.3 + netavark 1.14.0 has a structural rootless-netns bug (the shared per-UID pause namespace never bind-mounts a custom network's JSON config into its view for /var/lib-class graphroots, so podman run --network <custom> fails with "network not found" once the pause process is alive). The committed script's probe-gated pinned-netavark strategy cannot catch this — the probe network is created before the bug's precondition holds — so the deployed WIP replaces it with an unconditional SHA256-pinned podman-static v5.8.4 bundle in /usr/local, verified working on the host. Git needs to catch up to the deployed reality, and the WIP had known bugs to fix before committing.

Fixes over the deployed WIP

  • Temp-dir leak: trap ... RETURN at top level never fires (RETURN traps only run on function/sourced-script return), leaking the ~45MB extract dir every run. The download/install now lives in install_podman_static(), matching the deleted download_pinned() idiom that was correct.
  • STEPS=2 standalone killed the socket: step 2 stopped podman.service/podman.socket to swap binaries and relied on step 6 to re-enable them — breaking the header's promise of independently re-runnable steps. It now records what was active and restores exactly that before the step ends.
  • --storage-opt contradiction: step 8 passed --storage-opt size= through the wrapper (podman --remote), which rejects that flag — the very reason ContainerSandboxBackend drops it via storageQuotaDisabled(). Verification now mirrors the backend: omitted by default, opt-in via T3_SANDBOX_CONTAINER_STORAGE_QUOTA. The XFS prjquota volume quotas (proven enforced in 8b/8c) remain the real disk bound.
  • AppArmor: this host sets kernel.apparmor_restrict_unprivileged_userns=1 and stock profiles cover only /usr/bin/podman; moving execution to unprofiled /usr/local/bin/podman is exactly the trap the runbook warned about. Step 2 now installs a minimal userns profile (modeled on Ubuntu's stock podman profile) when AppArmor is active and the sysctl is 1, reloading only on content change. Step 8a is the assertion that catches a missing profile.
  • Idempotency / stale-binary cleanup: bundle files are recorded in a manifest and the previous bundle's files are removed before installing the new one (a bare cp -r merge can never delete); the old strategy's legacy netavark/aardvark-dns in /usr/local/libexec/podman and its appended helper_binaries_dir block in /etc/containers/containers.conf are cleaned up; a rootless ~/.config/containers/containers.conf for the service user (which would outrank every pin) is a loud failure.
  • Docs: docs/operations/sandbox-host.md rewritten to match — wrapper path, the real pinned-runtime rationale, step table and ordering, the AppArmor caveat now describing the installed profile, and a new "Supply chain" section recording the TOFU sign-off on the third-party mgoltzsche/podman-static bundle (hand-captured hashes, upstream signing-key fingerprint, re-verification procedure for when keyserver access is available).

Static verification only (bash -n both scripts; careful read-through): per the task constraints, nothing was run against the live host and no host state was touched.

Note: #47 also edits docs/operations/sandbox-host.md (around the §6b flip section) and merges first; this branch's doc edits deliberately avoid that region to keep the rebase clean.

Claude Fable 5 via Claude Code

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XL labels Aug 21, 2026
@github-actions

github-actions Bot commented Aug 21, 2026

Copy link
Copy Markdown

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ The exact PR base did not have a successful artifact. Baseline uses the latest successful main measurement shown below.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 13.6 KiB 13.6 KiB −6 B (−0.0%) 15.1 KiB
Codex Thread snapshot wire 6.9 KiB 6.9 KiB −2 B (−0.0%) 7.3 KiB
Codex Live turn WebSocket wire 6.7 KiB 6.7 KiB −4 B (−0.1%) 7.8 KiB
Codex Live turn WebSocket decoded 54.2 KiB 54.2 KiB 0 B (0.0%) 66.4 KiB
Codex Live turn messages 17 17 0 (0.0%) 21
Claude Total thread wire 13.7 KiB 13.7 KiB −8 B (−0.1%) 15.1 KiB
Claude Thread snapshot wire 7.0 KiB 7.0 KiB +2 B (+0.0%) 7.3 KiB
Claude Live turn WebSocket wire 6.7 KiB 6.7 KiB −10 B (−0.1%) 7.8 KiB
Claude Live turn WebSocket decoded 55.0 KiB 55.0 KiB 0 B (0.0%) 66.4 KiB
Claude Live turn messages 17 17 0 (0.0%) 21

Baseline: 520c72d · PR result: 991706a · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 107.8 KiB
  • Claude decoded thread snapshot: 108.6 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

…e-runs

Apt's podman 4.9.3 + netavark 1.14.0 has a structural rootless-netns bug:
the shared per-UID pause namespace never bind-mounts a custom network's
JSON config into its view for /var/lib-class graphroots, so `podman run
--network <custom>` fails with "network not found" once the pause process
is alive. No empirical probe can catch it (the probe network is created
before the bug's precondition holds), so the old probe-gated pinned
netavark strategy is replaced by an unconditional SHA256-pinned
podman-static v5.8.4 bundle installed into /usr/local, with the wrapper
execing /usr/local/bin/podman. This matches what is already deployed and
working on the production sandbox host; the script was never committed.

Hardening over the deployed WIP:

- Wrap the download/extract in a function so its RETURN trap actually
  fires; at top level RETURN traps never run and the ~45MB temp dir
  leaked every install.
- STEPS=2 standalone no longer leaves the socket down: record which of
  podman.socket/podman.service were active before the binary swap and
  restore exactly that at the end of the step.
- Manifest-tracked install (/usr/local/share/podman-static/manifest):
  remove the previous bundle's files before copying the new one, so a
  future bundle that drops a helper cannot leave a stale binary first on
  podman's search path.
- Remove the previous strategy's leftovers: the legacy pinned
  netavark/aardvark-dns in /usr/local/libexec/podman and the old
  appended helper_binaries_dir block in /etc/containers/containers.conf
  (only when the file is exactly that block; otherwise warn and rely on
  the conf.d drop-in that outranks it).
- Refuse to proceed if a rootless ~/.config/containers/containers.conf
  exists for the service user; it would outrank every pin the step writes.
- Install an AppArmor userns profile for /usr/local/bin/podman (modeled
  on Ubuntu's stock podman profile) when AppArmor is active and
  kernel.apparmor_restrict_unprivileged_userns=1; without it every
  rootless container fails with a permission error. Idempotent: parser
  reload only on content change.
- Step 8 no longer passes --storage-opt size= by default: podman
  --remote rejects it and the backend omits the pair on socket
  deployments (ContainerSandboxBackend storageQuotaDisabled), so the
  verification now mirrors the backend exactly. The XFS prjquota
  volume quotas proven in 8b/8c remain the real disk bound.

Rewrites docs/operations/sandbox-host.md to match: wrapper path, the real
rationale for the pinned runtime (replacing the netavark-DNS-probe
narrative), step table and ordering, the AppArmor caveat (now describing
the installed profile), and a new "Supply chain" section recording the
TOFU sign-off on the third-party podman-static bundle with the upstream
key fingerprint and re-verification procedure.

Note: scripts and docs land in one commit because the repo pre-commit
formatter rejects staged sets with no formatter-target files, which a
shell-only commit would be.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@awtprod
awtprod force-pushed the fix/podman-static-v58-bootstrap branch from 667e890 to 991706a Compare August 22, 2026 02:48
@github-actions github-actions Bot added size:L and removed size:XL labels Aug 22, 2026
@awtprod
awtprod merged commit 20bee75 into main Aug 22, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant