fix(sandbox): pin podman-static v5.8.4 host bootstrap and harden re-runs - #49
Merged
Conversation
Thread transfer impact✅ Thread transfer remains within every enforced ceiling.
Baseline: Scenario and decoded snapshot size10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.
Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed. |
…e-runs Apt's podman 4.9.3 + netavark 1.14.0 has a structural rootless-netns bug: the shared per-UID pause namespace never bind-mounts a custom network's JSON config into its view for /var/lib-class graphroots, so `podman run --network <custom>` fails with "network not found" once the pause process is alive. No empirical probe can catch it (the probe network is created before the bug's precondition holds), so the old probe-gated pinned netavark strategy is replaced by an unconditional SHA256-pinned podman-static v5.8.4 bundle installed into /usr/local, with the wrapper execing /usr/local/bin/podman. This matches what is already deployed and working on the production sandbox host; the script was never committed. Hardening over the deployed WIP: - Wrap the download/extract in a function so its RETURN trap actually fires; at top level RETURN traps never run and the ~45MB temp dir leaked every install. - STEPS=2 standalone no longer leaves the socket down: record which of podman.socket/podman.service were active before the binary swap and restore exactly that at the end of the step. - Manifest-tracked install (/usr/local/share/podman-static/manifest): remove the previous bundle's files before copying the new one, so a future bundle that drops a helper cannot leave a stale binary first on podman's search path. - Remove the previous strategy's leftovers: the legacy pinned netavark/aardvark-dns in /usr/local/libexec/podman and the old appended helper_binaries_dir block in /etc/containers/containers.conf (only when the file is exactly that block; otherwise warn and rely on the conf.d drop-in that outranks it). - Refuse to proceed if a rootless ~/.config/containers/containers.conf exists for the service user; it would outrank every pin the step writes. - Install an AppArmor userns profile for /usr/local/bin/podman (modeled on Ubuntu's stock podman profile) when AppArmor is active and kernel.apparmor_restrict_unprivileged_userns=1; without it every rootless container fails with a permission error. Idempotent: parser reload only on content change. - Step 8 no longer passes --storage-opt size= by default: podman --remote rejects it and the backend omits the pair on socket deployments (ContainerSandboxBackend storageQuotaDisabled), so the verification now mirrors the backend exactly. The XFS prjquota volume quotas proven in 8b/8c remain the real disk bound. Rewrites docs/operations/sandbox-host.md to match: wrapper path, the real rationale for the pinned runtime (replacing the netavark-DNS-probe narrative), step table and ordering, the AppArmor caveat (now describing the installed profile), and a new "Supply chain" section recording the TOFU sign-off on the third-party podman-static bundle with the upstream key fingerprint and re-verification procedure. Note: scripts and docs land in one commit because the repo pre-commit formatter rejects staged sets with no formatter-target files, which a shell-only commit would be. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
awtprod
force-pushed
the
fix/podman-static-v58-bootstrap
branch
from
August 22, 2026 02:48
667e890 to
991706a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The production sandbox host has been running an uncommitted rewrite of the bootstrap script for days: apt's podman 4.9.3 + netavark 1.14.0 has a structural rootless-netns bug (the shared per-UID pause namespace never bind-mounts a custom network's JSON config into its view for
/var/lib-class graphroots, sopodman run --network <custom>fails with "network not found" once the pause process is alive). The committed script's probe-gated pinned-netavark strategy cannot catch this — the probe network is created before the bug's precondition holds — so the deployed WIP replaces it with an unconditional SHA256-pinned podman-static v5.8.4 bundle in/usr/local, verified working on the host. Git needs to catch up to the deployed reality, and the WIP had known bugs to fix before committing.Fixes over the deployed WIP
trap ... RETURNat top level never fires (RETURN traps only run on function/sourced-script return), leaking the ~45MB extract dir every run. The download/install now lives ininstall_podman_static(), matching the deleteddownload_pinned()idiom that was correct.STEPS=2standalone killed the socket: step 2 stoppedpodman.service/podman.socketto swap binaries and relied on step 6 to re-enable them — breaking the header's promise of independently re-runnable steps. It now records what was active and restores exactly that before the step ends.--storage-optcontradiction: step 8 passed--storage-opt size=through the wrapper (podman --remote), which rejects that flag — the very reasonContainerSandboxBackenddrops it viastorageQuotaDisabled(). Verification now mirrors the backend: omitted by default, opt-in viaT3_SANDBOX_CONTAINER_STORAGE_QUOTA. The XFS prjquota volume quotas (proven enforced in 8b/8c) remain the real disk bound.kernel.apparmor_restrict_unprivileged_userns=1and stock profiles cover only/usr/bin/podman; moving execution to unprofiled/usr/local/bin/podmanis exactly the trap the runbook warned about. Step 2 now installs a minimalusernsprofile (modeled on Ubuntu's stock podman profile) when AppArmor is active and the sysctl is 1, reloading only on content change. Step 8a is the assertion that catches a missing profile.cp -rmerge can never delete); the old strategy's legacy netavark/aardvark-dns in/usr/local/libexec/podmanand its appendedhelper_binaries_dirblock in/etc/containers/containers.confare cleaned up; a rootless~/.config/containers/containers.conffor the service user (which would outrank every pin) is a loud failure.docs/operations/sandbox-host.mdrewritten to match — wrapper path, the real pinned-runtime rationale, step table and ordering, the AppArmor caveat now describing the installed profile, and a new "Supply chain" section recording the TOFU sign-off on the third-partymgoltzsche/podman-staticbundle (hand-captured hashes, upstream signing-key fingerprint, re-verification procedure for when keyserver access is available).Static verification only (
bash -nboth scripts; careful read-through): per the task constraints, nothing was run against the live host and no host state was touched.Note: #47 also edits
docs/operations/sandbox-host.md(around the §6b flip section) and merges first; this branch's doc edits deliberately avoid that region to keep the rebase clean.Claude Fable 5 via Claude Code