fix(server): harden sandbox lifecycle against restarts, deletion, and provision failures

[awtprod]

Six sandbox hardening fixes from the audit (P0 4-6, P1 8-10). Stacked on #50 (impl/sandbox-stop-wedge).

Problems, in plain language

1. Mixed docker/podman threads broke each other's previews. The desktop gateway held one global preview-proxy reference; each per-runtime manager overwrote it on provision. Since a proxy only knows its own runtime's containers, provisioning a podman thread instantly broke every docker thread's preview, CDP automation, and WebRTC signaling. Now keyed per thread, cleared by removeThread.

2. A server restart stranded every live container. Reconcile only adopted threads with an in-memory record, and restarts empty those records -- so every surviving container was reported missing, its thread marked failed, while the container kept running (it was still "expected", so orphan removal skipped it). Reconcile now verifies restart-discovered containers against the thread's label signature (same verification the export/teardown adoption path uses) and adopts the verified ones for accounting; an expected container that fails verification is stopped instead of left running unaccounted. Adoption is never cached: exec stays fail-closed.

3. Deleting a thread left its sandbox running and its transcripts on disk. The deletion reactor stopped the provider session and terminals but never dispatched sandbox.stop, and nothing deleted the exported branch bundle / provider conversation store. It now dispatches sandbox.stop for stoppable lifecycles and best-effort deletes the thread's artifacts (logged with cause on failure). The lifecycle reactor's stop path learned to find deleted threads (the detail query hides them) and skips the branch export for them -- transcripts must not outlive the thread.

4. Sidecars ran without resource limits. Egress, credential, and preview proxies plus service containers only had a pids ceiling. All now get conservative --memory/--cpus (256m/0.5 for proxies, 1g/1 for services), --memory-swap pinned to --memory (docker otherwise defaults swap to 2x memory -- also applied to the workspace container), and the egress sidecar gets a tmpfs /tmp (it is --read-only and had no writable scratch space).

5. A provision failing at desktop start leaked gateway state. Preview routes, service status, and credential grants (token values readable via the read-scoped status endpoint) were registered before desktop.start but never unwound on failure -- live routes and grants pointed at destroyed containers. Every unwind now clears the thread's gateway state first.

6. A sync throw in the provision generator skipped all unwinds. The "generated service credential ... is missing" check threw a bare Error from inside the Effect generator -- a defect that escapes tapError, leaking everything provisioned so far. Now a typed SandboxManagerError with its own unwind.

Follow-up (out of scope here)

The GET /api/thread-desktop/<id>/status endpoint returns service credential grant token values under a read scope. Fix 5 stops them from outliving a failed provision, but the payload/scope of that endpoint deserves its own review.

Verification

• vp check exit 0, vpr typecheck exit 0
• Full server suite: 3275 passed, 19 skipped (330 files, +2 skipped)
• New tests: per-thread proxy resolution across runtimes, label-verified reconcile adoption + fail-closed unverified stop, deletion-triggered stop + artifact removal (real temp dir), gateway-state unwind at desktop failure, typed grant failure with executor-log teardown assertions, run-arg assertions for every new limit flag

---

Follow-up: three defects found by actually running it

The fixes above were written against unit tests. Building an end-to-end canary
that drives real containers surfaced three more defects that no existing test
could have caught -- every sandbox test either stubbed the lifecycle reactors or
faked the runtime manager, so the path that actually provisions had no
coverage at all.

7. Sandboxes could never work on a rootless-podman host. The executor spawned
   every container command with env: { PATH: process.env.PATH }. Stripping the
   host environment is the point -- it keeps provider credentials out of
   container commands -- but it also strips XDG_RUNTIME_DIR, which is how
   rootless podman finds its socket and libpod runtime root. Every command died
   at podman info, so provisioning failed in #validateRootless and the
   sandbox went straight to failed. Measured directly: PATH-only fails,
   PATH+XDG_RUNTIME_DIR succeeds. The deployment had been working around it
   with a PATH wrapper that re-injects CONTAINER_HOST
   (deploy/openclaw/sandbox/podman-wrapper.sh, documented as constraint feat: build the Command Center agentic OS foundation #1 of
   the host design). The allowlist stays; the few entries that name the runtime
   rather than carry a secret now pass through.

8. The Provision button silently did nothing on any thread that had run once.
   sandbox.provision has two paths: one drives the sandbox to provisioning
   and emits sandbox.provisioning-started (read only by the projector --
   correct for a caller that provisions inline immediately after), the other
   emits sandbox.provision-requested, which is what SandboxLifecycleReactor
   listens for. The decider chose between them by whether a branch was
   resolvable -- but the UI sends { threadId } with no branch, and a thread
   that has run once keeps sandbox.branch across teardown. So every client
   re-provision skipped the request event and parked the thread in
   provisioning with no container, no reactor running, and no error. Branch
   presence cannot carry that meaning (ProviderCommandReactor provisions
   inline and omits the branch when re-provisioning), so the claim is now an
   explicit server-only provisionsInline, absent from
   ClientSandboxProvisionCommand -- the same shape as
   ThreadSessionStopCommand.canceledThroughSequence.

9. Container failures threw away their own diagnosis.
   SandboxRuntimeManager.attempt rebuilt every backend failure from
   cause.message, and SandboxRuntimeError keeps the runtime's stderr in a
   separate field -- so a thread's recorded failure, the lifecycle warning, and
   the UI all showed a bare "podman network failed". This is what made the two
   above take so long to find: the canary looked like a 67% flake for five runs
   until the real text ("network name t3-net- already used") named the
   cause instantly.

The canary

apps/server/integration/sandboxCanary.integration.ts, plus a
realSandboxReactors option on the integration harness that swaps the stubs for
SandboxLifecycleReactorLive, SandboxSettleCleanupReactorLive,
ThreadDeletionReactorLive, and SandboxRuntimeManagerLive. Off by default, so
decider and projection tests stay hermetic and never touch a container.

It drives provision -> settle -> re-provision -> delete and asserts against the
host, with the provider adapter still faked (none of this needs a provider
credential):

Phase 1  PASS  sandbox reached 'ready'; workspace container running on the host
Phase 2  PASS  settled to 'stopped' -- never wedged in 'stopping'   <- #47/#50
         PASS  containers reclaimed on settle
         PASS  export artifacts written on teardown
Phase 3  PASS  re-provisioned from the export                        <- #47
Phase 4  PASS  containers torn down on thread deletion               <- this PR
         PASS  transcript artifacts removed on deletion              <- #52

Three consecutive runs, 10/10 each, exit 0. Run it with:

T3_SANDBOX_RUNTIME=podman T3_SANDBOX_DESKTOP=disabled \
T3_SANDBOX_CONTAINER_STORAGE_QUOTA=disabled \
node apps/server/integration/sandboxCanary.integration.ts

Notes for whoever runs it: the host needs digest-pinned T3_SANDBOX_IMAGE and
T3_SANDBOX_PREVIEW_PROXY_IMAGE; T3_SANDBOX_RUNTIME defaults to docker,
which this host does not have; and podman here reports no project-quota support
for volumes despite the prjquota XFS graphroot, so volume quotas must be
disabled. T3_INTEGRATION_TMPDIR places the harness root outside a private
/tmp, which the runtime's mount namespace cannot see.

Verification (updated)

• Canary: 3 consecutive runs, 10/10 checks, exit 0, against real containers
• Full server suite: 3274 passed, 3 failed -- all three are CodexAdapter
  tests failing on "refuses linked Git metadata outside its managed worktree
  directory", reproduced identically on a worktree without any of these commits
• vpr typecheck exit 0; new tests for the executor allowlist and the decider's
  client path are revert-checked (each fails with its fix removed)

🤖 Generated with Claude Code (Claude Fable 5)

[github-actions]

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider	Metric	Main baseline	This PR	Impact	PR ceiling
Codex	Total thread wire	13.6 KiB	13.6 KiB	−12 B (−0.1%)	15.1 KiB	✅
Codex	Thread snapshot wire	6.9 KiB	6.9 KiB	0 B (0.0%)	7.3 KiB	✅
Codex	Live turn WebSocket wire	6.7 KiB	6.7 KiB	−12 B (−0.2%)	7.8 KiB	✅
Codex	Live turn WebSocket decoded	54.2 KiB	54.2 KiB	0 B (0.0%)	66.4 KiB	✅
Codex	Live turn messages	17	17	0 (0.0%)	21	✅
Claude	Total thread wire	13.7 KiB	13.7 KiB	−12 B (−0.1%)	15.1 KiB	✅
Claude	Thread snapshot wire	7.0 KiB	7.0 KiB	−6 B (−0.1%)	7.3 KiB	✅
Claude	Live turn WebSocket wire	6.7 KiB	6.7 KiB	−6 B (−0.1%)	7.8 KiB	✅
Claude	Live turn WebSocket decoded	55.0 KiB	55.0 KiB	0 B (0.0%)	66.4 KiB	✅
Claude	Live turn messages	17	17	0 (0.0%)	21	✅

Baseline: ceb12db · PR result: c27f869 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

• Codex decoded thread snapshot: 107.8 KiB
• Claude decoded thread snapshot: 108.6 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.