feat(opensandbox): add run-scoped cleanup job

[hemildesai]

Summary

• replace the benchmark-local, user-wide cleanup helper with a standalone OpenSandbox cleanup CLI that requires an exact run and user scope
• attach the Slurm run and user attribution to every sandbox created by the evaluation
• submit one CPU-only afterany cleanup job immediately after the evaluation job, giving cleanup its own allocation and time budget after success, failure, timeout, or cancellation
• paginate the full sandbox inventory, delete matches concurrently through one bounded connection pool, and repeat the sweep to handle teardown races
• keep credentials in the existing connection config; the dependent job receives only its path and the exact cleanup scope

Test plan

• focused cleanup and launcher tests: 37 passed locally, cleanup module at 100% coverage
• Bash 5 launcher behavior checks for workload failure, TERM cancellation, and server-first exit
• scoped pre-commit hooks, shell syntax check, and git diff --check
• live Slurm dependency probe: plain parent cancellation released the afterany CPU job and inherited GPU requests were overridden
• live cancellation E2E with real attributed rollouts plus controlled isolation fixtures: matching target deleted with HTTP 2xx, same-user wrong-run decoy preserved, cleanup completed successfully, and the final exact-scope audit found zero survivors
• fixture teardown verified no test fixtures remained

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[hemildesai]

Real Slurm cancellation smoke test: done (the unchecked test-plan box)

Two rounds against a live 500-rollout eval, scancel issued with the full sandbox population present. Net: the epilogue works, but trap placement is load-bearing.

Round 1 — trap inside the containerized eval command: FAILS. When the eval runs under ray symmetric-run (this launcher family's topology), Ray's teardown kills its managed children the instant the step is cancelled — the trap never fires, no epilogue output, all ~350 sandboxes orphaned. Any launcher that wraps the eval in Ray (or any supervisor that hard-kills children) will silently lose the epilogue.

Round 2 — trap in the batch shell (host side, no exec so the shell survives): WORKS. scancel with 498 live sandboxes → Slurm TERMs the batch shell → trap fires within the kill grace → CLEANUP-EPILOGUE → 497 × Deleted … HTTP 204 → CLEANUP-EPILOGUE-DONE, population 498 → 1 in ~95 s end-to-end (most of that was the workload shutting down; the 32-way concurrent reap itself took seconds).

CLI validation at scale: separately ran the CLI against round 1's 349 orphans — all reaped, zero remaining, correctly scoped to exact run+user.

Two suggestions:

1. Document (or assert in the script) that the trap must live in a shell Slurm signals directly — not inside ray symmetric-run or similar child-killing supervisors. The batch shell also needs a host python with aiohttp, which is worth a preflight check.
2. One sandbox out of 498 survived round 2's reap (created mid-wave, still listed at reap time) — likely list-pagination skew while the shutting-down workload's own deletes were mutating the set. A cheap fix: loop list→delete until the list comes back empty instead of a single pass.

🤖 Generated with Claude Code

[bxyu-nvidia]

/ok to test b5038e2