feat(opensandbox): keepalive-bounded aiohttp transport + create/command hardening

[hemildesai]

Summary

Fixes two sandbox-infrastructure failure classes that silently zero out rollout rewards in high-concurrency sandboxed agent evals (diagnosed on 4-node GB200 SWE-bench Verified runs at concurrency 300–500 against an EKS OpenSandbox deployment; the same signatures appear at concurrency 8, just less often).

1. Server disconnected without sending a response

The SDK's default httpx pool keeps idle connections for 30s (opensandbox.config.connection.with_transport_if_missing), but the OpenSandbox server's uvicorn keep-alive reaper closes idle sockets after ~5s. Agent workloads idle between sandbox commands (model think time), so commands routinely reuse a socket the server already closed — surfacing as httpx.RemoteProtocolError → SandboxInternalException, which permanently kills the rollout.

Fix: the provider injects a transport whose keepalive_expiry sits below the server's keep-alive timeout (default 3s, connection.keepalive_expiry_s), so pooled sockets are retired before the server can close them:

• connection.transport_backend: httpx (default) — stock httpx.AsyncHTTPTransport(limits=..., retries=connect_retries). No new required dependencies.
• connection.transport_backend: aiohttp (opt-in) — aiohttp ClientSession/TCPConnector pool under the SDK's httpx surface via httpx-aiohttp. Falls back to the httpx transport with a warning when the package is absent.
• keepalive_expiry_s: null disables injection entirely (SDK default transport).

2. 502 could not connect to the backend sandbox endpoint='<podIP>:44772'

With create.skip_health_check: true, Sandbox.create returns before the pod's exec daemon is listening; the first command races pod startup and the server proxy 502s. Under a create burst this killed 7–12 rollouts per run.

Fix: create.skip_health_check: false (create waits for readiness, bounded by the spec's ready_timeout_s; create.timeout_s lowered 1200→900 to stay coherent with a 600s ready timeout).

command_retries stays at 0. An earlier revision of this branch raised it to 2; that is reverted. Retrying a command the server may have already started would execute it twice, and agent commands are frequently mutating (file writes, patch application, test-state changes). The keepalive bound removes the stale-connection failures retries were compensating for, so the risk buys nothing. Raise it only for idempotent workloads.

3. Separate resource requests and limits

Sandbox.create accepts distinct resource (limits) and resource_requests maps; a lone resource map is applied by the server as requests=limits. This exposes the requests side via sandbox_spec.provider_options.resource_requests (same keys as SandboxSpec.resources).

Field motivation: SWE-bench test suites OOM-killed sandbox pods at 2Gi, but raising a single combined map to 8Gi would 4× the cluster reservation. With the split, limits go to 1 vCPU / 8Gi while requests stay 0.5 vCPU / 2Gi. Requires opensandbox>=0.1.15 (lower bound raised here).

Validation

Measured on the EKS deployment across paired 4-node runs at concurrency 300–500, identical except for the variable under test:

	2Gi requests=limits	8Gi limits / 2Gi requests
infra-failed rollouts	41/316 (13.0%)	19/288 (6.6%)
pass@1 (avg per instance)	58.0%	67.1%
mean reward excl. infra failures	0.647	0.706

With the full stack (keepalive bound + health-checked create + requests/limits split), a later run showed 3 infra failures in 70 rollouts (4.3%), zero Server disconnected events, and zero OOMKilled pods — all three failures traced to a single misbehaving worker node rather than the client. A fleet sweep of execd logs across 92 sandbox pods found 101 OnExecuteError events, all CommandExecError: 1 (the agent's own commands exiting nonzero) and zero exec-layer faults: no spawn failures, no timeouts, no OOM kills.

Relationship to #2020

Complementary, no overlap: #2020 adds job attribution metadata (team/user/workload/run labels); this PR covers transport reliability, create hardening, and the resource split. Field-validated together — the attribution labels are what make post-cancellation sandbox garbage collection safely scoped to a single job.

Testing

• tests/unit_tests/test_opensandbox_provider.py: new test_connection_transport_backends (httpx default with keepalive expiry, custom pool settings, fallback when httpx_aiohttp is unavailable, keepalive_expiry_s: null disables injection), test_connection_transport_backend_aiohttp_opt_in (importorskip-guarded), and test_direct_create_passes_resource_requests_to_sdk_create (requests/limits both reach the SDK, sanitized; bad-type and unknown-key rejection). test_connection_config_and_image_policy updated for the injected transport kwarg.
• Full file: 16 passed, both with and without httpx-aiohttp installed.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[hemildesai]

Superseded by #2212 — identical content (verified byte-identical tree), with Signed-off-by trailers added so the DCO check passes. Force-pushing this branch to add the trailers in place is blocked by the repository's non_fast_forward ruleset, so a fresh branch was the only route. Closing in favour of the new PR.