feat(opensandbox): keepalive-bounded transport + create hardening + resource requests/limits

[hemildesai]

Summary

Hardens the OpenSandbox provider and the mini_swe_agent_2 eval path so that large sandboxed agent evals (SWE-bench Verified scale: hundreds of concurrent rollouts against a Kubernetes-backed OpenSandbox deployment) run end-to-end reliably: every finished rollout is delivered and recorded, and sandbox-infrastructure noise is bounded instead of silently zeroing rewards. Diagnosed on multi-node runs at concurrency 300–1500; the same failure signatures appear at concurrency 8, just less often.

1. Server disconnected without sending a response

The SDK's default httpx pool keeps idle connections for 30s (opensandbox.config.connection.with_transport_if_missing), but the OpenSandbox server's uvicorn keep-alive reaper closes idle sockets after ~5s. Agent workloads idle between sandbox commands (model think time), so commands routinely reuse a socket the server already closed — surfacing as httpx.RemoteProtocolError → SandboxInternalException, which permanently kills the rollout.

Fix: the provider injects a transport whose keepalive_expiry sits below the server's keep-alive timeout (default 3s, connection.keepalive_expiry_s):

• connection.transport_backend: httpx (default) — stock httpx.AsyncHTTPTransport(limits=..., retries=connect_retries). No new required dependencies.
• connection.transport_backend: aiohttp (opt-in) — aiohttp pool under the SDK's httpx surface via httpx-aiohttp; falls back to the httpx transport with a warning when absent.
• keepalive_expiry_s: null disables injection entirely (SDK default transport).

2. 502 could not connect to the backend sandbox endpoint='<podIP>:44772' on first command

With create.skip_health_check: true, Sandbox.create returns before the pod's exec daemon is listening; the first command races pod startup and the server proxy 502s. Under a create burst this killed 7–12 rollouts per run.

Fix: create.skip_health_check: false (create waits for readiness, bounded by the spec's ready_timeout_s; create.timeout_s kept above the ready timeout).

command_retries stays at 0. Retrying a command the server may have already started would execute it twice, and agent commands are frequently mutating. The keepalive bound removes the stale-connection failures retries were compensating for. Raise it only for idempotent workloads.

3. Separate resource requests and limits

Sandbox.create accepts distinct resource (limits) and resource_requests maps; a lone resource map is applied by the server as requests=limits. This exposes the requests side via sandbox_spec.provider_options.resource_requests (same keys as SandboxSpec.resources). Motivation: SWE-bench test suites OOM-killed sandbox pods at 2Gi, but raising a single combined map to 8Gi would 4× the cluster reservation; with the split, limits rise while requests stay small and pods pack densely. Requires opensandbox>=0.1.15 (lower bound raised here).

4. Sandbox teardown off the rollout's critical path

env.cleanup() ran in the rollout Ray task's finally block, so a finished result only became fetchable after the sandbox DELETE returned — and a teardown failure re-raised over the successful eval, degrading it to a reward-0 error row. Teardown is also effectively one-shot (Sandbox.stop latches _closed before dispatch), so a single failed DELETE leaked the sandbox while still costing the rollout.

Fix: cleanup runs on a best-effort daemon thread; results return immediately, teardown errors are logged rather than raised, and orphans remain covered by the provider's sandbox TTL.

5. High-concurrency rollout delivery (mini_swe_agent_2)

Two coupled fixes so every finished rollout is delivered at high concurrency (note: at very high concurrency the single-process policy-model proxy remains a separate bottleneck — connection-refused storms confirmed by a controlled A/B — with its scaling fix tracked separately):

• await the Ray ObjectRef instead of asyncio.to_thread(ray.get, ...): the default executor caps at min(32, cpu+4) threads, each pinned for a full rollout, so delivery stalls at ~32 concurrent rollouts and finished tasks queue behind blocked ray.get calls.
• bounded litellm retries (num_retries=5, config-overridable) — no retry means one transient LLM-call failure kills a whole rollout; unbounded retries make failures look like hangs — and num_cpus=0.25 on the rollout Ray task so concurrency is not capped at cluster core count (rollouts are I/O-bound).

6. datasets declared in the base agent config

The struct-mode config merge rejects keys absent from the base config, so a benchmark config using _inherit_from could not add its dataset list (ConfigKeyError: Key 'datasets' is not in struct). Declared empty in the base server config, matching how other agents (e.g. swe_agents) expose it.

Validation

• Requests/limits split (paired multi-node runs, identical except the variable under test): infra-failed rollouts 41/316 (13.0%) at 2Gi requests=limits → 19/288 (6.6%) at 8Gi limits / 2Gi requests; pass@1 58.0% → 67.1%.
• Keepalive bound + health-checked create: zero Server disconnected events and zero create-race 502s across all subsequent runs (previously 2–12 per run).
• Full stack, end-to-end from this branch: single-pass SWE-bench Verified (500 instances, concurrency 500+) delivered >98% of rollouts with ~1.6% residual sandbox-infra failures (server-side proxy 502s on established sandboxes, tracked separately) and zero client-transport failures. Before the delivery fixes (Alias as Penguin #4/Comp-Coding Verifier #5), an identically shaped run lost the majority of completed evaluations at the wall — hundreds of finished evals, only tens recorded.

Relationship to #2020

Complementary, no overlap; #2020 (job attribution metadata) has since merged and this branch is updated on top of it. Field-validated together — the attribution labels are what make post-cancellation sandbox garbage collection safely scoped to a single job.

Testing

• tests/unit_tests/test_opensandbox_provider.py: transport-backend coverage (httpx default with keepalive expiry, custom pool settings, fallback when httpx_aiohttp is unavailable, null disables injection), aiohttp opt-in (importorskip-guarded), and requests/limits plumbing tests. 16 passed with and without httpx-aiohttp.
• responses_api_agents/mini_swe_agent_2/tests/test_app.py: updated for the awaited ObjectRef (awaitable FakeObjectRef); assertions strengthened to check the Ray call's params.
• CI green except a pre-existing tau2 failure on main (reproduces on unrelated PRs).

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[hemildesai]

/claude review

[hemildesai]

/ok to test 3eb2171

[claude]

SHIP WITH CARE — high-concurrency perf/reliability tuning for the mini_swe_agent_2 harness and OpenSandbox provider. No silent-scoring or async-hang defects found; the core changes match the repo's own prescribed patterns. One operability tradeoff worth sizing before merge (inline).

Verified good:

• result = await future on the Ray ObjectRef (app.py:868) is exactly the pattern CLAUDE.md prescribes and removes the asyncio.to_thread(ray.get, ...) executor cap (~32 in-flight) that serialized finished rollouts. Correct.
• File dumps (_dump_config, _dump_result) moved off the event loop via asyncio.to_thread — right call for a handler serving all concurrent rollouts; shared-FS mkdir+write was blocking the loop.
• ClientTimeout(sock_connect=30) on the shared global aiohttp client bounds connect only, deliberately leaving no read/total timeout so long-silent /run rollouts aren't aborted and re-POSTed as duplicates. Reasoning is sound and documented.
• num_retries=5 via setdefault (config wins) is a reasonable transient-LLM-failure guard.
• resource_requests plumbing: validated as a mapping in from_mapping, converted through SandboxResources.from_mapping, and covered by a new test asserting both requests and limits reach the SDK, plus the TypeError and unknown-key paths. Good coverage.
• Transport injection into the OpenSandbox SDK provides the aiohttp-adapter escape hatch CLAUDE.md recommends (transport_backend: aiohttp), guarded by ImportError→httpx fallback. Defaulting to httpx here is defensible: sandbox create/exec/delete is low-volume per provider, not the 16k-request path the O(n²) warning targets. Tests cover default, custom-pool, and fallback paths.

One item to weigh (inline, NOTE): sandbox teardown is now a fire-and-forget daemon thread. The finally-block motivation is correct, but the "orphans covered by TTL" backstop weakens on Ray worker churn — a daemon thread dying mid-cleanup() leaks the sandbox until ttl_s (18000s / 5h in the exemplar). Under sustained high concurrency that's a large orphan pool if the provider bills per live sandbox or enforces a pod quota. Size the TTL against expected worker churn, or add a bounded reaper.

opensandbox>=0.1.15 bump is correctly declared in both pyproject.toml and uv.lock, justified by the new resource_requests create arg.

[claude]

NOTE — sandbox teardown moved to a fire-and-forget daemon thread. Motivation is sound (a slow/hanging stop() in finally previously delayed every finished result and re-raised over it). But the leak backstop is weaker than the comment implies: on Ray worker churn (autoscale-down, task eviction, worker crash), the daemon thread dies mid-cleanup() and the sandbox leaks until TTL — ttl_s: 18000 (5h) in the exemplar config. Under high rollout concurrency that's a large steady-state pool of orphaned sandboxes. If the provider bills per live sandbox or has a pod quota, this can starve new rollouts. Consider a bounded reaper or a shorter TTL for eval runs. Deliberate tradeoff — author's call, but size the TTL against expected worker churn.

commit_id: 3eb2171

[hemildesai]

Deliberate tradeoff, keeping it as-is for this PR — but the analysis is fair and worth recording.

Why the daemon thread: the in-band stop() in finally ran before the task result became fetchable, so a slow or hanging teardown gated every finished rollout, and a raising teardown destroyed an otherwise successful result. Under high rollout concurrency that was the dominant source of both stalled delivery and lost work, which is what this PR is about.

On the leak window, you are right that it is not zero and not identical to the old behavior. With in-band cleanup, teardown finished before the task returned, so worker teardown could not interrupt it. Fire-and-forget genuinely widens the window: on autoscale-down, eviction, or worker crash the thread can die mid-cleanup(). Accepted, because the backstop is not the thread — orphans are reclaimed by the provider TTL plus an out-of-band sweep on the operator side, and a crashed worker was never going to complete an in-band cleanup either.

On TTL sizing: agreed that 5h is provisioned for long rollouts rather than for churn, and that it is the right knob if orphan pressure shows up. Leaving it unchanged here so this PR stays scoped to the transport and delivery path; sizing it against observed churn is a config-level follow-up for the author to make with real numbers rather than a guess.

[hemildesai]

/ok to test 3e8e5e6

[hemildesai]

/claude review

[claude]

SHIP WITH CARE — solid throughput/reliability work, well-commented and test-backed. No blockers; two things worth an explicit eyes-open before merge.

Reviewed against CLAUDE.md conventions:

• await future (app.py:868) replacing await asyncio.to_thread(ray.get, future) matches the documented Ray-in-async pattern exactly, and lifts the ~32-thread executor ceiling that was serializing finished rollouts. Good fix.
• The httpx.AsyncHTTPTransport in provider.py:_build_transport is not a regression against the "no httpx" rule: the OpenSandbox SDK is already httpx-based, so this only bounds the pool (max_connections=100, keepalive_expiry=3.0) to fix stale-socket reuse against uvicorn's ~5s idle close. The aiohttp bridge is offered as opt-in. Correct call.
• ClientTimeout(sock_connect=30) (server_utils.py) deliberately keeps no read/total timeout on the shared session — right, since a long-silent /run must not be aborted and re-POSTed as a duplicate rollout.

RISK — fire-and-forget sandbox cleanup can leak up to the TTL (app.py:~832, _run_mini_swe_v2 finally → threading.Thread(..., daemon=True)):

• WHAT BREAKS: cleanup now runs in a daemon thread that outlives the Ray task return. If the Ray worker process is recycled/torn down (idle autoscale-down, worker reuse churn) before the thread finishes env.cleanup(), the sandbox delete never lands.
• BLAST RADIUS: orphaned sandboxes persist until ttl_s (18000s = 5h in the exemplar). Under high-churn training this can accumulate many live sandboxes → provider resource/cost pressure, not a correctness bug. The comment acknowledges TTL coverage — flagging so it's a conscious tradeoff, not a surprise on the first big run.
• FIX (optional): if leak accumulation bites, lower ttl_s or move cleanup to a bounded provider-side reaper rather than per-task daemon threads.

NOTE — undeclared direct httpx import (provider.py:_build_transport, import httpx): httpx is only a transitive dep (via opensandbox/openai), not a declared direct dependency in pyproject.toml. It's guaranteed present wherever this sandbox-extra code runs, so this won't break today, but per the dependency-hygiene convention a direct import ideally gets a direct declaration.

Config changes (skip_health_check: false, create timeout_s: 1500 > ready_timeout_s: 1200, resource_requests, num_retries: 5, num_cpus=0.25) are internally consistent and the new resource_requests path is covered by test_direct_create_passes_resource_requests_to_sdk_create. Test refactor to FakeObjectRef correctly models the awaitable ObjectRef.

[hemildesai]

/ok to test 3698bcf

[hemildesai]

/claude review

[claude]

SHIP — reliability-hardening across the OpenSandbox provider, mini_swe_agent_2 harness, and the global aiohttp session. No verifier/scoring logic touched; the risky bits are async lifecycle and pooling, and they check out.

Verified:

• Ray await pattern (app.py:835): await future replacing asyncio.to_thread(ray.get, ...) matches CLAUDE.md's endorsed pattern and removes the ~32-worker default-executor ceiling on concurrent rollouts. Tests updated correctly via FakeObjectRef.__await__, covering both success and error paths.
• Shared transport lifecycle (provider.py:617-648): lazy _get_transport() build is check-then-assign with no await in between, so no event-loop race; the provider owns the transport and aclose() is reached from sandbox/api.py:84,132, so no pool leak. Injection is skipped cleanly when keepalive_expiry_s is null.
• httpx import (provider.py:625): guarded behind the sandbox extra and keepalive_expiry_s is not None; httpx is a transitive dep of opensandbox, so no undeclared import. httpx_aiohttp is optional with an ImportError→warning fallback and skip-guarded tests.
• Global aiohttp timeout (server_utils.py:177): ClientTimeout(sock_connect=30) strictly adds a connect bound to what was previously unbounded (ClientTimeout()); the no-read/no-total choice is correct for long-silent /run requests and the rationale is documented.
• resource_requests, transport backends, and shared-transport close are all covered by new unit tests; the opensandbox>=0.1.15 bump is reflected in uv.lock.

Notes (author's call, non-blocking):

• Fire-and-forget cleanup thread (app.py:588-247): daemon-thread env.cleanup() is unbounded per rollout and dies abruptly on process exit. The comment correctly notes orphans are covered by the provider sandbox TTL, so this is an acceptable tradeoff — just flagging that teardown is now best-effort and failures only surface via a print, not logs/metrics.
• command_retries: 0 and the keepalive/timeout tuning are well-commented; no action needed.

[hemildesai]

/ok to test 717de2a

[hemildesai]

/ok to test f3cdeb4