Comp-Coding Verifier by kbhardwaj-nvidia · Pull Request #5 · NVIDIA-NeMo/Gym

kbhardwaj-nvidia · 2025-09-05T18:41:31Z

No description provided.

copy-pr-bot · 2025-09-05T18:41:36Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Brian Yu <bxyu@nvidia.com> Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>

Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>

Signed-off-by: Brian Yu <bxyu@nvidia.com> Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>

Signed-off-by: Brian Yu <bxyu@nvidia.com> Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com> Co-authored-by: bxyu-nvidia <bxyu@nvidia.com> Signed-off-by: soares-f <soarescmsa@gmail.com>

…validator Fixes/lang validator

…validator Fixes/lang validator Signed-off-by: Abubakari Al-Waasiu <abubakari.a@turing.com>

…validator Fixes/lang validator Signed-off-by: Al-Waasiu Abubakari <abubakari.a@turing.com>

Signed-off-by: Brian Yu <bxyu@nvidia.com> Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com> Co-authored-by: bxyu-nvidia <bxyu@nvidia.com>

…validator Fixes/lang validator Signed-off-by: Al-Waasiu Abubakari <abubakari.a@turing.com>

Signed-off-by: Brian Yu <bxyu@nvidia.com> Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com> Co-authored-by: bxyu-nvidia <bxyu@nvidia.com>

…validator Fixes/lang validator Signed-off-by: Al-Waasiu Abubakari <abubakari.a@turing.com>

…audit) - (#4) Cache the static mcp_config (read once, reuse) instead of reading it from disk every rollout. - (NVIDIA-NeMo#5) Document that the per-rollout Gym entry overwrites a same-named static mcp_config server. - (NVIDIA-NeMo#6) Warn when seed MCP metadata has no headers (tool calls would otherwise be unauthenticated). - (NVIDIA-NeMo#7) Add an agent test asserting the session cookie is threaded seed_session -> verify. - (#8a) Move the example task ground truth (expected_city) under verifier_metadata, per convention. - (#8b) Use the message-list input format in the example dataset, like the other examples. - (NVIDIA-NeMo#9) Remove the dead 'url' branch in the rollout mcp_config (MCPServerMetadata has no 'url'); audited that every metadata field accessed (server_name/url_path/transport/headers) actually exists. - (#2v) Add a test that a token-less MCP tool call surfaces as a clean isError (HTTP 200), not a 401. Also patch only ensure_claude_code (not all of model_post_init) in the claude_code_agent test helper, so the model's private attributes initialize. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>

…esource requests/limits (#2212) ## Summary Hardens the OpenSandbox provider and the `mini_swe_agent_2` eval path so that large sandboxed agent evals (SWE-bench Verified scale: hundreds of concurrent rollouts against a Kubernetes-backed OpenSandbox deployment) run end-to-end reliably: every finished rollout is delivered and recorded, and sandbox-infrastructure noise is bounded instead of silently zeroing rewards. Diagnosed on multi-node runs at concurrency 300–1500; the same failure signatures appear at concurrency 8, just less often. ### 1. `Server disconnected without sending a response` The SDK's default httpx pool keeps idle connections for 30s (`opensandbox.config.connection.with_transport_if_missing`), but the OpenSandbox server's uvicorn keep-alive reaper closes idle sockets after ~5s. Agent workloads idle between sandbox commands (model think time), so commands routinely reuse a socket the server already closed — surfacing as `httpx.RemoteProtocolError` → `SandboxInternalException`, which permanently kills the rollout. **Fix:** the provider injects a transport whose `keepalive_expiry` sits *below* the server's keep-alive timeout (default 3s, `connection.keepalive_expiry_s`): - `connection.transport_backend: httpx` (default) — stock `httpx.AsyncHTTPTransport(limits=..., retries=connect_retries)`. No new required dependencies. - `connection.transport_backend: aiohttp` (opt-in) — aiohttp pool under the SDK's httpx surface via [`httpx-aiohttp`](https://github.com/karpetrosyan/httpx-aiohttp); falls back to the httpx transport with a warning when absent. - `keepalive_expiry_s: null` disables injection entirely (SDK default transport). ### 2. 502 `could not connect to the backend sandbox endpoint='<podIP>:44772'` on first command With `create.skip_health_check: true`, `Sandbox.create` returns before the pod's exec daemon is listening; the first command races pod startup and the server proxy 502s. Under a create burst this killed 7–12 rollouts per run. **Fix:** `create.skip_health_check: false` (create waits for readiness, bounded by the spec's `ready_timeout_s`; `create.timeout_s` kept above the ready timeout). **`command_retries` stays at 0.** Retrying a command the server may have already started would execute it twice, and agent commands are frequently mutating. The keepalive bound removes the stale-connection failures retries were compensating for. Raise it only for idempotent workloads. ### 3. Separate resource requests and limits `Sandbox.create` accepts distinct `resource` (limits) and `resource_requests` maps; a lone `resource` map is applied by the server as requests=limits. This exposes the requests side via `sandbox_spec.provider_options.resource_requests` (same keys as `SandboxSpec.resources`). Motivation: SWE-bench test suites OOM-killed sandbox pods at 2Gi, but raising a single combined map to 8Gi would 4× the cluster reservation; with the split, limits rise while requests stay small and pods pack densely. Requires `opensandbox>=0.1.15` (lower bound raised here). ### 4. Sandbox teardown off the rollout's critical path `env.cleanup()` ran in the rollout Ray task's `finally` block, so a finished result only became fetchable after the sandbox DELETE returned — and a teardown failure re-raised *over* the successful eval, degrading it to a reward-0 error row. Teardown is also effectively one-shot (`Sandbox.stop` latches `_closed` before dispatch), so a single failed DELETE leaked the sandbox while still costing the rollout. **Fix:** cleanup runs on a best-effort daemon thread; results return immediately, teardown errors are logged rather than raised, and orphans remain covered by the provider's sandbox TTL. ### 5. High-concurrency rollout delivery (`mini_swe_agent_2`) Two coupled fixes so every finished rollout is delivered at high concurrency (note: at very high concurrency the single-process policy-model proxy remains a separate bottleneck — connection-refused storms confirmed by a controlled A/B — with its scaling fix tracked separately): - **await the Ray ObjectRef** instead of `asyncio.to_thread(ray.get, ...)`: the default executor caps at `min(32, cpu+4)` threads, each pinned for a full rollout, so delivery stalls at ~32 concurrent rollouts and finished tasks queue behind blocked `ray.get` calls. - **bounded litellm retries (`num_retries=5`, config-overridable)** — no retry means one transient LLM-call failure kills a whole rollout; unbounded retries make failures look like hangs — and **`num_cpus=0.25`** on the rollout Ray task so concurrency is not capped at cluster core count (rollouts are I/O-bound). ### 6. `datasets` declared in the base agent config The struct-mode config merge rejects keys absent from the base config, so a benchmark config using `_inherit_from` could not add its dataset list (`ConfigKeyError: Key 'datasets' is not in struct`). Declared empty in the base server config, matching how other agents (e.g. `swe_agents`) expose it. ## Validation - Requests/limits split (paired multi-node runs, identical except the variable under test): infra-failed rollouts 41/316 (**13.0%**) at 2Gi requests=limits → 19/288 (**6.6%**) at 8Gi limits / 2Gi requests; pass@1 58.0% → **67.1%**. - Keepalive bound + health-checked create: zero `Server disconnected` events and zero create-race 502s across all subsequent runs (previously 2–12 per run). - Full stack, end-to-end from this branch: single-pass SWE-bench Verified (500 instances, concurrency 500+) delivered **>98% of rollouts** with ~1.6% residual sandbox-infra failures (server-side proxy 502s on established sandboxes, tracked separately) and zero client-transport failures. Before the delivery fixes (#4/#5), an identically shaped run lost the majority of *completed* evaluations at the wall — hundreds of finished evals, only tens recorded. ## Relationship to #2020 Complementary, no overlap; #2020 (job attribution metadata) has since merged and this branch is updated on top of it. Field-validated together — the attribution labels are what make post-cancellation sandbox garbage collection safely scoped to a single job. ## Testing - `tests/unit_tests/test_opensandbox_provider.py`: transport-backend coverage (httpx default with keepalive expiry, custom pool settings, fallback when `httpx_aiohttp` is unavailable, `null` disables injection), aiohttp opt-in (importorskip-guarded), and requests/limits plumbing tests. 16 passed with and without `httpx-aiohttp`. - `responses_api_agents/mini_swe_agent_2/tests/test_app.py`: updated for the awaited ObjectRef (awaitable `FakeObjectRef`); assertions strengthened to check the Ray call's params. - CI green except a pre-existing `tau2` failure on `main` (reproduces on unrelated PRs). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Hemil Desai <hemild@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

…esource requests/limits (NVIDIA-NeMo#2212) ## Summary Hardens the OpenSandbox provider and the `mini_swe_agent_2` eval path so that large sandboxed agent evals (SWE-bench Verified scale: hundreds of concurrent rollouts against a Kubernetes-backed OpenSandbox deployment) run end-to-end reliably: every finished rollout is delivered and recorded, and sandbox-infrastructure noise is bounded instead of silently zeroing rewards. Diagnosed on multi-node runs at concurrency 300–1500; the same failure signatures appear at concurrency 8, just less often. ### 1. `Server disconnected without sending a response` The SDK's default httpx pool keeps idle connections for 30s (`opensandbox.config.connection.with_transport_if_missing`), but the OpenSandbox server's uvicorn keep-alive reaper closes idle sockets after ~5s. Agent workloads idle between sandbox commands (model think time), so commands routinely reuse a socket the server already closed — surfacing as `httpx.RemoteProtocolError` → `SandboxInternalException`, which permanently kills the rollout. **Fix:** the provider injects a transport whose `keepalive_expiry` sits *below* the server's keep-alive timeout (default 3s, `connection.keepalive_expiry_s`): - `connection.transport_backend: httpx` (default) — stock `httpx.AsyncHTTPTransport(limits=..., retries=connect_retries)`. No new required dependencies. - `connection.transport_backend: aiohttp` (opt-in) — aiohttp pool under the SDK's httpx surface via [`httpx-aiohttp`](https://github.com/karpetrosyan/httpx-aiohttp); falls back to the httpx transport with a warning when absent. - `keepalive_expiry_s: null` disables injection entirely (SDK default transport). ### 2. 502 `could not connect to the backend sandbox endpoint='<podIP>:44772'` on first command With `create.skip_health_check: true`, `Sandbox.create` returns before the pod's exec daemon is listening; the first command races pod startup and the server proxy 502s. Under a create burst this killed 7–12 rollouts per run. **Fix:** `create.skip_health_check: false` (create waits for readiness, bounded by the spec's `ready_timeout_s`; `create.timeout_s` kept above the ready timeout). **`command_retries` stays at 0.** Retrying a command the server may have already started would execute it twice, and agent commands are frequently mutating. The keepalive bound removes the stale-connection failures retries were compensating for. Raise it only for idempotent workloads. ### 3. Separate resource requests and limits `Sandbox.create` accepts distinct `resource` (limits) and `resource_requests` maps; a lone `resource` map is applied by the server as requests=limits. This exposes the requests side via `sandbox_spec.provider_options.resource_requests` (same keys as `SandboxSpec.resources`). Motivation: SWE-bench test suites OOM-killed sandbox pods at 2Gi, but raising a single combined map to 8Gi would 4× the cluster reservation; with the split, limits rise while requests stay small and pods pack densely. Requires `opensandbox>=0.1.15` (lower bound raised here). ### 4. Sandbox teardown off the rollout's critical path `env.cleanup()` ran in the rollout Ray task's `finally` block, so a finished result only became fetchable after the sandbox DELETE returned — and a teardown failure re-raised *over* the successful eval, degrading it to a reward-0 error row. Teardown is also effectively one-shot (`Sandbox.stop` latches `_closed` before dispatch), so a single failed DELETE leaked the sandbox while still costing the rollout. **Fix:** cleanup runs on a best-effort daemon thread; results return immediately, teardown errors are logged rather than raised, and orphans remain covered by the provider's sandbox TTL. ### 5. High-concurrency rollout delivery (`mini_swe_agent_2`) Two coupled fixes so every finished rollout is delivered at high concurrency (note: at very high concurrency the single-process policy-model proxy remains a separate bottleneck — connection-refused storms confirmed by a controlled A/B — with its scaling fix tracked separately): - **await the Ray ObjectRef** instead of `asyncio.to_thread(ray.get, ...)`: the default executor caps at `min(32, cpu+4)` threads, each pinned for a full rollout, so delivery stalls at ~32 concurrent rollouts and finished tasks queue behind blocked `ray.get` calls. - **bounded litellm retries (`num_retries=5`, config-overridable)** — no retry means one transient LLM-call failure kills a whole rollout; unbounded retries make failures look like hangs — and **`num_cpus=0.25`** on the rollout Ray task so concurrency is not capped at cluster core count (rollouts are I/O-bound). ### 6. `datasets` declared in the base agent config The struct-mode config merge rejects keys absent from the base config, so a benchmark config using `_inherit_from` could not add its dataset list (`ConfigKeyError: Key 'datasets' is not in struct`). Declared empty in the base server config, matching how other agents (e.g. `swe_agents`) expose it. ## Validation - Requests/limits split (paired multi-node runs, identical except the variable under test): infra-failed rollouts 41/316 (**13.0%**) at 2Gi requests=limits → 19/288 (**6.6%**) at 8Gi limits / 2Gi requests; pass@1 58.0% → **67.1%**. - Keepalive bound + health-checked create: zero `Server disconnected` events and zero create-race 502s across all subsequent runs (previously 2–12 per run). - Full stack, end-to-end from this branch: single-pass SWE-bench Verified (500 instances, concurrency 500+) delivered **>98% of rollouts** with ~1.6% residual sandbox-infra failures (server-side proxy 502s on established sandboxes, tracked separately) and zero client-transport failures. Before the delivery fixes (NVIDIA-NeMo#4/NVIDIA-NeMo#5), an identically shaped run lost the majority of *completed* evaluations at the wall — hundreds of finished evals, only tens recorded. ## Relationship to NVIDIA-NeMo#2020 Complementary, no overlap; NVIDIA-NeMo#2020 (job attribution metadata) has since merged and this branch is updated on top of it. Field-validated together — the attribution labels are what make post-cancellation sandbox garbage collection safely scoped to a single job. ## Testing - `tests/unit_tests/test_opensandbox_provider.py`: transport-backend coverage (httpx default with keepalive expiry, custom pool settings, fallback when `httpx_aiohttp` is unavailable, `null` disables injection), aiohttp opt-in (importorskip-guarded), and requests/limits plumbing tests. 16 passed with and without `httpx-aiohttp`. - `responses_api_agents/mini_swe_agent_2/tests/test_app.py`: updated for the awaited ObjectRef (awaitable `FakeObjectRef`); assertions strengthened to check the Ray call's params. - CI green except a pre-existing `tau2` failure on `main` (reproduces on unrelated PRs). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Hemil Desai <hemild@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

kbhardwaj-nvidia requested a review from a team as a code owner September 5, 2025 19:03

bxyu-nvidia and others added 10 commits September 5, 2025 16:08

Update GitHub with Gitlab main (#3)

3524001

Signed-off-by: Brian Yu <bxyu@nvidia.com> Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>

migrate comp coding mr

73939fc

Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>

ruff

d264a65

Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>

added copyright info

0aef34f

Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>

added copyright info

10336e8

Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>

ruff revert

5704d3e

Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>

Alias as Penguin (#4)

13a3ca4

Signed-off-by: Brian Yu <bxyu@nvidia.com> Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>

Add Copyright docs README FAQ (#7)

d731837

Signed-off-by: Brian Yu <bxyu@nvidia.com> Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>

Dapo17k (#6)

e929d86

Signed-off-by: Brian Yu <bxyu@nvidia.com> Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>

Fix docs build failures (#8)

cb9d38a

Signed-off-by: Brian Yu <bxyu@nvidia.com> Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>

kbhardwaj-nvidia force-pushed the khushi/comp-coding-github branch from 514c5f1 to cb9d38a Compare September 5, 2025 23:08

eh

a446cd9

bxyu-nvidia merged commit 0cb45db into main Sep 5, 2025
5 checks passed

bxyu-nvidia deleted the khushi/comp-coding-github branch September 5, 2025 23:37

soares-f pushed a commit that referenced this pull request Sep 8, 2025

Comp-Coding Verifier (#5)

780b539

Signed-off-by: Brian Yu <bxyu@nvidia.com> Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com> Co-authored-by: bxyu-nvidia <bxyu@nvidia.com> Signed-off-by: soares-f <soarescmsa@gmail.com>

abubakaria56 added a commit to abubakaria56/Gym that referenced this pull request Feb 28, 2026

Merge pull request NVIDIA-NeMo#5 from dhrutisundar-turing/fixes/lang_…

f6699f7

…validator Fixes/lang validator

abubakaria56 added a commit to abubakaria56/Gym that referenced this pull request Mar 2, 2026

Merge pull request NVIDIA-NeMo#5 from dhrutisundar-turing/fixes/lang_…

bfc21de

…validator Fixes/lang validator Signed-off-by: Abubakari Al-Waasiu <abubakari.a@turing.com>

abubakaria56 added a commit to abubakaria56/Gym that referenced this pull request Mar 2, 2026

Merge pull request NVIDIA-NeMo#5 from dhrutisundar-turing/fixes/lang_…

b5ff81a

…validator Fixes/lang validator Signed-off-by: Al-Waasiu Abubakari <abubakari.a@turing.com>

abubakaria56 pushed a commit to abubakaria56/Gym that referenced this pull request Mar 2, 2026

Comp-Coding Verifier (NVIDIA-NeMo#5)

48212f8

Signed-off-by: Brian Yu <bxyu@nvidia.com> Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com> Co-authored-by: bxyu-nvidia <bxyu@nvidia.com>

abubakaria56 added a commit to abubakaria56/Gym that referenced this pull request Mar 2, 2026

Merge pull request NVIDIA-NeMo#5 from dhrutisundar-turing/fixes/lang_…

eadb625

…validator Fixes/lang validator Signed-off-by: Al-Waasiu Abubakari <abubakari.a@turing.com>

abubakaria56 pushed a commit to abubakaria56/Gym that referenced this pull request Mar 2, 2026

Comp-Coding Verifier (NVIDIA-NeMo#5)

2a1c419

Signed-off-by: Brian Yu <bxyu@nvidia.com> Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com> Co-authored-by: bxyu-nvidia <bxyu@nvidia.com>

abubakaria56 added a commit to abubakaria56/Gym that referenced this pull request Mar 2, 2026

Merge pull request NVIDIA-NeMo#5 from dhrutisundar-turing/fixes/lang_…

4304b28

…validator Fixes/lang validator Signed-off-by: Al-Waasiu Abubakari <abubakari.a@turing.com>

This was referenced Jun 25, 2026

[Epic] Continuation of configuration works #1726

Closed

epic: reduce configuration friction for environment authors and users #1205

Closed

wprazuch mentioned this pull request Jun 29, 2026

docs: use the unified gym CLI and name/flavor selectors across docs #1808

Closed

hemildesai mentioned this pull request Jul 31, 2026

feat(opensandbox): keepalive-bounded transport + create hardening + resource requests/limits #2212

Merged

Navigation Menu

Uh oh!

Comp-Coding Verifier - #5

Comp-Coding Verifier#5
bxyu-nvidia merged 11 commits into
mainfrom
khushi/comp-coding-github

kbhardwaj-nvidia commented Sep 5, 2025

Uh oh!

copy-pr-bot Bot commented Sep 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

kbhardwaj-nvidia commented Sep 5, 2025

Uh oh!

copy-pr-bot Bot commented Sep 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants