Skip to content

Add README for vibench_agent - #2906

Merged
cmunley1 merged 1 commit into
NVIDIA-NeMo:mainfrom
deepbuilder:vibench-agent-readme
Sep 1, 2026
Merged

Add README for vibench_agent#2906
cmunley1 merged 1 commit into
NVIDIA-NeMo:mainfrom
deepbuilder:vibench-agent-readme

Conversation

@deepbuilder

Copy link
Copy Markdown
Contributor

vibench_agent shipped without a README.md in #2674. ng_test_all filters testable components to directories containing one, then asserts the filtered count matches the discovered count — so discovery found 174 components and tested 173, failing that assertion in all eight server-suite shards even though every selected module test passed:

AssertionError: Mismatch on the number of total modules found (174) and the number of actual modules tested (173)!
Extra candidate paths:
- responses_api_agents/vibench_agent

nemo_gym/cli/env.py:1114, on main at a2c5222.

This adds the missing README. Docs-only — no code or config changes.

Contents

  • The build → harvest → verify flow, and that everything about driving OpenCode is inherited from opencode_sandboxed_agent (only sandbox acquisition and harvesting are overridden).
  • Why the agent owns its sandbox instead of sharing one: reaching into the agent's box needs serialize()/connect(), which only OpenSandbox implements, so the swebench shape cannot run on Docker, Apptainer or enroot. See Sandbox server to share sandbox states needed for rollouts #2082.
  • Why configs/docker.yaml must be used instead of the stock provider config — the stock 180s exec timeout kills long installs, and the harness is told the model is at http://127.0.0.1:<port>, which inside a bridged container is the container itself, so it makes zero LLM calls and exports an empty app.
  • An explicit warning not to "fix" that with network: host, and the 0.0.0.0 exposure caveat for the policy-model bind.

Setup, task-row generation, reward definition and grading are not duplicated here; they stay in the resources server README.

Note on the other CI failure

The Test Wheel Use failure in that run is independent of this change: the wheel test enables prereleases/unsafe-best-match, unbounded mlflow>=3.15.1 resolved to 3.15.2, which selected sqlalchemy 2.1.0rc1, whose source metadata uv rejects (duplicate normalized extra name 'mssql-pymssql'), aborting arc_agi_resources_server startup. That needs a SQLAlchemy pin below 2.1 or prerelease containment in transitive resolution — happy to open a separate issue or PR for it if useful.

🤖 Generated with Claude Code

`ng_test_all` filters testable components to directories containing a
README.md, then asserts the filtered count matches the discovered count.
vibench_agent shipped without one in NVIDIA-NeMo#2674, so discovery found 174
components and tested 173, failing that assertion in all eight server-suite
shards even though the selected module tests passed:

    AssertionError: Mismatch on the number of total modules found (174)
    and the number of actual modules tested (173)!
    Extra candidate paths:
    - responses_api_agents/vibench_agent

Documents the build/harvest flow, why the agent owns its sandbox rather
than sharing one, the bridged-network constraint that makes the stock
docker config export an empty app, and the 0.0.0.0 exposure caveat.
Setup and grading stay in the resources server README.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: raghavendran ramakrishnan <raramakrishn@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 1, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@nemo-automation-bot nemo-automation-bot Bot added the community-request Issue reported or requested by someone from the community label Sep 1, 2026
@cmunley1

cmunley1 commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

/ok to test 392047f

@cmunley1
cmunley1 merged commit 02d978f into NVIDIA-NeMo:main Sep 1, 2026
33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-request Issue reported or requested by someone from the community

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants