feat(mini_swe_agent_2): opt-in reuse of the shared verifier (#1249) [3/3] - #4
Closed
adil-a wants to merge 1 commit into
Closed
feat(mini_swe_agent_2): opt-in reuse of the shared verifier (#1249) [3/3]#4adil-a wants to merge 1 commit into
adil-a wants to merge 1 commit into
Conversation
…VIDIA-NeMo#1249) Adds an opt-in path to mini-swe-agent-2 that scores its patch through the shared swe_env verifier over HTTP (the same contract the OpenHands agent uses), proving the verifier is agent-agnostic. Off by default; in-process grading remains the default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: adil-a <adil.asif2000@hotmail.com>
Owner
Author
|
Superseded by the stacked PRs opened directly on NVIDIA-NeMo/Gym (NVIDIA-NeMo#1677, NVIDIA-NeMo#1678, NVIDIA-NeMo#1679). |
adil-a
pushed a commit
that referenced
this pull request
Jun 25, 2026
…audit) - (#4) Cache the static mcp_config (read once, reuse) instead of reading it from disk every rollout. - (NVIDIA-NeMo#5) Document that the per-rollout Gym entry overwrites a same-named static mcp_config server. - (NVIDIA-NeMo#6) Warn when seed MCP metadata has no headers (tool calls would otherwise be unauthenticated). - (NVIDIA-NeMo#7) Add an agent test asserting the session cookie is threaded seed_session -> verify. - (#8a) Move the example task ground truth (expected_city) under verifier_metadata, per convention. - (#8b) Use the message-list input format in the example dataset, like the other examples. - (NVIDIA-NeMo#9) Remove the dead 'url' branch in the rollout mcp_config (MCPServerMetadata has no 'url'); audited that every metadata field accessed (server_name/url_path/transport/headers) actually exists. - (#2v) Add a test that a token-less MCP tool call surfaces as a clean isError (HTTP 200), not a 401. Also patch only ensure_claude_code (not all of model_post_init) in the claude_code_agent test helper, so the model's private attributes initialize. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>
adil-a
pushed a commit
that referenced
this pull request
Jun 28, 2026
Reward-profiling evidence (review #4): the docker gold-patch census resolves 486/500 (after the PYTEST_ADDOPTS=-rA flat-grading fix; 445 before), in line with the apptainer/.sif reference 492/500; the empty patch resolves 0/500. The remaining ~14 are instance-specific (documented astropy/django upstream flaky gold-failures + a few where a single required test does not run in the container), not a systematic grader defect. Includes a docker/apptainer repro snippet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Opt-in verifier reuse for mini-swe-agent-2 (NVIDIA-NeMo#1249)
Stacked PR 3 of 3 · base:
stack/2-swe-agents-openhandsBuilds on PR 1 (library + verifier) and PR 2 (OpenHands cutover). This PR proves the verifier is genuinely agent-agnostic by wiring a second, independent agent to it.
What changes
mini_swe_agent_2gains an opt-in path that scores its patch through the sharedswe_envverifier over HTTP — the exact same contract the OpenHands agent uses in PR 2. It emits the normalized unified-diff patch field and POSTs a standard verify request.Tests
ruffclean. The full stack (PR1+PR2+PR3) reconstructs the verified tree exactly and all suites pass together (swe_env 168, verifier 14, swe_agents 82, mini 20).Review notes
🤖 Generated with Claude Code