chore: consolidate v1 root modules into utils - #2204
Merged
Conversation
Move scoring, push, artifacts, decorators, loaders, and retries from the v1 package root into verifiers/v1/utils, and inline the sample() helper at its two call sites (SEED now lives next to Taskset.shuffle). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mikasenghaas
requested review from
hallerite and
xeophon
and removed request for
hallerite
July 31, 2026 17:26
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
Contributor
ApprovabilityVerdict: Approved 98c0984 Mechanical file reorganization moving v1 root modules into utils subdirectory. All import paths updated consistently, public API preserved via re-exports, and the inlined sampling logic is behaviorally identical to the removed helper function. You can customize Macroscope's approvability policy. Learn more. |
…utils # Conflicts: # verifiers/v1/agent.py
xeophon
reviewed
Jul 31, 2026
| if TYPE_CHECKING: | ||
| from verifiers.v1.mcp import Toolset | ||
|
|
||
| SEED = 0 # fixed so `--shuffle` samples the same items every run (reproducible) |
This was referenced Aug 5, 2026
xeophon
added a commit
that referenced
this pull request
Aug 31, 2026
## Overview Consolidates v1 MCP client utilities and standalone harness code behind shared owning modules. ## Details - Exposes mcp_client, with_retry, connect_mcp, mcp_content_to_chat_content, and call_mcp from verifiers.v1.mcp. - Preserves the existing per-call client, timeout, teardown, naming, image conversion, and at-least-once retry behavior. - Embeds the public client source after PEP 723 metadata so standalone sandbox programs retain their declared dependency isolation. - Shares one minimal program between Null and Bash and centralizes the common host-side launch path. - Reuses the public client helper from NeMo Gym and removes the duplicate MCP implementations from Null, Bash, Browser Use, and NeMo Gym. ## Related work - #2474 moved first-party clients to MCP 2.0 and the 2026-07-28 protocol; this PR consolidates those implementations without changing their protocol behavior. - #2204 established the v1 package-organization pattern by consolidating root helpers into an owning utils package; this PR applies the same approach to MCP client utilities. - #2134 explored both a shared Null/Bash program and new lifecycle/replay rules; this PR carries forward the shared-program consolidation only and leaves lifecycle and retry semantics unchanged. Closes #2202 <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Touches eval harness launch paths and MCP tool wiring across Null, Bash, Browser Use, and NeMo Gym; behavioral differences between Null and Bash modes in the unified program warrant careful regression testing. > > **Overview** > Moves duplicated MCP HTTP client, retry, tool discovery, and call helpers into **`verifiers.v1.mcp.client`**, re-exported from **`verifiers.v1.mcp`**, and wires NeMo Gym’s upstream MCP calls through that shared **`mcp_client`** instead of a local session helper. > > Adds **`standalone.py`** with **`launch_chat_program`** (shared CLI/MCP/initial-messages launch path) and **`inline_mcp_client`** (splices the client module source into PEP 723 sandbox scripts). **Null** and **Bash** now share **`minimal/program.py`** (Bash passes **`--bash`** plus edit/search/interception flags); **`null/program.py`** is removed. **Browser Use** drops its inlined MCP copy and uses the same embed + **`launch_chat_program`** pattern. > > The shared minimal program keeps **Null**-specific behavior when **`--bash`** is off (60s MCP enumeration cap, graceful exit on context-overflow API errors) and **Bash**-specific behavior when on (longer HTTP timeouts, unbounded MCP connect wait, local tools gated by flags). > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 1944f34. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> <!-- Macroscope's pull request summary starts here --> <!-- Macroscope will only edit the content between these invisible markers, and the markers themselves will not be visible in the GitHub rendered markdown. --> <!-- If you delete either of the start / end markers from your PR's description, Macroscope will append its summary at the bottom of the description. --> > [!NOTE] > ### Consolidate v1 MCP client utilities into shared `client.py` and `launch_chat_program` > - Adds a reusable MCP client in [client.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2467/files#diff-00e3f7299d973207d20664447f3f10392038cc94faba8282d8fd7145f53606fe) with `mcp_client`, `with_retry`, `connect_mcp`, `mcp_content_to_chat_content`, and `call_mcp` helpers, replacing per-harness duplicates > - Adds `standalone.launch_chat_program` and `inline_mcp_client` in [standalone.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2467/files#diff-2c4721a1f03415dbed2e15a162cbc76345a56a9c3cdcbc6092e1f6968b20102b) to centralize wire-argument passing and embed the MCP client into PEP 723 scripts at build time > - Refactors the bash, browser_use, and null harnesses to call `launch_chat_program` with the shared `minimal.PROGRAM_SOURCE`; deletes the standalone null program > - Unifies `minimal/program.py` to serve both Null and Bash harnesses via a `--bash` flag that enables bash tooling, unbounded MCP enumeration, and longer model timeout; without it, bash tooling is disabled and context-overflow errors terminate cleanly > - Refactors `NeMoGymToolset.list_tools`/`call_tool` in [toolset.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2467/files#diff-fdaa3c526186955703d1d22244a24748ea125e780d399bd6be9b9c863fda6511) to use the shared `mcp_client` helper with explicit timeouts > - Risk: `minimal.program` now relies on inlined shared functions (`mcp_client`, `with_retry`, `connect_mcp`, `call_mcp`, `mcp_content_to_chat_content`) being present at runtime via `inline_mcp_client`; if `PROGRAM_SOURCE` is constructed without calling `inline_mcp_client`, the script will fail with `NameError` > > <!-- Macroscope's review summary starts here --> > > <sup><a href="https://app.macroscope.com">Macroscope</a> summarized 1944f34.</sup> > <!-- Macroscope's review summary ends here --> > <!-- Macroscope's pull request summary ends here -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
verifiers/v1package root intoverifiers/v1/utils/:scoring.py→utils/score.py,push.py→utils/platform.py,artifacts.py→utils/artifacts.py,decorators.py→utils/decorators.py,loaders.py→utils/loaders.py,retries.py→utils/retries.py.sample()helper (and with itutils/sampling.py), inlining the fixed-seed shuffle + head-slice at its two call sites (cli/eval/runner.py,legacy.py). The sharedSEEDconstant now lives intaskset.py, next toTaskset.shufflewhich already used it._as_messagesat its single call site (Interaction._turninagent.py) and remove it fromrollout.py— it normalized wire-dict user turns to typed messages and had no other consumer._serve_interceptionout ofrollout.pyinto the interception package as publicserve_interception(alongsiderequires_tunnel/make_interception, symmetric with mcp'sserve_tools).RolloutRun->Rollout.vf.*re-exports inverifiers/v1/__init__.pyare unchanged.Breaking
Deep-path imports of the moved modules break;
verifiers.v1re-exports are unaffected.verifiers.v1.scoring→verifiers.v1.utils.scoreverifiers.v1.push→verifiers.v1.utils.platformverifiers.v1.artifacts→verifiers.v1.utils.artifactsverifiers.v1.decorators→verifiers.v1.utils.decoratorsverifiers.v1.loaders→verifiers.v1.utils.loadersverifiers.v1.retries→verifiers.v1.utils.retriesverifiers.v1.rollout.RolloutRun->verifiers.v1.rollout.Rollout(not re-exported fromverifiers.v1; its only in-tree driver isAgent).verifiers.v1.utils.samplingis removed: importSEEDfromverifiers.v1.taskset;sample()has no replacement (inline the shuffle + slice).Verification
uv run ruff check,uv run pre-commit run --all-files, anduv run pytest tests/v1(66 passed, 66 skipped — skips needPRIME_API_KEY) all pass.🤖 Generated with Claude Code
Note
Medium Risk
Breaking for deep imports of moved modules and removed
utils.sampling; publicverifiers.v1API unchanged. Shuffle behavior should match priorsample()if semantics are preserved.Overview
Moves six former
verifiers/v1top-level modules underverifiers/v1/utils/(loaders,decorators,retries,artifacts,scoring→score,push→platform) and updates internal imports across CLI, env, rollout, tasksets, and tests.verifiers/v1/__init__.pystill re-exports the samevf.*surface from the new paths.utils/sampling.pyis removed:SEEDlives ontaskset.py(alongsideTaskset.shuffle), and legacy/server eval paths inlinerandom.Random(SEED).shuffle+ slice instead ofsample().state.pyonly gets a shorter module docstring.Reviewed by Cursor Bugbot for commit 98c0984. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Move v1 root utility modules into
verifiers/v1/utils/subpackageRelocates
artifacts,decorators,loaders,push,retries,scoringfromverifiers/v1/toverifiers/v1/utils/, withpushrenamed toplatformandscoringrenamed toscore. All internal imports across the codebase are updated to the new paths. Public re-exports inverifiers/v1/__init__.pyare updated to source from the new locations, keeping the external API unchanged.verifiers/v1/utils/sampling.pyis deleted;SEEDis moved totaskset.pyand callers ofsample()now inline equivalent deterministic shuffle-and-slice logic usingrandom.Random(SEED).verifiers.v1.utils.samplingdirectly will now get anImportError.Macroscope summarized 98c0984.