Adopt MCP 2026-07-28 - #2474
Merged
Merged
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7fe66ccee9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
ApprovabilityVerdict: Not approved Macroscope's review found this PR not approvable — This is a cross-cutting MCP 2.0 migration that changes shared server transport semantics, multiple harness client paths, dependency resolution, and NeMo Gym subprocess isolation. An unresolved concrete finding also reports that NeMo Gym MCP calls may fail because a raw timeout is assigned where the new HTTP stack expects a timeout object. You can add or adjust custom eligibility rules. Learn more. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit e253577. Configure here.
xeophon
enabled auto-merge (squash)
August 31, 2026 11:58
hallerite
reviewed
Aug 31, 2026
| @@ -122,7 +122,6 @@ openenv = [ | |||
| ] | |||
| nemo-gym = [ | |||
xeophon
added a commit
that referenced
this pull request
Aug 31, 2026
## Overview Consolidates v1 MCP client utilities and standalone harness code behind shared owning modules. ## Details - Exposes mcp_client, with_retry, connect_mcp, mcp_content_to_chat_content, and call_mcp from verifiers.v1.mcp. - Preserves the existing per-call client, timeout, teardown, naming, image conversion, and at-least-once retry behavior. - Embeds the public client source after PEP 723 metadata so standalone sandbox programs retain their declared dependency isolation. - Shares one minimal program between Null and Bash and centralizes the common host-side launch path. - Reuses the public client helper from NeMo Gym and removes the duplicate MCP implementations from Null, Bash, Browser Use, and NeMo Gym. ## Related work - #2474 moved first-party clients to MCP 2.0 and the 2026-07-28 protocol; this PR consolidates those implementations without changing their protocol behavior. - #2204 established the v1 package-organization pattern by consolidating root helpers into an owning utils package; this PR applies the same approach to MCP client utilities. - #2134 explored both a shared Null/Bash program and new lifecycle/replay rules; this PR carries forward the shared-program consolidation only and leaves lifecycle and retry semantics unchanged. Closes #2202 <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Touches eval harness launch paths and MCP tool wiring across Null, Bash, Browser Use, and NeMo Gym; behavioral differences between Null and Bash modes in the unified program warrant careful regression testing. > > **Overview** > Moves duplicated MCP HTTP client, retry, tool discovery, and call helpers into **`verifiers.v1.mcp.client`**, re-exported from **`verifiers.v1.mcp`**, and wires NeMo Gym’s upstream MCP calls through that shared **`mcp_client`** instead of a local session helper. > > Adds **`standalone.py`** with **`launch_chat_program`** (shared CLI/MCP/initial-messages launch path) and **`inline_mcp_client`** (splices the client module source into PEP 723 sandbox scripts). **Null** and **Bash** now share **`minimal/program.py`** (Bash passes **`--bash`** plus edit/search/interception flags); **`null/program.py`** is removed. **Browser Use** drops its inlined MCP copy and uses the same embed + **`launch_chat_program`** pattern. > > The shared minimal program keeps **Null**-specific behavior when **`--bash`** is off (60s MCP enumeration cap, graceful exit on context-overflow API errors) and **Bash**-specific behavior when on (longer HTTP timeouts, unbounded MCP connect wait, local tools gated by flags). > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 1944f34. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> <!-- Macroscope's pull request summary starts here --> <!-- Macroscope will only edit the content between these invisible markers, and the markers themselves will not be visible in the GitHub rendered markdown. --> <!-- If you delete either of the start / end markers from your PR's description, Macroscope will append its summary at the bottom of the description. --> > [!NOTE] > ### Consolidate v1 MCP client utilities into shared `client.py` and `launch_chat_program` > - Adds a reusable MCP client in [client.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2467/files#diff-00e3f7299d973207d20664447f3f10392038cc94faba8282d8fd7145f53606fe) with `mcp_client`, `with_retry`, `connect_mcp`, `mcp_content_to_chat_content`, and `call_mcp` helpers, replacing per-harness duplicates > - Adds `standalone.launch_chat_program` and `inline_mcp_client` in [standalone.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2467/files#diff-2c4721a1f03415dbed2e15a162cbc76345a56a9c3cdcbc6092e1f6968b20102b) to centralize wire-argument passing and embed the MCP client into PEP 723 scripts at build time > - Refactors the bash, browser_use, and null harnesses to call `launch_chat_program` with the shared `minimal.PROGRAM_SOURCE`; deletes the standalone null program > - Unifies `minimal/program.py` to serve both Null and Bash harnesses via a `--bash` flag that enables bash tooling, unbounded MCP enumeration, and longer model timeout; without it, bash tooling is disabled and context-overflow errors terminate cleanly > - Refactors `NeMoGymToolset.list_tools`/`call_tool` in [toolset.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2467/files#diff-fdaa3c526186955703d1d22244a24748ea125e780d399bd6be9b9c863fda6511) to use the shared `mcp_client` helper with explicit timeouts > - Risk: `minimal.program` now relies on inlined shared functions (`mcp_client`, `with_retry`, `connect_mcp`, `call_mcp`, `mcp_content_to_chat_content`) being present at runtime via `inline_mcp_client`; if `PROGRAM_SOURCE` is constructed without calling `inline_mcp_client`, the script will fail with `NameError` > > <!-- Macroscope's review summary starts here --> > > <sup><a href="https://app.macroscope.com">Macroscope</a> summarized 1944f34.</sup> > <!-- Macroscope's review summary ends here --> > <!-- Macroscope's pull request summary ends here -->
xeophon
pushed a commit
to PrimeIntellect-ai/prime-envs
that referenced
this pull request
Aug 31, 2026
## Summary Makes prime-envs compatible with verifiers main, which pins `mcp==2.0.0` (the MCP 2026-07-28 adoption, PrimeIntellect-ai/verifiers#2474) and removes `mcp.server.fastmcp`. Unblocks the verifiers submodule bump in prime-rl (PrimeIntellect-ai/prime-rl#3441). - **automationbench_env**: removed entirely rather than migrated (team decision — superseded by the Verified fork; its `mcp<2` pin was the hard workspace-lock conflict). Also drops its row from the tool_use README index. Nothing else in prime-envs or prime-rl references it. - **browsecomp_plus**: upstream #777 already dropped its `mcp>=1,<2` pin; on the merged tree this PR removes the stale FastMCP comment it left behind (changelog entry added). It imports nothing from mcp itself. - **enterprise_ops_gym / mcp_atlas**: both tool servers advertised dynamic, service-defined tool schemas by installing fastmcp `Tool` objects with a passthrough `FuncMetadata` into the private tool manager — runtime-broken under mcp 2. Migrated to the pattern verifiers' nemo_gym toolset established in #2474: `register` overrides `MCPServer.list_tools` / `call_tool` (the public methods its protocol handlers dispatch through at request time), serving `mcp.types.Tool` entries with the live schemas verbatim and wrapping forwarded results in `CallToolResult`. Argument validation stays with the backing services, as before. Both now declare `mcp>=2,<3` directly and are version-bumped (enterprise-ops-gym 0.2.0, mcp-atlas 0.4.0) for the breaking change. ## E2E results (hosted model via Prime Inference) Same tasks, same model (`openai/gpt-5.6-luna` via api.pinference.ai), same flags (`-n 2 -r 1`, max-turns 8, max-tokens 1024), prime VM runtimes. **main stack** = env from prime-envs main + released verifiers 0.3.1 (mcp 1.x, FastMCP path); **PR stack** = env from this branch + verifiers main (mcp 2, MCPServer path). | Run | main stack | PR stack | |---|---|---| | enterprise_ops_gym task 0 (calendar kickoff) | reward **1.0**, verifier pass rate 1.0 | reward **1.0**, verifier pass rate 1.0 | | enterprise_ops_gym task 1 (Helios roadmap) | reward **0.0**, pass rate 0.4 | reward **0.0**, pass rate 0.4 | The model received the dynamic tool catalog over the wire, called the gym MCP tools, mutated the seeded databases through them, and the SQL end-state verifiers scored **identically on both stacks** — including matching per-verifier pass rates on the failed task (a model/task failure, identical on both sides). This exercises the migrated `register`/`list_tools`/`call_tool` path end-to-end against live services. mcp_atlas (with the documented source installs for env + verifiers) completed on both stacks — healthy multi-turn tool loops (5–8 tool-call rounds per rollout, no tool errors), rollouts `ok`, judge-scored: | Run | main stack | PR stack | |---|---|---| | mcp_atlas task 0 (local-files QA) | coverage **1.0** (6 calls) | coverage **0.0** (8 calls) | | mcp_atlas task 1 (Santiago communes QA) | coverage **0.3** (7 calls) | coverage **0.5** (5 calls) | Unlike EOG's deterministic SQL verifiers, atlas scores are single-sample LLM-judge coverage under temperature sampling — the per-task deltas go both directions and there is no mechanical divergence in the traces (both stacks listed and called the dynamic Atlas tools over MCP without errors). Earlier atlas attempts failed **identically on both stacks** at infra layers before any MCP code (GHCR image rejection — since fixed upstream and merged in; then the source-install requirements). ## Verification - Functional smoke test on mcp 2.0.0: both migrated toolsets driven through a real `MCPServer`'s public `list_tools`/`call_tool` — dynamic schemas served verbatim (`input_schema` intact), forwarded results wrapped in `CallToolResult`, unknown tools raise. - mcp 2.0.0 source check (re Bugbot): the SDK's protocol handlers dispatch late through `self.list_tools()` / `self.call_tool(...)` — there is no `on_list_tools`/`on_call_tool` init binding — so the instance-attribute overrides are the supported interception point. - ruff check clean; root `uv lock --check` passes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Removes an entire taskset and changes how two stateful MCP proxies advertise and dispatch tools; behavior is intended to be unchanged, but this touches multi-turn grading paths. > > **Overview** > Updates prime-envs for **verifiers main**, which pins **MCP 2** and drops `mcp.server.fastmcp`. > > **`automationbench_env` is removed** (package, docs, and tool-use index row) because it could not be reconciled with the MCP 2 lock and is superseded elsewhere. > > **`enterprise_ops_gym`** and **`mcp_atlas`** no longer register dynamic tools via FastMCP internals. Their tool servers override **`MCPServer.list_tools` / `call_tool`** to expose live service schemas as `mcp.types.Tool` and return **`CallToolResult`**, matching the verifiers pattern for dynamic catalogs. Both declare **`mcp>=2,<3`** and are version-bumped (**0.2.0** / **0.4.0**). > > **`browsecomp_plus`** only drops a stale FastMCP-era dependency comment and notes MCP 2 in the changelog; it does not import MCP directly. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 4cb5269. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> <!-- Macroscope's pull request summary starts here --> <!-- Macroscope will only edit the content between these invisible markers, and the markers themselves will not be visible in the GitHub rendered markdown. --> <!-- If you delete either of the start / end markers from your PR's description, Macroscope will append its summary at the bottom of the description. --> > [!NOTE] > ### Migrate `enterprise_ops_gym` and `mcp_atlas` toolsets to MCP 2 and remove `automationbench_env` > - Replaces FastMCP internals (`FastMCP`, `Tool`, `ArgModelBase`, `_PassthroughMetadata`) in `enterprise_ops_gym` and `mcp_atlas` with MCP 2 `MCPServer` registration that overrides `list_tools` and `call_tool` directly on the server instance. > - `register` now assigns instance coroutines for dynamic tool listing and dispatch; `call_tool` returns `CallToolResult` with `TextContent` and raises `ValueError` for unknown tool names. > - Deletes the `automationbench_env` environment package and removes its taskset row from the `tool_use` README. > - Both updated packages bump versions and add a direct `mcp>=2,<3` dependency constraint in their `pyproject.toml`. > - Risk: installing these packages now requires MCP 2.x; consumers still on MCP 1.x / FastMCP will fail to resolve dependencies. The removed `_PassthroughMetadata` and `_EMPTY_ARG_MODEL` shims in [toolset.py](https://github.com/PrimeIntellect-ai/prime-envs/pull/789/files#diff-801ef4ed1395fe376f6de7d8884a88b72fdbbbddb93d8dff3e9e8c2588d2c5a9) and [toolset.py](https://github.com/PrimeIntellect-ai/prime-envs/pull/789/files#diff-e51faf9f76fd939f40f8e0f3256c2e72307821c3a3f306a03fb06e0bee62f8b4) mean any external code relying on those private symbols breaks. > > <!-- Macroscope's review summary starts here --> > > <sup><a href="https://app.macroscope.com">Macroscope</a> summarized 4cb5269.</sup> > <!-- Macroscope's review summary ends here --> > <!-- Macroscope's pull request summary ends here --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Move Verifiers tool servers and first-party MCP clients to the MCP 2026-07-28 protocol and its stateless HTTP model.
Details
Note
Medium Risk
Major MCP SDK and HTTP stack changes affect every tool call and MCP server path; NeMo Gym’s split subprocess adds a second MCP version boundary that could break if script isolation or negotiation regresses.
Overview
Upgrades the stack to MCP 2.0 (
mcp==2.0.0) and the 2026-07-28 stateless Streamable HTTP model across V1 tool servers, harness MCP clients, and related integrations.Clients replace
ClientSession+ manualinitialize()withmcp.Clienton streamable HTTP transports. Harness programs (null, bash, browser_use, compact) renamemcp_session→mcp_client, usehttpx2for MCP HTTP timeouts, and read tool metadata viainput_schema/ imagemime_type. The legacy experimental MCP env and NeMo Gym bridge follow the same client API.Servers move from
FastMCPtoMCPServer, servingstreamable_http_app(..., stateless_http=True)with a small ASGI wrapper so state coordinates still come from query params. Codex config prependsfeatures={mcp_2026_07_28=true}.Dependencies: pin MCP 2, bump
openai-agents>=0.20.0, refresh the lockfile (e.g. fastmcp 4.x beta,httpx2). NeMo Gym no longer pulls MCP 1.x into the main install—the resource server runs as an isolated uv script (server.py) with its ownmcp>=1.27,<2metadata, and the optionalnemo-gymextra no longer declares the package directly inpyproject.toml.Reviewed by Cursor Bugbot for commit e253577. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Upgrade MCP client and server to
mcp==2.0.0(MCP 2026-07-28)mcp.ClientSessionwithmcp.Clientusingstreamable_http_clientorstdio_clientandhttpx2timeouts across all harnessesFastMCPtoMCPServerusing a stateless streamable HTTP app and a contextvar for request query parametersinputSchematoinput_schemaand image block field frommimeTypetomime_typeuvscript in the sandbox, removingnemo-gymfrom harness core dependenciesmcpto==2.0.0and bumpsopenai-agentsto>=0.20.0. API changes inMCPServerandmcp.Clientwill break existingFastMCPserver extensions orClientSessioncallers.Macroscope summarized e253577.