Add WebVoyager benchmark and shared web runtime infrastructure

[JeffPengCoder]

Summary

This PR adds first-class WebVoyager support to NeMo Gym together with the shared web protocol used by the benchmark. It contains only the WebVoyager layer after splitting the former combined WebVoyager, WebArena, and VisualWebArena change into focused stacked follow-ups.

WebVoyager evaluates multimodal browser agents on live public websites and scores the final answer together with browser screenshots. NeMo Gym now exposes one model-neutral browser runtime, visual_browser, over the maintained 552-task population. Headed Chromium runs under Xvfb; Playwright owns browser lifecycle and navigation; PyAutoGUI performs visible coordinate input. Nano Omni and Qwen3.5 retain separate policy-output adapters but use the same tasks, browser behavior, proxy/CAPTCHA path, action executor, and Gemini judge.

The earlier BrowserGym-compatible 643-task route has been removed. The original Selenium runner is not copied into Gym. Large runtime images, model checkpoints, credentials, proxy endpoints, recordings, screenshots, and raw experiment artifacts remain outside this repository.

Scope split and follow-ups

The current dependency chain is:

upstream main
  -> #2295 WebVoyager plus shared web infrastructure
    -> #2917 WebArena runtime and benchmark
      -> #2918 VisualWebArena runtime and benchmark

• #2917 Add WebArena benchmark and local-site runtime
• #2918 Add VisualWebArena benchmark and evaluator

The follow-up PRs remain drafts until equivalent full-population rollout evidence is attached. PR #2295 originates from a personal fork, so an exact upstream mirror of its head is maintained as the base of #2917 while preserving this PR's review history.

Design

maintained 552-task dataset
            |
        web_agent
       /         \
Nano Omni       Qwen3.5
policy adapter  policy adapter
       \         /
       visual_browser
Playwright lifecycle and navigation
PyAutoGUI visible coordinate actions
proxy, CAPTCHA, screenshots, cleanup
            |
    webvoyager_judge

• The Responses agent owns prompt history, multimodal policy requests, timeout and retry policy, schema validation and bounded parse retries, and rollout history.
• The browser resource server owns browser supply, navigation, visible actions, screenshots, session identity, and cleanup.
• Browser leases are acquired and released asynchronously. Synchronous Playwright and PyAutoGUI operations remain on a session-affine worker thread, so they do not block the FastAPI event loop.
• The browser-supply seam supports local Chromium today and can admit a remote CDP or AgentEnv provider without changing benchmark tasks, actions, prompts, or evaluation.
• The judge is a separate resource server. Evidence is retained before browser close, enabling judge-only retry without keeping the browser alive.
• Policy failures, evaluator failures, infrastructure failures, and benchmark reward remain distinct signals. Fixed-denominator reconciliation prevents partial shards from being reported as complete scores.
• One visual-browser process owns one display and accepts one active session. Distributed runs scale with isolated processes or containers, each with its own display, browser profile, ports, writable roots, and rollout file.

Main changes

• Adds the maintained 552-task WebVoyager benchmark package, immutable source lock, dataset preparation, policy profiles, summary tooling, and fixed-denominator reconciliation.
• Adds the backend-neutral nemo_gym.web task, action, observation, artifact, session, resource-server, failure, task-image, browser-provider, and composed driver/evaluator contracts.
• Adds the headed-Chromium visual_browser server with Playwright lifecycle, PyAutoGUI coordinate actions, Unicode clipboard input, navigation recovery, proxy routing, CAPTCHA handling, redacted logs, recordings, and evidence retention.
• Adds the independent WebVoyager multimodal judge and judge-only re-verification path.
• Adds the shared multimodal web agent plus separate Nano Omni and Qwen output adapters, schema validation, bounded parse retries, screenshot-history controls, and explicit timeout envelopes.
• Adds provider lease cleanup and failure masking suitable for distributed evaluation and training: infrastructure failures become retryable masked samples rather than reward-zero policy examples.
• Adds model-recipe provenance locks for tokenizer, chat template, multimodal processor, reasoning parser, and tool-call parser expectations without committing model assets.

Standard preparation and execution

From a clean Linux checkout:

uv lock --check
uv sync --frozen --extra dev
uv run --project resources_servers/visual_browser playwright install chromium
./.venv/bin/gym eval prepare --benchmark webvoyager

./.venv/bin/python benchmarks/webvoyager/prepare.py \
  --profile qwen35_122b_a10b \
  --rollout-output "$PWD/results/webvoyager/qwen/rollouts.jsonl" \
  --force-env

cd benchmarks/webvoyager
../../.venv/bin/gym env prefetch
xvfb-run --auto-servernum --server-args="-screen 0 1920x1080x24" \
  ../../.venv/bin/gym env start

From a second terminal in benchmarks/webvoyager:

../../.venv/bin/gym eval run --no-serve --concurrency 1

Use --profile nano_omni for the Nano Omni policy adapter. Stop the foreground gym env start process with Ctrl-C; Gym currently has no separate env stop command. The Fern WebVoyager tutorial is the single end-to-end guide for external-service preflight, smoke, full-population execution, fixed-denominator reconciliation, and training failure semantics.

Validation

• 317 focused WebVoyager, shared-web, and modified server-utility tests pass after synchronization with current main.
• gym eval prepare --benchmark webvoyager downloads the immutable pinned source and produces exactly 552 tasks.
• The generated Qwen profile passes gym env validate --benchmark webvoyager and gym env resolve.
• The exact provenance-lock false positives pass the repository's detect-secrets hook against the committed baseline.
• uv lock --check, repository-wide pre-commit run --all-files, and diff checks pass. This PR does not change .gitignore, .pre-commit-config.yaml, or GitHub workflow definitions.
• The latest post-refactor Qwen3.5-122B-A10B-FP8 full run completed 300/552 tasks successfully, 54.35 percent, with 552 valid unique task results and no missing, invalid, or duplicate-valid tasks.
• Two parser-faithful Nano Omni tuned-checkpoint repetitions completed 415/552, 75.18 percent, and 420/552, 76.09 percent. Both produced 552 valid unique task results with no missing, malformed, or duplicate tasks.

All full runs used the same hash-pinned task population and visual_browser runtime. The two Nano Omni repetitions used the same frozen source, tuned checkpoint, parser configuration, external services, judge, and generation parameters; their mean was 417.5/552, or 75.63 percent, with a five-task difference. In the parser-faithful profile, Gym validates parser-produced structured calls but does not repair malformed JSON, complete delimiters, infer aliases, or silently clamp action arguments. Because WebVoyager uses live websites and external proxy, CAPTCHA, judge, and policy services, these numbers are integration evidence rather than stable leaderboard guarantees.

Operational requirements and non-goals

• Full WebVoyager evaluation requires Linux with Xvfb, Chromium, PyAutoGUI, and xclip; live-site access; US egress for proxy-scoped domains; a funded CAPTCHA provider; a configured multimodal judge; and policy-serving assets that match the selected recipe lock.
• Scores from different task populations, browser runtimes, checkpoints, live-site snapshots, judges, or serving recipes are not directly comparable.
• This PR does not include WebArena or VisualWebArena task preparation, local-site stacks, site reset/login behavior, or benchmark-specific evaluators. Those are isolated in the follow-up layers above.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[Glorf]

FYI @mpatel31415

[mpatel31415]

1. osworld_internal is mentioned in a few files (e.g. `benchmarks/webvoyager/native-v3.md), but actually it's not used and 2 new runtimes are implemented. From discussion with @Glorf i understood that osworld work can be reused here.

2. If only WebVoyager is validated we need to keep the changes only related to this benchmarks and add visualwebarena and webarena in the follow-up PRs, once they are validated too. Review done by CC mentioned some correctness issues for webarena, so I think we should be careful about it.

3. Can you clarify difference between 2 implemented runtimes? if legacy WebVoyager with browsergym_web is not validated maybe it's not needed now and we can just drop it? Adding not valdiated code paths to Gym is risky in my opinion.

4. Is it correct to assume that native_web should be used for Super 3.5 VL ? I see in benchmarks/webvoyager/README.md "The maintained 552-task Nano Omni", so it's a bit confusing which version is the one to be used for Super 3.5 VL

5. I think that we should not have versions of the models (v3) used to index markdown files or or e.g. "Native Nano Omni route" comments inside benchmark, they should be model-agnostic.

6. I'm attaching results of review done by CC, I'd be glad to take a deeper dive into the code myself once it contains only validated and needed runtimes and benchmarks.

pr_2295_review.md

[mpatel31415]

Would it be possible to implement the judge call in Gym's standard way, via nemo_gym.judge? Right now verify_webvoyager() calls self.server_client.post(...) directly (app.py:185) with a hand-rolled except Exception and retry loop, instead of using the shared judge-failure abstraction that ~20 other resources servers use (e.g. frontierscience_judge/app.py:287-322, arena_judge/app.py:265-281).

Using the shared helper would get us for free dealing with judge failures and rerunning them and would match this repo's convention for judge-based resources servers.

[JeffPengCoder]

Agreed. I will route WebVoyager judging through the standard /verify endpoint and use nemo_gym.judge.call_judge() for the model call. The current custom /verify_webvoyager route and agent-side retry loop will be removed.
Judge transport, timeout, authentication, HTTP, and response-schema failures will then be classified as judge_failed by judge_failsafe, allowing gym eval reverify --judge-failed-only to rerun only the judge.
Captured screenshots, page URLs, and the final answer will be retained in the rollout response so reverification does not repeat live-site interaction. A successfully received but unparseable verdict will remain a benchmark outcome rather than an infrastructure failure

[JeffPengCoder]

1. osworld_internal is mentioned in a few files (e.g. `benchmarks/webvoyager/native-v3.md), but actually it's not used and 2 new runtimes are implemented. From discussion with @Glorf i understood that osworld work can be reused here.

It might be introduced by Baseline directory name.  I disagree that we should do something refactor to make osworld and webbench share something in this web initial PR. That would introduce much more work on regression.

2. If only WebVoyager is validated we need to keep the changes only related to this benchmarks and add visualwebarena and webarena in the follow-up PRs, once they are validated too. Review done by CC mentioned some correctness issues for webarena, so I think we should be careful about it.

Just before I decide to align number with Baseline about one week ago, I get the information that webvoyager is the top1, so webarena/visualwebarena are already there.  They pass basic check but remain for further improvement. But with their existance, the design architecture would look more reasonable, which I need to brining in such shared modules in the end. 

3. Can you clarify difference between 2 implemented runtimes? if legacy WebVoyager with browsergym_web is not validated maybe it's not needed now and we can just drop it? Adding not valdiated code paths to Gym is risky in my opinion.

Browsergym is for other models, current native_gym only works for nemotron. legacy webvoyager is for pure omni v3 (without any change toward behavior like super3.5) and I could remove it if it is not needed at all.

4. Is it correct to assume that native_web should be used for Super 3.5 VL ? I see in benchmarks/webvoyager/README.md "The maintained 552-task Nano Omni", so it's a bit confusing which version is the one to be used for Super 3.5 VL

Yes, it would work with super 3.5 vl and it works with nano omni as well, we use customized tokenizer/chat-template, etc to make nano omni behave like super 3.5. 

5. I think that we should not have versions of the models (v3) used to index markdown files or or e.g. "Native Nano Omni route" comments inside benchmark, they should be model-agnostic.

Until our tokenizer alike component is stable enough.

6. I'm attaching results of review done by CC, I'd be glad to take a deeper dive into the code myself once it contains only validated and needed runtimes and benchmarks.

Let me make running webvoyager benchmark simply like this: gym eval prepare->gym eval run

pr_2295_review.md

[mpatel31415]

3. Can you clarify difference between 2 implemented runtimes? if legacy WebVoyager with browsergym_web is not validated maybe it's not needed now and we can just drop it? Adding not valdiated code paths to Gym is risky in my opinion.

Browsergym is for other models, current native_gym only works for nemotron. legacy webvoyager is for pure omni v3 (without any change toward behavior like super3.5) and I could remove it if it is not needed at all.

Could you please point me to any resources so I could understand e. g. "any change toward behavior like super3.5" or "we use customized tokenizer/chat-template, etc to make nano omni behave like super 3.5. ". What is this customised tokenizer? Our goal is to compare Qwen3.5-122B-A10B and Super 3.5 VL and I'm not sure that it's possible to do so with this version of WebVoyager, considering that "Browsergym is for other models" and " current native_gym only works for nemotron".

[mpatel31415]

@JeffPengCoder - just to let you know, that there is "Use of AI and LLM Tools" section in contribution guide in Gym's documentation. You might want to look into this section which describes when changes to a PR are required.

[JeffPengCoder]

3. Can you clarify difference between 2 implemented runtimes? if legacy WebVoyager with browsergym_web is not validated maybe it's not needed now and we can just drop it? Adding not valdiated code paths to Gym is risky in my opinion.

Browsergym is for other models, current native_gym only works for nemotron. legacy webvoyager is for pure omni v3 (without any change toward behavior like super3.5) and I could remove it if it is not needed at all.

Could you please point me to any resources so I could understand e. g. "any change toward behavior like super3.5" or "we use customized tokenizer/chat-template, etc to make nano omni behave like super 3.5. ". What is this customised tokenizer? Our goal is to compare Qwen3.5-122B-A10B and Super 3.5 VL and I'm not sure that it's possible to do so with this version of WebVoyager, considering that "Browsergym is for other models" and " current native_gym only works for nemotron".
========================
Thanks for pushing on this. My previous wording was misleading, so let me correct it.
The two paths are separated by their browser and action contracts, not by model family:

• browsergym_web uses BrowserGym’s compact SoM observation and high-level browser actions.
• The maintained 552-task path uses headed Chromium, screenshots, and the structured computer, navigate, tab, and terminate tool contract.
  Neither runtime is inherently restricted to a particular model. However, only the Nano-derived model profile has completed full reference-aligned validation on the maintained native route. A different model can
  use that route if its serving profile can reliably produce the same structured tools, but that still requires validation.

“Customized tokenizer” was also an imprecise description. It refers to model-serving assets used by the Nano alignment profile:

• tokenizer vocabulary and multimodal image special tokens;
• a Jinja chat template that serializes multimodal messages, tool schemas, and retained reasoning history;
• the nano_v3 reasoning parser and qwen3_coder tool-call parser.
  These assets do not modify model weights and do not make Nano Omni equivalent to Super 3.5. Any Super-like behavior comes from the checkpoint and training. The tokenizer, template, and parsers only define how
  requests and outputs are serialized.
  They are scoped to the Nano model profile
  (https://github.com/JeffPengCoder/Gym/blob/feature/webarena/responses_api_models/local_vllm_model/configs/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16-alignment.yaml), while Qwen has its own Qwen3.5
  serving profile (https://github.com/JeffPengCoder/Gym/blob/feature/webarena/responses_api_models/local_vllm_model/configs/Qwen/Qwen3.5-122B-A10B.yaml). The benchmark runtime contract is documented separately in
  native-v3.md (https://github.com/JeffPengCoder/Gym/blob/feature/webarena/benchmarks/webvoyager/native-v3.md).

For a fair comparison between Qwen3.5-122B-A10B and Super 3.5 VL, both models must use the same:

• task population and task text;
• observation and action contract;
• browser and runtime configuration;
• proxy and CAPTCHA policy;
• judge and result-accounting rules.
  Each model should still use its own tokenizer, chat template, and reasoning and tool parser. Comparing Qwen on browsergym_web against Super on the coordinate-action runtime would not be an apples-to-apples comparison.
  More specifically, these are vLLM output-protocol parsers rather than components that improve model reasoning. The reasoning parser separates the model’s native reasoning stream from its final content and exposes them as OpenAI-compatible reasoning_content and content fields. The tool-call parser recognizes the model’s native function-call syntax and converts it into standard tool_calls entries containing a function name and JSON arguments. Gym then independently validates those arguments against the benchmark’s browser tools before executing them.
  For the reference-aligned Nano Omni WebVoyager recipe, qwen3_coder is not inferred from the model name. The reference launcher explicitly configures --tool-call-parser qwen3_coder. The custom nano_v3 reasoning parser is loaded from nano_v3_reasoning_parser.py through --reasoning-parser-plugin and selected with --reasoning-parser nano_v3.
  In our containerized run, the host serving-assets directory is mounted read-only at /opt/nemotron-v3-assets. vLLM then loads /opt/nemotron-v3-assets/nano_v3_reasoning_parser.py. These are recipe-specific serving settings, not intrinsic requirements of every Nano Omni deployment.

We have completed an end-to-end Qwen3.5 diagnostic on the BrowserGym path, but we have not yet completed a same-runtime Qwen-versus-Super comparison. The PR also does not currently contain a pinned Super 3.5 VL model profile, but current infrastructure of Omin does apply for Super 3.5 on native web path.
I will update the wording to make that boundary explicit and treat the BrowserGym path as an upstream-compatible baseline rather than as a result directly comparable with the native route.

[JeffPengCoder]

@JeffPengCoder - just to let you know, that there is "Use of AI and LLM Tools" section in contribution guide in Gym's documentation. You might want to look into this section which describes when changes to a PR are required.

Get, let me try to use stacked PR

[github-actions]

🌿 Preview your docs: https://nvidia-preview-feature-webarena.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

• nemo/gym/tutorials/evaluation-tutorials
• nemo/gym/tutorials/evaluation-tutorials/webvoyager

[waple0820]

Nice split in composed_backend.py — WebBrowserDriver owning only the live browser is the same line we drew in #1865, so this might compose better than it looks.

You mentioned above that the headed Chromium lifecycle would stay in a shared native layer rather than move into webvoyager_browser. That's roughly what #1865 is. Might be worth a look before writing a second one.

Where it would help: native_browser.py:251 does sync_playwright().start() + chromium.launch(...), so the driver always owns a browser it started itself, and max_sessions: 1 fits that. Fine for eval — but it leaves no room for training, where we run one isolated cloud browser per rollout at 64-way concurrency and have to connect over CDP to a session a provider hands out. #1865 is that seam: BrowserBackend drives an existing browser, BrowserSessionProvider does acquire/release behind a registry, with local Chromium and remote CDP as the two shipped implementations. It would sit under WebBrowserDriver — WebTask, WebAction and the evaluators wouldn't move.

Separately, session_manager.py:484 grows its own close_session + _reaper_loop over session_ttl_seconds. That's now the third copy: #2612 puts it on SimpleResourcesServer, and #2886 merged a hand-written one into workplace_assistant yesterday. We came at it from the training side — orphaned sessions took 1172 of 1280 rollouts across steps 61–80 of a GB200 run, each landing in GRPO as a real reward=0.0.

One thing that would need settling either way: your driver protocol is sync and ours is async, since a CDP connect and a provider call are both I/O.

If it's useful I can put up the seam PR against this branch so you can see the shape. No opinions on the benchmark side from us — that's your call, we don't have one here.

[jayl940712]

Nice split in composed_backend.py — WebBrowserDriver owning only the live browser is the same line we drew in #1865, so this might compose better than it looks.

You mentioned above that the headed Chromium lifecycle would stay in a shared native layer rather than move into webvoyager_browser. That's roughly what #1865 is. Might be worth a look before writing a second one.

Where it would help: native_browser.py:251 does sync_playwright().start() + chromium.launch(...), so the driver always owns a browser it started itself, and max_sessions: 1 fits that. Fine for eval — but it leaves no room for training, where we run one isolated cloud browser per rollout at 64-way concurrency and have to connect over CDP to a session a provider hands out. #1865 is that seam: BrowserBackend drives an existing browser, BrowserSessionProvider does acquire/release behind a registry, with local Chromium and remote CDP as the two shipped implementations. It would sit under WebBrowserDriver — WebTask, WebAction and the evaluators wouldn't move.

Separately, session_manager.py:484 grows its own close_session + _reaper_loop over session_ttl_seconds. That's now the third copy: #2612 puts it on SimpleResourcesServer, and #2886 merged a hand-written one into workplace_assistant yesterday. We came at it from the training side — orphaned sessions took 1172 of 1280 rollouts across steps 61–80 of a GB200 run, each landing in GRPO as a real reward=0.0.

One thing that would need settling either way: your driver protocol is sync and ours is async, since a CDP connect and a provider call are both I/O.

If it's useful I can put up the seam PR against this branch so you can see the shape. No opinions on the benchmark side from us — that's your call, we don't have one here.

@waple0820 Thank you for your contributions. This PR like you have seen is more geared towards evaluations rather training in mind at the moment. It would be likely that we prioritize this current implementation first and if you can help us improve it for training needs. Sorry for the trouble and not full-filling your request sooner.

[waple0820]

Thanks @jayl940712, that's a clear answer and it helps.

Should we close #1865? If the plan is to land this implementation first, we'd rather not leave a second browser environment sitting in the queue — happy to close it and bring the remote-CDP backend and the session-provider seam back as a follow-up once this one is in. Just say the word either way.

Closing it wouldn't settle the layering question though, which is why I opened #2921. Short version: session lifecycle now has four separate implementations across open PRs and one merged agent, and whichever browser runtime lands first, that piece probably shouldn't live inside a benchmark's package.

The one thing worth deciding before this lands rather than after: WebEnvironmentBackend is sync (def reset, def step, def close). For eval that's fine. For training it isn't — a CDP connect and a provider acquire are both network calls, and at 64-way concurrency one rollout waiting on a browser handshake blocks every other rollout in the same process. That's not something a follow-up can add on top; it changes the protocol every backend implements. Making the driver methods async def now costs almost nothing while there is one implementation, and gets expensive once WebArena and VisualWebArena are on it too.

We're glad to do the training-side work — that's the half we actually have experience with, and Gym needs it eventually if these environments are going to be trained on and not only scored. It's just easier to build if the protocol leaves room for I/O.

[JeffPengCoder]

@waple0820 Thanks for your input.

Closing it wouldn't settle the layering question though, which is why I opened #2921. Short version: session lifecycle now has four separate implementations across open PRs and one merged agent, and whichever browser runtime lands first, that piece probably shouldn't live inside a benchmark's package.

[Jeff] understand there is definitely some difference between benchmarking and train, and that difference would impact architecture design. I would leave it to your PR to do such improvement given I get foundation of webbench correctly built up in this PR.

The one thing worth deciding before this lands rather than after: WebEnvironmentBackend is sync (def reset, def step, def close). For eval that's fine. For training it isn't — a CDP connect and a provider acquire are both network calls, and at 64-way concurrency one rollout waiting on a browser handshake blocks every other rollout in the same process. That's not something a follow-up can add on top; it changes the protocol every backend implements. Making the driver methods async def now costs almost nothing while there is one implementation, and gets expensive once WebArena and VisualWebArena are on it too.

[Jeff] I notice this synchronization issue, it is the same case for other bench package such as osworld. We do have some solution in term of de-lock or parellelism. And now I am using thread to partially cope with the issue and would pursuit more better solution. Thanks for your suggestion.

We're glad to do the training-side work — that's the half we actually have experience with, and Gym needs it eventually if these environments are going to be trained on and not only scored. It's just easier to build if the protocol leaves room for I/O.

[Jeff] Thanks and please help review.

[waple0820]

The new browser_session.py is the same seam as #1865 — same BrowserSessionSpec / BrowserSessionHandle / BrowserSessionProvider, async acquire/release, registry plus entry points. Yours adds lease_ttl_seconds and the renewable-lease heartbeat, and both are better than what we have; the note about enforcing the lease outside the Gym process is exactly the failure mode we hit in training.

So the interface question looks settled. What's left is that there are now two copies of it, which is what #2921 was about.

That file has nothing benchmark-specific in it — no WebVoyager, no task model, no evaluator — and it is currently sitting inside a 16k-line PR. Could it land on its own first, from either branch? Then this PR and #1865 both build on one copy instead of each shipping their own, and the question of which of the two merges first stops mattering.