Skip to content

Add WebArena benchmark and local-site runtime - #2917

Draft
JeffPengCoder wants to merge 1 commit into
feature/webarenafrom
feature/webarena-split
Draft

Add WebArena benchmark and local-site runtime#2917
JeffPengCoder wants to merge 1 commit into
feature/webarenafrom
feature/webarena-split

Conversation

@JeffPengCoder

@JeffPengCoder JeffPengCoder commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

This is the WebArena follow-up layer above #2295. It adds only the WebArena-specific benchmark and local-site behavior:

  • a hash-pinned, model-neutral preparation path for the maintained 812-task population
  • a webarena_browser specialization of the shared visual-browser runtime from Add WebVoyager benchmark and shared web runtime infrastructure #2295
  • deployment URL substitution, benchmark-account login, local-site navigation recovery, and mutable-site locking
  • collision-aware before-and-after snapshots for stateful evaluation
  • the pinned WebArena evaluator with explicit source provenance
  • focused benchmark, dataset, authentication, evaluator, and configuration tests

It does not add BrowserGym or a second browser interaction protocol. WebArena reuses #2295's headed Chromium lifecycle, Playwright control plane, PyAutoGUI coordinate input, computer_use action contract, browser-session provider, Responses API agent loop, artifacts, failures, cleanup, and distributed process model. VisualWebArena-specific data, images, and evaluation remain in #2918.

The base branch feature/webarena mirrors the exact current head of #2295 in NVIDIA-NeMo/Gym because GitHub cannot use a fork-originated PR directly as the base of a native PR stack.

Standard workflow

gym eval prepare --benchmark webarena
python benchmarks/webarena/prepare.py --force-env
gym env prefetch --config benchmarks/webarena/env.yaml
gym env start --config benchmarks/webarena/env.yaml

Then, from another terminal:

gym eval run --config benchmarks/webarena/env.yaml --no-serve

The prepare path downloads the public source at the revision recorded in benchmarks/webarena/source_lock.json, validates its SHA-256 and exact 812-row denominator, and produces model-neutral Gym rows. The benchmark still requires a compatible self-hosted WebArena deployment, reset/isolation policy, site URL environment variables, and judge credentials for model-scored evaluator cases.

Validation

  • gym eval prepare --benchmark webarena: passed and produced 812 of 812 rows from the pinned source
  • gym env validate --benchmark webarena: passed
  • generated environment composition: resolved successfully
  • isolated webarena_browser component suite: 12 of 12 passed
  • cumulative Web-focused unit, agent, benchmark, and resource tests after synchronization with current main: 314 of 314 passed
  • exact source-lock false positives: passed the repository's detect-secrets hook against the committed baseline
  • uv lock --check: passed
  • pre-commit run --all-files: passed
  • git diff --check: passed

This PR remains a draft until a full rollout on a qualified WebArena deployment is attached. The checks above establish source completeness, configuration integrity, component behavior, and stack isolation; they are not presented as a new WebArena success-rate result.

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated repository configuration is excluded.
  • Substantive tests cover the new preparation, runtime specialization, and evaluator behavior.
  • Standard Gym preparation and environment commands were exercised.
  • Pre-commit checks pass locally.
  • All commits have DCO sign-off.

@copy-pr-bot

copy-pr-bot Bot commented Sep 1, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@JeffPengCoder JeffPengCoder changed the title feature/webarena split Add WebArena benchmark runtimes Sep 1, 2026
@JeffPengCoder
JeffPengCoder force-pushed the feature/webarena-split branch 2 times, most recently from 75aae68 to f57f3db Compare September 4, 2026 08:07
Signed-off-by: Jeff Peng <jepeng@nvidia.com>
@JeffPengCoder JeffPengCoder changed the title Add WebArena benchmark runtimes Add WebArena benchmark and local-site runtime Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant