Skip to content

Add VisualWebArena benchmark and evaluator - #2918

Draft
JeffPengCoder wants to merge 1 commit into
feature/webarena-splitfrom
feature/visualwebarena-split
Draft

Add VisualWebArena benchmark and evaluator#2918
JeffPengCoder wants to merge 1 commit into
feature/webarena-splitfrom
feature/visualwebarena-split

Conversation

@JeffPengCoder

@JeffPengCoder JeffPengCoder commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

This is the VisualWebArena follow-up layer above #2917 and, transitively, #2295. It adds only the VisualWebArena-specific pieces:

  • a hash-pinned, model-neutral preparation path for the maintained 908-task population and its 346 reference images
  • VisualWebArena task-image materialization inside a confined read-only source root
  • the image-aware VisualWebArena evaluator and its optional fuzzy-image dependencies
  • VisualWebArena verifier routing on the existing webarena_browser local-site resource
  • focused source-integrity, image, dataset, evaluator, and configuration tests

It does not restore BrowserGym or introduce a second browser protocol. It reuses the visual-browser runtime, browser-session provider, computer_use actions, agent loop, and distributed process model from #2295, plus the WebArena-family site, login, collision, and evaluator lifecycle from #2917.

Standard workflow

gym eval prepare --benchmark visualwebarena
python benchmarks/visualwebarena/prepare.py --force-env
gym env prefetch --config benchmarks/visualwebarena/env.yaml
gym env start --config benchmarks/visualwebarena/env.yaml

Then, from another terminal:

gym eval run --config benchmarks/visualwebarena/env.yaml --no-serve

The prepare path downloads the public source at the revision recorded in benchmarks/visualwebarena/source_lock.json, validates its SHA-256, exact 908-row denominator, and every local reference image before emitting Gym rows. A real rollout additionally requires a compatible VisualWebArena deployment, reset/isolation policy, site URL environment variables, and judge credentials where the evaluator needs them.

Validation

  • gym eval prepare --benchmark visualwebarena: passed and produced 908 of 908 rows from the pinned source
  • all 346 referenced task images in that source were validated during preparation
  • gym env validate --benchmark visualwebarena: passed
  • generated environment composition: resolved successfully
  • isolated webarena_browser suite with the VWA layer: 15 of 15 passed
  • cumulative Web-focused unit, agent, benchmark, and resource tests after synchronization with current main: 321 of 321 passed
  • exact WebArena and VisualWebArena source-lock false positives: passed the repository's detect-secrets hook against the committed baseline
  • uv lock --check: passed
  • pre-commit run --all-files: passed
  • git diff --check: passed

This PR remains a draft until a full VisualWebArena rollout on a qualified deployment is attached. The current evidence proves preparation completeness, image integrity, configuration composition, component behavior, and stack isolation; it is not a new VisualWebArena success-rate result.

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated repository configuration is excluded.
  • Substantive tests cover the new source, image, evaluator, and configuration behavior.
  • Standard Gym preparation and environment commands were exercised.
  • Pre-commit checks pass locally.
  • All commits have DCO sign-off.

@copy-pr-bot

copy-pr-bot Bot commented Sep 1, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@JeffPengCoder JeffPengCoder changed the title feature/visualwebarena split Add VisualWebArena benchmark runtimes and evaluators Sep 1, 2026
@JeffPengCoder
JeffPengCoder force-pushed the feature/visualwebarena-split branch from 3a23899 to 61e30f3 Compare September 4, 2026 07:31
@JeffPengCoder
JeffPengCoder force-pushed the feature/visualwebarena-split branch from 61e30f3 to d2105e0 Compare September 4, 2026 08:07
Signed-off-by: Jeff Peng <jepeng@nvidia.com>
@JeffPengCoder
JeffPengCoder force-pushed the feature/visualwebarena-split branch from d2105e0 to 44a86f0 Compare September 4, 2026 08:25
@JeffPengCoder JeffPengCoder changed the title Add VisualWebArena benchmark runtimes and evaluators Add VisualWebArena benchmark and evaluator Sep 4, 2026
@anwithk
anwithk requested a review from mpatel31415 September 9, 2026 18:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant