Skip to content

feat(browser): add selector parameter for element-level screenshot in browser_vision - #9775

Open
hushicai wants to merge 1 commit into
NousResearch:mainfrom
hushicai:feature/browser-vision-selector
Open

hushicai wants to merge 1 commit into
NousResearch:mainfrom
hushicai:feature/browser-vision-selector

Conversation

@hushicai

Copy link
Copy Markdown

Summary

Add selector parameter to browser_vision tool for capturing specific DOM elements instead of full viewport, matching agent-browser's element screenshot capability.

Background

The browser_vision tool currently only supports full viewport screenshots (--full). However, agent-browser supports element-level screenshots via selector parameter, which is useful when you only need to capture a specific element (e.g., a modal, a tweet card, a form).

Changes

  • Add selector parameter to tool schema
  • Update browser_vision() function signature
  • Pass selector to agent-browser screenshot command — mutually exclusive with --full
  • Update registry handler to pass selector parameter

Documentation Evidence

From agent-browser docs (https://www.mintlify.com/vercel-labs/agent-browser/commands/screenshots):

agent-browser screenshot [selector] [path]  # Screenshot (--full for full page)
agent-browser screenshot @e1 element.png  # Screenshot specific element
agent-browser screenshot ".modal" dialog.png  # Screenshot by CSS selector

Selector is mutually exclusive with --full — when selector is provided, it captures that element instead of full page.

Test Results

Verified implementation with actual tests:

Command File Size Behavior
agent-browser screenshot @e1 test-element.png 2917 bytes Element screenshot
agent-browser screenshot --full test-full.png 15744 bytes Full page screenshot
agent-browser screenshot --full @e1 15744 bytes Ignored @e1, captured full page

Test confirms selector and --full are mutually exclusive (selector takes precedence when provided).

Hermes Agent in-browser test:

browser_vision(question="截取这个推文的内容", selector="@e38")

✅ Success —截图保存到 /Users/hushicai/.hermes/cache/screenshots/browser_screenshot_58ffa11312014b118931a9ad1da2c940.png

Usage

# Element screenshot
browser_vision(question="...", selector="@e1")

# Full viewport (default, backward compatible)
browser_vision(question="...")

# Annotated element screenshot
browser_vision(question="...", selector="@e1", annotate=True)

… browser_vision

Add selector parameter to browser_vision tool for capturing specific DOM
elements instead of full viewport, matching agent-browser's element
screenshot capability.

- Add selector parameter to tool schema
- Update browser_vision() function signature
- Pass selector to agent-browser screenshot command (mutually exclusive with --full)

Usage:
browser_vision(question='...', selector='@e1')  # element screenshot
browser_vision(question='...')  # full viewport (default)
@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have tool/browser Browser automation (CDP, Playwright) comp/tools Tool registry, model_tools, toolsets labels Apr 26, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the focused element-capture proposal. The capability is still missing on current main: browser_vision exposes no selector at tools/browser_tool.py:1929-1945 and unconditionally appends --full at tools/browser_tool.py:4017-4022.

Problems

  • The patch predates current backend routing. browser_vision dispatches Camofox at tools/browser_tool.py:3911-3913; camofox_vision accepts no selector at tools/browser_camofox.py:835-851, so Camofox would ignore the new parameter.
  • Lightpanda pre-routes capture through _chrome_fallback_screenshot at tools/browser_tool.py:3961-3968, before the normal screenshot argument construction. That fallback would likewise omit a selector.
  • No automated coverage is included. Existing command-argument tests live at tests/tools/test_browser_console.py:294-344.

Suggested changes

  • Port the change onto the current routing flow and share selector-aware screenshot arguments between the normal and Lightpanda fallback paths.
  • Implement or explicitly reject selector capture for Camofox.
  • Add tests for schema/registry forwarding and all supported backend paths.

Automated hermes-sweeper review.

Comment thread tools/browser_tool.py
if annotate:
screenshot_args.append("--annotate")
screenshot_args.append("--full")
if selector:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current main now has Camofox dispatch before this command construction (tools/browser_tool.py:3911) and a Lightpanda Chrome pre-route (tools/browser_tool.py:3961). Please carry selector behavior through both paths (or return an explicit unsupported-backend error for Camofox); otherwise this parameter silently captures the full page on those supported backends.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/tools Tool registry, model_tools, toolsets P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades tool/browser Browser automation (CDP, Playwright) type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants