Skip to content

feat(browser): store full snapshots on truncation; make eval denylist opt-in - #65923

Merged
teknium1 merged 3 commits into
mainfrom
hermes/hermes-8fcc18e8
Jul 17, 2026
Merged

feat(browser): store full snapshots on truncation; make eval denylist opt-in#65923
teknium1 merged 3 commits into
mainfrom
hermes/hermes-8fcc18e8

Conversation

@teknium1

@teknium1 teknium1 commented Jul 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Browser snapshots that exceed the truncation threshold (now 15,000 chars, aligned with web_extract's per-page budget) now save the complete accessibility tree to disk with a read_file pointer in the output (nothing is lost to truncation anymore), and the browser_console(expression=...) sensitive-primitive denylist is now opt-in instead of on-by-default.

Both changes address BU_Bench findings where lossy observation and a names-based eval denylist crippled the agent's browser performance: truncation could drop the exact element ref a task needed, and the denylist blocked legitimate DOM extraction like document.querySelector('#fetch-results') because the expression contained the word "fetch".

Changes

  • tools/browser_tool.py:
    • New _store_full_snapshot() — mirrors web_tools._store_full_text: writes the full snapshot to cache/web (mounted read-only into remote backends), force-redacts secrets before writing, caps stored copies at 2MB, dedupes by content hash.
    • _truncate_snapshot() — truncation note now includes the stored file path and a ready-to-use read_file path=... offset=... limit=200 call; falls back to the old note if storage fails. Output stays within max_chars (note budget is reserved, clamped for tiny caps).
    • _extract_relevant_content() — LLM summaries append a pointer to the stored full snapshot.
    • _enforce_browser_eval_policy() — the denylist now only applies when browser.restrict_evaluate: true is set. browser.allow_unsafe_evaluate keeps its existing meaning (bypass the denylist) for configs that already set it. The SSRF/private-URL egress guards in _browser_eval are independent of this policy and remain always-on.
    • browser_snapshot schema description mentions the saved-file pointer.
  • hermes_cli/config.py: new browser.restrict_evaluate: false default (no version bump — additive key).
  • website/docs/user-guide/features/browser.md: both behaviors documented.
  • SNAPSHOT_SUMMARIZE_THRESHOLD raised 8000 → 15000, equal to web_tools.DEFAULT_EXTRACT_CHAR_LIMIT, with an invariant test pinning the alignment; _truncate_snapshot default follows the constant.
  • Tests: 7 new (truncate-store pointer, storage-failure fallback, disk redaction, summary pointer, default-open eval, restrict opt-in config read); existing denylist tests updated to pin the opt-in path.

All truncation/summarization call sites route through these two functions — browser_navigate auto-snapshot, browser_snapshot, and both Camofox paths — so every consumer gets the stored-file pointer with no per-site changes.

Validation

Before After
Snapshot > 15,000 chars content beyond cut lost; refs unreachable full tree stored; output points at file + read_file call
LLM-summarized snapshot summary only summary + stored full-text pointer
expression="document.cookie" (default config) Blocked evaluates
expression=... w/ restrict_evaluate: true n/a denylist active (old behavior)
Private-URL egress via eval Blocked Blocked (unchanged)

E2E-tested against a temp HERMES_HOME with real file I/O and a real config.yaml: store/dedupe/redaction verified on disk, default-open + opt-in restrict + allow_unsafe override verified through the real config read path. Targeted suites: test_browser_hardening, test_browser_console, test_browser_secret_exfil, test_browser_eval_ssrf, test_browser_camofox, test_config — 256 tests, 0 failures.

Infographic

browser-harness-upgrade

@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/cli CLI entry point, hermes_cli/, setup wizard tool/browser Browser automation (CDP, Playwright) sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Jul 16, 2026
@alt-glitch

alt-glitch commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related to #56366 and #24009: this retains the private-URL egress guards while changing the primitive-name eval denylist to opt-in and adds a file-backed alternative to lossy snapshot truncation.

tonydwb
tonydwb reviewed Jul 16, 2026
View reviewed changes

@tonydwb tonydwb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Verdict: Comment

Looks Good

  • Browser hardening: store full snapshots on truncation, make eval denylist opt-in
  • Clean additions to browser_tool.py, config.py, and corresponding tests
  • No security concerns

Reviewed by Hermes Agent

teknium1 added 3 commits July 16, 2026 23:29
@teknium1
feat(browser): store full snapshots on truncation; make eval denylist…
27ae555 
… opt-in

Two harness fixes motivated by BU_Bench results where fixed-verb + lossy
observation cost Hermes heavily vs code-driven browser agents:

1. Snapshot truncation no longer loses content. When a snapshot exceeds
   the 8000-char threshold, the complete accessibility tree is saved to
   cache/web (same truncate-and-store pattern as web_extract) and the
   truncated view / LLM summary includes the file path plus a ready-made
   read_file call. Element refs beyond the cut are recoverable without
   re-snapshotting. Stored copies are force-redacted and capped at 2MB;
   content-hash filenames dedupe repeated snapshots of the same page.

2. The browser_console(expression=...) sensitive-primitive denylist is
   now opt-in via browser.restrict_evaluate (default false). The
   names-based denylist blocked legitimate DOM extraction — any selector
   or expression containing 'fetch', 'cookie', 'input', etc. — which
   crippled the agent's only programmatic page-inspection path. The
   SSRF/private-URL egress guards in _browser_eval are independent of
   this policy and remain always-on. browser.allow_unsafe_evaluate keeps
   its meaning (bypass the denylist) for configs that already set it.
@teknium1
test: update None-guard test for stored-snapshot pointer in _extract_…
f3bec5a 
…relevant_content

test_normal_content_returned pinned the exact return value; the summary
now carries a pointer to the stored full snapshot. Assert the summary
passes through and the pointer is present instead.
@teknium1
feat(browser): align snapshot threshold with web_extract's 15k char b…
7f63265 
…udget

SNAPSHOT_SUMMARIZE_THRESHOLD 8000 -> 15000, matching
web_tools.DEFAULT_EXTRACT_CHAR_LIMIT so the snapshot and web_extract
truncate-and-store paths give the model the same per-page budget.
_truncate_snapshot's default max_chars now follows the constant.
Invariant test added; docs (en+zh) and CLI tip updated.
@teknium1
teknium1 force-pushed the hermes/hermes-8fcc18e8 branch from 5ffe2d2 to 7f63265 Compare July 17, 2026 06:29
@teknium1
teknium1 merged commit 0f102fa into main Jul 17, 2026
32 of 33 checks passed
@teknium1
teknium1 deleted the hermes/hermes-8fcc18e8 branch July 17, 2026 06:41
@PRATHAMESH75 PRATHAMESH75 mentioned this pull request Jul 18, 2026
Open
rdnot added a commit to rdnot/hermes-agent-medical-research that referenced this pull request Jul 19, 2026
@rdnot
test(browser): skip upstream threshold-alignment invariant on fork
98fe5bb 
Upstream commit 0f102fa (PR NousResearch#65923) added an invariant test requiring
SNAPSHOT_SUMMARIZE_THRESHOLD (browser_tool.py) == DEFAULT_EXTRACT_CHAR_LIMIT
(web_tools.py). The fork intentionally diverges these:

- DEFAULT_EXTRACT_CHAR_LIMIT = 400_000 (fork #9, raised for medical article
  extraction — full PubMed/AHA article bodies returned whole)
- SNAPSHOT_SUMMARIZE_THRESHOLD = 15_000 (unchanged from upstream — browser
  snapshots are interactive session context, not research documents)

Mark the test as skipped with a documented fork-divergence reason rather
than aligning the thresholds, per user decision.

This was caught by upstream CI on merge NousResearch#56 (8952d7b) after local
verification missed it (test file was not in the merge-changed filter set).
This was referenced Jul 20, 2026
Open
Merged
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
@teknium1
feat(browser): store full snapshots on truncation; make eval denylist…
f8eca49 
… opt-in (NousResearch#65923)

* feat(browser): store full snapshots on truncation; make eval denylist opt-in

Two harness fixes motivated by BU_Bench results where fixed-verb + lossy
observation cost Hermes heavily vs code-driven browser agents:

1. Snapshot truncation no longer loses content. When a snapshot exceeds
   the 8000-char threshold, the complete accessibility tree is saved to
   cache/web (same truncate-and-store pattern as web_extract) and the
   truncated view / LLM summary includes the file path plus a ready-made
   read_file call. Element refs beyond the cut are recoverable without
   re-snapshotting. Stored copies are force-redacted and capped at 2MB;
   content-hash filenames dedupe repeated snapshots of the same page.

2. The browser_console(expression=...) sensitive-primitive denylist is
   now opt-in via browser.restrict_evaluate (default false). The
   names-based denylist blocked legitimate DOM extraction — any selector
   or expression containing 'fetch', 'cookie', 'input', etc. — which
   crippled the agent's only programmatic page-inspection path. The
   SSRF/private-URL egress guards in _browser_eval are independent of
   this policy and remain always-on. browser.allow_unsafe_evaluate keeps
   its meaning (bypass the denylist) for configs that already set it.

* test: update None-guard test for stored-snapshot pointer in _extract_relevant_content

test_normal_content_returned pinned the exact return value; the summary
now carries a pointer to the stored full snapshot. Assert the summary
passes through and the pointer is present instead.

* feat(browser): align snapshot threshold with web_extract's 15k char budget

SNAPSHOT_SUMMARIZE_THRESHOLD 8000 -> 15000, matching
web_tools.DEFAULT_EXTRACT_CHAR_LIMIT so the snapshot and web_extract
truncate-and-store paths give the model the same per-page budget.
_truncate_snapshot's default max_chars now follows the constant.
Invariant test added; docs (en+zh) and CLI tip updated.
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
@teknium1
feat(browser): store full snapshots on truncation; make eval denylist…
aabefba 
… opt-in (NousResearch#65923)

* feat(browser): store full snapshots on truncation; make eval denylist opt-in

Two harness fixes motivated by BU_Bench results where fixed-verb + lossy
observation cost Hermes heavily vs code-driven browser agents:

1. Snapshot truncation no longer loses content. When a snapshot exceeds
   the 8000-char threshold, the complete accessibility tree is saved to
   cache/web (same truncate-and-store pattern as web_extract) and the
   truncated view / LLM summary includes the file path plus a ready-made
   read_file call. Element refs beyond the cut are recoverable without
   re-snapshotting. Stored copies are force-redacted and capped at 2MB;
   content-hash filenames dedupe repeated snapshots of the same page.

2. The browser_console(expression=...) sensitive-primitive denylist is
   now opt-in via browser.restrict_evaluate (default false). The
   names-based denylist blocked legitimate DOM extraction — any selector
   or expression containing 'fetch', 'cookie', 'input', etc. — which
   crippled the agent's only programmatic page-inspection path. The
   SSRF/private-URL egress guards in _browser_eval are independent of
   this policy and remain always-on. browser.allow_unsafe_evaluate keeps
   its meaning (bypass the denylist) for configs that already set it.

* test: update None-guard test for stored-snapshot pointer in _extract_relevant_content

test_normal_content_returned pinned the exact return value; the summary
now carries a pointer to the stored full snapshot. Assert the summary
passes through and the pointer is present instead.

* feat(browser): align snapshot threshold with web_extract's 15k char budget

SNAPSHOT_SUMMARIZE_THRESHOLD 8000 -> 15000, matching
web_tools.DEFAULT_EXTRACT_CHAR_LIMIT so the snapshot and web_extract
truncate-and-store paths give the model the same per-page budget.
_truncate_snapshot's default max_chars now follows the constant.
Invariant test added; docs (en+zh) and CLI tip updated.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@tonydwb tonydwb tonydwb left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades tool/browser Browser automation (CDP, Playwright) type/feature New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@teknium1 @alt-glitch @tonydwb