feat(browser): add authenticated extension controller

[abundantbeing]

What does this PR do?

Adds an opt-in browser-extension controller lane so Hermes can route existing browser_* tools to the exact authenticated browser session that opened the conversation.

The implementation has two layers:

1. A transport-neutral broker with principal/profile/session/controller/browser-profile/transport scoping, one-shot WebSocket tickets, capability allowlisting, command lifecycle, cancellation, timeout, reconnect, and owner-scoped detach.
2. Request-bound tool routing for the seven existing Browser Use registry schemas. Generic requests preserve the existing backend; once the gateway binds controller identity, the extension lane is authoritative and fails closed if that exact controller disappears or cannot execute the action.

Local API and authenticated dashboard/cloud transports use the same protocol and real-action allowlist:

• browser_navigate
• browser_snapshot
• browser_screenshot
• browser_click
• browser_type
• browser_press
• browser_scroll
• browser_back
• browser_list_tabs
• browser_activate_tab

Raw CDP, arbitrary evaluation, console, file upload, vision, and image extraction are not admitted.

Reconnect and detach semantics

Unexpected transport loss is recoverable: the controller is hidden from new dispatch while already-started commands remain pending until their original deadline. A reconnect with the same stable identity refreshes the transport and negotiated capabilities, flushes deferred cancels before new work, and can complete the original command.

An authenticated browser.controller.detach frame/RPC remains immediately terminal. A different controller id or browser profile in the same authenticated session lane is also a hard replacement: the old controller's pending work is cancelled before the successor becomes routable. Inbound heartbeat, result, cancel, and detach frames are admitted only from the current owner.

Slow but live cross-thread WebSocket writes remain in flight under the broker command deadline instead of being misclassified as failed sends. Real send failures still surface immediately.

Existing user compatibility

• browser.extension_control.enabled defaults to false.
• Feature off: the broker is never queried and the current Browser Use path is unchanged.
• Feature on with no server-bound controller identity: the current backend still handles the tool.
• Feature on with bound controller identity: missing, ambiguous, disconnected, or incapable controllers fail closed instead of switching browsers.
• Controller tools are exposed only inside the matching request/session context; availability is never cached process-wide.
• Once a request is bound to the controller lane, failures do not silently jump to another browser.
• Server-bound browser-control identity variables are excluded from shared shell snapshots, preventing cross-session routing metadata persistence.

Tests

A real browser_snapshot journey covers the aiohttp route table, Bearer-authenticated registration, one-shot subprotocol ticket, controller WebSocket command/result exchange, router serialization, and zero calls to the legacy backend.

Reconnect coverage includes socket close and dashboard transport loss, same-identity capability renegotiation, deferred-cancel ordering and bounds, stale-owner rejection, explicit detach, different-identity hard replacement, timeout, late completion, and slow WebSocket send waits.

The diff is 23 files with 4,873 additions; 2,553 additions are tests and 101 are docs/config.

Resolves the Chrome-extension backend portion of #84000. This uses the existing authenticated API/dashboard transports instead of adding a second native-messaging server inside Hermes.

Related issue(s)

#84000

Checklist

• I have performed a self-review of my changes
• I have commented my code, particularly in hard-to-understand areas
• I have added or updated documentation as needed
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have run code quality checks locally and they pass
• I have not committed sensitive data or credentials
• My commit messages are clear and descriptive
• I have credited co-authors where applicable
• Any generated or vendored files are reproducible and justified in the PR description

Testing evidence

• Controller/broker/API/cloud/router plus snapshot-identity and bound-route authority regressions: 95 passed, 1 POSIX-only integration skipped on Windows
• Registry/cache/model/API compatibility: 93 passed
• RED → GREEN: 6 strict admission failures → all green
• RED → GREEN: real browser_snapshot WebSocket journey failed when the action was removed, then passed with 0 legacy-backend calls
• RED → GREEN: 4 reconnect lifecycle contracts failed on the old broker, then passed
• RED → GREEN: stale-scope cancel, completed-send TimeoutError, and zombie-controller replacement regressions
• RED → GREEN: two bound-controller authority cases failed by invoking legacy fallback, then passed fail-closed; a real schema-build → disconnect → dispatch regression proves zero fallback calls
• First public CI head exposed HERMES_BROWSER_CONTROL_* snapshot exclusion drift; reproduced locally and fixed in the fourth commit
• ruff check on every touched Python module/test: clean
• compileall on every touched Python module/test: clean
• Docusaurus English production build: successful (only the repo's pre-existing /docs/llms.txt and /docs/llms-full.txt root-page warnings)
• Config example parses with the feature disabled
• GitNexus change analysis: low risk, no affected existing process flow
• Added-line secret scan and git diff --check: clean

Additional context

The feature is deliberately opt-in and fail-closed on identity/capability mismatch. Registration validates protocol version 1 strictly (booleans are rejected), requires at least one permitted capability, and uses a 30-second one-shot ticket carried only in Sec-WebSocket-Protocol. Query-string tickets are rejected.

This branch is based on current main. The first CI head's only failing Python slice was the new browser-control ContextVars missing from shared terminal snapshot exclusions; the corrected head includes that regression fix.

[alt-glitch]

This was generated by AI during triage.

Related to #84000: this PR provides an opt-in authenticated extension-controller lane for the requested shared visible-browser workflow.

[Enough1122]

AI code review — automated review for reference, author can ignore or act on any point.

feat(browser): add authenticated extension controller

1. Clock mismatch in the advertised ticket expiry: the broker's clock is time.monotonic by default (gateway/browser_control_broker.py BrowserControlBroker.__init__), but _handle_browser_control_register advertises ticket_expires_at = time.time() + ticket_ttl (gateway/platforms/api_server.py). Both advance at the same rate so ticket_expires_in_seconds is accurate, but the absolute ticket_expires_at is wall-clock while the broker enforces expiry on monotonic — an NTP step or manual clock change makes the advertised deadline diverge from the enforced one. Either expose the broker's own deadline (ticket.expires_at is monotonic-based too) or document that the advertised absolute time is best-effort.

2. Ticket minting is unthrottled and each mint is O(n): every valid authenticated POST /v1/browser-control/register mints a ticket with no per-identity rate limit, and mint_ticket runs _prune_tickets, which scans all live tickets — so sustained minting from a leaked/compromised API key yields O(n²) prune work and unbounded (TTL-bounded) ticket accumulation. Consider a per-principal in-flight ticket cap (e.g. reject when a principal already holds N un-consumed tickets).

3. browser_control_enabled re-reads config on every call: both the broker helper and the adapter's _browser_control_enabled() call load_config() per request. If load_config does a YAML merge per call, every /v1/browser-control/* request pays a config load. Consider a TTL cache (or reading the value once at adapter init, re-checked on a shorter cadence) — the feature is off by default, so this path is cold unless enabled, but it's still per-request work on the enabled path.

4. Capability renegotiation vs. pending completion (worth a test): ControllerScope.__eq__ includes capabilities, and attach reassigns pending.scope to the new scope on reconnect. The reconnect test covers same-capability reconnects; a reconnect with a changed capability set (extension renegotiates mid-command) would make the second socket's complete(scope=…) fail its scope equality check against the reassigned pending scope and drop the result. Verify this path is intended (the identity contract says capabilities aren't identity, but completion scope-checks them) and add a test.

[abundantbeing]

CI note for the maintainers: the failing slice on this head is unrelated to this PR's diff and reproduces on clean main (b5455fdd1):

• tests/tools/test_image_generation.py::TestFalCatalog::test_upscale_defaults_are_all_off — reproduced locally against the branch's code, which does not touch test_image_generation.py or the FAL catalog; the assertion fails on the xai/grok-imagine-image/v2.0 catalog entry itself.
• Clean main's own CI also reports a failing Python-tests slice (slice 12/12 on b5455fdd1).
• Other flakes observed in the same runs: test_transcription_tools.py::TestRunCommandSttIdleTimeout (timing), test_goal_continuation_drain.py (queue key), and a GitHub 429 during checkout.

A CI re-run (maintainer-only; the fork token gets 403 on actions/rerun) should clear it. Happy to force-push a no-op refresh if that's easier.

[andrexibiza]

Reviewed exact head cb2e881ccd32d3262120abb83c3845083f2f025a against its base 00e5a361b60f621ae2246dffdd0a0252895d8493 and current main 6c6c17e0fc1fdb48cd4f7b50d5e263ec86acee5d.

The controller/broker direction is strong: server-derived principals, exact session/controller/browser-profile/transport identity, one-shot WS tickets, fail-closed bound routing, owner-aware reconnect/detach, and capability filtering are the right primitives. I also checked the earlier automated reconnect/capability-renegotiation concern: attach() rebinds matching pending commands to the successor scope, so I am not repeating that advisory as a blocker.

I found two merge blockers, both in the new artifact boundary:

1. HTTP-uploaded artifacts and broker-dispatched artifacts are bound to different scope keys

artifact_scope_key() hashes (principal_id, session_id, transport_family). The API upload/download handlers store and load with _ArtifactScopeFacade(principal, transport_family=...), i.e. empty session_id. But the broker's _validate_artifact_reference() validates the same artifact with the full ControllerScope, whose session_id is necessarily non-empty because controller registration requires an existing session.

That means the intended end-to-end path cannot compose: an artifact successfully uploaded through /v1/artifacts/upload is principal/family-scoped, then browser_artifact_upload / browser_artifact_download validates it under principal/session/family and gets ArtifactScopeMismatch. The tests currently prove two separate contracts rather than the real journey: the HTTP round-trip never enters broker validation, while the broker test seeds the store directly with a session-bearing _Scope() that already matches _broker_scope().

Required: choose one canonical artifact ownership projection and use it on both sides. If artifacts are session-owned (which matches the broker identity model and the file's own docs), bind the HTTP artifact request to a server-owned session and persist that exact scope. If they are intentionally principal/family-owned, the broker must validate against that same projection and the security/docs need to say so. Please add the missing real regression: authenticated HTTP upload → registered controller scope → broker artifact dispatch (and the reverse download path).

2. The “profile-scoped” artifact store is actually first-profile-wins process state

APIServerAdapter owns a single _browser_control_artifacts. _artifact_store_for(profile) returns that singleton as soon as it exists; only the first call resolves get_profile_dir(profile) and constructs <profile>/artifacts/browser-control. The same first store is then attached process-wide to the global broker.

On a multiplex API listener, profile A touching the artifact route first therefore pins profile B to A's physical artifact root. The principal hash prevents a simple logical cross-read, but the bytes are still written into the wrong profile home and B's broker dispatch is still validating against A's store. This is the same frozen-handle class Hermes already repaired for session storage in merged #88734 (salvaging @jackulau's #88632): a context/profile-sensitive path cannot be resolved once and cached process-wide.

Required: cache stores by resolved profile/home, not once per adapter, and make broker artifact lookup select the correct store from the controller/profile scope rather than holding one global store. Add an A/B multiplex regression proving distinct physical roots and successful broker dispatch for both profiles regardless of which profile touches the endpoint first.

Topology / credit

• #84000 by @SolshineCode is the source feature request; #85351 by @abundantbeing is a substantially broader authenticated implementation rather than a literal native-messaging backend.
• #88203 by @abundantbeing is complementary loopback pairing/token UX on the same api_server.py surface. It is independently mergeable, but whichever lands second needs semantic composition around auth/profile admission; it does not repair the artifact ownership defects above.
• #88734 by @teknium1, preserving @jackulau's #88632 authorship, is the directly relevant per-profile physical-storage invariant: one active profile must not inherit another profile's cached home-bound handle.

CI truth

Exact-head Docker and Nix are green. Hosted CI executed the real matrix: lint, supply-chain, E2E, Windows-only and macOS-only lanes are green. The returned Python slice 9/12 completed 2,595 passed / 1 failed / 14 skipped; the sole failure is the unrelated current image catalog assertion that xai/grok-imagine-image/v2.0/text-to-image defaults upscale=True, in tools/image_generation_tool.py, which this PR does not touch. I am not treating that as a #85351 regression.

Re-review gate: fix the canonical artifact-scope composition and per-profile store ownership, add HTTP→broker and multiplex A/B regressions, compose with #88203 if it lands first, then run a fresh exact-head matrix.

[kshitijk4poor]

Salvaged via #91535 (auto-merge armed) — all six of your commits cherry-picked onto current main with your authorship preserved (rebase merge). This is a genuinely strong feature: the WS auth core (one-shot subprotocol tickets, never-reflected, server-derived principals), the exact-identity broker contract, and the fail-closed capability allowlist all verified clean under review.

We added follow-up commits on top addressing review findings — summarizing so the dispositions are on the record here:

1. Bind at controller registration, not transport auth — the router treated any server-stamped principal as a bound lane, so flag ON + authenticated dashboard/API session + no extension registered lost the legacy browser backend entirely. New broker.lane_registered(): a never-registered lane keeps legacy tools; a registered-but-offline lane stays fail-closed exactly as you designed.
2. Artifact boundary composition (@andrexibiza's blocker 1) — artifact_scope_key hashed session_id, but HTTP uploads carry no session while broker dispatch always does, so the upload→dispatch journey always died with ArtifactScopeMismatch (reproduced first). Ownership is now canonically principal/transport-family; new composition regression is mutation-checked.
3. Per-profile artifact stores (@andrexibiza's blocker 2) — the adapter singleton was first-profile-wins; stores are now cached by resolved profile and the broker selects by ControllerScope.profile_id. A/B multiplex regression added.
4. Event-loop safety — broker lock acquisitions from loop context (WS finally/frame handling) offloaded via asyncio.to_thread so a teardown racing an in-flight command send can't stall the gateway loop.
5. Smaller items: browser.extension_control declared in DEFAULT_CONFIG; broker's TicketInvalid renamed ControllerTicketInvalid (name collision with ws_tickets.TicketInvalid in the same auth flow); server-internal sentinel imported from its canonical definition; flag reads via load_config_readonly(); ticket_expires_at documented as best-effort wall clock vs the broker's monotonic enforcement (@Enough1122's point 1).

Closing this PR in favor of #91535. Thanks for the substantial, well-tested contribution, @abundantbeing — and thanks @andrexibiza and @Enough1122 for reviews that materially improved the merge.

[kshitijk4poor]

Dispositions for the remaining automated-review points (1 and 3 were addressed in #91535 — expiry documented as best-effort wall clock; flag reads switched to the deepcopy-free readonly config path):

• Unthrottled minting / O(n) prune (point 2) — real but bounded: registration requires the server's Bearer API key, tickets expire in 30s, and prune runs under the same lock as mint. A leaked API key already grants far more than ticket-table growth. A per-principal in-flight cap is a sensible hardening follow-up rather than a merge blocker.
• Capability renegotiation vs pending completion (point 4) — traced: attach() rebinds pending.scope to the successor scope, so the new socket's complete() compares against the scope it itself registered — including its (possibly changed) capability set. A changed-capability reconnect therefore completes, not drops. Agreed it deserves an explicit test; noted as follow-up.