feat(computer_use): follow cua-driver's verify → escalate ladder (#67052) - #67123
Merged
Conversation
Hermes' computer_use wrapper dropped cua-driver's structured action verdicts, exposed no delivery_mode, and injected background-only guidance — so the agent reported unverified no-ops as success and concluded cua-driver 'cannot drive' Electron/Chromium surfaces (observed live on tldraw offline). Fixes #67052. Phase A — preserve the result contract: - ActionResult carries verified/effect/escalation/path/degraded/code/delivery_mode - CuaDriverBackend._action() reads structuredContent (was data-only); a helper normalizes it, additive and None-safe on old drivers - _text_response surfaces the fields additively (ok stays transport-only) Phase B — bounded, model-reachable foreground: - delivery_mode (background|foreground) + bring_to_front on the schema, dispatcher, ABC, and all input methods - foreground is capability-gated (input.delivery_mode); old drivers get a structured foreground_unsupported refusal, never a silent background downgrade - no automatic/hidden foreground retry — the model selects it from the signal Phase C — guidance + isolation: - system prompt (prompt_builder) and bundled skills/computer-use/SKILL.md go from background-ONLY to background-FIRST, teaching the AX→PX→foreground ladder driven by returned effect/escalation, not predicted from the app being Electron - foreground approval scoped by (action, delivery_mode): a background approval never silently authorizes foreground - approval state keyed per session_id so concurrent gateway runs don't leak unlocks Tests: tests/tools/test_computer_use_delivery_ladder.py (15) cover confirmed/ unverifiable/suspected_noop/degraded/old-driver verdicts, delivery_mode gating + foreground_unsupported, and session-scoped foreground approval. Existing 265 computer_use tests still green. Live E2E (real cua-driver 0.8.3 + tldraw offline on Linux/X11): a background click returned effect='unverifiable'/path='ax' (no fabricated success), and a foreground request returned code='foreground_unsupported' — correct on a driver that predates the input.delivery_mode capability.
alt-glitch
added
type/bug
This was referenced Jul 18, 2026
teknium1
added a commit
that referenced
this pull request
Jul 18, 2026
…ging (#67138) Bug 1 of #55048: when the MCP connection dropped (driver crash / restart), _lifecycle_coro exited but left _started=True, so the next list_apps/capture passed _require_started() and then operated on a None session — hanging forever instead of reconnecting. - _lifecycle_coro's finally now resets _started=False on ANY exit, so a dead session is re-enterable (idempotent no-op on the normal stop() path; atomic bool write, safe from the bridge-loop thread without the lock stop() holds). - call_tool() re-enters start() when the session isn't active, rebuilding it before the call. The start_session/end_session handshake (driven by start()/ stop() themselves) is exempted so bootstrap doesn't recurse. Tests: two cases in test_computer_use_delivery_ladder.py — finally resets _started, and call_tool restarts a dead session exactly once. Full computer_use suite green (233). Refs #55048 (Bug 1). Bug 2 (expose foreground dispatch) is covered by the delivery_mode work in #67123.
19 tasks
This was referenced Jul 20, 2026
randlee
pushed a commit
to randlee/hermes-agent
that referenced
this pull request
Aug 11, 2026
…sResearch#67123) Hermes' computer_use wrapper dropped cua-driver's structured action verdicts, exposed no delivery_mode, and injected background-only guidance — so the agent reported unverified no-ops as success and concluded cua-driver 'cannot drive' Electron/Chromium surfaces (observed live on tldraw offline). Fixes NousResearch#67052. Phase A — preserve the result contract: - ActionResult carries verified/effect/escalation/path/degraded/code/delivery_mode - CuaDriverBackend._action() reads structuredContent (was data-only); a helper normalizes it, additive and None-safe on old drivers - _text_response surfaces the fields additively (ok stays transport-only) Phase B — bounded, model-reachable foreground: - delivery_mode (background|foreground) + bring_to_front on the schema, dispatcher, ABC, and all input methods - foreground is capability-gated (input.delivery_mode); old drivers get a structured foreground_unsupported refusal, never a silent background downgrade - no automatic/hidden foreground retry — the model selects it from the signal Phase C — guidance + isolation: - system prompt (prompt_builder) and bundled skills/computer-use/SKILL.md go from background-ONLY to background-FIRST, teaching the AX→PX→foreground ladder driven by returned effect/escalation, not predicted from the app being Electron - foreground approval scoped by (action, delivery_mode): a background approval never silently authorizes foreground - approval state keyed per session_id so concurrent gateway runs don't leak unlocks Tests: tests/tools/test_computer_use_delivery_ladder.py (15) cover confirmed/ unverifiable/suspected_noop/degraded/old-driver verdicts, delivery_mode gating + foreground_unsupported, and session-scoped foreground approval. Existing 265 computer_use tests still green. Live E2E (real cua-driver 0.8.3 + tldraw offline on Linux/X11): a background click returned effect='unverifiable'/path='ax' (no fabricated success), and a foreground request returned code='foreground_unsupported' — correct on a driver that predates the input.delivery_mode capability.
randlee
pushed a commit
to randlee/hermes-agent
that referenced
this pull request
Aug 11, 2026
…ging (NousResearch#67138) Bug 1 of NousResearch#55048: when the MCP connection dropped (driver crash / restart), _lifecycle_coro exited but left _started=True, so the next list_apps/capture passed _require_started() and then operated on a None session — hanging forever instead of reconnecting. - _lifecycle_coro's finally now resets _started=False on ANY exit, so a dead session is re-enterable (idempotent no-op on the normal stop() path; atomic bool write, safe from the bridge-loop thread without the lock stop() holds). - call_tool() re-enters start() when the session isn't active, rebuilding it before the call. The start_session/end_session handshake (driven by start()/ stop() themselves) is exempted so bootstrap doesn't recurse. Tests: two cases in test_computer_use_delivery_ladder.py — finally resets _started, and call_tool restarts a dead session exactly once. Full computer_use suite green (233). Refs NousResearch#55048 (Bug 1). Bug 2 (expose foreground dispatch) is covered by the delivery_mode work in NousResearch#67123.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hermes'
computer_usewrapper now follows cua-driver's documented verify → escalate ladder instead of assuming absolute background operation. The agent no longer reports unverified no-ops as success, and no longer concludes cua-driver "cannot drive" Electron/Chromium — it climbs from background to foreground on a real returned signal. Fixes #67052.Root cause
The wrapper discarded cua-driver's structured action outcomes (
verified/effect/escalation/code), exposed nodelivery_modeto the model, and injected background-only guidance. So a background click that didn't land looked like success, and foreground — cua-driver's own documented fallback — was unreachable. (This is exactly what produced the wrong "CUA can't click Electron, needs xdotool" conclusion during tldraw offline testing.)Changes
Phase A — preserve the result contract
ActionResultcarriesverified,effect,escalation,path,degraded,code,delivery_mode.CuaDriverBackend._action()readsstructuredContent(wasdata-only) via a normalizer that is additive and None-safe on old drivers._text_response()surfaces the fields additively;okstays transport-only, not the semantic verdict.Phase B — bounded, model-reachable foreground
delivery_mode(background|foreground) +bring_to_fronton the schema, dispatcher, ABC, and every input method.input.delivery_mode); an old driver returns a structuredforeground_unsupportedrefusal, never a silent background downgrade.Phase C — guidance + isolation
prompt_builder) and bundledskills/computer-use/SKILL.md: background-only → background-first, teaching the AX→PX→foreground ladder driven by returnedeffect/escalation, not predicted from the app being Electron.(action, delivery_mode)— a background approval never silently authorizes foreground.session_idso concurrent gateway runs don't leak unlocks.Validation
test_computer_use_delivery_ladder.py)effect:"unverifiable",path:"ax"(no fabricated success); foreground request →code:"foreground_unsupported"(correct on a driver predating the capability)Note on the live E2E: the installed driver (0.8.3) does not yet advertise
input.delivery_mode, so the successful foreground-click-lands path (newer driver) was not exercised here — but the capability gate makes the wrapper correct on both driver generations, which the E2E confirms.Infographic