chore(cua-driver): add CLI-first agent helper #1426
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||
|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,109 @@ | ||||||||
| #!/usr/bin/env bash | ||||||||
| set -euo pipefail | ||||||||
|
|
||||||||
| CUA_DRIVER_BIN="${CUA_DRIVER_BIN:-cua-driver}" | ||||||||
| SHOT_DIR="${CUA_SCREENSHOT_DIR:-/tmp/cua-screenshots}" | ||||||||
|
|
||||||||
| usage() { | ||||||||
| cat <<'EOF' | ||||||||
| Usage: cua-driver-agent <command> [args] | ||||||||
|
|
||||||||
| CLI-first helper for agentic Cua Driver workflows. Install by putting this | ||||||||
| script on PATH as `cua-driver-agent` or `cua`. | ||||||||
|
|
||||||||
| Commands: | ||||||||
| driver <args...> Pass through to cua-driver | ||||||||
| call <tool> [json] [flags...] Pass through to cua-driver call | ||||||||
| serve Start one headless Cua Driver daemon | ||||||||
| status Show version, config, permissions, process state | ||||||||
| fast Automation mode: cursor off, capture_mode=ax | ||||||||
| demo Visible cursor mode: cursor on, fast motion | ||||||||
| ax | vision | som Set capture_mode | ||||||||
| apps cua-driver call list_apps '{}' | ||||||||
| windows <pid> cua-driver call list_windows '{"pid":pid}' | ||||||||
| state <pid> <window_id> [query] | ||||||||
| AX get_window_state; optional query filter | ||||||||
| state-shot <pid> <window_id> [out.png] | ||||||||
| som get_window_state with screenshot_out_file, then restore ax | ||||||||
| screenshot <window_id> [out.png] | ||||||||
| Window screenshot. Tries CUA, falls back to macOS screencapture -l | ||||||||
| screen [out.png] Full-display screenshot via macOS screencapture | ||||||||
| calc Launch Calculator and print pid/window_id candidates | ||||||||
| EOF | ||||||||
| } | ||||||||
|
|
||||||||
| ensure_dir() { mkdir -p "$(dirname "$1")"; } | ||||||||
| json_escape() { python3 -c 'import json,sys; print(json.dumps(sys.argv[1]))' "$1"; } | ||||||||
|
|
||||||||
| case "${1:-}" in | ||||||||
| ""|-h|--help|help) usage ;; | ||||||||
| driver) shift; exec "$CUA_DRIVER_BIN" "$@" ;; | ||||||||
| call) shift; exec "$CUA_DRIVER_BIN" call "$@" ;; | ||||||||
| serve) open -n -g -a CuaDriver --args serve ;; | ||||||||
| status) | ||||||||
| echo "== version =="; "$CUA_DRIVER_BIN" --version || true | ||||||||
| echo "== cursor =="; "$CUA_DRIVER_BIN" call get_agent_cursor_state '{}' || true | ||||||||
| echo "== config =="; "$CUA_DRIVER_BIN" call get_config '{}' || true | ||||||||
| echo "== permissions =="; "$CUA_DRIVER_BIN" call check_permissions '{}' || true | ||||||||
| echo "== processes ==" | ||||||||
| ps -axo pid,ppid,stat,lstart,command | awk '/CuaDriver|cua-driver/ && !/awk/ {print}' | ||||||||
| ;; | ||||||||
| fast) | ||||||||
| "$CUA_DRIVER_BIN" call set_agent_cursor_enabled '{"enabled":false}' | ||||||||
| "$CUA_DRIVER_BIN" call set_agent_cursor_motion '{"glide_duration_ms":50,"dwell_after_click_ms":0,"idle_hide_ms":1000}' | ||||||||
| "$CUA_DRIVER_BIN" config set capture_mode ax | ||||||||
| ;; | ||||||||
| demo) | ||||||||
| "$CUA_DRIVER_BIN" call set_agent_cursor_enabled '{"enabled":true}' | ||||||||
| "$CUA_DRIVER_BIN" call set_agent_cursor_motion '{"glide_duration_ms":50,"dwell_after_click_ms":0,"idle_hide_ms":1000}' | ||||||||
| ;; | ||||||||
| ax|vision|som) "$CUA_DRIVER_BIN" config set capture_mode "$1" ;; | ||||||||
| apps) "$CUA_DRIVER_BIN" call list_apps '{}' ;; | ||||||||
| windows) | ||||||||
| pid="${2:?usage: cua-driver-agent windows <pid>}" | ||||||||
| "$CUA_DRIVER_BIN" call list_windows "{\"pid\":${pid}}" | ||||||||
| ;; | ||||||||
| state) | ||||||||
| pid="${2:?usage: cua-driver-agent state <pid> <window_id> [query]}" | ||||||||
| wid="${3:?usage: cua-driver-agent state <pid> <window_id> [query]}" | ||||||||
| if [[ $# -ge 4 ]]; then | ||||||||
| q=$(json_escape "$4") | ||||||||
| "$CUA_DRIVER_BIN" call get_window_state "{\"pid\":${pid},\"window_id\":${wid},\"query\":${q}}" | ||||||||
| else | ||||||||
| "$CUA_DRIVER_BIN" call get_window_state "{\"pid\":${pid},\"window_id\":${wid}}" | ||||||||
| fi | ||||||||
| ;; | ||||||||
| state-shot) | ||||||||
| pid="${2:?usage: cua-driver-agent state-shot <pid> <window_id> [out.png]}" | ||||||||
| wid="${3:?usage: cua-driver-agent state-shot <pid> <window_id> [out.png]}" | ||||||||
| out="${4:-$SHOT_DIR/cua-state-${pid}-${wid}-$(date +%s).png}" | ||||||||
| ensure_dir "$out" | ||||||||
| "$CUA_DRIVER_BIN" config set capture_mode som >/dev/null | ||||||||
| cleanup() { "$CUA_DRIVER_BIN" config set capture_mode ax >/dev/null 2>&1 || true; } | ||||||||
| trap cleanup EXIT | ||||||||
| "$CUA_DRIVER_BIN" call get_window_state "{\"pid\":${pid},\"window_id\":${wid},\"screenshot_out_file\":\"${out}\"}" | ||||||||
|
Contributor
There was a problem hiding this comment. User-supplied output path is not JSON-escaped, so paths with Line 84 embeds 🐛 Proposed fix: use
|
||||||||
| "$CUA_DRIVER_BIN" call get_window_state "{\"pid\":${pid},\"window_id\":${wid},\"screenshot_out_file\":\"${out}\"}" | |
| out_json=$(json_escape "$out") | |
| "$CUA_DRIVER_BIN" call get_window_state "{\"pid\":${pid},\"window_id\":${wid},\"screenshot_out_file\":${out_json}}" |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@libs/cua-driver/scripts/agent-cli-helper.sh` at line 84, The get_window_state
invocation embeds the user-supplied `${out}` path into a JSON string without
escaping, so use the existing json_escape helper to produce a safe JSON string;
update the call that runs "$CUA_DRIVER_BIN" call get_window_state
"{\"pid\":${pid},\"window_id\":${wid},\"screenshot_out_file\":\"${out}\"}" to
pass the escaped path (via json_escape) instead of raw ${out}—mirror how the
state command uses json_escape at the earlier invocation to ensure quotes and
backslashes are properly escaped.
Comment on lines
+92
to
+98
Contributor
There was a problem hiding this comment.
timeout is not available on stock macOS, silently bypassing the cua-driver screenshot path entirely.
"macOS does not include the GNU Coreutils timeout command by default." Homebrew installs it prefixed as gtimeout, not timeout.
With stderr redirected to the log file (2>&1), the bash: timeout: command not found error is silently swallowed, test -s "$out" fails (nothing was written), and the else branch runs unconditionally on every stock macOS install. The entire if arm — the primary cua-driver screenshot path — is dead code for users without GNU coreutils.
🐛 Proposed fix: resolve a timeout command at the top of the script
CUA_DRIVER_BIN="${CUA_DRIVER_BIN:-cua-driver}"
SHOT_DIR="${CUA_SCREENSHOT_DIR:-/tmp/cua-screenshots}"
+TIMEOUT_CMD=$(command -v timeout 2>/dev/null || command -v gtimeout 2>/dev/null || echo "")Then in the screenshot case:
- if timeout 20 "$CUA_DRIVER_BIN" call screenshot "{\"window_id\":${wid},\"format\":\"png\"}" --screenshot-out-file "$out" >/tmp/cua-driver-agent-screenshot.log 2>&1 && test -s "$out"; then
+ if [[ -n "$TIMEOUT_CMD" ]] && "$TIMEOUT_CMD" 20 "$CUA_DRIVER_BIN" call screenshot "{\"window_id\":${wid},\"format\":\"png\"}" --screenshot-out-file "$out" >/tmp/cua-driver-agent-screenshot.log 2>&1 && test -s "$out"; then
+ # or if no timeout available, run without it:
+ # if "$CUA_DRIVER_BIN" call screenshot ... >/tmp/cua-driver-agent-screenshot.log 2>&1 && test -s "$out"; thenOr use a portable wrapper:
+_timeout() {
+ local t="$1"; shift
+ if command -v timeout >/dev/null 2>&1; then timeout "$t" "$@"
+ elif command -v gtimeout >/dev/null 2>&1; then gtimeout "$t" "$@"
+ else "$@"
+ fi
+}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@libs/cua-driver/scripts/agent-cli-helper.sh` around lines 92 - 98, The
script's screenshot branch uses the non-portable `timeout` command (in the if
that runs `timeout 20 "$CUA_DRIVER_BIN" call screenshot ...`) which is missing
on stock macOS; add a small detection at script startup to set a TIMEOUT_CMD
variable to the available binary (prefer gtimeout if present, then timeout, else
empty) and then replace the hardcoded `timeout` invocation in the screenshot
if-condition with "${TIMEOUT_CMD:+$TIMEOUT_CMD 20 }" or equivalent so the script
falls back gracefully to calling "$CUA_DRIVER_BIN" without timeout when neither
`gtimeout` nor `timeout` exist; update the if-condition that invokes
`"$CUA_DRIVER_BIN" call screenshot` to use this TIMEOUT_CMD variable so the
primary cua-driver path isn't silently skipped on macOS.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
state-shotcleanup unconditionally clobbers any pre-existing capture mode toax.If the caller was in
somorvisionmode before invokingstate-shot, thetrap cleanup EXITsilently resets their config toax. The fix is to save the current mode before switching and restore it on cleanup.🐛 Proposed fix: save and restore the previous capture mode
📝 Committable suggestion
🤖 Prompt for AI Agents