Skip to content

fix(computer-use): enforce background-only policy - #14

Merged
BenSheridanEdwards merged 2 commits into
mainfrom
neo/computer-use-background-only-perf-20260821
Aug 21, 2026
Merged

BenSheridanEdwards merged 2 commits into
mainfrom
neo/computer-use-background-only-perf-20260821

Conversation

@BenSheridanEdwards

Copy link
Copy Markdown
Owner

Why does this feature exist?

Native computer-use should not be able to take over an actively used desktop when an agent/profile is configured for background-only operation.

Hermes previously treated background delivery as a default rather than an enforceable policy. Explicit foreground delivery, bring-to-front, or focus_app(..., raise_window=True) could still reach approval and backend dispatch.

What changed?

  • Add computer_use.background_only as a fail-closed policy boundary for foreground intent.
  • Refuse delivery_mode="foreground", bring_to_front=true, and focus_app(..., raise_window=true) before approval and before backend startup.
  • Preserve read-only capture and ordinary background semantic operations.
  • Keep foreground behavior unchanged when the policy is disabled.
  • Read the policy through load_config_readonly() so config remains mtime-aware without deep-copying the full config on every computer-use request.
  • Add regression coverage for foreground refusal, backend/approval non-dispatch, background allowance, and hot-path loader selection.

Behavioural Proof (with video and screenshots)

No visual proof is required for this change. The diff changes dispatcher policy and tests only. It does not alter runtime UI, layout, copy, screenshots, capture rendering, or overlay presentation.

Behavior is proven by deterministic dispatcher tests:

  • all known foreground paths return code: foreground_disabled;
  • backend construction is not called;
  • approval is not requested;
  • background delivery still reaches the noop backend;
  • the gate uses the read-only config loader.

A live foreground demonstration would deliberately perform the behavior this PR is designed to prohibit and would disturb the active desktop, so it is intentionally excluded.

Verification Summary

Focused tests

env -u PYTHONPATH uv run --project . --extra dev python -m pytest tests/tools/test_computer_use_delivery_ladder.py -q
22 passed in 0.53s
env -u PYTHONPATH uv run --project . --extra dev ruff check tools/computer_use/tool.py tests/tools/test_computer_use_delivery_ladder.py
All checks passed!
git diff --check
passed

Broader computer-use suite

env -u PYTHONPATH uv run --project . --extra dev python -m pytest tests/tools/test_computer_use.py -q
72 passed, 1 failed

The sole failure is TestCaptureAppFilterNoMatch::test_linux_default_capture_skips_gnome_shell_helper, a Linux/GNOME window-selection test run on macOS. The exact test fails identically on the feature parent, so it is baseline/platform debt rather than an introduced regression.

Performance gate

100,000 calls per implementation, same process, same cached config:

implementation median p95
initial load_config() gate 369.22 µs 836.50 µs
optimized load_config_readonly() gate 20.54 µs 99.76 µs
raw dictionary lookup control 0.03 µs 0.09 µs

The optimized gate removes 348.68 µs median from the initial implementation, a 94.4% reduction. Its remaining 20.54 µs median overhead is negligible relative to IPC, capture, AX traversal, or browser operations.

Rollout plan

  1. Merge only after hosted checks and fresh-context QA pass.
  2. Deploy to one non-critical Fleet canary with a sealed runtime worktree and rollback receipt.
  3. Verify gateway health, Telegram liveness, read-only capture, background semantic dispatch, foreground refusal, CPU/RSS, latency, and frontmost-app stability.
  4. Expand in waves only if the canary shows no material regression.
  5. Apply profile policy:
computer_use:
  background_only: true
  no_overlay: true
  cua_telemetry: false

No gateway restarts are performed by this PR.

@BenSheridanEdwards
BenSheridanEdwards merged commit 386ad82 into main Aug 21, 2026
37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant