Skip to content

Companion on a real Mac: navigation goals type the address, steer instead of repeating, stale grants by signature (Microphone too) - #96

Merged
jayleaton merged 4 commits into
mainfrom
fix/mac-real-run
Oct 25, 2026
Merged

jayleaton merged 4 commits into
mainfrom
fix/mac-real-run

Conversation

@jayleaton

@jayleaton jayleaton commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

Fixes from the first real-Mac run of the companion (JJ's MacBook, signed build at 4bbd728, "open Chrome and go to example.com"). On main.

Summary

 controller.run(goal "open Chrome and go to example.com", text null)
-  no text → "type" never offered → decider clicks the focused "Address and search bar" → same again → stuck (×3 tasks)
+  text = navigationText(goal) = "example.com"   (the goal's own words, right after the verb; the decider still writes nothing)
+  click chosen on a focused text box → type into it
+  same action again on an unchanged screen → not done; left out of the next question + triedWithoutEffect; stuck only when nothing else fits
 permissions (macOS)
-  stale = macOS says yes && osascript preflight says no  ||  seen in another version@mtime
+  truth = this process's answer (AX trust; Electron's in-process TCC status for Screen Recording, Microphone)
+  stale = Accessibility yes && AX probe fails  ||  missing now && seen working under another signing identity (codesign -d -r-)
+  Microphone in the stale card, with "Reset the Microphone" (tccutil reset Microphone + macOS asks again)
+  Microphone denied (an old build's entry with no recorded history, like JJ's): its Allow button resets it first, then macOS asks

Evidence

  • Before (on JJ's Mac): three computer_use tasks ended stuck with "It kept doing the same thing (clicked the textbox “Address and search bar” containing “https://github.com/jayleaton/t3code/pull/138” (focused)) without the screen changing." Settings showed "Screen Recording kept from an older build" while TCC said ScreenCapture: Allowed (System Set).
  • After (Linux dev box): helper type-check ✓, 138 passed. Companion type-check ✓, lint ✓, 295 passed, build ✓.
  • New tests:
    • A replay of the Chrome run: the focused address bar showing the GitHub URL, and the decider choosing it. The grants now asked for are type "example.com" and then press Enter with the field. This test fails on the old code.
    • Steering: the repeated action is missing from the next question, and triedWithoutEffect is set. After steering, the decider's next pick runs. With nothing left, the task ends stuck.
    • navigationText: URLs, hosts with paths, and no false matches ("open the email from paypal.com…", email addresses).
    • Grants: Screen Recording allowed in-process is granted whatever another process says. A same-certificate update isn't stale. Accessibility on with a failing AX read is stale. A grant missing under the ad-hoc signature is stale for all three permissions, Microphone included. Legacy version@mtime entries never call anything stale.
  • Also kept: the scrolled-list test from Computer use on the Mac: safe typing, stops that end the action, display-aware corners, screen reads #88, which the steering edit had dropped (restored in this PR).
  • Needs the Mac again: the stale card and Microphone reset on JJ's machine, and a rerun of "go to example.com".

Merge Danger

Door: two-way. The settings' grants entries change format. Old entries are ignored, never called stale.

Blast Radius: app. Affects the controller's step choice on every OS, and macOS permission states in Settings.

🤖 Generated with Claude Code

jayleaton and others added 4 commits October 25, 2026 10:10
…g instead of repeating, stale grants by signature (Microphone too)

From the real run at 4bbd728 ("open Chrome and go to example.com"):

1. Three tasks ended stuck clicking Chrome's focused address bar. The goal carried the
   address but `text` was null, so typing was never offered. Now:
   - a navigation goal's address, taken literally right after the verb, is the text to type;
   - a click chosen on a text box that already has the focus types into it;
   - the same action chosen again on an unchanged screen is not done again: it leaves the
     next question and the decider is told it did nothing (triedWithoutEffect); "stuck"
     only when nothing else fits.
2. "Screen Recording kept from an older build" while TCC said Allowed: the check trusted an
   osascript preflight over the app's own. Screen Recording and the Microphone are now what
   macOS answers this process; only Accessibility's AX probe can call a "yes" stale. A
   grant's history is kept by signing identity (designated requirement), not version and
   file time, so an update signed by the same certificate never looks stale.
3. The Microphone joins the stale card, with a reset (tccutil reset Microphone) that asks again.

Checks: apps/desktop-helper type-check, test (138); apps/companion type-check, lint,
test (295), build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… of a fixed sleep

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e dropped

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…OS asks again

A microphone grant never recorded under any signature (JJ's: the ad-hoc build's TCC row) cannot be
told stale from history; macOS reports it denied and never prompts again by itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jayleaton
jayleaton merged commit da9b87e into main Oct 25, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant