Skip to content

docs(skills): document reference images for act and assert - #41

Draft
codeestX wants to merge 1 commit into
web-infra-dev:mainfrom
codeestX:feat/act-reference-images
Draft

docs(skills): document reference images for act and assert#41
codeestX wants to merge 1 commit into
web-infra-dev:mainfrom
codeestX:feat/act-reference-images

Conversation

@codeestX

Copy link
Copy Markdown

Draft — blocked on a midscene release. act --image is not published yet: it lands in web-infra-dev/midscene#2871, and the newest release is 1.10.7. These skills run npx -y @midscene/<pkg>@1, and the midscene CLI validates options with .strict(), so an unreleased flag fails the whole command (Unknown option "--image") rather than being ignored. Ready to merge once a release includes it — no changes to this PR needed.

What

Every platform skill told the agent to prefer tap --locate instead of a generic act --prompt when a reference image is provided. That framing existed because act could not take images. It now can, so the wording actively steers agents away from the capability.

Updated all five platform skills — android-automation, browser, computer-automation, harmony-automation, ios-automation — with the same three edits each:

  1. Section prose: tap --locate is now recommended for a single tap on the target, rather than "instead of act --prompt".
  2. New act example using --image / --image-name / --convert-http-image2-base64.
  3. The numbered rule: single tap → tap --locate; multi-step action or image-based assertion → --image / --image-name on act or assert.

vitest-midscene-e2e is untouched — it documents SDK usage, not the CLI.

Why all five

The capability comes from the shared generateCommonTools in @midscene/shared, and all five platform packages extend BaseMidsceneTools, so act --image works on every platform at once.

Notes

The docs spell out two things that are easy to get wrong and are enforced by the CLI:

  • every --image needs a matching --image-name (mismatched counts throw), and
  • the prompt should refer to the image by that name — the model receives each image labelled this is the reference image named '<name>', so the name is what connects the wording to the picture.

assert --image has worked since v1.8; only act is new.

🤖 Generated with Claude Code

`tap --locate` used to be the only way to target a reference image, so every
platform skill told the agent to prefer it "instead of a generic act --prompt".
`act` now accepts reference images as well, which makes that framing steer
agents away from the capability.

For all five platform skills (android, browser, computer, harmony, ios):

- keep `tap --locate` as the recommendation for a single tap on a visual target
- document `--image` / `--image-name` on `act` and `assert` for multi-step
  actions and image-based assertions
- note that the pairs must match and that the prompt should refer to the image
  by its name, which is how the model connects wording to picture

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@codeestX
codeestX force-pushed the feat/act-reference-images branch from 482e06b to 2ce89f0 Compare July 28, 2026 03:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant