docs(skills): document reference images for act and assert - #41
Draft
codeestX wants to merge 1 commit into
Draft
Conversation
`tap --locate` used to be the only way to target a reference image, so every platform skill told the agent to prefer it "instead of a generic act --prompt". `act` now accepts reference images as well, which makes that framing steer agents away from the capability. For all five platform skills (android, browser, computer, harmony, ios): - keep `tap --locate` as the recommendation for a single tap on a visual target - document `--image` / `--image-name` on `act` and `assert` for multi-step actions and image-based assertions - note that the pairs must match and that the prompt should refer to the image by its name, which is how the model connects wording to picture Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
codeestX
force-pushed
the
feat/act-reference-images
branch
from
July 28, 2026 03:53
482e06b to
2ce89f0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Every platform skill told the agent to prefer
tap --locateinstead of a genericact --promptwhen a reference image is provided. That framing existed becauseactcould not take images. It now can, so the wording actively steers agents away from the capability.Updated all five platform skills —
android-automation,browser,computer-automation,harmony-automation,ios-automation— with the same three edits each:tap --locateis now recommended for a single tap on the target, rather than "instead ofact --prompt".actexample using--image/--image-name/--convert-http-image2-base64.tap --locate; multi-step action or image-based assertion →--image/--image-nameonactorassert.vitest-midscene-e2eis untouched — it documents SDK usage, not the CLI.Why all five
The capability comes from the shared
generateCommonToolsin@midscene/shared, and all five platform packages extendBaseMidsceneTools, soact --imageworks on every platform at once.Notes
The docs spell out two things that are easy to get wrong and are enforced by the CLI:
--imageneeds a matching--image-name(mismatched counts throw), andthis is the reference image named '<name>', so the name is what connects the wording to the picture.assert --imagehas worked since v1.8; onlyactis new.🤖 Generated with Claude Code