feat(shared): support reference images in the act CLI tool - #2871
Merged
Conversation
The `act` common tool accepted only a text prompt, while `assert` already exposed `--image` / `--image-name` / `--convert-http-image2-base64`. Core `aiAct`/`aiAction` already forwards reference images to the planner (TaskExecutor.action -> planImpl(referenceImageMessages)), so this only wires the same multimodal entry point into the act tool: - add `promptInputExtraSchema` to the act tool schema - compose a `UserPromptLike` via `composeUserPrompt` before calling `aiAction` - widen `BaseAgent.aiAction` to accept `UserPromptLike` to match Image references support local paths, base64 and remote URLs, mirroring the existing `assert` behavior. Plan-cache correctness is preserved because the cache key is `JSON.stringify(prompt)`, which already includes image name/url. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`aiAct` already forwards reference images to the planner, and the CLI `act` tool now exposes the same image flags, but the docs only listed image prompting for the locate / insight / assert methods. Add aiAct to the image-prompting docs so site readers can discover it: - api.mdx: widen the aiAct signature to `string | Object`, link its `prompt` param to "Prompting with images", and add an aiAct example. - automate-with-scripts-in-yaml.mdx: document the `aiAct` object form with `images` / `convertHttpImage2Base64`, already covered by yaml-doc-usage.test.ts. Bilingual (en + zh). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Deploying midscene with
|
| Latest commit: |
8ec1cbb
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://5ed3c54c.midscene.pages.dev |
| Branch Preview URL: | https://feat-cli-act-image-prompt.midscene.pages.dev |
…ompt # Conflicts: # apps/site/docs/en/api.mdx # apps/site/docs/zh/api.mdx
The main branch consolidated the API docs from api.mdx into reference/index.mdx after this branch first documented aiAct image prompting. Re-apply the same change on the new file: widen the aiAct signature to `string | object`, link its `prompt` param to "Prompting with images", and add an aiAct image example. Bilingual (en + zh). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
aiAct attaches images with the same object form as the insight steps (`prompt` + `images` under the action key), so a separate example was redundant. List aiAct (and its `ai` shorthand) in the existing sentence — kept distinct from the "insight steps" wording since aiAct is an action, not a Q&A step — and drop the duplicate example. Bilingual (en + zh). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
quanru
approved these changes
Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds reference-image input to the
actcommon tool, so platform CLIs(
@midscene/web,@midscene/android,@midscene/ios, …) can pass images toactthe same way they already can toassert:npx @midscene/web act \ --url https://example.com \ --prompt "tap the icon that matches the reference image" \ --image ./target.png \ --image-name target--image/--image-namemay be repeated (counts must match) and remote URLscan be inlined via
--convert-http-image2-base64 true.Why
actwas the odd one out: onlyassertexposed the image flags at the CLIsurface, even though core already supports multimodal prompts everywhere.
aiAct/aiActionaccept aTUserPromptand forward reference images to theplanner (
TaskExecutor.action→planImpl({ referenceImageMessages }),packages/core/src/agent/tasks.ts). So this is purely CLI wiring — no corechange was needed.
How
promptInputExtraSchema(image/imageName/convertHttpImage2Base64)to the
acttool schemaUserPromptLikeviacomposeUserPrompt(...)before callingaiAction, mirroringassertBaseAgent.aiActionto acceptUserPromptLikePlan-cache correctness is preserved: the plan cache key is
JSON.stringify(prompt), which already incorporates each image name/url, so"same text, different image" does not hit a stale cached plan.
aiWaitForintentionally stays string-only (no multimodal entry point yet); thestale comment in
types.tsis updated accordingly.Validation
npx nx test @midscene/shared -- tests/unit-test/tool-generator.test.ts— 45 passed(new
generateCommonTools — act image promptscases: forwards images as aTUserPrompt object, forwards a local-path url verbatim, no-image prompt stays a
bare string; updated schema assertions)
npx nx build @midscene/shared— passed with the type checker enablednpx nx build @midscene/android— passed (downstream consumer of the widenedBaseAgent.aiActiontype)lint-staged(biome) — passed on the changed files🤖 Generated with Claude Code