feat(shared): support reference images in the act CLI tool

apps/site/docs/en/automate-with-scripts-in-yaml.mdx

@@ -768,7 +768,7 @@ tasks:
 
 > The legacy nested format (where `locate` is indented under the action key, e.g. `aiTap: \n    locate: ...`) is still supported but not recommended.
 
-For insight steps like `aiAsk`, `aiQuery`, `aiBoolean`, `aiNumber`, `aiString`, and `aiAssert`, you can set the `prompt` and `images` fields directly.
+For `aiAct` (and its `ai` shorthand), and for insight steps like `aiAsk`, `aiQuery`, `aiBoolean`, `aiNumber`, `aiString`, and `aiAssert`, you can set the `prompt` and `images` fields directly under the action key.
 
 ```yaml
 tasks:

apps/site/docs/en/reference/index.mdx

@@ -130,7 +130,7 @@ This method was previously named `aiAction()` in earlier versions. The current v
 
 ```typescript
 function aiAct(
-  prompt: string,
+  prompt: string | object,
   options?: {
     cacheable?: boolean;
     deepThink?: 'unset' | true | false;
@@ -144,7 +144,7 @@ function ai(prompt: string): Promise<string | undefined>; // shorthand form
 
 - Parameters:
 
-  - `prompt: string` — A natural language description of the UI steps.
+  - `prompt: string | object` — A natural language description of the UI steps, or [prompting with images](#prompting-with-images).
   - `options?: object` — Optional configuration:
     - `cacheable?: boolean` — Whether this call can use the [cache](../caching.mdx). Default: `true`.
     - `deepThink?: 'unset' | true | false` — Controls the planning implementation used by Midscene when `aiAct` performs planning. When enabled, `aiAct` focuses more on task decomposition and separates task planning from UI element locating into different model calls. For compatibility, `'unset'` is accepted and treated the same as `false`. [Learn more about deepThink](../model-strategy#about-the-deepthink-option-in-aiact).
@@ -1502,6 +1502,20 @@ await agent.aiAssert({
 });
 ```
 
+- Example 3: use images to guide an action (`aiAct`).
+
+```javascript
+await agent.aiAct({
+  prompt: 'Tap the icon that matches the reference logo',
+  images: [
+    {
+      name: 'The specific logo',
+      url: 'https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png',
+    },
+  ],
+});
+```
+
 **Image size**
 
 Follow your model provider's image size and dimension limits. Oversized or very small images may be rejected. Check the provider documentation for exact limits.

apps/site/docs/zh/automate-with-scripts-in-yaml.mdx

@@ -776,7 +776,7 @@ tasks:
 
 > 旧的嵌套写法（将 `locate` 缩进到操作指令内部，例如 `aiTap: \n    locate: ...`）仍然支持，但不推荐使用。
 
-对于视觉问答类步骤，例如 `aiAsk`、`aiQuery`、`aiBoolean`、`aiNumber`、`aiString`、`aiAssert`，可以直接设置 `prompt` 和 `images` 字段。
+对于 `aiAct`（及其简写 `ai`），以及 `aiAsk`、`aiQuery`、`aiBoolean`、`aiNumber`、`aiString`、`aiAssert` 等视觉问答类步骤，都可以直接在操作指令下设置 `prompt` 和 `images` 字段。
 
 ```yaml
 tasks:

apps/site/docs/zh/reference/index.mdx

@@ -128,7 +128,7 @@ Midscene 针对每个不同环境都有对应的 Agent。每个 Agent 的构造
 
 ```typescript
 function aiAct(
-  prompt: string,
+  prompt: string | object,
   options?: {
     cacheable?: boolean;
     deepThink?: 'unset' | true | false;
@@ -142,7 +142,7 @@ function ai(prompt: string): Promise<string | undefined>; // 简写形式
 
 - 参数：
 
-  - `prompt: string` - 用自然语言描述的操作内容
+  - `prompt: string | object` - 用自然语言描述的操作内容，或[使用图片作为提示词](#使用图片作为提示词)。
   - `options?: object` - 可选，一个配置对象，包含：
     - `cacheable?: boolean` - 当启用 [缓存功能](../caching.mdx) 时，是否允许缓存当前 API 调用结果。默认值为 true
     - `deepThink?: 'unset' | true | false` - 控制 Midscene 在 `aiAct` 执行规划时的具体实现。开启后，`aiAct` 会更注重任务拆解，并将任务 Planning 和 UI 元素定位拆解为不同的模型调用。为了兼容旧写法，`'unset'` 仍然可以传入，并会被按 `false` 处理。[详情参阅 deepThink 说明](../model-strategy#关于-aiact-方法的-deepthink-参数)。
@@ -1476,6 +1476,20 @@ await agent.aiAssert({
 });
 ```
 
+- 示例三：使用图片引导操作（`aiAct`）
+
+```javascript
+await agent.aiAct({
+  prompt: '点击与参考 logo 一致的图标',
+  images: [
+    {
+      name: '指定 logo',
+      url: 'https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png',
+    },
+  ],
+});
+```
+
 **图片尺寸的注意事项**
 
 请遵守模型提供商对图片体积和尺寸的限制。过大或过小的图片都可能被拒绝，准确限制请以模型提供商的文档为准。

packages/shared/src/agent-tools/tool-generator.ts

@@ -712,6 +712,7 @@ export function generateCommonTools(
           .describe(
             'Plan this action with deep thinking (richer context and sub-goal decomposition). Helps with complex multi-step instructions at the cost of speed. Defaults to the server --deep-think setting.',
           ),
+        ...promptInputExtraSchema,
         ...initArgSchema,
       },
       cli: mergeToolCliMetadata(undefined, initArgCliMetadata),
@@ -744,7 +745,13 @@ export function generateCommonTools(
             if (args.deepThink !== undefined) {
               actOptions.deepThink = args.deepThink;
             }
-            const result = await agent.aiAction(prompt, actOptions);
+            const userPrompt = composeUserPrompt({
+              prompt,
+              image: args.image,
+              imageName: args.imageName,
+              convertHttpImage2Base64: args.convertHttpImage2Base64,
+            });
+            const result = await agent.aiAction(userPrompt, actOptions);
             return await captureScreenshotResult(agent, 'act', result);
           } finally {
             unsubscribeVerbose();

packages/shared/src/agent-tools/types.ts

@@ -84,9 +84,10 @@ export interface ActionSpaceItem {
  * Structural shape compatible with @midscene/core `TUserPrompt`.
  * Declared locally to avoid a circular dep on `@midscene/core` from `@midscene/shared`.
  *
- * Currently consumed only by the `assert` tool in `generateCommonTools`.
- * `aiAction` and `aiWaitFor` stay string-only at the CLI surface because the
- * tools generator does not yet expose multimodal entry points for them.
+ * Consumed by the `assert` and `act` tools in `generateCommonTools`, both of
+ * which forward reference images to core (`aiAssert` / `aiAct`). `aiWaitFor`
+ * stays string-only at the CLI surface because the tools generator does not
+ * yet expose a multimodal entry point for it.
  */
 export type UserPromptLike =
   | string
@@ -148,7 +149,7 @@ export interface BaseAgent {
     params?: unknown,
   ) => Promise<unknown>;
   aiAction?: (
-    description: string,
+    description: UserPromptLike,
     params?: Record<string, unknown>,
   ) => Promise<unknown>;
   aiWaitFor?: (

packages/shared/tests/unit-test/tool-generator.test.ts

@@ -620,15 +620,88 @@ describe('generateCommonTools — assert image prompts', () => {
     expect(assertSchema).not.toHaveProperty('images');
     expect(assertSchema).not.toHaveProperty('imageFiles');
 
-    // act schema stays string-only because the underlying core aiAct
-    // does not yet parse multimodal prompts.
+    // act mirrors assert: it exposes the same reference-image flags, which
+    // core aiAct forwards to the planner as reference images.
     const actSchema = tools.find((t) => t.name === 'act')!.schema;
     expect(actSchema).toHaveProperty('prompt');
+    expect(actSchema).toHaveProperty('image');
+    expect(actSchema).toHaveProperty('imageName');
+    expect(actSchema).toHaveProperty('convertHttpImage2Base64');
     expect(actSchema).not.toHaveProperty('images');
     expect(actSchema).not.toHaveProperty('imageFiles');
   });
 });
 
+describe('generateCommonTools — act image prompts', () => {
+  const screenshotBase64 = 'data:image/png;base64,Zm9v';
+
+  it('passes the prompt through unchanged when no images are supplied', async () => {
+    const aiAction = vi.fn().mockResolvedValue(undefined);
+    const tools = generateCommonTools(async () => ({
+      aiAction,
+      getActionSpace: vi.fn().mockResolvedValue([]),
+      page: { screenshotBase64: vi.fn().mockResolvedValue(screenshotBase64) },
+    }));
+
+    const act = tools.find((t) => t.name === 'act')!;
+    await act.handler({ prompt: 'click the login button' });
+
+    expect(aiAction).toHaveBeenCalledWith('click the login button', {
+      deepThink: false,
+    });
+  });
+
+  it('forwards images to aiAction as a TUserPrompt-style object', async () => {
+    const aiAction = vi.fn().mockResolvedValue(undefined);
+    const tools = generateCommonTools(async () => ({
+      aiAction,
+      getActionSpace: vi.fn().mockResolvedValue([]),
+      page: { screenshotBase64: vi.fn().mockResolvedValue(screenshotBase64) },
+    }));
+
+    const act = tools.find((t) => t.name === 'act')!;
+    await act.handler({
+      prompt: 'tap the icon that matches the reference image',
+      image: 'https://example.com/icon.png',
+      imageName: 'target',
+    });
+
+    expect(aiAction).toHaveBeenCalledWith(
+      {
+        prompt: 'tap the icon that matches the reference image',
+        images: [{ name: 'target', url: 'https://example.com/icon.png' }],
+      },
+      { deepThink: false },
+    );
+  });
+
+  it('forwards a local-path url verbatim so core can resolve it', async () => {
+    const aiAction = vi.fn().mockResolvedValue(undefined);
+    const tools = generateCommonTools(async () => ({
+      aiAction,
+      getActionSpace: vi.fn().mockResolvedValue([]),
+      page: { screenshotBase64: vi.fn().mockResolvedValue(screenshotBase64) },
+    }));
+
+    const act = tools.find((t) => t.name === 'act')!;
+    await act.handler({
+      prompt: 'tap the icon that matches the supplied image',
+      image: './fixtures/icon.png',
+      imageName: 'icon',
+      convertHttpImage2Base64: true,
+    });
+
+    expect(aiAction).toHaveBeenCalledWith(
+      {
+        prompt: 'tap the icon that matches the supplied image',
+        images: [{ name: 'icon', url: './fixtures/icon.png' }],
+        convertHttpImage2Base64: true,
+      },
+      { deepThink: false },
+    );
+  });
+});
+
 describe('toolDefaults (deep locate / deep think)', () => {
   it('defaults locate.deepLocate to true for action tools when enabled', async () => {
     const callActionInActionSpace = vi.fn().mockResolvedValue(undefined);