Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/btw-image-support.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@moonshot-ai/kimi-code": minor
---

Add image support to /btw: pasted images now reach the side agent, which can also view image and video files.
5 changes: 5 additions & 0 deletions .changeset/btw-readonly-tools.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@moonshot-ai/kimi-code": minor
---

Add read-only tools to the /btw side agent.
79 changes: 65 additions & 14 deletions apps/kimi-code/src/tui/controllers/btw-panel.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ import { Spacer } from '@moonshot-ai/pi-tui';
import type {
Event,
KimiHarness,
PromptInput,
Session,
TurnEndedEvent,
} from '@moonshot-ai/kimi-code-sdk';
Expand All @@ -14,14 +15,40 @@ import { createMarkdownTheme } from '../theme/pi-tui-theme';
import type { InlineSkillActivation } from '../types';
import type { TUIState } from '../tui-state';

import type { StagingLease } from './staging-leases';

const BTW_BUSY_NOTICE = 'Wait for /btw to finish before sending another question.';

export interface BtwPreparedPrompt {
/** Media-expanded RPC input; undefined means send the plain text prompt. */
readonly input?: PromptInput;
/** Staged-media lease and its exact-binding submission id (plain prompts only). */
readonly lease?: StagingLease;
readonly submissionId?: string;
}

export interface BtwPanelHost {
state: TUIState;
session: Session | undefined;
readonly harness: KimiHarness;

showError(msg: string): void;
/**
* Expand pasted image/video placeholders into daemon file-ref parts for the
* side agent (the /btw counterpart of the main send path's media
* preparation). Returns undefined when preparation failed — the error was
* already shown.
*/
prepareBtwPrompt(
text: string,
opts: { readonly stage: boolean },
): Promise<BtwPreparedPrompt | undefined>;
/** Track a prompt dispatch carrying staged media; see StagingLeaseTracker.trackDispatch. */
trackBtwDispatch(
lease: StagingLease | undefined,
request: Promise<unknown>,
onError: (error: unknown) => void,
): void;
}

export class BtwPanelController {
Expand Down Expand Up @@ -180,27 +207,51 @@ export class BtwPanelController {
panel: BtwPanelComponent,
inlineSkillActivations?: readonly InlineSkillActivation[],
): void {
void this.prepareAndPrompt(agentId, prompt, panel, inlineSkillActivations);
}

private async prepareAndPrompt(
agentId: string,
prompt: string,
panel: BtwPanelComponent,
inlineSkillActivations?: readonly InlineSkillActivation[],
): Promise<void> {
const session = this.host.session;
if (session === undefined) {
panel.markFailed(NO_ACTIVE_SESSION_MESSAGE);
this.host.state.ui.requestRender();
return;
}
const send =
inlineSkillActivations !== undefined && inlineSkillActivations.length > 0
? () =>
session.promptWithSkills(
prompt,
inlineSkillActivations.map((activation) => ({
name: activation.skillName,
args: activation.args,
})),
)
: () => session.prompt(prompt);
void this.withInteractiveAgent(agentId, send).catch((error: unknown) => {
panel.markFailed(`Failed to send /btw prompt: ${formatErrorMessage(error)}`);
const useSkills = inlineSkillActivations !== undefined && inlineSkillActivations.length > 0;
// Skill bundles have no prompt-id channel, so they match the main turn's
// inline-skill path: media rides along without a staged lease.
const prepared = await this.host.prepareBtwPrompt(prompt, { stage: !useSkills });

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Abort preparation when the BTW panel is closed

If a just-pasted image or video is still being ingested, this await can last up to two seconds. During that interval, Esc or Ctrl-C closes/unregisters the panel and calls session.cancel() while the child is still idle, but the continuation never checks whether the panel remains active and subsequently dispatches the prompt anyway. This produces an invisible side-agent request after the user explicitly canceled it (and can overlap a newly opened /btw panel); cancellation or panel identity should be rechecked before dispatch, with any prepared lease released on abandonment.

Useful? React with 👍 / 👎.

if (prepared === undefined) {
panel.markFailed('Failed to prepare the media attachment.');
this.host.state.ui.requestRender();
});
return;
}
const input = prepared.input ?? prompt;
const send = useSkills
? () =>
session.promptWithSkills(
input,
inlineSkillActivations.map((activation) => ({
name: activation.skillName,
args: activation.args,
})),
)
: prepared.submissionId !== undefined
? () => session.prompt(input, { promptId: prepared.submissionId })
: () => session.prompt(input);
this.host.trackBtwDispatch(
prepared.lease,
this.withInteractiveAgent(agentId, send),
Comment on lines +247 to +249

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Route side-agent turn events through the lease tracker

When an actual /btw turn starts or ends, its agentId is the side-agent ID, so SessionEventHandler.handleEvent returns through routeChildAgentEvent before invoking the host's staging handleTurnStarted or handleTurnEnded hooks. Consequently, the lease tracked here is never bound or released after successful side prompts, and each pasted image/video remains retained in daemon staging until the session is closed or its TTL expires. The new test misses this because emitTurn emits both events for agentId: 'main'; the side-agent routing path must notify the staging tracker as well.

Useful? React with 👍 / 👎.

(error: unknown) => {
panel.markFailed(`Failed to send /btw prompt: ${formatErrorMessage(error)}`);
this.host.state.ui.requestRender();
},
);
}

private async cancelAgent(agentId: string): Promise<void> {
Expand Down
62 changes: 61 additions & 1 deletion apps/kimi-code/src/tui/kimi-tui.ts
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ import { MEDIA_INGESTION_SUBMIT_WAIT_MS } from './constant/media';
import { CHROME_GUTTER } from './constant/rendering';
import { MAX_TERMINAL_TITLE_LENGTH } from './constant/terminal';
import { AuthFlowController } from './controllers/auth-flow';
import { BtwPanelController } from './controllers/btw-panel';
import { BtwPanelController, type BtwPreparedPrompt } from './controllers/btw-panel';
import { ClipboardImageHintController } from './controllers/clipboard-image-hint';
import { EditorKeyboardController } from './controllers/editor-keyboard';
import { SessionEventHandler } from './controllers/session-event-handler';
Expand Down Expand Up @@ -1521,6 +1521,66 @@ export class KimiTUI {
});
}

/**
* /btw counterpart of the send path's media preparation (see
* sendNormalUserInput): expands pasted image/video placeholders into daemon
* file-ref parts for the side agent. The side panel has no queue or
* cache-hint interception, so this stops at extraction + validation; staged
* prompts also get a media lease with an exact-binding submission id, while
* unstaged ones (skill bundles) match the main inline-skill path.
*/
async prepareBtwPrompt(
text: string,
opts: { readonly stage: boolean },
): Promise<BtwPreparedPrompt | undefined> {
const ingestionWait = pendingMediaIngestions(
text,
this.imageStore,
MEDIA_INGESTION_SUBMIT_WAIT_MS,
);
if (ingestionWait !== undefined) await ingestionWait;
let extraction: ReturnType<typeof extractMediaAttachments>;
try {
extraction = extractMediaAttachments(text, this.imageStore);
} catch (error) {
this.showError(`Failed to prepare media attachment: ${formatErrorMessage(error)}`);
return undefined;
}
if (!extraction.hasMedia) return {};
const stagingLease = opts.stage
? this.staging.create(
// One retain per unique id per extraction, exactly like the main
// send path's lease (see sendNormalUserInput).
[...new Set([...extraction.imageAttachmentIds, ...extraction.videoAttachmentIds])],
[],
'user',
randomUUID(),
)
: undefined;
if (!this.validateMediaCapabilities(extraction)) {
this.staging.release(stagingLease);
return undefined;
}
return {
input: resolveOriginalCaptions(
extraction.parts,
extraction.imageAttachmentIds,
this.imageStore,
originalsDirForSession(this.session),
),
lease: stagingLease,
submissionId: stagingLease?.submissionId,
};
}

trackBtwDispatch(
lease: StagingLease | undefined,
request: Promise<unknown>,
onError: (error: unknown) => void,
): void {
this.staging.trackDispatch(lease, request, onError);
}

validateMediaCapabilities(extraction: {
hasMedia: boolean;
imageAttachmentIds: readonly number[];
Expand Down
50 changes: 50 additions & 0 deletions apps/kimi-code/test/tui/kimi-tui-message-flow.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4708,6 +4708,56 @@ command = "vim"
expect(stripSgr(renderBtwPanel(driver))).toContain('Q: What are you working on right now?');
});

it('sends a pasted image in the initial /btw prompt as daemon file-ref parts', async () => {
const session = makeSession();
const { driver, harness } = await makeDriver(session);
const imageStore = (driver as unknown as { imageStore: ImageAttachmentStore }).imageStore;
const attachment = stagedImage(imageStore, 'file-btw');

driver.handleUserInput(`/btw describe ${attachment.placeholder}`);

await vi.waitFor(() => {
expect(session.prompt).toHaveBeenCalledWith(
[
{ type: 'text', text: 'describe ' },
{ type: 'image_url', imageUrl: { url: 'kimi-file://file-btw' } },
],
{ promptId: expect.any(String) },
);
});
expect(harness.deleteFile).not.toHaveBeenCalled();
emitTurn(driver, 1, () => {
expect(harness.deleteFile).not.toHaveBeenCalled();
});
await vi.waitFor(() => {
expect(harness.deleteFile).toHaveBeenCalledWith('file-btw');
});
});

it('sends a pasted image in follow-up /btw panel input as daemon file-ref parts', async () => {
const session = makeSession();
const { driver } = await makeDriver(session);
driver.handleUserInput('/btw');
await vi.waitFor(() => {
expect(session.startBtw).toHaveBeenCalled();
expect(driver.state.btwPanelContainer.children).toHaveLength(2);
});
const imageStore = (driver as unknown as { imageStore: ImageAttachmentStore }).imageStore;
const attachment = stagedImage(imageStore, 'file-btw-2');

driver.handleUserInput(`look at ${attachment.placeholder}`);

await vi.waitFor(() => {
expect(session.prompt).toHaveBeenCalledWith(
[
{ type: 'text', text: 'look at ' },
{ type: 'image_url', imageUrl: { url: 'kimi-file://file-btw-2' } },
],
{ promptId: expect.any(String) },
);
});
});

it('sends /btw panel input with inline skills via promptWithSkills (v2 engine)', async () => {
const session = makeSession({
id: 'ses-lazy',
Expand Down
2 changes: 1 addition & 1 deletion docs/en/reference/server-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -751,7 +751,7 @@ On success, `data` is `{ aborted: true }`.

#### `POST /api/v1/sessions/{session_id}:btw`

Starts a "by the way" side conversation: forks the main agent into a child agent whose tool calls are disabled, so quick side questions run in isolation without touching the working context. Requires a usable model configuration.
Starts a "by the way" side conversation: forks the main agent into a child agent whose tool calls are limited to the read-only tools `Read`, `Grep`, `Glob`, and `ReadMediaFile`, so quick side questions run in isolation without touching the working context. Requires a usable model configuration.

On success, `data` is `{ agent_id }` — the id of the new child agent.

Expand Down
2 changes: 1 addition & 1 deletion docs/zh/reference/server-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -751,7 +751,7 @@ schema 还接受 `agent_config` 内的 `system_prompt`、`tools`、`mcp_servers`

#### `POST /api/v1/sessions/{session_id}:btw`

开启一个 `"by the way"` 旁路对话:把 main agent fork 成一个禁用工具调用的子 Agent,让快速的临时问题在隔离环境中运行,不触碰工作上下文。需要可用的模型配置。
开启一个 `"by the way"` 旁路对话:把 main agent fork 成一个仅可使用只读工具(`Read`、`Grep`、`Glob`、`ReadMediaFile`)的子 Agent,让快速的临时问题在隔离环境中运行,不触碰工作上下文。需要可用的模型配置。

成功时,`data` 为 `{ agent_id }`——新子 Agent 的 id。

Expand Down
19 changes: 12 additions & 7 deletions packages/agent-core-v2/src/features/btw/btw.ts
Original file line number Diff line number Diff line change
@@ -1,19 +1,24 @@
import { createDecorator, type ServiceIdentifier } from '#/_base/di/instantiation';

export const BTW_READONLY_TOOLS = new Set(['Read', 'Grep', 'Glob', 'ReadMediaFile']);

export const TOOL_CALL_DISABLED_MESSAGE =
'Tool calls are disabled for side questions. Answer with text only.';
'Only the read-only tools Read, Grep, Glob, and ReadMediaFile are available for side questions. Other tool calls are disabled.';

export const SIDE_QUESTION_SYSTEM_REMINDER = `
This is a side-channel conversation with the user. You should answer user questions directly based on what you already know.
This is a side-channel conversation with the user. You should answer user questions directly.

IMPORTANT:
- You are a separate, lightweight instance.
- The main agent continues independently; do not reference being interrupted.
- Do not call any tools. All tool calls are disabled and will be rejected.
Even though tool definitions are visible in this request, they exist only
for technical reasons (prompt cache). You must not use them.
- Respond only with text based on what you already know from the conversation
and this side-channel conversation.
- You may use the read-only tools Read, Grep, and Glob to inspect files when
the answer depends on current file contents, and ReadMediaFile to view an
image or video file (including images the user attached to their message).
All other tools are disabled and will be rejected, even though their
definitions are visible in this request (they exist only for technical
reasons — prompt cache).
- Prefer answering from what you already know from the conversation and this
side-channel conversation; reach for the read-only tools only when needed.
- Follow-up turns may happen in this side-channel conversation.
- If you do not know the answer, say so directly.
`.trim();
Expand Down
11 changes: 9 additions & 2 deletions packages/agent-core-v2/src/features/btw/btwService.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,12 @@ import { IAgentScopeContext } from '#/agent/scopeContext/scopeContext';
import { ErrorCodes, Error2 } from '#/errors';
import { IAgentLifecycleService, MAIN_AGENT_ID } from '#/session/agentLifecycle/agentLifecycle';

import { ISessionBtwService, SIDE_QUESTION_SYSTEM_REMINDER, TOOL_CALL_DISABLED_MESSAGE } from './btw';
import {
BTW_READONLY_TOOLS,
ISessionBtwService,
SIDE_QUESTION_SYSTEM_REMINDER,
TOOL_CALL_DISABLED_MESSAGE,
} from './btw';

export class SessionBtwService implements ISessionBtwService {
declare readonly _serviceBrand: undefined;
Expand All @@ -32,7 +37,9 @@ export class SessionBtwService implements ISessionBtwService {
child.accessor
.get(IAgentToolExecutorService)
?.onBeforeExecuteTool((event) => {
event.veto(denyToolExecution(reason));
if (!BTW_READONLY_TOOLS.has(event.toolCall.name)) {
event.veto(denyToolExecution(reason));
}
});
return childContext.agentId;
}
Expand Down
Loading
Loading