diff --git a/docs/design/2026-07-31-tool-execution-status.md b/docs/design/2026-07-31-tool-execution-status.md new file mode 100644 index 00000000000..4796ce0bf68 --- /dev/null +++ b/docs/design/2026-07-31-tool-execution-status.md @@ -0,0 +1,128 @@ +# Tool Execution Status + +## Motivation + +The terminal tool-call status describes whether the overall call succeeded, +failed, or was cancelled. It does not say whether the dispatcher actually +entered `invocation.execute()`. Validation failures, permission rejection, +execution failures, and post-execution failures therefore need a separate +execution outcome before they can be measured accurately. + +## Contract + +`ToolCallResponseInfo` carries an optional `executionStatus` for source and +recording compatibility: + +```ts +type ToolExecutionStatus = 'not_started' | 'success' | 'error' | 'cancelled'; +``` + +The Core scheduler (`CoreToolScheduler`) and ACP `Session.runTool` always set +the field. Missing values from older recordings, third-party producers, and +subagent result projections (the non-interactive `buildResponse` path, which +replays another agent's reported outcome) become `unknown` only at the +telemetry boundary and are never inferred from the terminal call status. + +The terminal and execution axes are intentionally independent: + +| Terminal status | Execution status | Example | +| --------------- | ---------------- | ------------------------------------------------------------------------------------ | +| `success` | `success` | Normal tool completion | +| `success` | `not_started` | Protocol-level synthetic sibling response | +| `error` | any value | Pre-execution denial, execution error, post-processing error, or batch-hook override | +| `cancelled` | any value | Cancellation before, during, or after execution | + +Reading each row as a (terminal, execution) pair, the only invalid combinations are `success/error` and `success/cancelled`: a call that terminates `success` can only carry execution status `success` or `not_started`. +Execution status freezes when `invocation.execute()` settles; hooks, result +bridging, persistence, and batch processing cannot overwrite it. +PostToolBatch enablement and its parent tool span are snapshotted when a +scheduler batch starts, so runtime hook reconfiguration affects the next +batch rather than changing completion behavior for an in-flight batch. + +## Telemetry + +The normalized `tool_call` event adds `call_id` and `execution_status`. +Normalization occurs once before all sinks: + +- empty tool names become `unknown_tool`; +- `success` is recomputed from terminal `status`; +- terminal errors without an error type use `unknown`; +- success and cancellation omit call-level error fields; +- missing execution status becomes `unknown`. + +The terminal `status` dimension on `qwen-code.tool.call.count`, established by +the terminal telemetry contract, is unchanged by this design. A new +`qwen-code.tool.execution.count` counter uses only `execution_status` and +`tool_type` event-specific dimensions. Globally configured common metric +attributes, such as the opt-in `session.id`, may also be present. The execution +failure rate is: + +```text +execution_status = error +──────────────────────────────────────── +execution_status in {success, error} +``` + +Cancellation, `not_started`, and `unknown` are excluded. Error type, function +name, call ID, messages, and MCP server names remain in logs or spans rather +than metric labels. The counter deliberately omits `function_name`, so an +execution-failure rate cannot be attributed to a specific tool from the metric +alone; drill down through the `tool_call` logs, which carry both `call_id` and +`function_name`. + +An execution span exists only after the dispatcher attempts `execute()`. +It records the tool identity, frozen execution status, and execution error +type. Parent tool spans continue to represent the terminal call status, and +cancelled spans remain unset rather than error. Core opens the parent span +after tool resolution and invocation validation; earlier terminal paths are +covered by the normalized event and execution counter and do not synthesize a +span from an unresolved request name. + +QwenLogger receives the normalized terminal status, execution status, call ID, +and tool type, but not MCP server names or function arguments. MCP server names +remain outside QwenLogger and are available to configured telemetry log and +span exporters. + +## Compatibility and Scope + +The public response and event fields stay optional. Built-in producers use an +internal required shape, while old JSONL recordings are not migrated or +backfilled. New JSONL recordings include `executionStatus` on recorded tool +results; the field is additive, so replay readers that ignore unknown fields +are unaffected. Manual recording projections in Core, ACP, TUI, and +non-interactive modes copy the new scalar without exposing it in user-facing +JSON output. A call cancelled before tool resolution can omit `tool` and +`invocation` from the public `CancelledToolCall` variant, so consumers of that +variant must guard those fields before use. +When such a pre-resolution cancel is emitted through telemetry, `tool_type` +defaults to `"native"` because the tool identity is not yet resolved; this +is a known skew in the `tool_type` dimension for pre-validation cancels. + +Per-call execution errors no longer reject `CoreToolScheduler.schedule()`; +the outcome is delivered through the existing update and completion callbacks +as a terminal `error` call, so one tool's failure does not abort its siblings. +The method still returns `Promise` and can reject for scheduler-level +setup or queue failures. `handleConfirmationResponse()` terminalizes +confirmation-flow errors before rethrowing them, preserving its existing +failure signal without leaving a call in `awaiting_approval`. Embedders should +read terminal `status` and `executionStatus` from callback-delivered calls, +not expect either public entry point to return completed calls. + +The first release covers `CoreToolScheduler` and ACP `Session.runTool`. +Speculation, direct `/fork` execution, MCP-internal retries, provisional +subagent result reconciliation, shell exit metadata, retryability, ownership, +and generic failure phases remain out of scope. + +Core and ACP must ship together. Dashboards should cut over by deployment time +or `service.version`, monitor `unknown` separately, and never use the legacy +`success` metric as the execution-failure SLI. + +## Known Maintenance Hazards + +The pre-execution cancellation invariant ("every `await` in the pre-execution +path is followed by an abort check") is enforced by hand-placed checks at each +call site in `CoreToolScheduler` and `Session.runTool` rather than by a +structural mechanism. Adding a new `await` to either path without a following +check silently reintroduces the stale-execution bug this design fixes. A +future refactor should wrap the awaits in a guarded helper; until then, +reviewers of those paths should verify the invariant manually. diff --git a/docs/developers/development/telemetry.md b/docs/developers/development/telemetry.md index a82a45b4c22..a88a2347bd3 100644 --- a/docs/developers/development/telemetry.md +++ b/docs/developers/development/telemetry.md @@ -576,8 +576,8 @@ The following events are logged: #### Tool Events -- `qwen-code.tool_call`: Each function/tool call. Terminal events are normalized so `status` is authoritative: success and cancelled events omit error fields, while error events always have a non-empty `error_type` (`unknown` when the producer did not classify the error). Blank tool names are emitted as `unknown_tool`. - - **Attributes**: `function_name` (string), `function_args` (object), `duration_ms` (int), `status` (string: "success", "error", or "cancelled"), `success` (boolean), `decision` (string: "accept", "reject", "auto_accept", or "modify", optional), `error` (string, optional), `error_type` (string, present for error events), `prompt_id` (string), `response_id` (string, optional), `content_length` (int, optional), `tool_type` (string: "native" or "mcp"), `mcp_server_name` (string, optional), `metadata` (object, optional — for file-writing tools contains `model_added_lines`, `model_removed_lines`, `user_added_lines`, `user_removed_lines`, `model_added_chars`, `model_removed_chars`, `user_added_chars`, `user_removed_chars`) +- `qwen-code.tool_call`: Each function/tool call. Terminal events are normalized so `status` is authoritative: success and cancelled events omit error fields, while error events always have a non-empty `error_type` (`unknown` when the producer did not classify the error). Blank tool names are emitted as `unknown_tool`. A missing `execution_status` is normalized to `unknown` and is never inferred from the terminal `status`. + - **Attributes**: `function_name` (string), `function_args` (object), `call_id` (string, optional), `duration_ms` (int), `status` (string: "success", "error", or "cancelled"), `execution_status` (string: "not_started", "success", "error", "cancelled", or "unknown"), `success` (boolean), `decision` (string: "accept", "reject", "auto_accept", or "modify", optional), `error` (string, optional), `error_type` (string, present for error events), `prompt_id` (string), `response_id` (string, optional), `content_length` (int, optional), `tool_type` (string: "native" or "mcp"), `mcp_server_name` (string, optional), `metadata` (object, optional — for file-writing tools contains `model_added_lines`, `model_removed_lines`, `user_added_lines`, `user_removed_lines`, `model_added_chars`, `model_removed_chars`, `user_added_chars`, `user_removed_chars`) - `qwen-code.file_operation`: Each file operation. - **Attributes**: `tool_name` (string), `operation` (string: "create", "read", "update"), `lines` (int, optional), `mimetype` (string, optional), `extension` (string, optional), `programming_language` (string, optional) @@ -723,6 +723,9 @@ Metrics are numerical measurements of behavior over time. Metric names use the ` - `qwen-code.tool.call.count` (Counter, Int): Counts tool calls. - **Attributes**: `function_name`, `status` ("success"/"error"/"cancelled"), `success` (boolean, retained for compatibility), `decision` ("accept"/"reject"/"auto_accept"/"modify", optional), `tool_type` ("mcp"/"native", optional) +- `qwen-code.tool.execution.count` (Counter, Int): Counts tool execution outcomes. Deliberately carries no `function_name` dimension to stay low-cardinality, so an execution-failure rate cannot be attributed to a specific tool without dropping to the `qwen-code.tool_call` logs; exclude `unknown`, `not_started`, and `cancelled` when computing execution-failure ratios (denominator is `success` + `error`). + - **Attributes**: `execution_status` ("not_started"/"success"/"error"/"cancelled"/"unknown"), `tool_type` ("mcp"/"native"), plus globally configured common metric attributes such as the opt-in `session.id` + - `qwen-code.tool.call.latency` (Histogram, ms): Measures tool call latency. - **Attributes**: `function_name` (string) @@ -866,10 +869,10 @@ Distributed tracing spans form a tree rooted at `qwen-code.interaction`. Each in - Streaming requests emit `gen_ai.request.stream=true`. `gen_ai.response.time_to_first_chunk` measures seconds from the provider call to the first normalized response yielded by the provider adapter, which may differ from the first raw network frame. Non-streaming requests omit both standard streaming attributes because an absent `gen_ai.request.stream` means non-streaming in the semantic convention. - `qwen-code.tool`: Wraps the full tool lifecycle (approval wait + execution). - - **Attributes**: `session.id`, optional ARMS extension `gen_ai.user.id`, `gen_ai.operation.name` (`execute_tool`), `gen_ai.tool.name`, `gen_ai.tool.type` (`function`), `gen_ai.tool.call.id`, `tool.call_id`, `duration_ms`, `success`, `error` + - **Attributes**: `session.id`, optional ARMS extension `gen_ai.user.id`, `gen_ai.operation.name` (`execute_tool`), `gen_ai.tool.name`, `gen_ai.tool.type` (`function`), `gen_ai.tool.call.id`, `tool.call_id`, `duration_ms`, `success`, `error`, `tool.failure_kind` (string, optional — the specific failure reason, e.g. "cancelled", "tool_error", "tool_exception", "timeout", "permission_denied", "pre_hook_blocked") -- `qwen-code.tool.execution`: Wraps the tool execution phase (after approval). - - **Attributes**: `session.id`, `duration_ms`, `success`, `error` +- `qwen-code.tool.execution`: Wraps the tool execution phase (after approval). Emitted only for attempted executions. + - **Attributes**: `session.id`, `gen_ai.tool.name` (optional), `tool.call_id` (optional), `duration_ms`, `success`, `error`, `execution_status` ("success"/"error"/"cancelled"), `error_type`, `error.type` - `qwen-code.tool.blocked_on_user`: Time a tool spends waiting on user approval. - **Attributes**: `session.id`, `tool.name`, `tool.call_id`, `duration_ms`, `decision` ("proceed_once"/"proceed_always"/"cancel"/"aborted"/"auto_approved"/"error"), `source` ("cli"/"ide"/"hook"/"auto"/"system") diff --git a/packages/cli/src/acp-integration/session/Session.test.ts b/packages/cli/src/acp-integration/session/Session.test.ts index 03ed8821c5a..b7e9498ff8e 100644 --- a/packages/cli/src/acp-integration/session/Session.test.ts +++ b/packages/cli/src/acp-integration/session/Session.test.ts @@ -59,6 +59,7 @@ const transcribeVoiceAudioSpy = vi.hoisted(() => vi.fn()); const startToolSpanSpy = vi.hoisted(() => vi.fn()); const addToolArgumentsAttributesSpy = vi.hoisted(() => vi.fn()); const addToolCallResultAttributesSpy = vi.hoisted(() => vi.fn()); +const logLoopDetectedSpy = vi.hoisted(() => vi.fn()); const TODO_STOP_GUARD_CONTINUATION_CLAIM_METHOD = 'craft/claimTodoStopGuardContinuation'; // Records every LoopTickResolver construction's deps so a test can assert what @@ -97,6 +98,10 @@ vi.mock('@qwen-code/qwen-code-core', async (importOriginal) => { addToolCallResultAttributesSpy(...args); return actual.addToolCallResultAttributes(...args); }, + logLoopDetected: (...args: Parameters) => { + logLoopDetectedSpy(...args); + return actual.logLoopDetected(...args); + }, // Transparent recording wrapper: records the constructor deps, then behaves // exactly like the real resolver (subclass → instanceof + methods preserved). LoopTickResolver: class extends actual.LoopTickResolver { @@ -494,6 +499,7 @@ describe('Session', () => { startToolSpanSpy.mockClear(); addToolArgumentsAttributesSpy.mockClear(); addToolCallResultAttributesSpy.mockClear(); + logLoopDetectedSpy.mockReset(); runVisionBridgeSpy.mockReset(); bridgeToolResultImagesSpy.mockReset(); bridgeToolResultImagesSpy.mockImplementation( @@ -5838,6 +5844,15 @@ describe('Session', () => { expect(result.parts[4].functionResponse?.response?.['error']).toEqual( 'Skipped because loop detection stopped the current turn before this tool call could run.', ); + // Loop-detection skips are not approval-mode denials: they must record + // UNKNOWN, not EXECUTION_DENIED, so the denial metric stays accurate. + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + expect.anything(), + expect.objectContaining({ + callId: 'read_after_loop', + errorType: core.ToolErrorType.UNKNOWN, + }), + ); expect(debugLoggerWarnSpy).toHaveBeenCalledWith( expect.stringContaining( 'Stopping ACP turn after repeated tool parameter errors from missing_tool', @@ -5845,6 +5860,59 @@ describe('Session', () => { ); }); + it('keeps loop telemetry failures from duplicating terminal results', async () => { + mockConfig.getApprovalMode = vi.fn().mockReturnValue(ApprovalMode.YOLO); + mockToolRegistry.getTool.mockReturnValue(undefined); + logLoopDetectedSpy.mockImplementationOnce(() => { + throw new Error('loop telemetry unavailable'); + }); + const functionCalls: FunctionCall[] = [ + { id: 'missing_1', name: 'missing_tool', args: {} }, + { id: 'missing_2', name: 'missing_tool', args: {} }, + { id: 'missing_3', name: 'missing_tool', args: {} }, + ]; + const toolLoopState = { + totalToolCalls: 0, + invalidToolParamErrors: new Map(), + loopDetected: false, + }; + + const result = await ( + session as unknown as { + runToolCalls: ( + abortSignal: AbortSignal, + promptId: string, + calls: FunctionCall[], + loopState: typeof toolLoopState, + ) => Promise<{ + parts: Part[]; + stopAfterPermissionCancel: boolean; + loopDetected?: boolean; + }>; + } + ).runToolCalls( + new AbortController().signal, + 'prompt-loop-telemetry-failure', + functionCalls, + toolLoopState, + ); + + expect(result.loopDetected).toBe(true); + expect(result.parts.map((part) => part.functionResponse?.id)).toEqual([ + 'missing_1', + 'missing_2', + 'missing_3', + ]); + expect(logLoopDetectedSpy).toHaveBeenCalledTimes(1); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledTimes( + 3, + ); + expect(debugLoggerDebugSpy).toHaveBeenCalledWith( + '[Session] Failed to record loop detection telemetry', + expect.objectContaining({ message: 'loop telemetry unavailable' }), + ); + }); + it('stops an ACP prompt after exceeding the daemon tool-call cap', async () => { mockConfig.getApprovalMode = vi.fn().mockReturnValue(ApprovalMode.YOLO); // Pin the cap via the config mock — the daemon halts at whatever the @@ -6322,14 +6390,29 @@ describe('Session', () => { .map( ([, ev]) => ev as { + call_id?: string; function_name?: string; status?: string; + execution_status?: string; success?: boolean; + error_type?: string; }, ) .find((ev) => ev.function_name === 'read_file'); + expect(toolEvent?.call_id).toBe('call-1'); expect(toolEvent?.status).toBe('error'); + expect(toolEvent?.execution_status).toBe('error'); expect(toolEvent?.success).toBe(false); + expect(toolEvent?.error_type).toBe(core.ToolErrorType.UNKNOWN); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + expect.anything(), + expect.objectContaining({ + callId: 'call-1', + status: 'error', + executionStatus: 'error', + errorType: core.ToolErrorType.UNKNOWN, + }), + ); const followUp = vi.mocked(mockChat.sendMessageStream).mock .calls[1][1] as { message: Part[]; @@ -15108,6 +15191,7 @@ describe('Session', () => { expect.objectContaining({ callId: 'call-guard', status: 'error', + executionStatus: 'not_started', errorType: core.ToolErrorType.EXECUTION_DENIED, }), ); @@ -15226,6 +15310,7 @@ describe('Session', () => { expect.objectContaining({ callId: 'call-guard-aborted', status: 'cancelled', + executionStatus: 'not_started', }), ); }); @@ -15329,10 +15414,19 @@ describe('Session', () => { it('stops execution when PostToolUse hook returns shouldStop', async () => { const messageBus = { - request: vi.fn().mockResolvedValue({ - success: true, - output: { shouldStop: true, reason: 'Stopping per hook request' }, - }), + request: vi + .fn() + .mockImplementation(async (request: { eventName?: string }) => ({ + success: true, + output: + request.eventName === 'PostToolUse' + ? { + decision: 'allow', + continue: false, + stopReason: 'Stopping per hook request', + } + : { decision: 'allow' }, + })), }; mockConfig.getMessageBus = vi.fn().mockReturnValue(messageBus); mockConfig.getDisableAllHooks = vi.fn().mockReturnValue(false); @@ -15388,11 +15482,28 @@ describe('Session', () => { }), expect.anything(), ); + // The stop must produce an observable error result + expect( + mockChatRecordingService.recordToolResult, + ).toHaveBeenCalledWith( + expect.anything(), + expect.objectContaining({ + callId: 'call-1', + status: 'error', + executionStatus: 'success', + errorType: core.ToolErrorType.EXECUTION_DENIED, + }), + ); }); }); describe('PostToolUseFailure hook', () => { it('fires PostToolUseFailure hook when tool execution fails', async () => { + const startExecutionSpanSpy = vi.spyOn( + core, + 'startToolExecutionSpan', + ); + const endExecutionSpanSpy = vi.spyOn(core, 'endToolExecutionSpan'); const messageBus = { request: vi .fn() @@ -15465,6 +15576,30 @@ describe('Session', () => { }), expect.anything(), ); + expect( + mockChatRecordingService.recordToolResult, + ).toHaveBeenCalledWith( + expect.anything(), + expect.objectContaining({ + callId: 'call-1', + status: 'error', + executionStatus: 'error', + errorType: core.ToolErrorType.UNHANDLED_EXCEPTION, + }), + ); + expect(startExecutionSpanSpy).toHaveBeenCalledOnce(); + expect(startExecutionSpanSpy).toHaveBeenCalledWith({ + toolName: 'read_file', + callId: 'call-1', + }); + expect(endExecutionSpanSpy).toHaveBeenCalledOnce(); + expect(endExecutionSpanSpy).toHaveBeenCalledWith( + expect.anything(), + expect.objectContaining({ + executionStatus: 'error', + errorType: core.ToolErrorType.UNHANDLED_EXCEPTION, + }), + ); expect(mockClient.extNotification).toHaveBeenCalledWith( 'qwen/notify/session/artifact-event', expect.objectContaining({ @@ -16392,130 +16527,1227 @@ describe('Session', () => { }; } - it('publishes a persisted Todo plan when cancellation races completion', async () => { - const controller = new AbortController(); - const execute = vi.fn().mockImplementation(async () => { - controller.abort(); - return { - llmContent: 'updated', - returnDisplay: { - type: 'todo_list', - planId: 'plan-1', - todos: [{ id: '1', content: 'Ship', status: 'pending' }], - }, - }; + it('records a missing tool name as a pre-execution failure', async () => { + const logToolCallSpy = vi + .spyOn(core, 'logToolCall') + .mockImplementation(() => {}); + + const result = await ( + session as unknown as ToolCallInternals + ).runToolCalls(new AbortController().signal, 'prompt-missing-name', [ + { + id: 'missing_name_call', + args: {}, + }, + ]); + + expect(mockToolRegistry.getTool).not.toHaveBeenCalled(); + expect(result.parts[0].functionResponse).toMatchObject({ + id: 'missing_name_call', + name: 'unknown_tool', + response: { error: 'Missing function name' }, }); - mockToolRegistry.getTool.mockReturnValue( - mockAllowedTool(core.ToolNames.TODO_WRITE, execute), + expect(logToolCallSpy).toHaveBeenCalledWith( + mockConfig, + expect.objectContaining({ + call_id: 'missing_name_call', + function_name: 'unknown_tool', + status: 'error', + execution_status: 'not_started', + error_type: core.ToolErrorType.INVALID_TOOL_PARAMS, + }), ); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + result.parts, + expect.objectContaining({ + callId: 'missing_name_call', + status: 'error', + executionStatus: 'not_started', + errorType: core.ToolErrorType.INVALID_TOOL_PARAMS, + }), + ); + }); - await (session as unknown as ToolCallInternals).runToolCalls( - controller.signal, - 'prompt-cancelled-todo', + it('preserves cancellation when tool enablement resolves after abort', async () => { + const logToolCallSpy = vi + .spyOn(core, 'logToolCall') + .mockImplementation(() => {}); + const endToolSpanSpy = vi.spyOn(core, 'endToolSpan'); + const abortController = new AbortController(); + let resolveEnabled!: (enabled: boolean) => void; + const isToolEnabled = vi.fn( + () => + new Promise((resolve) => { + resolveEnabled = resolve; + }), + ); + mockConfig.getPermissionManager = vi.fn().mockReturnValue({ + isToolEnabled, + }); + const execute = vi.fn(); + const build = vi.fn().mockReturnValue({ + params: {}, + execute, + getDefaultPermission: vi.fn(), + getDescription: vi.fn().mockReturnValue('enablement_tool'), + toolLocations: vi.fn().mockReturnValue([]), + }); + mockToolRegistry.getTool.mockReturnValue({ + name: 'enablement_tool', + kind: core.Kind.Read, + displayName: 'enablement_tool', + description: 'enablement_tool', + build, + canUpdateOutput: false, + isOutputMarkdown: true, + }); + + const runPromise = (session as unknown as ToolCallInternals).runToolCalls( + abortController.signal, + 'prompt-enablement-cancel', [ { - id: 'todo-call', - name: core.ToolNames.TODO_WRITE, - args: { todos: [{ id: '1', content: 'Ship', status: 'pending' }] }, + id: 'enablement_cancel_call', + name: 'enablement_tool', + args: {}, }, ], ); - expect(mockClient.sessionUpdate).toHaveBeenCalledWith( + await vi.waitFor(() => expect(isToolEnabled).toHaveBeenCalledOnce()); + abortController.abort(); + resolveEnabled(false); + const result = await runPromise; + + expect(build).not.toHaveBeenCalled(); + expect(execute).not.toHaveBeenCalled(); + expect(result.parts[0].functionResponse?.response).toEqual({ + error: 'Tool call was cancelled before execution.', + }); + expect(logToolCallSpy).toHaveBeenCalledWith( + mockConfig, expect.objectContaining({ - update: expect.objectContaining({ - sessionUpdate: 'plan', - entries: [ - expect.objectContaining({ - content: 'Ship', - _meta: { qwenTodo: { id: '1' } }, - }), - ], - }), + call_id: 'enablement_cancel_call', + status: 'cancelled', + execution_status: 'not_started', }), ); - }); - - it('lets normalized tool images select a full-turn model', async () => { - const execute = vi.fn().mockResolvedValue({ - llmContent: [ - { text: 'captured screen' }, - { - inlineData: { - mimeType: 'image/png', - data: 'aW1hZ2U=', - }, - }, - ], - returnDisplay: 'captured screen', + expect(endToolSpanSpy).toHaveBeenCalledWith(expect.anything(), { + success: false, + cancelled: true, }); - mockToolRegistry.getTool.mockReturnValue( - mockAllowedTool('screenshot_tool', execute), + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + result.parts, + expect.objectContaining({ + callId: 'enablement_cancel_call', + status: 'cancelled', + executionStatus: 'not_started', + error: undefined, + errorType: undefined, + }), ); - mockConfig.getApprovalMode = vi.fn().mockReturnValue(ApprovalMode.YOLO); - const onFullTurnModel = vi.fn().mockReturnValue(true); - bridgeToolResultImagesSpy.mockImplementationOnce( - async ({ - responseParts, - onFullTurnModel: selectFullTurnModel, - onVisionBridgeNotice, - }: { - responseParts: Part[]; - onFullTurnModel?: (model: string) => boolean; - onVisionBridgeNotice?: (notice: string) => void; - }) => { - expect(selectFullTurnModel?.('qwen3-vl-plus\0')).toBe(true); - onVisionBridgeNotice?.('Routing this image turn to qwen3-vl-plus.'); - return responseParts; - }, + }); + + it('preserves cancellation when permission flow resolves deny after abort', async () => { + const logToolCallSpy = vi + .spyOn(core, 'logToolCall') + .mockImplementation(() => {}); + const abortController = new AbortController(); + let resolvePermission!: (permission: 'deny') => void; + const getDefaultPermission = vi.fn( + () => + new Promise<'deny'>((resolve) => { + resolvePermission = resolve; + }), ); + const execute = vi.fn(); + const build = vi.fn().mockReturnValue({ + params: {}, + execute, + getDefaultPermission, + getDescription: vi.fn().mockReturnValue('permission_flow_tool'), + toolLocations: vi.fn().mockReturnValue([]), + }); + mockToolRegistry.getTool.mockReturnValue({ + name: 'permission_flow_tool', + kind: core.Kind.Read, + displayName: 'permission_flow_tool', + description: 'permission_flow_tool', + build, + canUpdateOutput: false, + isOutputMarkdown: true, + }); - const result = await ( - session as unknown as ToolCallInternals - ).runToolCalls( - new AbortController().signal, - 'prompt-tool-image', + const runPromise = (session as unknown as ToolCallInternals).runToolCalls( + abortController.signal, + 'prompt-permission-flow-cancel', [ { - id: 'call-screen', - name: 'screenshot_tool', + id: 'permission_flow_cancel_call', + name: 'permission_flow_tool', args: {}, }, ], - undefined, - onFullTurnModel, ); - expect(execute).toHaveBeenCalledOnce(); - expect(bridgeToolResultImagesSpy).toHaveBeenCalledWith( + await vi.waitFor(() => + expect(getDefaultPermission).toHaveBeenCalledOnce(), + ); + abortController.abort(); + resolvePermission('deny'); + const result = await runPromise; + + expect(build).toHaveBeenCalledOnce(); + expect(execute).not.toHaveBeenCalled(); + expect(result.parts[0].functionResponse?.response).toEqual({ + error: 'Tool call was cancelled before execution.', + }); + expect(logToolCallSpy).toHaveBeenCalledWith( + mockConfig, expect.objectContaining({ - config: mockConfig, - responseParts: [ - expect.objectContaining({ - functionResponse: expect.objectContaining({ - id: 'call-screen', - name: 'screenshot_tool', - parts: [ - expect.objectContaining({ - inlineData: expect.objectContaining({ - mimeType: 'image/png', - }), - }), - ], - }), - }), - ], - signal: expect.any(AbortSignal), - onFullTurnModel, + call_id: 'permission_flow_cancel_call', + status: 'cancelled', + execution_status: 'not_started', }), ); - expect(onFullTurnModel).toHaveBeenCalledWith('qwen3-vl-plus\0'); - expect(result.parts[0].functionResponse?.parts).toHaveLength(1); - expect(agentMessageChunks()).toContain( - 'Routing this image turn to qwen3-vl-plus.', - ); - }); - + const event = logToolCallSpy.mock.calls.at(-1)?.[1]; + expect(event).not.toHaveProperty('error'); + expect(event).not.toHaveProperty('error_type'); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + result.parts, + expect.objectContaining({ + callId: 'permission_flow_cancel_call', + status: 'cancelled', + executionStatus: 'not_started', + error: undefined, + errorType: undefined, + }), + ); + }); + + it('records one successful terminal when ACP updates fail', async () => { + const logToolCallSpy = vi + .spyOn(core, 'logToolCall') + .mockImplementation(() => {}); + vi.mocked(mockClient.sessionUpdate).mockRejectedValue( + new Error('ACP update unavailable'), + ); + mockConfig.getDisableAllHooks = vi.fn().mockReturnValue(true); + mockConfig.getApprovalMode = vi.fn().mockReturnValue(ApprovalMode.YOLO); + const execute = vi.fn().mockResolvedValue({ + llmContent: 'completed', + returnDisplay: 'completed', + }); + mockToolRegistry.getTool.mockReturnValue( + mockAllowedTool('success_tool', execute), + ); + + const result = await ( + session as unknown as ToolCallInternals + ).runToolCalls(new AbortController().signal, 'prompt-success', [ + { id: 'success_call', name: 'success_tool', args: {} }, + ]); + + expect(result.parts[0].functionResponse?.response).toEqual({ + output: 'completed', + }); + expect(logToolCallSpy).toHaveBeenCalledTimes(1); + expect(logToolCallSpy).toHaveBeenCalledWith( + mockConfig, + expect.objectContaining({ + call_id: 'success_call', + status: 'success', + execution_status: 'success', + }), + ); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledTimes( + 1, + ); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + result.parts, + expect.objectContaining({ + callId: 'success_call', + status: 'success', + executionStatus: 'success', + }), + ); + }); + + it('records execution-stage cancellation on both axes', async () => { + const logToolCallSpy = vi + .spyOn(core, 'logToolCall') + .mockImplementation(() => {}); + const endExecutionSpanSpy = vi.spyOn(core, 'endToolExecutionSpan'); + const endToolSpanSpy = vi.spyOn(core, 'endToolSpan'); + const abortController = new AbortController(); + mockConfig.getDisableAllHooks = vi.fn().mockReturnValue(true); + mockConfig.getApprovalMode = vi.fn().mockReturnValue(ApprovalMode.YOLO); + const execute = vi.fn().mockImplementation(async () => { + abortController.abort(); + return { + llmContent: 'partial output', + returnDisplay: 'partial output', + }; + }); + mockToolRegistry.getTool.mockReturnValue( + mockAllowedTool('cancel_tool', execute), + ); + + const result = await ( + session as unknown as ToolCallInternals + ).runToolCalls(abortController.signal, 'prompt-execution-cancel', [ + { id: 'execution_cancel_call', name: 'cancel_tool', args: {} }, + ]); + + expect(result.parts[0].functionResponse?.response).toEqual({ + error: 'The tool had already completed; its output was discarded.', + }); + expect(endExecutionSpanSpy).toHaveBeenCalledWith( + expect.anything(), + expect.objectContaining({ + executionStatus: 'cancelled', + cancelled: true, + }), + ); + expect(endToolSpanSpy).toHaveBeenCalledWith(expect.anything(), { + success: false, + cancelled: true, + }); + expect(logToolCallSpy).toHaveBeenCalledWith( + mockConfig, + expect.objectContaining({ + call_id: 'execution_cancel_call', + status: 'cancelled', + execution_status: 'cancelled', + }), + ); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + result.parts, + expect.objectContaining({ + callId: 'execution_cancel_call', + status: 'cancelled', + executionStatus: 'cancelled', + error: undefined, + errorType: undefined, + }), + ); + }); + + it('does not execute after cancellation settles during PreToolUse', async () => { + const logToolCallSpy = vi + .spyOn(core, 'logToolCall') + .mockImplementation(() => {}); + const abortController = new AbortController(); + let resolvePreHook: + | ((value: { success: true; output: { decision: 'allow' } }) => void) + | undefined; + const preHookPromise = new Promise<{ + success: true; + output: { decision: 'allow' }; + }>((resolve) => { + resolvePreHook = resolve; + }); + const messageBus = { + request: vi.fn().mockReturnValue(preHookPromise), + }; + mockConfig.getMessageBus = vi.fn().mockReturnValue(messageBus); + mockConfig.getDisableAllHooks = vi.fn().mockReturnValue(false); + mockConfig.getApprovalMode = vi.fn().mockReturnValue(ApprovalMode.YOLO); + const execute = vi.fn().mockResolvedValue({ + llmContent: 'should not execute', + returnDisplay: 'should not execute', + }); + mockToolRegistry.getTool.mockReturnValue( + mockAllowedTool('pre_hook_tool', execute), + ); + + const runPromise = (session as unknown as ToolCallInternals).runToolCalls( + abortController.signal, + 'prompt-pre-hook-cancel', + [ + { + id: 'pre_hook_cancel_call', + name: 'pre_hook_tool', + args: {}, + }, + ], + ); + + await vi.waitFor(() => expect(messageBus.request).toHaveBeenCalledOnce()); + abortController.abort(); + resolvePreHook?.({ + success: true, + output: { decision: 'allow' }, + }); + const result = await runPromise; + + expect(execute).not.toHaveBeenCalled(); + expect(result.parts[0].functionResponse?.response).toEqual({ + error: 'Tool call was cancelled before execution.', + }); + expect(logToolCallSpy).toHaveBeenCalledWith( + mockConfig, + expect.objectContaining({ + call_id: 'pre_hook_cancel_call', + status: 'cancelled', + execution_status: 'not_started', + }), + ); + const event = logToolCallSpy.mock.calls.at(-1)?.[1]; + expect(event).not.toHaveProperty('error'); + expect(event).not.toHaveProperty('error_type'); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + result.parts, + expect.objectContaining({ + callId: 'pre_hook_cancel_call', + status: 'cancelled', + executionStatus: 'not_started', + error: undefined, + errorType: undefined, + }), + ); + }); + + it('preserves cancellation while emitting a PreToolUse block message', async () => { + const logToolCallSpy = vi + .spyOn(core, 'logToolCall') + .mockImplementation(() => {}); + const abortController = new AbortController(); + let markBlockMessageStarted!: () => void; + const blockMessageStarted = new Promise((resolve) => { + markBlockMessageStarted = resolve; + }); + let releaseBlockMessage!: () => void; + const blockMessageRelease = new Promise((resolve) => { + releaseBlockMessage = resolve; + }); + vi.mocked(mockClient.sessionUpdate).mockImplementation(async (params) => { + if ( + params.update.sessionUpdate === 'agent_message_chunk' && + params.update.content.type === 'text' && + params.update.content.text.includes('PreToolUse blocked') + ) { + markBlockMessageStarted(); + await blockMessageRelease; + } + }); + const messageBus = { + request: vi.fn().mockResolvedValue({ + success: true, + output: { decision: 'deny', reason: 'blocked by test hook' }, + }), + }; + mockConfig.getMessageBus = vi.fn().mockReturnValue(messageBus); + mockConfig.getDisableAllHooks = vi.fn().mockReturnValue(false); + mockConfig.getApprovalMode = vi.fn().mockReturnValue(ApprovalMode.YOLO); + const execute = vi.fn(); + mockToolRegistry.getTool.mockReturnValue( + mockAllowedTool('blocked_tool', execute), + ); + + const runPromise = (session as unknown as ToolCallInternals).runToolCalls( + abortController.signal, + 'prompt-block-message-cancel', + [ + { + id: 'block_message_cancel_call', + name: 'blocked_tool', + args: {}, + }, + ], + ); + + await blockMessageStarted; + abortController.abort(); + releaseBlockMessage(); + const result = await runPromise; + + expect(execute).not.toHaveBeenCalled(); + expect(result.parts[0].functionResponse?.response).toEqual({ + error: 'Tool call was cancelled before execution.', + }); + expect(logToolCallSpy).toHaveBeenCalledWith( + mockConfig, + expect.objectContaining({ + call_id: 'block_message_cancel_call', + status: 'cancelled', + execution_status: 'not_started', + }), + ); + const event = logToolCallSpy.mock.calls.at(-1)?.[1]; + expect(event).not.toHaveProperty('error'); + expect(event).not.toHaveProperty('error_type'); + }); + + it('does not execute after cancellation settles during the start emitter', async () => { + const logToolCallSpy = vi + .spyOn(core, 'logToolCall') + .mockImplementation(() => {}); + const startExecutionSpanSpy = vi.spyOn(core, 'startToolExecutionSpan'); + const abortController = new AbortController(); + let resolveStart: (() => void) | undefined; + vi.mocked(mockClient.sessionUpdate).mockImplementationOnce( + () => + new Promise((resolve) => { + resolveStart = resolve; + }), + ); + mockConfig.getDisableAllHooks = vi.fn().mockReturnValue(true); + mockConfig.getApprovalMode = vi.fn().mockReturnValue(ApprovalMode.YOLO); + const execute = vi.fn().mockResolvedValue({ + llmContent: 'should not execute', + returnDisplay: 'should not execute', + }); + mockToolRegistry.getTool.mockReturnValue( + mockAllowedTool('start_emitter_tool', execute), + ); + + const runPromise = (session as unknown as ToolCallInternals).runToolCalls( + abortController.signal, + 'prompt-start-emitter-cancel', + [ + { + id: 'start_emitter_cancel_call', + name: 'start_emitter_tool', + args: {}, + }, + ], + ); + + await vi.waitFor(() => + expect(mockClient.sessionUpdate).toHaveBeenCalledOnce(), + ); + abortController.abort(); + resolveStart?.(); + const result = await runPromise; + + expect(execute).not.toHaveBeenCalled(); + expect(startExecutionSpanSpy).not.toHaveBeenCalled(); + expect(result.parts[0].functionResponse?.response).toEqual({ + error: 'Tool call was cancelled before execution.', + }); + expect(logToolCallSpy).toHaveBeenCalledWith( + mockConfig, + expect.objectContaining({ + call_id: 'start_emitter_cancel_call', + status: 'cancelled', + execution_status: 'not_started', + }), + ); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + result.parts, + expect.objectContaining({ + callId: 'start_emitter_cancel_call', + status: 'cancelled', + executionStatus: 'not_started', + }), + ); + }); + + it('keeps a structured timeout exception ahead of a later parent abort', async () => { + const logToolCallSpy = vi + .spyOn(core, 'logToolCall') + .mockImplementation(() => {}); + const endExecutionSpanSpy = vi.spyOn(core, 'endToolExecutionSpan'); + const abortController = new AbortController(); + const timeoutError = Object.assign(new Error('MCP request timed out'), { + errorType: core.ToolErrorType.EXECUTION_TIMEOUT, + }); + const execute = vi.fn().mockImplementation( + () => + new Promise((_resolve, reject) => { + reject(timeoutError); + abortController.abort(); + }), + ); + mockToolRegistry.getTool.mockReturnValue( + mockAllowedTool('timeout_tool', execute), + ); + mockConfig.getApprovalMode = vi.fn().mockReturnValue(ApprovalMode.YOLO); + const messageBus = { + request: vi + .fn() + .mockImplementation(async (request: { eventName?: string }) => ({ + success: true, + output: + request.eventName === 'PreToolUse' ? { decision: 'allow' } : {}, + })), + }; + mockConfig.getMessageBus = vi.fn().mockReturnValue(messageBus); + mockConfig.getDisableAllHooks = vi.fn().mockReturnValue(false); + + const result = await ( + session as unknown as ToolCallInternals + ).runToolCalls(abortController.signal, 'prompt-structured-timeout', [ + { + id: 'structured_timeout_call', + name: 'timeout_tool', + args: {}, + }, + ]); + + expect(result.parts[0].functionResponse?.response).toEqual({ + error: 'MCP request timed out', + }); + expect(endExecutionSpanSpy).toHaveBeenCalledWith( + expect.anything(), + expect.objectContaining({ + executionStatus: 'error', + errorType: core.ToolErrorType.EXECUTION_TIMEOUT, + cancelled: false, + }), + ); + expect(logToolCallSpy).toHaveBeenCalledWith( + mockConfig, + expect.objectContaining({ + call_id: 'structured_timeout_call', + status: 'error', + execution_status: 'error', + error_type: core.ToolErrorType.EXECUTION_TIMEOUT, + }), + ); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + result.parts, + expect.objectContaining({ + callId: 'structured_timeout_call', + status: 'error', + executionStatus: 'error', + errorType: core.ToolErrorType.EXECUTION_TIMEOUT, + }), + ); + expect(messageBus.request).toHaveBeenCalledWith( + expect.objectContaining({ + eventName: 'PostToolUseFailure', + input: expect.objectContaining({ is_interrupt: false }), + }), + expect.anything(), + ); + }); + + it('records PostToolUse stop as an error after successful execution', async () => { + const logToolCallSpy = vi + .spyOn(core, 'logToolCall') + .mockImplementation(() => {}); + const messageBus = { + request: vi + .fn() + .mockImplementation(async (request: { eventName?: string }) => ({ + success: true, + output: + request.eventName === 'PostToolUse' + ? { + decision: 'allow', + continue: false, + stopReason: 'Stopped by hook', + } + : { decision: 'allow' }, + })), + }; + mockConfig.getMessageBus = vi.fn().mockReturnValue(messageBus); + mockConfig.getDisableAllHooks = vi.fn().mockReturnValue(false); + mockConfig.getApprovalMode = vi.fn().mockReturnValue(ApprovalMode.YOLO); + const execute = vi.fn().mockResolvedValue({ + llmContent: 'completed', + returnDisplay: 'completed', + }); + mockToolRegistry.getTool.mockReturnValue( + mockAllowedTool('post_stop_tool', execute), + ); + + const result = await ( + session as unknown as ToolCallInternals + ).runToolCalls(new AbortController().signal, 'prompt-post-stop', [ + { id: 'post_stop_call', name: 'post_stop_tool', args: {} }, + ]); + + expect(result.parts[0].functionResponse?.response).toEqual({ + error: 'Stopped by hook', + }); + expect(logToolCallSpy).toHaveBeenCalledWith( + mockConfig, + expect.objectContaining({ + call_id: 'post_stop_call', + status: 'error', + execution_status: 'success', + error_type: core.ToolErrorType.EXECUTION_DENIED, + }), + ); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + result.parts, + expect.objectContaining({ + callId: 'post_stop_call', + status: 'error', + executionStatus: 'success', + errorType: core.ToolErrorType.EXECUTION_DENIED, + }), + ); + }); + + it('records postprocessing failure after successful execution', async () => { + const logToolCallSpy = vi + .spyOn(core, 'logToolCall') + .mockImplementation(() => {}); + const execute = vi.fn().mockResolvedValue({ + llmContent: 'completed', + returnDisplay: 'completed', + }); + mockToolRegistry.getTool.mockReturnValue( + mockAllowedTool('postprocess_tool', execute), + ); + mockConfig.getDisableAllHooks = vi.fn().mockReturnValue(true); + mockConfig.getApprovalMode = vi.fn().mockReturnValue(ApprovalMode.YOLO); + bridgeToolResultImagesSpy.mockRejectedValueOnce( + new Error('image postprocessing failed'), + ); + + const result = await ( + session as unknown as ToolCallInternals + ).runToolCalls(new AbortController().signal, 'prompt-postprocess-fail', [ + { + id: 'postprocess_success_call', + name: 'postprocess_tool', + args: {}, + }, + ]); + + expect(result.parts[0].functionResponse?.response).toEqual({ + error: 'image postprocessing failed', + }); + expect(logToolCallSpy).toHaveBeenCalledWith( + mockConfig, + expect.objectContaining({ + call_id: 'postprocess_success_call', + status: 'error', + execution_status: 'success', + error_type: core.ToolErrorType.UNHANDLED_EXCEPTION, + }), + ); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + result.parts, + expect.objectContaining({ + callId: 'postprocess_success_call', + status: 'error', + executionStatus: 'success', + errorType: core.ToolErrorType.UNHANDLED_EXCEPTION, + }), + ); + }); + + it('preserves a structured postprocessing error type after successful execution', async () => { + const logToolCallSpy = vi + .spyOn(core, 'logToolCall') + .mockImplementation(() => {}); + const execute = vi.fn().mockResolvedValue({ + llmContent: 'completed', + returnDisplay: 'completed', + }); + mockToolRegistry.getTool.mockReturnValue( + mockAllowedTool('structured_postprocess_tool', execute), + ); + mockConfig.getDisableAllHooks = vi.fn().mockReturnValue(true); + mockConfig.getApprovalMode = vi.fn().mockReturnValue(ApprovalMode.YOLO); + bridgeToolResultImagesSpy.mockRejectedValueOnce( + Object.assign(new Error('structured postprocessing failure'), { + errorType: core.ToolErrorType.EXECUTION_FAILED, + }), + ); + + const result = await ( + session as unknown as ToolCallInternals + ).runToolCalls(new AbortController().signal, 'prompt-structured-post', [ + { + id: 'structured_postprocess_call', + name: 'structured_postprocess_tool', + args: {}, + }, + ]); + + expect(logToolCallSpy).toHaveBeenCalledWith( + mockConfig, + expect.objectContaining({ + call_id: 'structured_postprocess_call', + status: 'error', + execution_status: 'success', + error_type: core.ToolErrorType.EXECUTION_FAILED, + }), + ); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + result.parts, + expect.objectContaining({ + status: 'error', + executionStatus: 'success', + errorType: core.ToolErrorType.EXECUTION_FAILED, + }), + ); + }); + + it('records cancellation when abort arrives during exception failure hooks', async () => { + const logToolCallSpy = vi + .spyOn(core, 'logToolCall') + .mockImplementation(() => {}); + const abortController = new AbortController(); + const messageBus = { + request: vi + .fn() + .mockImplementation(async (request: { eventName?: string }) => { + if (request.eventName === 'PostToolUseFailure') { + abortController.abort(); + } + return { + success: true, + output: + request.eventName === 'PreToolUse' ? { decision: 'allow' } : {}, + }; + }), + }; + mockConfig.getMessageBus = vi.fn().mockReturnValue(messageBus); + mockConfig.getDisableAllHooks = vi.fn().mockReturnValue(false); + mockConfig.getApprovalMode = vi.fn().mockReturnValue(ApprovalMode.YOLO); + mockToolRegistry.getTool.mockReturnValue( + mockAllowedTool( + 'cancel_during_failure_hook_tool', + vi.fn().mockRejectedValue(new Error('tool failed')), + ), + ); + + const result = await ( + session as unknown as ToolCallInternals + ).runToolCalls(abortController.signal, 'prompt-failure-hook-cancel', [ + { + id: 'failure_hook_cancel_call', + name: 'cancel_during_failure_hook_tool', + args: {}, + }, + ]); + + expect(logToolCallSpy).toHaveBeenCalledWith( + mockConfig, + expect.objectContaining({ + call_id: 'failure_hook_cancel_call', + status: 'cancelled', + execution_status: 'error', + }), + ); + expect(logToolCallSpy.mock.calls[0][1]).not.toHaveProperty('error_type'); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + result.parts, + expect.objectContaining({ + status: 'cancelled', + executionStatus: 'error', + errorType: undefined, + }), + ); + }); + + it('records one terminal when failure hooks and ACP updates fail', async () => { + const logToolCallSpy = vi + .spyOn(core, 'logToolCall') + .mockImplementation(() => {}); + vi.mocked(mockClient.sessionUpdate).mockRejectedValue( + new Error('ACP update unavailable'), + ); + const messageBus = { + request: vi + .fn() + .mockImplementation(async (request: { eventName?: string }) => { + if (request.eventName === 'PostToolUseFailure') { + throw new Error('failure hook unavailable'); + } + return { success: true, output: { decision: 'allow' } }; + }), + }; + mockConfig.getMessageBus = vi.fn().mockReturnValue(messageBus); + mockConfig.getDisableAllHooks = vi.fn().mockReturnValue(false); + mockConfig.getApprovalMode = vi.fn().mockReturnValue(ApprovalMode.YOLO); + const execute = vi.fn().mockRejectedValue(new Error('tool failed')); + mockToolRegistry.getTool.mockReturnValue( + mockAllowedTool('failing_tool', execute), + ); + + const result = await ( + session as unknown as ToolCallInternals + ).runToolCalls(new AbortController().signal, 'prompt-hook-fail', [ + { id: 'hook_fail_call', name: 'failing_tool', args: {} }, + ]); + + expect(result.parts[0].functionResponse?.response).toEqual({ + error: 'tool failed', + }); + expect(logToolCallSpy).toHaveBeenCalledTimes(1); + expect(logToolCallSpy).toHaveBeenCalledWith( + mockConfig, + expect.objectContaining({ + call_id: 'hook_fail_call', + status: 'error', + execution_status: 'error', + error_type: core.ToolErrorType.UNHANDLED_EXCEPTION, + }), + ); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledTimes( + 1, + ); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + result.parts, + expect.objectContaining({ + callId: 'hook_fail_call', + status: 'error', + executionStatus: 'error', + errorType: core.ToolErrorType.UNHANDLED_EXCEPTION, + }), + ); + }); + + it('classifies postprocessing failures independently from a settled soft error', async () => { + const logToolCallSpy = vi + .spyOn(core, 'logToolCall') + .mockImplementation(() => {}); + const endExecutionSpanSpy = vi.spyOn(core, 'endToolExecutionSpan'); + const execute = vi.fn().mockResolvedValue({ + llmContent: 'timed out', + returnDisplay: 'timed out', + error: { + message: 'execution timed out', + type: core.ToolErrorType.EXECUTION_TIMEOUT, + }, + }); + mockToolRegistry.getTool.mockReturnValue( + mockAllowedTool('timeout_tool', execute), + ); + mockConfig.getApprovalMode = vi.fn().mockReturnValue(ApprovalMode.YOLO); + bridgeToolResultImagesSpy.mockRejectedValueOnce( + new Error('image postprocessing failed'), + ); + + const result = await ( + session as unknown as ToolCallInternals + ).runToolCalls(new AbortController().signal, 'prompt-postprocess-error', [ + { + id: 'postprocess_error_call', + name: 'timeout_tool', + args: {}, + }, + ]); + + expect(result.parts[0].functionResponse?.response).toEqual({ + error: 'image postprocessing failed', + }); + expect(endExecutionSpanSpy).toHaveBeenCalledWith( + expect.anything(), + expect.objectContaining({ + executionStatus: 'error', + errorType: core.ToolErrorType.EXECUTION_TIMEOUT, + }), + ); + expect(logToolCallSpy).toHaveBeenCalledWith( + mockConfig, + expect.objectContaining({ + call_id: 'postprocess_error_call', + status: 'error', + execution_status: 'error', + error: 'image postprocessing failed', + error_type: core.ToolErrorType.UNHANDLED_EXCEPTION, + }), + ); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + result.parts, + expect.objectContaining({ + callId: 'postprocess_error_call', + status: 'error', + executionStatus: 'error', + errorType: core.ToolErrorType.UNHANDLED_EXCEPTION, + }), + ); + }); + + it.each([ + { + label: 'successful execution', + toolResult: { + llmContent: 'completed output', + returnDisplay: 'completed output', + }, + executionStatus: 'success' as const, + }, + { + label: 'soft execution failure', + toolResult: { + llmContent: 'failed output', + returnDisplay: 'failed output', + error: { + message: 'tool failed', + type: core.ToolErrorType.EXECUTION_FAILED, + }, + }, + executionStatus: 'error' as const, + }, + { + label: 'soft execution timeout', + toolResult: { + llmContent: 'timed out output', + returnDisplay: 'timed out output', + error: { + message: 'tool timed out', + type: core.ToolErrorType.EXECUTION_TIMEOUT, + }, + }, + executionStatus: 'error' as const, + }, + ])( + 'replaces $label output when image postprocessing is cancelled', + async ({ toolResult, executionStatus }) => { + const logToolCallSpy = vi + .spyOn(core, 'logToolCall') + .mockImplementation(() => {}); + const abortController = new AbortController(); + const execute = vi.fn().mockResolvedValue(toolResult); + mockToolRegistry.getTool.mockReturnValue( + mockAllowedTool('bridge_cancel_tool', execute), + ); + mockConfig.getApprovalMode = vi.fn().mockReturnValue(ApprovalMode.YOLO); + mockConfig.getDisableAllHooks = vi.fn().mockReturnValue(true); + bridgeToolResultImagesSpy.mockImplementationOnce( + async ({ responseParts }: { responseParts: Part[] }) => { + abortController.abort(); + return responseParts; + }, + ); + + const result = await ( + session as unknown as ToolCallInternals + ).runToolCalls( + abortController.signal, + `prompt-bridge-cancel-${executionStatus}`, + [ + { + id: `bridge_cancel_${executionStatus}`, + name: 'bridge_cancel_tool', + args: {}, + }, + ], + ); + + expect(result.parts[0].functionResponse?.response).toEqual({ + error: 'The tool had already completed; its output was discarded.', + }); + expect(logToolCallSpy).toHaveBeenCalledWith( + mockConfig, + expect.objectContaining({ + call_id: `bridge_cancel_${executionStatus}`, + status: 'cancelled', + execution_status: executionStatus, + }), + ); + const event = logToolCallSpy.mock.calls.at(-1)?.[1]; + expect(event).not.toHaveProperty('error'); + expect(event).not.toHaveProperty('error_type'); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + result.parts, + expect.objectContaining({ + callId: `bridge_cancel_${executionStatus}`, + status: 'cancelled', + executionStatus, + error: undefined, + errorType: undefined, + }), + ); + }, + ); + + it('prefers cancellation when PostToolUse stops after execution settles', async () => { + const logToolCallSpy = vi + .spyOn(core, 'logToolCall') + .mockImplementation(() => {}); + const abortController = new AbortController(); + const messageBus = { + request: vi + .fn() + .mockImplementation(async (request: { eventName?: string }) => { + if (request.eventName === 'PostToolUse') { + abortController.abort(); + return { + success: true, + output: { + decision: 'allow', + continue: false, + stopReason: 'Stopping per hook request', + }, + }; + } + return { success: true, output: { decision: 'allow' } }; + }), + }; + mockConfig.getMessageBus = vi.fn().mockReturnValue(messageBus); + mockConfig.getDisableAllHooks = vi.fn().mockReturnValue(false); + mockConfig.getApprovalMode = vi.fn().mockReturnValue(ApprovalMode.YOLO); + const execute = vi.fn().mockResolvedValue({ + llmContent: 'completed', + returnDisplay: 'completed', + }); + mockToolRegistry.getTool.mockReturnValue( + mockAllowedTool('post_hook_tool', execute), + ); + + const result = await ( + session as unknown as ToolCallInternals + ).runToolCalls(abortController.signal, 'prompt-post-hook-cancel', [ + { + id: 'post_hook_cancel_call', + name: 'post_hook_tool', + args: {}, + }, + ]); + + expect(execute).toHaveBeenCalledOnce(); + expect(result.parts[0].functionResponse?.response).toEqual({ + error: 'The tool had already completed; its output was discarded.', + }); + expect(logToolCallSpy).toHaveBeenCalledWith( + mockConfig, + expect.objectContaining({ + call_id: 'post_hook_cancel_call', + status: 'cancelled', + execution_status: 'success', + }), + ); + const event = logToolCallSpy.mock.calls.at(-1)?.[1]; + expect(event).not.toHaveProperty('error'); + expect(event).not.toHaveProperty('error_type'); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + result.parts, + expect.objectContaining({ + callId: 'post_hook_cancel_call', + status: 'cancelled', + executionStatus: 'success', + error: undefined, + errorType: undefined, + }), + ); + }); + + it('publishes a persisted Todo plan when cancellation races completion', async () => { + const controller = new AbortController(); + const execute = vi.fn().mockImplementation(async () => { + controller.abort(); + return { + llmContent: 'updated', + returnDisplay: { + type: 'todo_list', + planId: 'plan-1', + todos: [{ id: '1', content: 'Ship', status: 'pending' }], + }, + }; + }); + mockToolRegistry.getTool.mockReturnValue( + mockAllowedTool(core.ToolNames.TODO_WRITE, execute), + ); + + await (session as unknown as ToolCallInternals).runToolCalls( + controller.signal, + 'prompt-cancelled-todo', + [ + { + id: 'todo-call', + name: core.ToolNames.TODO_WRITE, + args: { todos: [{ id: '1', content: 'Ship', status: 'pending' }] }, + }, + ], + ); + + expect(mockClient.sessionUpdate).toHaveBeenCalledWith( + expect.objectContaining({ + update: expect.objectContaining({ + sessionUpdate: 'plan', + entries: [ + expect.objectContaining({ + content: 'Ship', + _meta: { qwenTodo: { id: '1' } }, + }), + ], + }), + }), + ); + }); + + it('lets normalized tool images select a full-turn model', async () => { + const execute = vi.fn().mockResolvedValue({ + llmContent: [ + { text: 'captured screen' }, + { + inlineData: { + mimeType: 'image/png', + data: 'aW1hZ2U=', + }, + }, + ], + returnDisplay: 'captured screen', + }); + mockToolRegistry.getTool.mockReturnValue( + mockAllowedTool('screenshot_tool', execute), + ); + mockConfig.getApprovalMode = vi.fn().mockReturnValue(ApprovalMode.YOLO); + const onFullTurnModel = vi.fn().mockReturnValue(true); + bridgeToolResultImagesSpy.mockImplementationOnce( + async ({ + responseParts, + onFullTurnModel: selectFullTurnModel, + onVisionBridgeNotice, + }: { + responseParts: Part[]; + onFullTurnModel?: (model: string) => boolean; + onVisionBridgeNotice?: (notice: string) => void; + }) => { + expect(selectFullTurnModel?.('qwen3-vl-plus\0')).toBe(true); + onVisionBridgeNotice?.('Routing this image turn to qwen3-vl-plus.'); + return responseParts; + }, + ); + + const result = await ( + session as unknown as ToolCallInternals + ).runToolCalls( + new AbortController().signal, + 'prompt-tool-image', + [ + { + id: 'call-screen', + name: 'screenshot_tool', + args: {}, + }, + ], + undefined, + onFullTurnModel, + ); + + expect(execute).toHaveBeenCalledOnce(); + expect(bridgeToolResultImagesSpy).toHaveBeenCalledWith( + expect.objectContaining({ + config: mockConfig, + responseParts: [ + expect.objectContaining({ + functionResponse: expect.objectContaining({ + id: 'call-screen', + name: 'screenshot_tool', + parts: [ + expect.objectContaining({ + inlineData: expect.objectContaining({ + mimeType: 'image/png', + }), + }), + ], + }), + }), + ], + signal: expect.any(AbortSignal), + onFullTurnModel, + }), + ); + expect(onFullTurnModel).toHaveBeenCalledWith('qwen3-vl-plus\0'); + expect(result.parts[0].functionResponse?.parts).toHaveLength(1); + expect(agentMessageChunks()).toContain( + 'Routing this image turn to qwen3-vl-plus.', + ); + }); + it('keeps a full-turn model selected across parallel tool images', async () => { mockConfig.getApprovalMode = vi.fn().mockReturnValue(ApprovalMode.YOLO); const execute = vi.fn().mockResolvedValue({ @@ -17316,6 +18548,23 @@ describe('Session', () => { }); expect(cancelledExecute).not.toHaveBeenCalled(); expect(laterExecute).not.toHaveBeenCalled(); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + [result.parts[0]], + expect.objectContaining({ + callId: 'shell_call', + status: 'cancelled', + executionStatus: 'not_started', + error: undefined, + errorType: undefined, + }), + ); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + [result.parts[1]], + expect.objectContaining({ + callId: 'read_call', + executionStatus: 'not_started', + }), + ); }); it('skips later pre-loop tools after non-question permission cancellation', async () => { @@ -17494,6 +18743,9 @@ describe('Session', () => { }); it('skips later tools after non-question permission request failure', async () => { + const logToolCallSpy = vi + .spyOn(core, 'logToolCall') + .mockImplementation(() => {}); const failedPermissionExecute = vi.fn(); const laterExecute = vi.fn().mockResolvedValue({ llmContent: 'should not execute', @@ -17541,6 +18793,92 @@ describe('Session', () => { }); expect(failedPermissionExecute).not.toHaveBeenCalled(); expect(laterExecute).not.toHaveBeenCalled(); + expect(logToolCallSpy).toHaveBeenCalledWith( + mockConfig, + expect.objectContaining({ + call_id: 'shell_call', + status: 'error', + execution_status: 'not_started', + error_type: core.ToolErrorType.UNHANDLED_EXCEPTION, + }), + ); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + [result.parts[0]], + expect.objectContaining({ + callId: 'shell_call', + status: 'error', + executionStatus: 'not_started', + errorType: core.ToolErrorType.UNHANDLED_EXCEPTION, + }), + ); + }); + + it('does not treat a parent abort during permission as explicit rejection', async () => { + const permissionExecute = vi.fn(); + const laterExecute = vi.fn().mockResolvedValue({ + llmContent: 'should not execute', + returnDisplay: 'should not execute', + }); + mockToolRegistry.getTool.mockImplementation((name: string) => + name === core.ToolNames.SHELL + ? mockConfirmingTool(name, permissionExecute, 'exec') + : mockAllowedTool(name, laterExecute), + ); + vi.mocked(mockClient.requestPermission).mockReturnValueOnce( + new Promise(() => { + // requestPermissionWithAbort owns cancellation of this pending call. + }), + ); + const abortController = new AbortController(); + + const runPromise = (session as unknown as ToolCallInternals).runToolCalls( + abortController.signal, + 'prompt-shell-permission-parent-abort', + [ + { + id: 'shell_call', + name: core.ToolNames.SHELL, + args: { command: 'echo denied' }, + }, + { + id: 'read_call', + name: core.ToolNames.READ_FILE, + args: { file_path: '/tmp/should-not-run' }, + }, + ], + ); + + await vi.waitFor(() => + expect(mockClient.requestPermission).toHaveBeenCalledOnce(), + ); + abortController.abort(); + const result = await runPromise; + + expect(result.stopAfterPermissionCancel).toBe(false); + expect( + result.parts.map((part) => part.functionResponse?.response), + ).toEqual([ + { error: 'Tool call was cancelled before execution.' }, + { error: 'Tool call was cancelled before execution.' }, + ]); + expect(permissionExecute).not.toHaveBeenCalled(); + expect(laterExecute).not.toHaveBeenCalled(); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + [result.parts[0]], + expect.objectContaining({ + callId: 'shell_call', + status: 'cancelled', + executionStatus: 'not_started', + }), + ); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + [result.parts[1]], + expect.objectContaining({ + callId: 'read_call', + status: 'cancelled', + executionStatus: 'not_started', + }), + ); }); it('keeps plan mode and gives manual guidance when switch_mode approval is unavailable', async () => { @@ -18014,7 +19352,118 @@ describe('Session', () => { expect(secondSiblingSignal?.aborted).toBe(true); }); - it('passes an already-aborted parent signal to Agent batches', async () => { + it('does not treat parent abort during nested Agent permission as explicit rejection', async () => { + const previousMaxConcurrency = + process.env['QWEN_CODE_MAX_TOOL_CONCURRENCY']; + process.env['QWEN_CODE_MAX_TOOL_CONCURRENCY'] = '1'; + try { + const eventEmitter = new EventEmitter(); + const respond = vi.fn().mockResolvedValue(undefined); + const firstExecute = vi + .fn() + .mockImplementation(async (signal: AbortSignal) => { + emitNestedInfoPermission(eventEmitter, respond); + await vi.waitFor(() => { + expect(signal.aborted).toBe(true); + }); + return { + llmContent: 'agent stopped', + returnDisplay: 'agent stopped', + }; + }); + const secondExecute = vi.fn(); + mockToolRegistry.getTool.mockReturnValue({ + name: core.ToolNames.AGENT, + kind: core.Kind.Think, + displayName: 'Agent', + description: 'Agent', + build: vi + .fn() + .mockImplementation((args: Record) => ({ + params: { subagent_type: 'explore', ...args }, + eventEmitter, + execute: + args['_test_id'] === 'first' ? firstExecute : secondExecute, + getDefaultPermission: vi.fn().mockResolvedValue('allow'), + getDescription: vi.fn().mockReturnValue('Agent'), + toolLocations: vi.fn().mockReturnValue([]), + })), + canUpdateOutput: false, + isOutputMarkdown: true, + }); + vi.mocked(mockClient.requestPermission).mockReturnValueOnce( + new Promise(() => {}), + ); + const abortController = new AbortController(); + + const runPromise = ( + session as unknown as ToolCallInternals + ).runToolCalls( + abortController.signal, + 'prompt-agent-nested-permission-parent-abort', + [ + { + id: 'agent_first', + name: core.ToolNames.AGENT, + args: { _test_id: 'first', subagent_type: 'explore' }, + }, + { + id: 'agent_second', + name: core.ToolNames.AGENT, + args: { _test_id: 'second', subagent_type: 'explore' }, + }, + ], + ); + + await vi.waitFor(() => { + expect(mockClient.requestPermission).toHaveBeenCalledOnce(); + }); + abortController.abort(); + const result = await runPromise; + + expect(result.stopAfterPermissionCancel).toBe(false); + expect(firstExecute).toHaveBeenCalledOnce(); + expect(secondExecute).not.toHaveBeenCalled(); + expect( + result.parts.map((part) => part.functionResponse?.response), + ).toEqual([ + { + error: 'The tool had already completed; its output was discarded.', + }, + { error: 'Tool call was cancelled before execution.' }, + ]); + await vi.waitFor(() => { + expect(respond).toHaveBeenCalledWith( + core.ToolConfirmationOutcome.Cancel, + ); + }); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + [result.parts[0]], + expect.objectContaining({ + callId: 'agent_first', + status: 'cancelled', + executionStatus: 'cancelled', + }), + ); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + [result.parts[1]], + expect.objectContaining({ + callId: 'agent_second', + status: 'cancelled', + executionStatus: 'not_started', + }), + ); + } finally { + if (previousMaxConcurrency === undefined) { + delete process.env['QWEN_CODE_MAX_TOOL_CONCURRENCY']; + } else { + process.env['QWEN_CODE_MAX_TOOL_CONCURRENCY'] = + previousMaxConcurrency; + } + } + }); + + it('does not start Agent batches with an already-aborted parent signal', async () => { const eventEmitter = new EventEmitter(); const receivedAbortStates: boolean[] = []; const execute = vi @@ -18061,8 +19510,30 @@ describe('Session', () => { ]); expect(result.stopAfterPermissionCancel).toBe(false); - expect(execute).toHaveBeenCalledTimes(2); - expect(receivedAbortStates).toEqual([true, true]); + expect(execute).not.toHaveBeenCalled(); + expect(receivedAbortStates).toEqual([]); + expect( + result.parts.map((part) => part.functionResponse?.response), + ).toEqual([ + { error: 'Tool call was cancelled before execution.' }, + { error: 'Tool call was cancelled before execution.' }, + ]); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + [result.parts[0]], + expect.objectContaining({ + callId: 'agent_first', + status: 'cancelled', + executionStatus: 'not_started', + }), + ); + expect(mockChatRecordingService.recordToolResult).toHaveBeenCalledWith( + [result.parts[1]], + expect.objectContaining({ + callId: 'agent_second', + status: 'cancelled', + executionStatus: 'not_started', + }), + ); }); it('skips unstarted Agent calls after nested ask_user_question cancellation', async () => { diff --git a/packages/cli/src/acp-integration/session/Session.ts b/packages/cli/src/acp-integration/session/Session.ts index 42e6e20d92d..60ba8faf4ee 100644 --- a/packages/cli/src/acp-integration/session/Session.ts +++ b/packages/cli/src/acp-integration/session/Session.ts @@ -37,6 +37,7 @@ import type { GoalTerminalEvent, ToolCallRequestInfo, ToolCallResponseInfo, + ToolExecutionStatus, LoopTickResult, ToolArtifact, VisionBridgeResult, @@ -421,8 +422,9 @@ type PendingToolResultRecord = { toolName: string; responseParts: Part[]; persistedOutputFiles?: string[]; - metadata: Partial & { + metadata: Omit, 'executionStatus'> & { status: 'success' | 'error' | 'cancelled'; + executionStatus: ToolExecutionStatus; }; }; @@ -445,6 +447,9 @@ const LOOP_DETECTED_SKIP_MESSAGE = 'Skipped because loop detection stopped the current turn before this tool call could run.'; const LOOP_DETECTED_CONTEXT_MESSAGE = 'System: this turn was terminated because the model exceeded tool-call safety limits. Try a different approach on the next turn.'; +const TOOL_EXECUTION_CANCELLED_MESSAGE = 'Tool execution was cancelled.'; +const TOOL_POST_EXECUTION_CANCELLED_MESSAGE = + 'The tool had already completed; its output was discarded.'; function createDaemonToolLoopState(): DaemonToolLoopState { return { @@ -464,7 +469,14 @@ function recordDaemonLoopDetected( if (!loopState.loopDetected) { loopState.loopDetected = true; debugLogger.warn(message); - logLoopDetected(config, new LoopDetectedEvent(loopType, promptId)); + try { + logLoopDetected(config, new LoopDetectedEvent(loopType, promptId)); + } catch (error) { + debugLogger.debug( + '[Session] Failed to record loop detection telemetry', + error, + ); + } } return true; } @@ -3352,7 +3364,10 @@ export class Session implements SessionContext { onFullTurnModel, ), ); - if (toolRun.stopAfterPermissionCancel) { + if ( + toolRun.stopAfterPermissionCancel || + pendingSend.signal.aborted + ) { this.todoStopGuard.suspend(); await this.#preserveStoppedToolRun( toolRun, @@ -4299,7 +4314,11 @@ export class Session implements SessionContext { options.onFullTurnModel, ), ); - if (toolRun.stopAfterPermissionCancel || toolRun.loopDetected) { + if ( + toolRun.stopAfterPermissionCancel || + toolRun.loopDetected || + pendingSend.signal.aborted + ) { this.todoStopGuard.suspend(); await this.#preserveStoppedToolRun(toolRun, pendingSend.signal); return { @@ -5658,7 +5677,7 @@ export class Session implements SessionContext { functionCalls, toolLoopState, ); - if (toolRun.stopAfterPermissionCancel) { + if (toolRun.stopAfterPermissionCancel || ac.signal.aborted) { this.todoStopGuard.suspend(); await this.#preserveStoppedToolRun(toolRun, ac.signal); return; @@ -6171,7 +6190,7 @@ export class Session implements SessionContext { functionCalls, toolLoopState, ); - if (toolRun.stopAfterPermissionCancel) { + if (toolRun.stopAfterPermissionCancel || ac.signal.aborted) { this.todoStopGuard.suspend(); await this.#preserveStoppedToolRun(toolRun, ac.signal); await this.#emitBackgroundNotificationEndTurn( @@ -6734,9 +6753,10 @@ export class Session implements SessionContext { metadata: { callId, status: 'error', + executionStatus: 'not_started', resultDisplay: undefined, error, - errorType, + errorType: errorType ?? ToolErrorType.EXECUTION_DENIED, }, }); if (emitStart) { @@ -6765,7 +6785,12 @@ export class Session implements SessionContext { return await finalizeRunToolResult({ parts: await Promise.all( dedupedFunctionCalls.map((fc) => - recordSkippedToolCall(fc, LOOP_DETECTED_SKIP_MESSAGE, false), + recordSkippedToolCall( + fc, + LOOP_DETECTED_SKIP_MESSAGE, + false, + ToolErrorType.UNKNOWN, + ), ), ), stopAfterPermissionCancel: false, @@ -6830,38 +6855,48 @@ export class Session implements SessionContext { const emitDuplicateBatch = async (batch: DuplicateBatch): Promise => { const { request, response } = batch; - if (request.name === ToolNames.TODO_WRITE) { - const provenance = ToolCallEmitter.resolveToolProvenance(request.name); - await this.sendUpdate({ - sessionUpdate: 'tool_call_update', - toolCallId: response.callId, - status: 'failed', - content: [ - { - type: 'content', - content: { - type: 'text', - text: response.error?.message ?? String(response.resultDisplay), + try { + if (request.name === ToolNames.TODO_WRITE) { + const provenance = ToolCallEmitter.resolveToolProvenance( + request.name, + ); + await this.sendUpdate({ + sessionUpdate: 'tool_call_update', + toolCallId: response.callId, + status: 'failed', + content: [ + { + type: 'content', + content: { + type: 'text', + text: + response.error?.message ?? String(response.resultDisplay), + }, }, + ], + rawOutput: response.resultDisplay, + _meta: { + toolName: request.name, + provenance: provenance.provenance, + ...(provenance.serverId ? { serverId: provenance.serverId } : {}), }, - ], - rawOutput: response.resultDisplay, - _meta: { + }); + } else { + await this.toolCallEmitter.emitResult({ + callId: response.callId, toolName: request.name, - provenance: provenance.provenance, - ...(provenance.serverId ? { serverId: provenance.serverId } : {}), - }, - }); - } else { - await this.toolCallEmitter.emitResult({ - callId: response.callId, - toolName: request.name, - args: request.args, - message: response.responseParts, - resultDisplay: response.resultDisplay, - error: response.error, - success: false, - }); + args: request.args, + message: response.responseParts, + resultDisplay: response.resultDisplay, + error: response.error, + success: false, + }); + } + } catch (emitError) { + debugLogger.debug( + '[Session.runToolCalls] Failed to emit duplicate tool update', + emitError, + ); } queueToolResultRecord(batch.fc, { callId: response.callId, @@ -6871,6 +6906,7 @@ export class Session implements SessionContext { metadata: { callId: response.callId, status: 'error', + executionStatus: response.executionStatus ?? 'not_started', resultDisplay: response.resultDisplay, error: response.error, errorType: response.errorType, @@ -6920,10 +6956,13 @@ export class Session implements SessionContext { parts: Part[], fc: FunctionCall, message = PERMISSION_CANCEL_SKIP_MESSAGE, + errorType?: ToolErrorType, ) => { const startIndex = dedupedFunctionCalls.indexOf(fc) + 1; for (const remainingCall of dedupedFunctionCalls.slice(startIndex)) { - parts.push(await recordSkippedToolCall(remainingCall, message)); + parts.push( + await recordSkippedToolCall(remainingCall, message, true, errorType), + ); } }; const memoryWriteCandidates: MemoryWriteCandidate[] = []; @@ -6965,7 +7004,12 @@ export class Session implements SessionContext { if (results[i]) continue; results[i] = { parts: [ - await recordSkippedToolCall(calls[i], LOOP_DETECTED_SKIP_MESSAGE), + await recordSkippedToolCall( + calls[i], + LOOP_DETECTED_SKIP_MESSAGE, + true, + ToolErrorType.UNKNOWN, + ), ], stopAfterPermissionCancel: false, loopDetected: true, @@ -7141,6 +7185,7 @@ export class Session implements SessionContext { parts, batch.calls[batch.calls.length - 1], LOOP_DETECTED_SKIP_MESSAGE, + ToolErrorType.UNKNOWN, ); return await finalizeRunToolResult({ parts, @@ -7177,7 +7222,12 @@ export class Session implements SessionContext { parts.push(...r.parts); collectMemoryWriteCandidates(r); if (r.loopDetected) { - await appendSkippedAfter(parts, fc, LOOP_DETECTED_SKIP_MESSAGE); + await appendSkippedAfter( + parts, + fc, + LOOP_DETECTED_SKIP_MESSAGE, + ToolErrorType.UNKNOWN, + ); return await finalizeRunToolResult({ parts, stopAfterPermissionCancel: false, @@ -7253,6 +7303,7 @@ export class Session implements SessionContext { fc: FunctionCall, message?: string, emitStart?: boolean, + errorType?: ToolErrorType, ) => Promise, queueToolResultRecord?: QueueToolResultRecord, generatedCallId?: string, @@ -7260,11 +7311,22 @@ export class Session implements SessionContext { ): Promise { const callId = fc.id ?? generatedCallId ?? `${fc.name}-${Date.now()}`; let args = (fc.args ?? {}) as Record; + let executionStatus: ToolExecutionStatus = 'not_started'; + let executionErrorType: ToolErrorType | undefined; + let executeReturned = false; + let terminalStatus: 'success' | 'error' | 'cancelled' | undefined; + let toolType: 'native' | 'mcp' = 'native'; + let mcpServerName: string | undefined = undefined; if (toolLoopState?.loopDetected) { return { parts: [ recordSkippedToolCall - ? await recordSkippedToolCall(fc, LOOP_DETECTED_SKIP_MESSAGE, false) + ? await recordSkippedToolCall( + fc, + LOOP_DETECTED_SKIP_MESSAGE, + false, + ToolErrorType.UNKNOWN, + ) : { functionResponse: { id: callId, @@ -7293,30 +7355,46 @@ export class Session implements SessionContext { removeAgentToolAbortPropagation = undefined; }; - const errorResponse = (error: Error) => { + const errorResponse = ( + error: Error, + toolName: string, + status: 'error' | 'cancelled', + errorType: ToolErrorType | undefined, + ) => { const durationMs = Date.now() - startTime; - logToolCall(this.config, { - 'event.name': 'tool_call', - 'event.timestamp': new Date().toISOString(), - prompt_id: promptId, - function_name: fc.name ?? '', - function_args: args, - duration_ms: durationMs, - // An aborted signal means the call was cancelled, not a genuine error. - status: activeToolAbortSignal.aborted ? 'cancelled' : 'error', - success: false, - error: error.message, - tool_type: - typeof tool !== 'undefined' && tool instanceof DiscoveredMCPTool - ? 'mcp' - : 'native', - }); + try { + logToolCall(this.config, { + 'event.name': 'tool_call', + 'event.timestamp': new Date().toISOString(), + call_id: callId, + prompt_id: promptId, + function_name: toolName, + function_args: args, + duration_ms: durationMs, + status, + execution_status: executionStatus, + success: false, + ...(status === 'error' + ? { + error: error.message, + error_type: errorType, + } + : {}), + tool_type: toolType, + mcp_server_name: mcpServerName, + }); + } catch (telemetryError) { + debugLogger.debug( + '[Session.runTool] Failed to record terminal tool telemetry', + telemetryError, + ); + } return [ { functionResponse: { id: callId, - name: fc.name ?? '', + name: toolName, response: { error: error.message }, }, }, @@ -7326,36 +7404,54 @@ export class Session implements SessionContext { const earlyErrorResponse = async ( error: Error, toolName = fc.name ?? 'unknown_tool', - opts?: { - errorType?: ToolErrorType; + opts: { + status: 'error' | 'cancelled'; + errorType: ToolErrorType | undefined; + executionStatus: ToolExecutionStatus; recordInvalidToolParams?: boolean; - status?: 'error' | 'cancelled'; stopAfterPermissionCancel?: boolean; }, ) => { - spanError = error.message; + executionStatus = opts.executionStatus; + terminalStatus = opts.status; + spanError = opts.status === 'error' ? error.message : undefined; cleanupAgentToolResources(); if (toolName !== ToolNames.TODO_WRITE) { - await this.toolCallEmitter.emitError(callId, toolName, error); + try { + await this.toolCallEmitter.emitError(callId, toolName, error); + } catch (emitError) { + debugLogger.debug( + '[Session.runTool] Failed to emit terminal tool update', + emitError, + ); + } } - const errorParts = errorResponse(error); + const errorParts = errorResponse( + error, + toolName, + opts.status, + opts.errorType, + ); queueToolResultRecord?.(fc, { callId, toolName, responseParts: errorParts, metadata: { callId, - status: opts?.status ?? 'error', + status: opts.status, + executionStatus, resultDisplay: undefined, - error, - errorType: opts?.errorType, + error: opts.status === 'error' ? error : undefined, + errorType: opts.status === 'error' ? opts.errorType : undefined, }, }); const loopDetected = - opts?.recordInvalidToolParams === true && + opts.recordInvalidToolParams === true && !activeToolAbortSignal.aborted && - !opts?.stopAfterPermissionCancel && + // A permission cancellation is the user declining, not the model + // re-sending invalid params, so it must not feed loop detection. + !opts.stopAfterPermissionCancel && recordDaemonInvalidToolParams( this.config, promptId, @@ -7365,15 +7461,40 @@ export class Session implements SessionContext { ); return { parts: errorParts, - stopAfterPermissionCancel: opts?.stopAfterPermissionCancel ?? false, + stopAfterPermissionCancel: opts.stopAfterPermissionCancel ?? false, loopDetected, }; }; + const cancelBeforeExecutionIfAborted = ( + toolName = fc.name ?? 'unknown_tool', + ) => + activeToolAbortSignal.aborted + ? earlyErrorResponse( + new Error('Tool call was cancelled before execution.'), + toolName, + { + status: 'cancelled', + errorType: undefined, + executionStatus: 'not_started', + }, + ) + : undefined; + + const initialCancellation = cancelBeforeExecutionIfAborted(); + if (initialCancellation) return initialCancellation; + if (!fc.name) { - return earlyErrorResponse(new Error('Missing function name'), undefined, { - recordInvalidToolParams: true, - }); + return earlyErrorResponse( + new Error('Missing function name'), + 'unknown_tool', + { + status: 'error', + errorType: ToolErrorType.INVALID_TOOL_PARAMS, + executionStatus: 'not_started', + recordInvalidToolParams: true, + }, + ); } const toolName = fc.name; @@ -7384,9 +7505,17 @@ export class Session implements SessionContext { return earlyErrorResponse( new Error(`Tool "${toolName}" not found in registry.`), toolName, - { recordInvalidToolParams: true }, + { + status: 'error', + errorType: ToolErrorType.TOOL_NOT_REGISTERED, + executionStatus: 'not_started', + recordInvalidToolParams: true, + }, ); } + toolType = tool instanceof DiscoveredMCPTool ? 'mcp' : 'native'; + mcpServerName = + tool instanceof DiscoveredMCPTool ? tool.serverName : undefined; const policyToolName = tool.name; const originalPolicyRequestArgs = policyToolName === ToolNames.SHELL || policyToolName === ToolNames.MONITOR @@ -7407,16 +7536,25 @@ export class Session implements SessionContext { tool.description, promptId, ); - let spanSuccess = false; - try { return await runInToolSpanContext(toolSpan, async () => { + const entryCancellation = cancelBeforeExecutionIfAborted(toolName); + if (entryCancellation) return entryCancellation; + // ---- L1: Tool enablement check ---- const pm = this.config.getPermissionManager?.(); - if (pm && !(await pm.isToolEnabled(policyToolName))) { + const toolEnabled = pm ? await pm.isToolEnabled(policyToolName) : true; + const enablementCancellation = cancelBeforeExecutionIfAborted(toolName); + if (enablementCancellation) return enablementCancellation; + if (pm && !toolEnabled) { return earlyErrorResponse( new Error(`Tool "${toolName}" is disabled.`), toolName, + { + status: 'error', + errorType: ToolErrorType.EXECUTION_DENIED, + executionStatus: 'not_started', + }, ); } @@ -7521,6 +7659,9 @@ export class Session implements SessionContext { policyToolName, toolParams, ); + const permissionFlowCancellation = + cancelBeforeExecutionIfAborted(toolName); + if (permissionFlowCancellation) return permissionFlowCancellation; const { finalPermission, pmForcedAsk, @@ -7541,6 +7682,11 @@ export class Session implements SessionContext { return earlyErrorResponse( new Error(denyMessage ?? `Tool "${toolName}" is denied.`), toolName, + { + status: 'error', + errorType: ToolErrorType.EXECUTION_DENIED, + executionStatus: 'not_started', + }, ); } @@ -7572,6 +7718,9 @@ export class Session implements SessionContext { signal: activeToolAbortSignal, }) : ({ classification: 'not-applicable' } as const); + const planPolicyCancellation = + cancelBeforeExecutionIfAborted(toolName); + if (planPolicyCancellation) return planPolicyCancellation; if (planShellDecision.classification !== 'not-applicable') { const initialPlanShellError = await validatePlanModeShellContext({ config: this.config, @@ -7580,10 +7729,20 @@ export class Session implements SessionContext { invocationParams: invocation.params as Record, signal: activeToolAbortSignal, }); + const initialPlanValidationCancellation = + cancelBeforeExecutionIfAborted(toolName); + if (initialPlanValidationCancellation) { + return initialPlanValidationCancellation; + } if (initialPlanShellError) { return earlyErrorResponse( new Error(initialPlanShellError), toolName, + { + status: 'error', + errorType: ToolErrorType.EXECUTION_DENIED, + executionStatus: 'not_started', + }, ); } } @@ -7591,6 +7750,11 @@ export class Session implements SessionContext { return earlyErrorResponse( new Error(planShellDecision.writeBlockMessage), toolName, + { + status: 'error', + errorType: ToolErrorType.EXECUTION_DENIED, + executionStatus: 'not_started', + }, ); } const planShellRequiresConfirmation = @@ -7660,6 +7824,9 @@ export class Session implements SessionContext { ? fallback.reason : undefined, }); + const autoModeCancellation = + cancelBeforeExecutionIfAborted(toolName); + if (autoModeCancellation) return autoModeCancellation; // Apply decision via shared helper — eliminates ~40 lines of // line-for-line duplication with coreToolScheduler.ts and makes @@ -7680,6 +7847,11 @@ export class Session implements SessionContext { callId, abortSignal, ); + const permissionDeniedHookCancellation = + cancelBeforeExecutionIfAborted(toolName); + if (permissionDeniedHookCancellation) { + return permissionDeniedHookCancellation; + } switch (outcome.kind) { case 'approved': autoModeAllowed = true; @@ -7692,6 +7864,11 @@ export class Session implements SessionContext { return earlyErrorResponse( new Error(outcome.errorMessage), toolName, + { + status: 'error', + errorType: ToolErrorType.EXECUTION_DENIED, + executionStatus: 'not_started', + }, ); case 'fallback': // Drop through to the manual-approval flow below. @@ -7755,6 +7932,11 @@ export class Session implements SessionContext { confirmationDetails = await invocation.getConfirmationDetails( activeToolAbortSignal, ); + const confirmationDetailsCancellation = + cancelBeforeExecutionIfAborted(toolName); + if (confirmationDetailsCancellation) { + return confirmationDetailsCancellation; + } if (autoModeFallbackMessage) { confirmationDetails = decorateClassifierUnavailableConfirmation( @@ -7775,10 +7957,20 @@ export class Session implements SessionContext { >, signal: activeToolAbortSignal, }); + const preDisplayValidationCancellation = + cancelBeforeExecutionIfAborted(toolName); + if (preDisplayValidationCancellation) { + return preDisplayValidationCancellation; + } if (preDisplayPlanShellError) { return earlyErrorResponse( new Error(preDisplayPlanShellError), toolName, + { + status: 'error', + errorType: ToolErrorType.EXECUTION_DENIED, + executionStatus: 'not_started', + }, ); } } @@ -7793,6 +7985,11 @@ export class Session implements SessionContext { return earlyErrorResponse( new Error(planShellDecision.noApprovalMessage), toolName, + { + status: 'error', + errorType: ToolErrorType.EXECUTION_DENIED, + executionStatus: 'not_started', + }, ); } throw new Error('Unable to prepare shell confirmation.'); @@ -7817,6 +8014,11 @@ export class Session implements SessionContext { 'Please use the exit_plan_mode tool to present your plan and exit plan mode before making changes.', ), toolName, + { + status: 'error', + errorType: ToolErrorType.EXECUTION_DENIED, + executionStatus: 'not_started', + }, ); } @@ -7833,6 +8035,11 @@ export class Session implements SessionContext { undefined, activeToolAbortSignal, ); + const permissionHookCancellation = + cancelBeforeExecutionIfAborted(toolName); + if (permissionHookCancellation) { + return permissionHookCancellation; + } if ( hookResult.hasDecision && @@ -7855,10 +8062,20 @@ export class Session implements SessionContext { ? { updatedInput: hookResult.updatedInput } : undefined, }); + const hookPlanApprovalCancellation = + cancelBeforeExecutionIfAborted(toolName); + if (hookPlanApprovalCancellation) { + return hookPlanApprovalCancellation; + } await confirmationDetails.onConfirm( approval.outcome, approval.payload, ); + const hookPlanConfirmationCancellation = + cancelBeforeExecutionIfAborted(toolName); + if (hookPlanConfirmationCancellation) { + return hookPlanConfirmationCancellation; + } if (approval.outcome === ToolConfirmationOutcome.Cancel) { return earlyErrorResponse( new Error( @@ -7866,6 +8083,11 @@ export class Session implements SessionContext { planShellDecision.noApprovalMessage, ), toolName, + { + status: 'error', + errorType: ToolErrorType.EXECUTION_DENIED, + executionStatus: 'not_started', + }, ); } recordAutoModeFallbackResolution(approval.outcome); @@ -7879,6 +8101,11 @@ export class Session implements SessionContext { await confirmationDetails.onConfirm( ToolConfirmationOutcome.ProceedOnce, ); + const hookConfirmationCancellation = + cancelBeforeExecutionIfAborted(toolName); + if (hookConfirmationCancellation) { + return hookConfirmationCancellation; + } recordAutoModeFallbackResolution( ToolConfirmationOutcome.ProceedOnce, ); @@ -7890,6 +8117,11 @@ export class Session implements SessionContext { `Permission denied by hook for "${toolName}"`, ), toolName, + { + status: 'error', + errorType: ToolErrorType.EXECUTION_DENIED, + executionStatus: 'not_started', + }, ); } } @@ -7918,10 +8150,20 @@ export class Session implements SessionContext { >, signal: activeToolAbortSignal, }); + const finalPlanValidationCancellation = + cancelBeforeExecutionIfAborted(toolName); + if (finalPlanValidationCancellation) { + return finalPlanValidationCancellation; + } if (finalPreDisplayPlanShellError) { return earlyErrorResponse( new Error(finalPreDisplayPlanShellError), toolName, + { + status: 'error', + errorType: ToolErrorType.EXECUTION_DENIED, + executionStatus: 'not_started', + }, ); } } @@ -7981,7 +8223,12 @@ export class Session implements SessionContext { message ?? `Tool "${toolName}" was canceled by the user.`, ), toolName, - { stopAfterPermissionCancel: true }, + { + status: 'cancelled', + errorType: undefined, + executionStatus: 'not_started', + stopAfterPermissionCancel: true, + }, ); }; @@ -7996,6 +8243,11 @@ export class Session implements SessionContext { )) as RequestPermissionResponse & { answers?: Record; }; + const permissionRequestCancellation = + cancelBeforeExecutionIfAborted(toolName); + if (permissionRequestCancellation) { + return permissionRequestCancellation; + } outcome = resolvePermissionOutcome( output, offeredPermissionOptions, @@ -8015,7 +8267,10 @@ export class Session implements SessionContext { confirmError, ); } - onStopAfterPermissionCancel?.(); + const wasAborted = activeToolAbortSignal.aborted; + if (!wasAborted) { + onStopAfterPermissionCancel?.(); + } const permissionFailureMessage = isExitPlanModeTool ? 'The host could not present plan-exit approval. Plan mode remains active; use the host mode selector or /plan exit to leave plan mode.' : planShellDecision.classification === 'unknown' @@ -8026,9 +8281,20 @@ export class Session implements SessionContext { error, )}`; return earlyErrorResponse( - new Error(permissionFailureMessage), + new Error( + wasAborted + ? 'Tool call was cancelled before execution.' + : permissionFailureMessage, + ), toolName, - { stopAfterPermissionCancel: true }, + { + status: wasAborted ? 'cancelled' : 'error', + errorType: wasAborted + ? undefined + : ToolErrorType.UNHANDLED_EXCEPTION, + executionStatus: 'not_started', + stopAfterPermissionCancel: !wasAborted, + }, ); } @@ -8048,6 +8314,11 @@ export class Session implements SessionContext { outcome, payload: confirmationPayload, }); + const planApprovalCancellation = + cancelBeforeExecutionIfAborted(toolName); + if (planApprovalCancellation) { + return planApprovalCancellation; + } outcome = approval.outcome; confirmationPayload = approval.payload; } @@ -8064,6 +8335,11 @@ export class Session implements SessionContext { outcome, confirmationPayload, ); + const confirmationCancellation = + cancelBeforeExecutionIfAborted(toolName); + if (confirmationCancellation) { + return confirmationCancellation; + } } catch (error) { if (outcome !== ToolConfirmationOutcome.Cancel) { throw error; @@ -8078,6 +8354,9 @@ export class Session implements SessionContext { if (shouldSwitchToDefault) { this.config.setApprovalMode(ApprovalMode.DEFAULT); await this.sendCurrentModeUpdateNotification(); + const modeUpdateCancellation = + cancelBeforeExecutionIfAborted(toolName); + if (modeUpdateCancellation) return modeUpdateCancellation; } // Persist permission rules when user explicitly chose "Always Allow". @@ -8096,6 +8375,11 @@ export class Session implements SessionContext { this.config.getPermissionManager?.(), confirmationPayload, ); + const permissionPersistenceCancellation = + cancelBeforeExecutionIfAborted(toolName); + if (permissionPersistenceCancellation) { + return permissionPersistenceCancellation; + } } // After edit tool ProceedAlways, notify the client about mode change @@ -8104,6 +8388,11 @@ export class Session implements SessionContext { outcome === ToolConfirmationOutcome.ProceedAlways ) { await this.sendCurrentModeUpdateNotification(); + const editModeUpdateCancellation = + cancelBeforeExecutionIfAborted(toolName); + if (editModeUpdateCancellation) { + return editModeUpdateCancellation; + } } switch (outcome) { @@ -8112,10 +8401,9 @@ export class Session implements SessionContext { 'Switch-to-Default outcome must be normalized before execution.', ); case ToolConfirmationOutcome.Cancel: - // Route through earlyErrorResponse so spanError carries the - // cancellation reason (plain errorResponse leaves it unset, - // which makes endToolSpan fall back to the generic 'tool - // error' message) and the declined call is still recorded. + // Route through the terminal helper so the declined call is + // emitted and recorded consistently without marking its span + // as an error. return stopAfterPermissionCancel( confirmationPayload?.cancelMessage, ); @@ -8145,7 +8433,17 @@ export class Session implements SessionContext { args, status: 'in_progress', }; - await this.toolCallEmitter.emitStart(startParams); + try { + await this.toolCallEmitter.emitStart(startParams); + } catch (emitError) { + debugLogger.debug( + '[Session.runTool] Failed to emit tool start update', + emitError, + ); + } + const startEmissionCancellation = + cancelBeforeExecutionIfAborted(toolName); + if (startEmissionCancellation) return startEmissionCancellation; } // Fire PreToolUse hook (aligned with core path in coreToolScheduler.ts) @@ -8163,15 +8461,32 @@ export class Session implements SessionContext { activeToolAbortSignal, callId, ); + const preHookCancellation = + cancelBeforeExecutionIfAborted(toolName); + if (preHookCancellation) return preHookCancellation; if (!preHookResult.shouldProceed) { // Hook blocked the tool execution - send notification to UI const blockReason = preHookResult.blockReason || 'Blocked by PreToolUse hook'; - await this.messageEmitter.emitAgentMessage( - `✗ **PreToolUse blocked**: ${toolName} - ${blockReason}`, - ); - return earlyErrorResponse(new Error(blockReason), toolName); + try { + await this.messageEmitter.emitAgentMessage( + `✗ **PreToolUse blocked**: ${toolName} - ${blockReason}`, + ); + } catch (emitError) { + debugLogger.debug( + '[Session.runTool] Failed to emit PreToolUse block message', + emitError, + ); + } + const blockMessageCancellation = + cancelBeforeExecutionIfAborted(toolName); + if (blockMessageCancellation) return blockMessageCancellation; + return earlyErrorResponse(new Error(blockReason), toolName, { + status: 'error', + errorType: ToolErrorType.EXECUTION_DENIED, + executionStatus: 'not_started', + }); } // Add additional context from PreToolUse hook if provided @@ -8199,18 +8514,32 @@ export class Session implements SessionContext { return earlyErrorResponse( new Error('Tool invocation was cancelled'), toolName, - { status: 'cancelled' }, + { + status: 'cancelled', + errorType: undefined, + executionStatus: 'not_started', + }, ); } if (!guardDecision.allowed) { return earlyErrorResponse( new Error(guardDecision.reason), toolName, - { errorType: ToolErrorType.EXECUTION_DENIED }, + { + status: 'error', + errorType: ToolErrorType.EXECUTION_DENIED, + executionStatus: 'not_started', + }, ); } } + const executionBoundaryCancellation = + cancelBeforeExecutionIfAborted(toolName); + if (executionBoundaryCancellation) { + return executionBoundaryCancellation; + } + const continuedAgentId = toolName === ToolNames.SEND_MESSAGE && typeof args['task_id'] === 'string' && @@ -8253,9 +8582,9 @@ export class Session implements SessionContext { } }; - const execSpan = startToolExecutionSpan(); let toolResult: ToolResult; let isExecutionTimeout = false; + let parentAbortedAtExecutionSettle = false; let aborted = false; // Shell liveness heartbeats: forwarded to the client as meta-only // tool_call_update frames so a headless gateway can tell a silent @@ -8294,58 +8623,102 @@ export class Session implements SessionContext { }, } : undefined; + const sleepInhibitorHandle = acquireSleepInhibitor( + this.config, + `Qwen Code is executing tool ${toolName}`, + ); try { - const sleepInhibitorHandle = acquireSleepInhibitor( - this.config, - `Qwen Code is executing tool ${toolName}`, - ); try { - try { - addToolArgumentsAttributes( - this.config, - toolSpan, - invocation.params, - ); - } catch { - debugLogger.debug( - '[Session.runTool] Failed to record tool arguments telemetry', - ); - } + addToolArgumentsAttributes( + this.config, + toolSpan, + invocation.params, + ); + } catch { + debugLogger.debug( + '[Session.runTool] Failed to record tool arguments telemetry', + ); + } + + const execSpan = startToolExecutionSpan({ + toolName: policyToolName, + callId, + }); + // Set the attempted outcome immediately before calling execute so + // synchronous throws are classified as execution failures. + executionStatus = 'error'; + try { toolResult = await invocation.execute( activeToolAbortSignal, onToolProgress, ); - } finally { - toolSettled = true; - sleepInhibitorHandle.release(); + executeReturned = true; + parentAbortedAtExecutionSettle = activeToolAbortSignal.aborted; + isExecutionTimeout = + toolResult.error?.type === ToolErrorType.EXECUTION_TIMEOUT; + aborted = parentAbortedAtExecutionSettle && !isExecutionTimeout; + executionStatus = aborted + ? 'cancelled' + : toolResult.error + ? 'error' + : 'success'; + executionErrorType = toolResult.error + ? (toolResult.error.type ?? + (toolType === 'mcp' + ? ToolErrorType.MCP_TOOL_ERROR + : ToolErrorType.UNKNOWN)) + : undefined; + settleRelatedAgent(executionStatus === 'success'); + endToolExecutionSpan(execSpan, { + success: executionStatus === 'success', + error: aborted + ? 'tool_cancelled' + : isExecutionTimeout + ? 'tool_timeout' + : toolResult.error + ? 'tool_error' + : undefined, + cancelled: aborted, + executionStatus, + errorType: executionErrorType, + ...heartbeatSpanAttributes(), + }); + } catch (execError) { + const explicitErrorType = ( + execError as { errorType?: ToolErrorType } | undefined + )?.errorType; + const executionTimedOut = + explicitErrorType === ToolErrorType.EXECUTION_TIMEOUT; + executionStatus = + activeToolAbortSignal.aborted && !executionTimedOut + ? 'cancelled' + : 'error'; + executionErrorType = + executionStatus === 'error' + ? (explicitErrorType ?? + (toolType === 'mcp' + ? ToolErrorType.MCP_TOOL_ERROR + : ToolErrorType.UNHANDLED_EXCEPTION)) + : undefined; + settleRelatedAgent(false); + endToolExecutionSpan(execSpan, { + success: false, + error: + executionStatus === 'cancelled' + ? 'tool_cancelled' + : executionTimedOut + ? 'tool_timeout' + : 'tool_exception', + cancelled: executionStatus === 'cancelled', + executionStatus, + errorType: executionErrorType, + ...heartbeatSpanAttributes(), + }); + throw execError; } - isExecutionTimeout = - toolResult.error?.type === ToolErrorType.EXECUTION_TIMEOUT; - aborted = activeToolAbortSignal.aborted && !isExecutionTimeout; - settleRelatedAgent(!toolResult.error && !aborted); - endToolExecutionSpan(execSpan, { - success: !toolResult.error && !aborted, - error: aborted - ? 'tool_cancelled' - : isExecutionTimeout - ? 'tool_timeout' - : toolResult.error - ? 'tool_error' - : undefined, - cancelled: aborted, - ...heartbeatSpanAttributes(), - }); - } catch (execError) { - settleRelatedAgent(false); - endToolExecutionSpan(execSpan, { - success: false, - error: activeToolAbortSignal.aborted - ? 'tool_cancelled' - : 'tool_exception', - cancelled: activeToolAbortSignal.aborted, - ...heartbeatSpanAttributes(), - }); - throw execError; + } finally { + toolSettled = true; + sleepInhibitorHandle.release(); } // Clean up event listeners @@ -8365,18 +8738,25 @@ export class Session implements SessionContext { } // Create response parts first (needed for emitResult and recordToolResult) - let responseParts = toolResult.error + let responseParts = aborted ? convertToFunctionErrorResponse( toolName, callId, - toolResult.llmContent, - toolResult.error.message, + TOOL_EXECUTION_CANCELLED_MESSAGE, + TOOL_EXECUTION_CANCELLED_MESSAGE, ) - : convertToFunctionResponse( - toolName, - callId, - toolResult.llmContent, - ); + : toolResult.error + ? convertToFunctionErrorResponse( + toolName, + callId, + toolResult.llmContent, + toolResult.error.message, + ) + : convertToFunctionResponse( + toolName, + callId, + toolResult.llmContent, + ); // A tool can fail "softly" by returning toolResult.error without // throwing, and can be cancelled mid-flight. Compute the real outcome @@ -8385,17 +8765,11 @@ export class Session implements SessionContext { // hardcoding success — otherwise failed/cancelled daemon/ACP tools // are mislabeled as successful in telemetry, session replay, and the // client UI. - const status: 'success' | 'error' | 'cancelled' = aborted + let status: 'success' | 'error' | 'cancelled' = aborted ? 'cancelled' : toolResult.error ? 'error' : 'success'; - const succeeded = status === 'success'; - const responseError = toolResult.error - ? new Error(toolResult.error.message) - : aborted - ? new Error('Tool execution was cancelled') - : undefined; if (isTrustedTodoWriteTool && !toolResult.error) { this.todoStopGuard.observeTodoWrite( @@ -8429,6 +8803,18 @@ export class Session implements SessionContext { callId, ); + if (activeToolAbortSignal.aborted) { + return earlyErrorResponse( + new Error(TOOL_POST_EXECUTION_CANCELLED_MESSAGE), + toolName, + { + status: 'cancelled', + errorType: undefined, + executionStatus, + }, + ); + } + // If hook indicates to stop, return an error response if (postHookResult.shouldStop) { const stopMessage = @@ -8438,7 +8824,11 @@ export class Session implements SessionContext { `PostToolUse hook requested stop for ${toolName}: ${stopMessage}`, ); this.todoStopGuard.suspend(); - return earlyErrorResponse(new Error(stopMessage), toolName); + return earlyErrorResponse(new Error(stopMessage), toolName, { + status: 'error', + errorType: ToolErrorType.EXECUTION_DENIED, + executionStatus, + }); } // Add additional context from PostToolUse hook if provided @@ -8460,30 +8850,35 @@ export class Session implements SessionContext { ) { const isInterrupt = aborted; // Fire PostToolUseFailure hook when a tool errors or resolves after cancellation. - const failureHookResult = await firePostToolUseFailureHook( - messageBusForTool, - toolUseId, - policyToolName, - args, - toolResult.error?.message ?? 'Tool execution was cancelled', - isInterrupt, - permissionMode, - activeToolAbortSignal, - callId, - ); - - // Log additional context if provided - if (failureHookResult.additionalContext) { + try { + const failureHookResult = await firePostToolUseFailureHook( + messageBusForTool, + toolUseId, + policyToolName, + args, + toolResult.error?.message ?? TOOL_EXECUTION_CANCELLED_MESSAGE, + isInterrupt, + permissionMode, + activeToolAbortSignal, + callId, + ); + if (failureHookResult.additionalContext) { + debugLogger.debug( + `PostToolUseFailure hook additional context for ${toolName}: ${failureHookResult.additionalContext}`, + ); + } + await this.emitHookArtifactsNotification({ + hookEventName: 'PostToolUseFailure', + toolName, + toolCallId: callId, + artifacts: failureHookResult.artifacts, + }); + } catch (hookError) { debugLogger.debug( - `PostToolUseFailure hook additional context for ${toolName}: ${failureHookResult.additionalContext}`, + '[Session.runTool] PostToolUseFailure hook failed', + hookError, ); } - await this.emitHookArtifactsNotification({ - hookEventName: 'PostToolUseFailure', - toolName, - toolCallId: callId, - artifacts: failureHookResult.artifacts, - }); } const visionBridgeNotices: string[] = []; @@ -8499,7 +8894,38 @@ export class Session implements SessionContext { ? visionBridgeNotices.join('\n') : undefined; if (visionBridgeNotice) { - await this.messageEmitter.emitAgentMessage(visionBridgeNotice); + try { + await this.messageEmitter.emitAgentMessage(visionBridgeNotice); + } catch (emitError) { + debugLogger.debug( + '[Session.runTool] Failed to emit vision bridge notice', + emitError, + ); + } + } + + if ( + activeToolAbortSignal.aborted && + !(isExecutionTimeout && parentAbortedAtExecutionSettle) + ) { + status = 'cancelled'; + responseParts = convertToFunctionErrorResponse( + toolName, + callId, + TOOL_POST_EXECUTION_CANCELLED_MESSAGE, + TOOL_POST_EXECUTION_CANCELLED_MESSAGE, + ); + } + terminalStatus = status; + const succeeded = status === 'success'; + const responseError = + status === 'error' && toolResult.error + ? new Error(toolResult.error.message) + : status === 'cancelled' + ? new Error(TOOL_POST_EXECUTION_CANCELLED_MESSAGE) + : undefined; + if (isTrustedTodoWriteTool && status === 'cancelled') { + this.todoStopGuard.suspend(); } // Handle TodoWriteTool: extract todos and send plan update @@ -8514,42 +8940,67 @@ export class Session implements SessionContext { plan && (plan.todos.length > 0 || Array.isArray(args['todos'])) ) { - await this.planEmitter.emitPlan(plan, callId); + try { + await this.planEmitter.emitPlan(plan, callId); + } catch (emitError) { + debugLogger.debug( + '[Session.runTool] Failed to emit plan update', + emitError, + ); + } } // Skip tool_call_update event for TodoWriteTool // Still log and return function response for LLM } else if (!isTodoWriteTool) { // Normal tool handling: emit result using ToolCallEmitter - await this.toolCallEmitter.emitResult({ - callId, - toolName, - args, - message: responseParts, - resultDisplay: toolResult.returnDisplay, - error: responseError, - success: succeeded, - artifacts: toolResult.artifacts, - }); + try { + await this.toolCallEmitter.emitResult({ + callId, + toolName, + args, + message: responseParts, + resultDisplay: toolResult.returnDisplay, + error: responseError, + success: succeeded, + artifacts: toolResult.artifacts, + }); + } catch (emitError) { + debugLogger.debug( + '[Session.runTool] Failed to emit terminal tool update', + emitError, + ); + } } const durationMs = Date.now() - startTime; - logToolCall(this.config, { - 'event.name': 'tool_call', - 'event.timestamp': new Date().toISOString(), - function_name: toolName, - function_args: args, - duration_ms: durationMs, - status, - success: succeeded, - error: toolResult.error?.message, - error_type: toolResult.error?.type, - prompt_id: promptId, - tool_type: - typeof tool !== 'undefined' && tool instanceof DiscoveredMCPTool - ? 'mcp' - : 'native', - }); + try { + logToolCall(this.config, { + 'event.name': 'tool_call', + 'event.timestamp': new Date().toISOString(), + call_id: callId, + function_name: toolName, + function_args: args, + duration_ms: durationMs, + status, + execution_status: executionStatus, + success: succeeded, + ...(status === 'error' + ? { + error: toolResult.error?.message, + error_type: executionErrorType, + } + : {}), + prompt_id: promptId, + tool_type: toolType, + mcp_server_name: mcpServerName, + }); + } catch (telemetryError) { + debugLogger.debug( + '[Session.runTool] Failed to record terminal tool telemetry', + telemetryError, + ); + } queueToolResultRecord?.(fc, { callId, @@ -8559,18 +9010,19 @@ export class Session implements SessionContext { metadata: { callId, status, + executionStatus, resultDisplay: toolResult.returnDisplay, ...(visionBridgeNotice !== undefined ? { visionBridgeNotice } : {}), - error: toolResult.error - ? new Error(toolResult.error.message) - : undefined, - errorType: toolResult.error?.type, + error: + status === 'error' && toolResult.error + ? new Error(toolResult.error.message) + : undefined, + errorType: status === 'error' ? executionErrorType : undefined, }, }); - spanSuccess = succeeded; if (succeeded && !nestedPermissionCancelled) { const result = responseParts.find( (part) => part.functionResponse !== undefined, @@ -8585,10 +9037,8 @@ export class Session implements SessionContext { } } } - if (toolResult.error) { + if (status === 'error' && toolResult.error) { spanError = toolResult.error.message; - } else if (aborted) { - spanError = 'Tool execution was cancelled'; } return { parts: responseParts, @@ -8605,80 +9055,95 @@ export class Session implements SessionContext { : undefined, }; } catch (e) { - // Ensure cleanup on error - cleanupAgentToolResources(); - const error = e instanceof Error ? e : new Error(String(e)); - spanError = error.message; - - // Fire PostToolUseFailure hook (aligned with core path in coreToolScheduler.ts) const hooksEnabledForError = !this.config.getDisableAllHooks?.(); const messageBusForError = this.config.getMessageBus?.(); - const isInterrupt = activeToolAbortSignal.aborted; + const executionTimeoutException = + !executeReturned && + executionErrorType === ToolErrorType.EXECUTION_TIMEOUT; + let status: 'cancelled' | 'error' = + executionStatus === 'cancelled' || + (activeToolAbortSignal.aborted && !executionTimeoutException) + ? 'cancelled' + : 'error'; + const isInterrupt = status === 'cancelled'; if (hooksEnabledForError && messageBusForError) { - const failureHookResult = await firePostToolUseFailureHook( - messageBusForError, - toolUseId, - policyToolName, - args, - error.message, - isInterrupt, - String(approvalMode), - activeToolAbortSignal, - callId, - ); - - // Log additional context if provided - if (failureHookResult.additionalContext) { + try { + const failureHookResult = await firePostToolUseFailureHook( + messageBusForError, + toolUseId, + policyToolName, + args, + error.message, + isInterrupt, + String(approvalMode), + activeToolAbortSignal, + callId, + ); + if (failureHookResult.additionalContext) { + debugLogger.debug( + `PostToolUseFailure hook additional context for ${toolName}: ${failureHookResult.additionalContext}`, + ); + } + await this.emitHookArtifactsNotification({ + hookEventName: 'PostToolUseFailure', + toolName, + toolCallId: callId, + artifacts: failureHookResult.artifacts, + }); + } catch (hookError) { debugLogger.debug( - `PostToolUseFailure hook additional context for ${toolName}: ${failureHookResult.additionalContext}`, + '[Session.runTool] PostToolUseFailure hook failed', + hookError, ); } - await this.emitHookArtifactsNotification({ - hookEventName: 'PostToolUseFailure', - toolName, - toolCallId: callId, - artifacts: failureHookResult.artifacts, - }); } - // Use ToolCallEmitter for error handling - await this.toolCallEmitter.emitError(callId, toolName, error); - - const loopDetected = - !activeToolAbortSignal.aborted && - !toolBuildSucceeded && - recordDaemonInvalidToolParams( - this.config, - promptId, - toolLoopState, - toolName, - error, - ); + if (activeToolAbortSignal.aborted && !executionTimeoutException) { + status = 'cancelled'; + } - const responseParts = errorResponse(error); - queueToolResultRecord?.(fc, { - callId, - toolName, - responseParts, - metadata: { - callId, - status: activeToolAbortSignal.aborted ? 'cancelled' : 'error', - resultDisplay: undefined, - error, - errorType: undefined, - }, - }); - return { - parts: responseParts, + const explicitErrorType = ( + e as { errorType?: ToolErrorType } | undefined + )?.errorType; + const errorType = + status === 'cancelled' + ? undefined + : (explicitErrorType ?? + (executeReturned + ? ToolErrorType.UNHANDLED_EXCEPTION + : (executionErrorType ?? + (!toolBuildSucceeded + ? ToolErrorType.INVALID_TOOL_PARAMS + : ToolErrorType.UNHANDLED_EXCEPTION)))); + return earlyErrorResponse(error, toolName, { + status, + errorType, + executionStatus, + recordInvalidToolParams: !toolBuildSucceeded, stopAfterPermissionCancel: nestedPermissionCancelled, - loopDetected, - }; + }); } }); // end runInToolSpanContext + } catch (e) { + const error = e instanceof Error ? e : new Error(String(e)); + const status = activeToolAbortSignal.aborted ? 'cancelled' : 'error'; + return await earlyErrorResponse(error, toolName, { + status, + errorType: + status === 'error' ? ToolErrorType.UNHANDLED_EXCEPTION : undefined, + executionStatus, + }); } finally { - endToolSpan(toolSpan, { success: spanSuccess, error: spanError }); + if (terminalStatus === 'cancelled') { + endToolSpan(toolSpan, { success: false, cancelled: true }); + } else { + endToolSpan(toolSpan, { + success: terminalStatus === 'success', + error: spanError, + }); + } } } diff --git a/packages/cli/src/acp-integration/session/SubAgentTracker.test.ts b/packages/cli/src/acp-integration/session/SubAgentTracker.test.ts index d4856a68cb3..45f2c8c5d53 100644 --- a/packages/cli/src/acp-integration/session/SubAgentTracker.test.ts +++ b/packages/cli/src/acp-integration/session/SubAgentTracker.test.ts @@ -15,6 +15,7 @@ import type { AgentToolResultEvent, AgentApprovalRequestEvent, AgentStreamTextEvent, + AgentUsageEvent, ToolEditConfirmationDetails, ToolInfoConfirmationDetails, } from '@qwen-code/qwen-code-core'; @@ -352,6 +353,34 @@ describe('SubAgentTracker', () => { }); }); + it('treats rejected nested tool updates as best-effort', async () => { + sendUpdateSpy.mockRejectedValue(new Error('client unavailable')); + tracker.setup(eventEmitter, abortController.signal); + + eventEmitter.emit( + AgentEventType.TOOL_CALL, + createToolCallEvent({ + name: 'read_file', + callId: 'call-best-effort', + args: { path: '/test.ts' }, + }), + ); + eventEmitter.emit( + AgentEventType.TOOL_RESULT, + createToolResultEvent({ + name: 'read_file', + callId: 'call-best-effort', + success: true, + resultDisplay: 'contents', + }), + ); + + await vi.waitFor(() => { + expect(sendUpdateSpy).toHaveBeenCalledTimes(2); + }); + await Promise.resolve(); + }); + // Subagent todo state is isolated from the parent session plan: a // subagent's TodoWrite result must not promote into a session-level // plan update. The guard lives in ToolCallEmitter.emitResult, keyed on @@ -687,6 +716,40 @@ describe('SubAgentTracker', () => { ); }); + it('does not report parent abort as an explicit nested permission cancellation', async () => { + requestPermissionSpy.mockReturnValue(new Promise(() => {})); + const onPermissionCancel = vi.fn(); + tracker = new SubAgentTracker( + mockContext, + mockClient, + 'parent-call-123', + 'test-subagent', + onPermissionCancel, + ); + tracker.setup(eventEmitter, abortController.signal); + + const respondSpy = vi.fn().mockResolvedValue(undefined); + eventEmitter.emit( + AgentEventType.TOOL_WAITING_APPROVAL, + createApprovalEvent({ + name: 'shell', + callId: 'call-shell', + confirmationDetails: createInfoConfirmation(), + respond: respondSpy, + }), + ); + + await vi.waitFor(() => { + expect(requestPermissionSpy).toHaveBeenCalledOnce(); + }); + abortController.abort(); + await vi.waitFor(() => { + expect(respondSpy).toHaveBeenCalledWith(ToolConfirmationOutcome.Cancel); + }); + + expect(onPermissionCancel).not.toHaveBeenCalled(); + }); + it('notifies when nested permission failure cannot respond', async () => { requestPermissionSpy.mockRejectedValue(new Error('Network error')); const onPermissionCancel = vi.fn(); @@ -874,6 +937,39 @@ describe('SubAgentTracker', () => { }); describe('stream text handling', () => { + it.each([ + [ + 'stream text', + AgentEventType.STREAM_TEXT, + () => + createStreamTextEvent({ + text: 'best-effort stream text', + }), + ], + [ + 'usage metadata', + AgentEventType.USAGE_METADATA, + () => + ({ + subagentId: 'test-subagent', + round: 1, + timestamp: Date.now(), + usage: { promptTokenCount: 1 }, + durationMs: 5, + }) satisfies AgentUsageEvent, + ], + ])('treats rejected %s updates as best-effort', async (_, type, event) => { + sendUpdateSpy.mockRejectedValue(new Error('client unavailable')); + tracker.setup(eventEmitter, abortController.signal); + + eventEmitter.emit(type, event()); + + await vi.waitFor(() => { + expect(sendUpdateSpy).toHaveBeenCalledOnce(); + }); + await Promise.resolve(); + }); + it('should emit agent_message_chunk on STREAM_TEXT event', async () => { tracker.setup(eventEmitter, abortController.signal); diff --git a/packages/cli/src/acp-integration/session/SubAgentTracker.ts b/packages/cli/src/acp-integration/session/SubAgentTracker.ts index 172497b892f..c141b906fea 100644 --- a/packages/cli/src/acp-integration/session/SubAgentTracker.ts +++ b/packages/cli/src/acp-integration/session/SubAgentTracker.ts @@ -150,12 +150,19 @@ export class SubAgentTracker { }); // Use unified emitter - handles TodoWriteTool skipping internally - void this.toolCallEmitter.emitStart({ - toolName: event.name, - callId: event.callId, - args: event.args, - subagentMeta: this.subagentMeta, - }); + void this.toolCallEmitter + .emitStart({ + toolName: event.name, + callId: event.callId, + args: event.args, + subagentMeta: this.subagentMeta, + }) + .catch((error) => { + debugLogger.debug( + `Failed to emit subagent tool start for ${event.name}:`, + error, + ); + }); }; } @@ -172,15 +179,22 @@ export class SubAgentTracker { const state = this.toolStates.get(event.callId); // Use unified emitter - handles TodoWriteTool plan updates internally - void this.toolCallEmitter.emitResult({ - toolName: event.name, - callId: event.callId, - success: event.success, - message: event.responseParts ?? [], - resultDisplay: event.resultDisplay, - args: state?.args, - subagentMeta: this.subagentMeta, - }); + void this.toolCallEmitter + .emitResult({ + toolName: event.name, + callId: event.callId, + success: event.success, + message: event.responseParts ?? [], + resultDisplay: event.resultDisplay, + args: state?.args, + subagentMeta: this.subagentMeta, + }) + .catch((error) => { + debugLogger.debug( + `Failed to emit subagent tool result for ${event.name}:`, + error, + ); + }); // Clean up state this.toolStates.delete(event.callId); @@ -251,7 +265,10 @@ export class SubAgentTracker { ? (output.answers as Record | undefined) : undefined, }); - if (outcome === ToolConfirmationOutcome.Cancel) { + if ( + outcome === ToolConfirmationOutcome.Cancel && + !abortSignal.aborted + ) { this.onPermissionCancel?.(); } } catch (error) { @@ -263,7 +280,9 @@ export class SubAgentTracker { // Fail closed: if the client cannot answer a nested permission // request, stop the parent turn instead of letting later tools run // without the required user input. - this.onPermissionCancel?.(); + if (!abortSignal.aborted) { + this.onPermissionCancel?.(); + } try { await event.respond(ToolConfirmationOutcome.Cancel); } catch (respondError) { @@ -286,12 +305,11 @@ export class SubAgentTracker { const event = args[0] as AgentUsageEvent; if (abortSignal.aborted) return; - this.messageEmitter.emitUsageMetadata( - event.usage, - '', - event.durationMs, - this.subagentMeta, - ); + void this.messageEmitter + .emitUsageMetadata(event.usage, '', event.durationMs, this.subagentMeta) + .catch((error) => { + debugLogger.debug('Failed to emit subagent usage metadata:', error); + }); }; } @@ -307,13 +325,17 @@ export class SubAgentTracker { if (abortSignal.aborted) return; // Emit streamed text as agent message or thought based on the flag - void this.messageEmitter.emitMessage( - event.text, - 'assistant', - event.thought ?? false, - undefined, - this.subagentMeta, - ); + void this.messageEmitter + .emitMessage( + event.text, + 'assistant', + event.thought ?? false, + undefined, + this.subagentMeta, + ) + .catch((error) => { + debugLogger.debug('Failed to emit subagent stream text:', error); + }); }; } } diff --git a/packages/cli/src/nonInteractiveCli.test.ts b/packages/cli/src/nonInteractiveCli.test.ts index 03d024ee960..21bd699958b 100644 --- a/packages/cli/src/nonInteractiveCli.test.ts +++ b/packages/cli/src/nonInteractiveCli.test.ts @@ -1336,7 +1336,10 @@ describe('runNonInteractive', () => { }, }; const toolResponse: Part[] = [{ text: 'Tool response' }]; - mockCoreExecuteToolCall.mockResolvedValue({ responseParts: toolResponse }); + mockCoreExecuteToolCall.mockResolvedValue({ + responseParts: toolResponse, + executionStatus: 'success', + }); const firstCallEvents: ServerGeminiStreamEvent[] = [toolCallEvent]; const secondCallEvents: ServerGeminiStreamEvent[] = [ @@ -1703,6 +1706,20 @@ describe('runNonInteractive', () => { it('isolates enter_plan_mode from headless siblings without charging skipped calls to the budget', async () => { setupMetricsMock(); + const recordToolResult = vi.fn(); + ( + mockConfig as Config & { + getChatRecordingService: () => { + recordToolResult: typeof recordToolResult; + finalize: ReturnType; + flush: ReturnType; + }; + } + ).getChatRecordingService = () => ({ + recordToolResult, + finalize: vi.fn(), + flush: vi.fn().mockResolvedValue(undefined), + }); vi.mocked(mockConfig.getMaxToolCalls).mockReturnValue(1); vi.mocked(mockToolRegistry.getTool).mockImplementation( (name: string) => @@ -1787,6 +1804,12 @@ describe('runNonInteractive', () => { expect(nextTurnParts[3].functionResponse?.response).toEqual({ error: PLAN_MODE_ENTRY_SIBLING_SKIP_MESSAGE, }); + expect( + recordToolResult.mock.calls + .map((call) => call[1]) + .filter((metadata) => metadata.callId !== 'enter-plan') + .map((metadata) => metadata.executionStatus), + ).toEqual(['not_started', 'not_started', 'not_started']); }); it('runs a batch of concurrency-safe tool calls concurrently', async () => { @@ -2528,6 +2551,7 @@ describe('runNonInteractive', () => { resultDisplay: 'Tool response', error: undefined, errorType: undefined, + executionStatus: 'success', }; await options.onAllToolCallsComplete?.([ { request, response, status: 'success' }, @@ -2575,6 +2599,9 @@ describe('runNonInteractive', () => { 'success', 'error', ]); + expect( + recordToolResult.mock.calls.map((call) => call[1].executionStatus), + ).toEqual(['success', 'not_started']); expect(processStdoutSpy).toHaveBeenCalledWith('Final answer\n'); }); @@ -4804,6 +4831,8 @@ describe('runNonInteractive', () => { expect(toolResultBlock?.tool_use_id).toBe('tool-1'); expect(toolResultBlock?.is_error).toBe(false); expect(toolResultBlock?.content).toBe('Tool executed successfully'); + expect(writes.join('')).not.toContain('executionStatus'); + expect(writes.join('')).not.toContain('execution_status'); }); it('should emit tool errors in tool_result blocks in stream-json format', async () => { @@ -4834,6 +4863,7 @@ describe('runNonInteractive', () => { mockCoreExecuteToolCall.mockResolvedValue({ error: new Error('Tool execution failed'), errorType: ToolErrorType.EXECUTION_FAILED, + executionStatus: 'error', responseParts: [ { functionResponse: { @@ -4898,6 +4928,8 @@ describe('runNonInteractive', () => { ); expect(toolResultBlock?.tool_use_id).toBe('tool-error'); expect(toolResultBlock?.is_error).toBe(true); + expect(writes.join('')).not.toContain('executionStatus'); + expect(writes.join('')).not.toContain('execution_status'); }); it('should emit partial messages when includePartialMessages is true', async () => { diff --git a/packages/cli/src/nonInteractiveCli.ts b/packages/cli/src/nonInteractiveCli.ts index 9d62ab38709..1535fab0b4a 100644 --- a/packages/cli/src/nonInteractiveCli.ts +++ b/packages/cli/src/nonInteractiveCli.ts @@ -1512,6 +1512,7 @@ export async function runNonInteractive( resultDisplay: error.message, error, errorType: ToolErrorType.EXECUTION_DENIED, + executionStatus: 'not_started', }); responseByRequest.set(requestInfo, { callId: requestInfo.callId, @@ -1519,6 +1520,7 @@ export async function runNonInteractive( resultDisplay: error.message, error, errorType: ToolErrorType.EXECUTION_DENIED, + executionStatus: 'not_started', }); executedRequests.add(requestInfo); }; @@ -1670,6 +1672,7 @@ export async function runNonInteractive( resultDisplay: skippedOutput, error: undefined, errorType: undefined, + executionStatus: 'not_started', }; adapter.emitToolResult(call, toolResponse); responseByRequest.set(call, toolResponse); @@ -1718,6 +1721,7 @@ export async function runNonInteractive( resultDisplay: response.resultDisplay, error: response.error, errorType: response.errorType, + executionStatus: response.executionStatus, }); } diff --git a/packages/cli/src/ui/hooks/useGeminiStream.test.tsx b/packages/cli/src/ui/hooks/useGeminiStream.test.tsx index 3dea6b5c00c..c13624f1356 100644 --- a/packages/cli/src/ui/hooks/useGeminiStream.test.tsx +++ b/packages/cli/src/ui/hooks/useGeminiStream.test.tsx @@ -4481,6 +4481,7 @@ describe('useGeminiStream', () => { resultDisplay: 'first', error: undefined, errorType: undefined, + executionStatus: 'success', persistedOutputFiles: [], }, tool: { @@ -4525,6 +4526,9 @@ describe('useGeminiStream', () => { expect(recordToolResult.mock.calls.flatMap((call) => call[0])).toEqual( toolResultParts, ); + expect( + recordToolResult.mock.calls.map((call) => call[1].executionStatus), + ).toEqual(expect.arrayContaining(['success', 'not_started'])); expect(client.recordCompletedToolCall).toHaveBeenCalledTimes(1); }); diff --git a/packages/cli/src/ui/hooks/useGeminiStream.ts b/packages/cli/src/ui/hooks/useGeminiStream.ts index 0837efb7518..6245513d2ec 100644 --- a/packages/cli/src/ui/hooks/useGeminiStream.ts +++ b/packages/cli/src/ui/hooks/useGeminiStream.ts @@ -3336,6 +3336,7 @@ export const useGeminiStream = ( resultDisplay: response.resultDisplay, error: response.error, errorType: response.errorType, + executionStatus: response.executionStatus, }, goalContext ? request.name === ToolNames.GET_GOAL || @@ -3985,6 +3986,7 @@ export const useGeminiStream = ( resultDisplay: response.resultDisplay, error: response.error, errorType: response.errorType, + executionStatus: response.executionStatus, }, goalContext ? request.name === ToolNames.GET_GOAL || diff --git a/packages/cli/src/ui/hooks/useReactToolScheduler.ts b/packages/cli/src/ui/hooks/useReactToolScheduler.ts index 9ac33108558..bd822e1f9c9 100644 --- a/packages/cli/src/ui/hooks/useReactToolScheduler.ts +++ b/packages/cli/src/ui/hooks/useReactToolScheduler.ts @@ -263,6 +263,7 @@ export function useReactToolScheduler( resultDisplay: message, error: toolError, errorType: ToolErrorType.UNHANDLED_EXCEPTION, + executionStatus: 'not_started', contentLength: message.length, }, }; @@ -355,7 +356,11 @@ export function mapToDisplay( let description: string; let renderOutputAsMarkdown = false; - if (trackedCall.status === 'error') { + if ( + trackedCall.status === 'error' || + trackedCall.tool === undefined || + trackedCall.invocation === undefined + ) { displayName = trackedCall.tool === undefined ? trackedCall.request.name diff --git a/packages/cli/src/ui/hooks/useToolScheduler.test.ts b/packages/cli/src/ui/hooks/useToolScheduler.test.ts index 0a701be3e2c..48093c8c294 100644 --- a/packages/cli/src/ui/hooks/useToolScheduler.test.ts +++ b/packages/cli/src/ui/hooks/useToolScheduler.test.ts @@ -451,6 +451,7 @@ describe('useReactToolScheduler', () => { error: expect.objectContaining({ message: expect.stringContaining('tool was not executed'), }), + executionStatus: 'not_started', }), }), ]); @@ -499,6 +500,7 @@ describe('useReactToolScheduler', () => { error: expect.objectContaining({ message: expect.stringContaining('tool was not executed'), }), + executionStatus: 'not_started', }), }), ]); @@ -962,6 +964,20 @@ describe('mapToDisplay', () => { expectedName: baseTool.displayName, expectedDescription: baseInvocation.getDescription(), }, + { + name: 'cancelled before tool resolution', + status: 'cancelled', + extraProps: { + response: { + ...baseResponse, + resultDisplay: 'Cancelled before resolution', + }, + }, + expectedStatus: ToolCallStatus.Canceled, + expectedResultDisplay: 'Cancelled before resolution', + expectedName: baseRequest.name, + expectedDescription: JSON.stringify(baseRequest.args), + }, ]; testCases.forEach( diff --git a/packages/core/src/core/coreToolScheduler.test.ts b/packages/core/src/core/coreToolScheduler.test.ts index 7f6f1857c82..93abb9dea23 100644 --- a/packages/core/src/core/coreToolScheduler.test.ts +++ b/packages/core/src/core/coreToolScheduler.test.ts @@ -12,9 +12,11 @@ import type { ChatRecordingService, Config, ToolCallConfirmationDetails, + ToolCallRequestInfo, ToolConfirmationPayload, ToolInvocation, ToolInvocationGuard, + ToolExecutionStatus, ToolResult, ToolResultDisplay, ToolRegistry, @@ -52,7 +54,7 @@ import { extractToolFilePaths, isToolCallConcurrencySafe, } from './coreToolScheduler.js'; -import type { Part, PartListUnion } from '@google/genai'; +import type { CallableTool, Part, PartListUnion } from '@google/genai'; import { MockModifiableTool, MockTool, @@ -69,6 +71,7 @@ import type { MessageBus } from '../confirmation-bus/message-bus.js'; import { IdeClient } from '../ide/ide-client.js'; import { WriteFileTool } from '../tools/write-file.js'; import { ShellTool, ShellToolInvocation } from '../tools/shell.js'; +import { DiscoveredMCPTool } from '../tools/mcp-tool.js'; import type { ShellToolParams } from '../tools/shell.js'; import type { ShellExecutionConfig } from '../services/shellExecutionService.js'; import { @@ -100,7 +103,13 @@ type ToolSpanRecord = { * Metadata passed to endToolSpan / endToolExecutionSpan — captured so * tests can assert success/error/cancelled values are forwarded correctly. */ - endMetadata?: { success?: boolean; error?: string; cancelled?: boolean }; + endMetadata?: { + success?: boolean; + error?: string; + cancelled?: boolean; + executionStatus?: ToolExecutionStatus; + errorType?: string; + }; /** Metadata passed to endToolBlockedOnUserSpan. */ blockedMetadata?: { decision?: string; source?: string }; /** Metadata passed to endHookSpan. */ @@ -132,6 +141,14 @@ const debugLoggerWarnSpy = vi.hoisted(() => vi.fn()); const debugLoggerInfoSpy = vi.hoisted(() => vi.fn()); const runSideQueryMock = vi.hoisted(() => vi.fn()); const mockTelemetrySdkState = vi.hoisted(() => ({ initialized: false })); +const modifyWithEditorOverride = vi.hoisted(() => ({ + value: undefined as + | (() => Promise<{ + updatedParams: Record; + updatedDiff: string; + }>) + | undefined, +})); vi.mock('../utils/debugLogger.js', async (importOriginal) => { const actual = @@ -168,6 +185,16 @@ vi.mock('../utils/sideQuery.js', () => ({ runSideQuery: (...args: unknown[]) => runSideQueryMock(...args), })); +vi.mock('../tools/modifiable-tool.js', async (importOriginal) => { + const actual = + await importOriginal(); + return { + ...actual, + modifyWithEditor: (...args: Parameters) => + modifyWithEditorOverride.value?.() ?? actual.modifyWithEditor(...args), + }; +}); + vi.mock('../telemetry/sdk.js', async (importOriginal) => { const actual = await importOriginal(); return { @@ -248,11 +275,23 @@ vi.mock('../telemetry/session-tracing.js', () => ({ }, ), runInToolSpanContext: vi.fn((_span: unknown, fn: () => T): T => fn()), - startToolExecutionSpan: vi.fn(() => createMockToolSpan('tool.execution', {})), + startToolExecutionSpan: vi.fn( + (options?: { toolName?: string; callId?: string }) => + createMockToolSpan('tool.execution', { + ...(options?.toolName ? { 'gen_ai.tool.name': options.toolName } : {}), + ...(options?.callId ? { 'tool.call_id': options.callId } : {}), + }), + ), endToolExecutionSpan: vi.fn( ( span: ToolSpanRecord & ReturnType, - metadata?: { success?: boolean; error?: string; cancelled?: boolean }, + metadata?: { + success?: boolean; + error?: string; + cancelled?: boolean; + executionStatus?: ToolExecutionStatus; + errorType?: string; + }, ) => { if (metadata) { span.endMetadata = metadata; @@ -585,12 +624,14 @@ describe('CoreToolScheduler', () => { beforeEach(() => { debugLoggerInfoSpy.mockClear(); runSideQueryMock.mockReset(); + modifyWithEditorOverride.value = undefined; }); type SchedulerDenialTrackingInternals = { toolCalls: ToolCall[]; autoModeFallbackCallIds: Set; drainSpansForBatch: (callIds: Iterable) => void; + finalizeToolSpan: (callId: string, force?: boolean) => void; _handleConfirmationResponseInner: ( callId: string, toolCall: ToolCall, @@ -728,6 +769,15 @@ describe('CoreToolScheduler', () => { } }); + it('cleans denialTracking fallback call ids when finalizeToolSpan runs', () => { + const { internals } = createSchedulerForDenialTrackingApprovalTest(); + internals.autoModeFallbackCallIds.add('call-1'); + + internals.finalizeToolSpan('call-1'); + + expect(internals.autoModeFallbackCallIds.has('call-1')).toBe(false); + }); + function createSchedulerForLegacyToolTests(options: { toolsByName: Map; approvalMode?: ApprovalMode; @@ -737,6 +787,7 @@ describe('CoreToolScheduler', () => { firePermissionDeniedEvent: ReturnType; }; disableHooks?: boolean; + hooksEnabled?: () => boolean; autoModeDenialState?: { consecutiveBlock: number; consecutiveUnavailable: number; @@ -746,6 +797,7 @@ describe('CoreToolScheduler', () => { setAutoModeDenialState?: ReturnType; setApprovalMode?: ReturnType; onAllToolCallsComplete?: ReturnType; + disableCompletionCallback?: boolean; onToolCallsUpdate?: ReturnType; memoryMonitor?: { scheduleCheck: () => void }; toolOutputBatchBudget?: number; @@ -833,11 +885,13 @@ describe('CoreToolScheduler', () => { getChatRecordingService: () => undefined, getMemoryPressureMonitor: () => options.memoryMonitor, getMessageBus: vi.fn().mockReturnValue(options.messageBus), - hasHooksForEvent: vi.fn().mockReturnValue(!options.disableHooks), + hasHooksForEvent: vi.fn( + () => options.hooksEnabled?.() ?? !(options.disableHooks ?? true), + ), getHookSystem: vi.fn().mockReturnValue(options.hookSystem), - getDisableAllHooks: vi - .fn() - .mockReturnValue(options.disableHooks ?? true), + getDisableAllHooks: vi.fn( + () => !(options.hooksEnabled?.() ?? !(options.disableHooks ?? true)), + ), getAutoModeDenialState: () => options.autoModeDenialState ?? { consecutiveBlock: 0, @@ -855,7 +909,9 @@ describe('CoreToolScheduler', () => { getExperimentalZedIntegration: () => false, getActiveTodoWorkChainOwner: options.getActiveTodoWorkChainOwner, } as unknown as Config, - onAllToolCallsComplete, + onAllToolCallsComplete: options.disableCompletionCallback + ? undefined + : onAllToolCallsComplete, onToolCallsUpdate, getPreferredEditor: () => 'vscode', onEditorClose: vi.fn(), @@ -1308,6 +1364,72 @@ describe('CoreToolScheduler', () => { expect(fnCall.args!['originalRequest']).toBe('please plan this'); }); + it('redacts an approved plan before post-processing cancellation completes', async () => { + const bigPlan = '## Plan\n\nprivate implementation details'; + const planFile = path.join( + os.tmpdir(), + `qwen-plan-cancel-${process.pid}-${Math.random().toString(16).slice(2)}.md`, + ); + fsSync.writeFileSync(planFile, bigPlan, 'utf-8'); + const chat = createChatWithPlanCall('plan-call-cancel', bigPlan); + const abortController = new AbortController(); + const messageBus = { + request: vi.fn(async (request: { eventName: string }) => { + if (request.eventName === 'PostToolUse') { + abortController.abort(); + } + return { + type: MessageBusType.HOOK_EXECUTION_RESPONSE, + correlationId: `${request.eventName}-hook`, + success: true, + output: { decision: 'allow' }, + }; + }), + }; + const tool = new MockTool({ + name: ToolNames.EXIT_PLAN_MODE, + execute: vi.fn().mockResolvedValue({ + llmContent: + 'User approved. You can now start coding. Start with updating your todo list if applicable.', + returnDisplay: 'User approved.', + }), + }); + const onAllToolCallsComplete = vi.fn(); + const { scheduler } = createSchedulerForLegacyToolTests({ + toolsByName: new Map([[ToolNames.EXIT_PLAN_MODE, tool]]), + approvalMode: ApprovalMode.YOLO, + messageBus, + disableHooks: false, + onAllToolCallsComplete, + getGeminiClient: () => ({ getChat: () => chat }), + getPlanFilePath: () => planFile, + }); + + await scheduler.schedule( + [ + { + callId: 'plan-call-cancel', + name: ToolNames.EXIT_PLAN_MODE, + args: { plan: bigPlan }, + isClientInitiated: false, + prompt_id: 'prompt-plan-cancel', + }, + ], + abortController.signal, + ); + await vi.waitFor(() => expect(onAllToolCallsComplete).toHaveBeenCalled()); + + const completedCall = onAllToolCallsComplete.mock.calls[0][0][0]; + expect(completedCall).toMatchObject({ + status: 'cancelled', + response: { executionStatus: 'success' }, + }); + const plan = chat.getHistory()[1]!.parts![1]!.functionCall!.args!['plan']; + expect(plan).not.toContain('private implementation details'); + expect(plan).toContain(`Plan approved and saved to ${planFile}`); + fsSync.unlinkSync(planFile); + }); + it('redacts the plan for a leader-approved (teammate) exit_plan_mode', async () => { const bigPlan = '## Plan\n\nleader path fixture'; const planFile = path.join( @@ -1734,6 +1856,75 @@ describe('CoreToolScheduler', () => { expect(completedCalls.every((call) => call.status === 'success')).toBe( true, ); + expect( + completedCalls.every( + (call) => + (call as CompletedToolCall).response.executionStatus === 'success', + ), + ).toBe(true); + }); + + it('resolves rather than rejects when a tool execution throws (#8180)', async () => { + // A per-call terminal error is reported on the returned call; schedule() + // must resolve, not reject, so one tool's failure cannot abort its + // siblings (load-bearing contract change, see the design doc). + const execute = vi.fn(async (): Promise => { + throw new Error('execution blew up'); + }); + const healthyExecute = vi.fn().mockResolvedValue({ + llmContent: 'healthy', + returnDisplay: 'healthy', + }); + const toolsByName = new Map([ + ['read_file', new MockTool({ name: 'read_file', execute })], + [ + 'healthy_tool', + new MockTool({ name: 'healthy_tool', execute: healthyExecute }), + ], + ]); + const { scheduler, onAllToolCallsComplete } = + createSchedulerForLegacyToolTests({ toolsByName }); + + await expect( + scheduler.schedule( + [ + { + callId: 'throws-1', + name: 'read_file', + args: { file_path: 'a.ts' }, + isClientInitiated: false, + prompt_id: 'prompt-throws', + }, + { + callId: 'healthy-1', + name: 'healthy_tool', + args: {}, + isClientInitiated: false, + prompt_id: 'prompt-throws', + }, + ], + new AbortController().signal, + ), + ).resolves.toBeUndefined(); + + const completedCalls = onAllToolCallsComplete.mock + .calls[0][0] as ToolCall[]; + const failedCall = completedCalls.find( + (c) => c.request.callId === 'throws-1', + ); + expect(failedCall?.status).toBe('error'); + if (failedCall?.status === 'error') { + expect(failedCall.response.executionStatus).toBe('error'); + expect(failedCall.response.error?.message).toContain('execution blew up'); + } + expect(healthyExecute).toHaveBeenCalledOnce(); + const healthyCall = completedCalls.find( + (c) => c.request.callId === 'healthy-1', + ); + expect(healthyCall?.status).toBe('success'); + if (healthyCall?.status === 'success') { + expect(healthyCall.response.executionStatus).toBe('success'); + } }); it('aborts and fails a tool call that exceeds the execution timeout', async () => { @@ -1785,6 +1976,7 @@ describe('CoreToolScheduler', () => { )[0]; expect(completedCall.status).toBe('error'); if (completedCall.status === 'error') { + expect(completedCall.response.executionStatus).toBe('error'); expect(completedCall.response.errorType).toBe( ToolErrorType.EXECUTION_TIMEOUT, ); @@ -2602,6 +2794,11 @@ describe('CoreToolScheduler', () => { 'response' in call ? call.response.responseParts : [], ), ); + expect( + recordToolResult.mock.calls.every( + ([, result]) => result.executionStatus === 'success', + ), + ).toBe(true); }); it('hard-caps a batch whose producer outputs already carry truncation markers', async () => { @@ -3086,71 +3283,162 @@ describe('CoreToolScheduler', () => { expect(ensureTool).not.toHaveBeenCalled(); }); - it('fires PermissionDenied hooks for AUTO classifier blocks', async () => { + it('preserves cancellation when permission evaluation resolves after abort', async () => { + toolSpanRecords.length = 0; + const abortController = new AbortController(); + const execute = vi.fn(); + const tool = new MockTool({ + name: 'abort-during-permission', + getDefaultPermission: async () => { + abortController.abort(); + return 'deny'; + }, + execute, + }); + const { scheduler, onAllToolCallsComplete } = + createSchedulerForLegacyToolTests({ + toolsByName: new Map([[tool.name, tool]]), + }); + + await scheduler.schedule( + { + callId: 'abort-during-permission', + name: tool.name, + args: {}, + isClientInitiated: false, + prompt_id: 'prompt-abort-during-permission', + }, + abortController.signal, + ); + + await vi.waitFor(() => + expect(onAllToolCallsComplete).toHaveBeenCalledOnce(), + ); + const completedCall = onAllToolCallsComplete.mock + .calls[0][0][0] as CompletedToolCall; + expect(completedCall.status).toBe('cancelled'); + expect(completedCall.response.executionStatus).toBe('not_started'); + expect(execute).not.toHaveBeenCalled(); + const toolSpan = toolSpanRecords.findLast( + (record) => + record.name === `tool.${tool.name}` && + record.attributes['tool.call_id'] === 'abort-during-permission', + ); + expect(toolSpan?.spanAttributes['tool.failure_kind']).toBe('cancelled'); + expect(toolSpan?.statusCalls.at(-1)?.code).toBe(SpanStatusCode.UNSET); + expect(toolSpan?.ended).toBe(true); + }); + + it('preserves cancellation when AUTO classification resolves after abort', async () => { + toolSpanRecords.length = 0; + const abortController = new AbortController(); runSideQueryMock .mockResolvedValueOnce({ shouldBlock: true }) - .mockResolvedValueOnce({ - shouldBlock: true, - reason: 'dangerous shell command', + .mockImplementationOnce(async () => { + abortController.abort(); + return { + shouldBlock: true, + reason: 'dangerous shell command', + }; }); - const execute = vi.fn().mockResolvedValue({ - llmContent: 'should not execute', - returnDisplay: 'should not execute', + const execute = vi.fn(); + const tool = new MockTool({ + name: ToolNames.SHELL, + getDefaultPermission: MOCK_TOOL_GET_DEFAULT_PERMISSION, + getConfirmationDetails: MOCK_TOOL_GET_CONFIRMATION_DETAILS, + execute, }); - const toolsByName = new Map([ - [ - ToolNames.SHELL, - new MockTool({ - name: ToolNames.SHELL, - getDefaultPermission: MOCK_TOOL_GET_DEFAULT_PERMISSION, - getConfirmationDetails: MOCK_TOOL_GET_CONFIRMATION_DETAILS, - execute, - }), - ], - ]); const hookSystem = { firePermissionDeniedEvent: vi.fn().mockResolvedValue(undefined), }; const { scheduler, onAllToolCallsComplete } = createSchedulerForLegacyToolTests({ - toolsByName, + toolsByName: new Map([[tool.name, tool]]), approvalMode: ApprovalMode.AUTO, hookSystem, disableHooks: false, }); - const abortController = new AbortController(); await scheduler.schedule( - [ - { - callId: 'auto-denied', - name: ToolNames.SHELL, - args: { command: 'rm -rf /tmp/example' }, - isClientInitiated: false, - prompt_id: 'prompt-auto-denied', - }, - ], + { + callId: 'abort-during-auto', + name: tool.name, + args: { command: 'rm -rf /tmp/example' }, + isClientInitiated: false, + prompt_id: 'prompt-abort-during-auto', + }, abortController.signal, ); - await vi.waitFor(() => { - expect(onAllToolCallsComplete).toHaveBeenCalled(); + await vi.waitFor(() => + expect(onAllToolCallsComplete).toHaveBeenCalledOnce(), + ); + const completedCall = onAllToolCallsComplete.mock + .calls[0][0][0] as CompletedToolCall; + expect(completedCall.status).toBe('cancelled'); + expect(completedCall.response.executionStatus).toBe('not_started'); + expect(hookSystem.firePermissionDeniedEvent).not.toHaveBeenCalled(); + expect(execute).not.toHaveBeenCalled(); + const toolSpan = toolSpanRecords.findLast( + (record) => + record.name === `tool.${tool.name}` && + record.attributes['tool.call_id'] === 'abort-during-auto', + ); + expect(toolSpan?.spanAttributes['tool.failure_kind']).toBe('cancelled'); + expect(toolSpan?.statusCalls.at(-1)?.code).toBe(SpanStatusCode.UNSET); + expect(toolSpan?.ended).toBe(true); + }); + + it('cleans AUTO fallback state when confirmation preparation is cancelled', async () => { + const abortController = new AbortController(); + runSideQueryMock + .mockResolvedValueOnce({ shouldBlock: true }) + .mockRejectedValueOnce(new Error('classifier unavailable')); + const execute = vi.fn(); + const tool = new MockTool({ + name: ToolNames.SHELL, + getDefaultPermission: MOCK_TOOL_GET_DEFAULT_PERMISSION, + getConfirmationDetails: async () => { + abortController.abort(); + return MOCK_TOOL_GET_CONFIRMATION_DETAILS(); + }, + execute, }); - expect(hookSystem.firePermissionDeniedEvent).toHaveBeenCalledWith( - ToolNames.SHELL, - { command: 'rm -rf /tmp/example' }, - 'auto-denied', - 'classifier_blocked', + const { scheduler, onAllToolCallsComplete } = + createSchedulerForLegacyToolTests({ + toolsByName: new Map([[tool.name, tool]]), + approvalMode: ApprovalMode.AUTO, + }); + + await scheduler.schedule( + { + callId: 'cancelled-auto-fallback', + name: tool.name, + args: { command: 'touch /tmp/example' }, + isClientInitiated: false, + prompt_id: 'prompt-cancelled-auto-fallback', + }, abortController.signal, - 'auto-denied', ); + + await vi.waitFor(() => + expect(onAllToolCallsComplete).toHaveBeenCalledOnce(), + ); + const completedCall = onAllToolCallsComplete.mock + .calls[0][0][0] as CompletedToolCall; + expect(completedCall.status).toBe('cancelled'); + expect(completedCall.response.executionStatus).toBe('not_started'); expect(execute).not.toHaveBeenCalled(); - const completedCalls = onAllToolCallsComplete.mock - .calls[0][0] as ToolCall[]; - expect(completedCalls[0].status).toBe('error'); + expect( + ( + scheduler as unknown as { + autoModeFallbackCallIds: Set; + } + ).autoModeFallbackCallIds.has('cancelled-auto-fallback'), + ).toBe(false); }); - it('continues AUTO block handling when PermissionDenied hook fails', async () => { + it('fires PermissionDenied hooks for AUTO classifier blocks', async () => { runSideQueryMock .mockResolvedValueOnce({ shouldBlock: true }) .mockResolvedValueOnce({ @@ -3173,9 +3461,7 @@ describe('CoreToolScheduler', () => { ], ]); const hookSystem = { - firePermissionDeniedEvent: vi - .fn() - .mockRejectedValueOnce(new Error('hook failed')), + firePermissionDeniedEvent: vi.fn().mockResolvedValue(undefined), }; const { scheduler, onAllToolCallsComplete } = createSchedulerForLegacyToolTests({ @@ -3184,31 +3470,214 @@ describe('CoreToolScheduler', () => { hookSystem, disableHooks: false, }); + const abortController = new AbortController(); await scheduler.schedule( [ { - callId: 'auto-denied-hook-fails', + callId: 'auto-denied', name: ToolNames.SHELL, args: { command: 'rm -rf /tmp/example' }, isClientInitiated: false, - prompt_id: 'prompt-auto-denied-hook-fails', + prompt_id: 'prompt-auto-denied', }, ], - new AbortController().signal, + abortController.signal, ); await vi.waitFor(() => { expect(onAllToolCallsComplete).toHaveBeenCalled(); }); - expect(hookSystem.firePermissionDeniedEvent).toHaveBeenCalled(); - expect(execute).not.toHaveBeenCalled(); - const completedCalls = onAllToolCallsComplete.mock - .calls[0][0] as ToolCall[]; - const completedCall = completedCalls[0]; - expect(completedCall.status).toBe('error'); - if (completedCall.status === 'error') { - expect(completedCall.response.errorType).toBe( + expect(hookSystem.firePermissionDeniedEvent).toHaveBeenCalledWith( + ToolNames.SHELL, + { command: 'rm -rf /tmp/example' }, + 'auto-denied', + 'classifier_blocked', + abortController.signal, + 'auto-denied', + ); + expect(execute).not.toHaveBeenCalled(); + const completedCalls = onAllToolCallsComplete.mock + .calls[0][0] as ToolCall[]; + expect(completedCalls[0].status).toBe('error'); + const toolSpan = toolSpanRecords.findLast( + (record) => + record.name === `tool.${ToolNames.SHELL}` && + record.attributes['tool.call_id'] === 'auto-denied', + ); + expect(toolSpan?.spanAttributes['success']).toBe(false); + expect(toolSpan?.spanAttributes['tool.failure_kind']).toBe( + 'permission_denied', + ); + expect(toolSpan?.statusCalls.at(-1)?.code).toBe(SpanStatusCode.ERROR); + expect(toolSpan?.ended).toBe(true); + }); + + it('marks invalid PermissionRequest rewrites as pre-execution span failures', async () => { + const execute = vi.fn(); + const onConfirm = vi.fn().mockResolvedValue(undefined); + const tool = new MockTool({ + name: 'rewrite-target', + kind: Kind.Edit, + params: { + type: 'object', + properties: { value: { type: 'string' } }, + required: ['value'], + additionalProperties: false, + }, + getDefaultPermission: MOCK_TOOL_GET_DEFAULT_PERMISSION, + getConfirmationDetails: async () => ({ + type: 'exec', + title: 'Confirm rewrite-target', + command: 'rewrite-target', + rootCommand: 'rewrite-target', + onConfirm, + }), + execute, + }); + const build = tool.build.bind(tool); + const buildSpy = vi.spyOn(tool, 'build').mockImplementation((params) => { + if ('unexpected' in params) { + throw new Error('invalid permission rewrite'); + } + return build(params); + }); + const messageBus = { + request: vi.fn().mockImplementation( + async (request: { + eventName: string; + }): Promise => ({ + type: MessageBusType.HOOK_EXECUTION_RESPONSE, + correlationId: `${request.eventName}-hook`, + success: true, + output: + request.eventName === 'PermissionRequest' + ? { + hookSpecificOutput: { + decision: { + behavior: 'allow', + updatedInput: { unexpected: true }, + }, + }, + } + : { decision: 'allow' }, + }), + ), + }; + const { scheduler, onAllToolCallsComplete } = + createSchedulerForLegacyToolTests({ + toolsByName: new Map([[tool.name, tool]]), + approvalMode: ApprovalMode.DEFAULT, + messageBus, + disableHooks: false, + }); + + await scheduler.schedule( + [ + { + callId: 'invalid-permission-rewrite', + name: tool.name, + args: { value: 'original' }, + isClientInitiated: false, + prompt_id: 'prompt-invalid-permission-rewrite', + }, + ], + new AbortController().signal, + ); + + await vi.waitFor(() => { + expect(onAllToolCallsComplete).toHaveBeenCalledOnce(); + }); + const completedCalls = onAllToolCallsComplete.mock + .calls[0][0] as CompletedToolCall[]; + expect(buildSpy.mock.calls).toEqual([ + [{ value: 'original' }], + [{ unexpected: true }], + ]); + expect(completedCalls[0]?.status).toBe('error'); + expect(completedCalls[0]?.response.error?.message).toBe( + 'invalid permission rewrite', + ); + expect(completedCalls[0]?.response.errorType).toBe( + ToolErrorType.INVALID_TOOL_PARAMS, + ); + expect(completedCalls[0]?.response.executionStatus).toBe('not_started'); + expect(completedCalls[0]?.outcome).toBeUndefined(); + expect(onConfirm).not.toHaveBeenCalled(); + expect(execute).not.toHaveBeenCalled(); + + const toolSpan = toolSpanRecords.findLast( + (record) => + record.name === `tool.${tool.name}` && + record.attributes['tool.call_id'] === 'invalid-permission-rewrite', + ); + expect(toolSpan?.spanAttributes['success']).toBe(false); + expect(toolSpan?.spanAttributes['tool.failure_kind']).toBe( + 'tool_exception', + ); + expect(toolSpan?.statusCalls.at(-1)?.code).toBe(SpanStatusCode.ERROR); + expect(toolSpan?.ended).toBe(true); + }); + + it('continues AUTO block handling when PermissionDenied hook fails', async () => { + runSideQueryMock + .mockResolvedValueOnce({ shouldBlock: true }) + .mockResolvedValueOnce({ + shouldBlock: true, + reason: 'dangerous shell command', + }); + const execute = vi.fn().mockResolvedValue({ + llmContent: 'should not execute', + returnDisplay: 'should not execute', + }); + const toolsByName = new Map([ + [ + ToolNames.SHELL, + new MockTool({ + name: ToolNames.SHELL, + getDefaultPermission: MOCK_TOOL_GET_DEFAULT_PERMISSION, + getConfirmationDetails: MOCK_TOOL_GET_CONFIRMATION_DETAILS, + execute, + }), + ], + ]); + const hookSystem = { + firePermissionDeniedEvent: vi + .fn() + .mockRejectedValueOnce(new Error('hook failed')), + }; + const { scheduler, onAllToolCallsComplete } = + createSchedulerForLegacyToolTests({ + toolsByName, + approvalMode: ApprovalMode.AUTO, + hookSystem, + disableHooks: false, + }); + + await scheduler.schedule( + [ + { + callId: 'auto-denied-hook-fails', + name: ToolNames.SHELL, + args: { command: 'rm -rf /tmp/example' }, + isClientInitiated: false, + prompt_id: 'prompt-auto-denied-hook-fails', + }, + ], + new AbortController().signal, + ); + + await vi.waitFor(() => { + expect(onAllToolCallsComplete).toHaveBeenCalled(); + }); + expect(hookSystem.firePermissionDeniedEvent).toHaveBeenCalled(); + expect(execute).not.toHaveBeenCalled(); + const completedCalls = onAllToolCallsComplete.mock + .calls[0][0] as ToolCall[]; + const completedCall = completedCalls[0]; + expect(completedCall.status).toBe('error'); + if (completedCall.status === 'error') { + expect(completedCall.response.errorType).toBe( ToolErrorType.EXECUTION_DENIED, ); } @@ -3754,6 +4223,252 @@ describe('CoreToolScheduler', () => { ).toBe(0); }); + it('keeps a valid batch parent span open when the last response has no span', async () => { + const execute = vi.fn().mockResolvedValue({ + llmContent: 'alpha output', + returnDisplay: 'alpha output', + }); + let releasePostToolBatch: + | ((response: HookExecutionResponse) => void) + | undefined; + const messageBus = { + request: vi + .fn() + .mockImplementation( + (request: { eventName: string }): Promise => { + if (request.eventName !== 'PostToolBatch') { + return Promise.resolve({ + type: MessageBusType.HOOK_EXECUTION_RESPONSE, + correlationId: `${request.eventName}-hook`, + success: true, + output: { decision: 'allow' }, + }); + } + return new Promise((resolve) => { + releasePostToolBatch = resolve; + }); + }, + ), + }; + const onAllToolCallsComplete = vi.fn(); + const { scheduler } = createSchedulerForLegacyToolTests({ + toolsByName: new Map([ + [ + 'alpha', + new MockTool({ + name: 'alpha', + kind: Kind.Read, + execute, + }), + ], + ]), + messageBus, + disableHooks: false, + onAllToolCallsComplete, + }); + + await scheduler.schedule( + [ + { + callId: 'mixed-alpha', + name: 'alpha', + args: {}, + isClientInitiated: false, + prompt_id: 'prompt-mixed-batch', + }, + { + callId: 'mixed-invalid-tail', + name: 'missing_tool', + args: {}, + isClientInitiated: false, + prompt_id: 'prompt-mixed-batch', + }, + ], + new AbortController().signal, + ); + + await vi.waitFor(() => { + expect( + messageBus.request.mock.calls.some( + ([request]) => request.eventName === 'PostToolBatch', + ), + ).toBe(true); + }); + const alphaSpan = toolSpanRecords.findLast( + (record) => + record.name === 'tool.alpha' && + record.attributes['tool.call_id'] === 'mixed-alpha', + ); + expect(alphaSpan?.spanAttributes['success']).toBe(true); + expect(alphaSpan?.ended).toBe(false); + expect(onAllToolCallsComplete).not.toHaveBeenCalled(); + + releasePostToolBatch?.({ + type: MessageBusType.HOOK_EXECUTION_RESPONSE, + correlationId: 'PostToolBatch-hook', + success: true, + output: { decision: 'allow' }, + }); + await vi.waitFor(() => { + expect(onAllToolCallsComplete).toHaveBeenCalledOnce(); + expect(alphaSpan?.ended).toBe(true); + }); + }); + + it('passes the scheduling abort signal to an invalid-only PostToolBatch hook', async () => { + const abortController = new AbortController(); + const onAllToolCallsComplete = vi.fn(); + const messageBus = { + request: vi + .fn() + .mockImplementation( + (request: { + eventName: string; + signal?: AbortSignal; + }): Promise => { + if (request.eventName !== 'PostToolBatch') { + return Promise.resolve({ + type: MessageBusType.HOOK_EXECUTION_RESPONSE, + correlationId: `${request.eventName}-hook`, + success: true, + output: { decision: 'allow' }, + }); + } + return new Promise((resolve) => { + const finish = () => + resolve({ + type: MessageBusType.HOOK_EXECUTION_RESPONSE, + correlationId: 'PostToolBatch-hook', + success: true, + output: { decision: 'allow' }, + }); + if (request.signal?.aborted) { + finish(); + } else { + request.signal?.addEventListener('abort', finish, { + once: true, + }); + } + }); + }, + ), + }; + const { scheduler } = createSchedulerForLegacyToolTests({ + toolsByName: new Map(), + messageBus, + disableHooks: false, + onAllToolCallsComplete, + }); + + await scheduler.schedule( + [ + { + callId: 'invalid-only', + name: 'missing_tool', + args: {}, + isClientInitiated: false, + prompt_id: 'prompt-invalid-only', + }, + ], + abortController.signal, + ); + + await vi.waitFor(() => { + expect( + messageBus.request.mock.calls.some( + ([request]) => request.eventName === 'PostToolBatch', + ), + ).toBe(true); + }); + const batchRequest = messageBus.request.mock.calls.find( + ([request]) => request.eventName === 'PostToolBatch', + )?.[0]; + expect(batchRequest.signal).toBe(abortController.signal); + expect(onAllToolCallsComplete).not.toHaveBeenCalled(); + + abortController.abort(); + await vi.waitFor(() => { + expect(onAllToolCallsComplete).toHaveBeenCalledOnce(); + }); + }); + + it('snapshots PostToolBatch enablement at the batch boundary', async () => { + let hooksEnabled = false; + let resolveExecution!: (result: { + llmContent: string; + returnDisplay: string; + }) => void; + const execution = new Promise<{ + llmContent: string; + returnDisplay: string; + }>((resolve) => { + resolveExecution = resolve; + }); + const execute = vi.fn().mockReturnValue(execution); + const messageBus = { + request: vi.fn().mockResolvedValue({ + type: MessageBusType.HOOK_EXECUTION_RESPONSE, + correlationId: 'hook', + success: true, + output: { decision: 'allow' }, + }), + }; + const onAllToolCallsComplete = vi.fn(); + const { scheduler } = createSchedulerForLegacyToolTests({ + toolsByName: new Map([ + [ + 'alpha', + new MockTool({ + name: 'alpha', + kind: Kind.Read, + execute, + }), + ], + ]), + messageBus, + hooksEnabled: () => hooksEnabled, + onAllToolCallsComplete, + }); + + const schedulePromise = scheduler.schedule( + [ + { + callId: 'hook-snapshot-alpha', + name: 'alpha', + args: {}, + isClientInitiated: false, + prompt_id: 'prompt-hook-snapshot', + }, + ], + new AbortController().signal, + ); + await vi.waitFor(() => { + expect(execute).toHaveBeenCalledOnce(); + }); + + hooksEnabled = true; + resolveExecution({ + llmContent: 'alpha output', + returnDisplay: 'alpha output', + }); + await schedulePromise; + await vi.waitFor(() => { + expect(onAllToolCallsComplete).toHaveBeenCalledOnce(); + }); + + expect( + messageBus.request.mock.calls.some( + ([request]) => request.eventName === 'PostToolBatch', + ), + ).toBe(false); + const alphaSpan = toolSpanRecords.findLast( + (record) => + record.name === 'tool.alpha' && + record.attributes['tool.call_id'] === 'hook-snapshot-alpha', + ); + expect(alphaSpan?.ended).toBe(true); + }); + it('bridges image tool results before completing the tool call', async () => { runSideQueryMock.mockResolvedValue({ text: 'Screen says READY' }); const execute = vi.fn().mockResolvedValue({ @@ -4132,6 +4847,7 @@ describe('CoreToolScheduler', () => { tool_response: expect.objectContaining({ error: undefined, error_type: undefined, + execution_status: 'success', }), }), expect.objectContaining({ @@ -4140,6 +4856,7 @@ describe('CoreToolScheduler', () => { tool_response: expect.objectContaining({ error: 'beta failed', error_type: ToolErrorType.UNHANDLED_EXCEPTION, + execution_status: 'error', }), }), expect.objectContaining({ @@ -4148,6 +4865,7 @@ describe('CoreToolScheduler', () => { tool_response: expect.objectContaining({ error: 'gamma failed', error_type: ToolErrorType.UNKNOWN, + execution_status: 'error', }), }), ], @@ -4382,6 +5100,137 @@ describe('CoreToolScheduler', () => { }); }); + it('waits for scheduling to unwind before draining an early terminal queue', async () => { + const deniedTool = new MockTool({ + name: 'denied', + getDefaultPermission: async () => 'deny', + }); + const secondTool = new MockTool({ name: 'second' }); + const thirdTool = new MockTool({ name: 'third' }); + const toolsByName = new Map([ + [deniedTool.name, deniedTool], + [secondTool.name, secondTool], + [thirdTool.name, thirdTool], + ]); + const { scheduler, ensureTool } = createSchedulerForLegacyToolTests({ + toolsByName, + disableCompletionCallback: true, + }); + + let markSecondLookupStarted!: () => void; + const secondLookupStarted = new Promise((resolve) => { + markSecondLookupStarted = resolve; + }); + let releaseSecondLookup!: () => void; + const secondLookupRelease = new Promise((resolve) => { + releaseSecondLookup = resolve; + }); + const thirdLookup = vi.fn(); + ensureTool.mockImplementation(async (name: string) => { + if (name === secondTool.name) { + markSecondLookupStarted(); + await secondLookupRelease; + } else if (name === thirdTool.name) { + thirdLookup(); + } + const tool = toolsByName.get(name); + if (!tool) { + throw new Error(`Missing test tool: ${name}`); + } + return tool; + }); + + const request = (callId: string, name: string): ToolCallRequestInfo => ({ + callId, + name, + args: {}, + isClientInitiated: false, + prompt_id: `prompt-${callId}`, + }); + const firstSchedule = scheduler.schedule( + request('first-call', deniedTool.name), + new AbortController().signal, + ); + const secondSchedule = scheduler.schedule( + request('second-call', secondTool.name), + new AbortController().signal, + ); + + await firstSchedule; + await secondLookupStarted; + const thirdSchedule = scheduler.schedule( + request('third-call', thirdTool.name), + new AbortController().signal, + ); + await Promise.resolve(); + expect(thirdLookup).not.toHaveBeenCalled(); + + releaseSecondLookup(); + await Promise.all([secondSchedule, thirdSchedule]); + expect(thirdLookup).toHaveBeenCalledOnce(); + }); + + it.each([ + { + name: 'errors', + abortDuringLookup: false, + rejectLookup: true, + status: 'error' as const, + errorType: ToolErrorType.UNHANDLED_EXCEPTION, + }, + { + name: 'cancels', + abortDuringLookup: true, + rejectLookup: true, + status: 'cancelled' as const, + errorType: undefined, + }, + { + name: 'cancels after a normal resolution', + abortDuringLookup: true, + rejectLookup: false, + status: 'cancelled' as const, + errorType: undefined, + }, + ])( + '$name a tool call during lazy tool resolution', + async ({ abortDuringLookup, rejectLookup, status, errorType }) => { + const abortController = new AbortController(); + const { scheduler, ensureTool, onAllToolCallsComplete } = + createSchedulerForLegacyToolTests({ toolsByName: new Map() }); + const resolvedTool = new MockTool({ name: 'lazy-tool' }); + const build = vi.spyOn(resolvedTool, 'build'); + ensureTool.mockImplementation(async () => { + if (abortDuringLookup) abortController.abort(); + if (!rejectLookup) return resolvedTool; + throw new Error('lazy tool resolution failed'); + }); + + await expect( + scheduler.schedule( + { + callId: `lazy-${status}`, + name: 'lazy-tool', + args: {}, + isClientInitiated: false, + prompt_id: `prompt-lazy-${status}`, + }, + abortController.signal, + ), + ).resolves.toBeUndefined(); + + await vi.waitFor(() => + expect(onAllToolCallsComplete).toHaveBeenCalledOnce(), + ); + const completedCall = onAllToolCallsComplete.mock + .calls[0][0][0] as CompletedToolCall; + expect(completedCall.status).toBe(status); + expect(completedCall.response.executionStatus).toBe('not_started'); + expect(completedCall.response.errorType).toBe(errorType); + expect(build).not.toHaveBeenCalled(); + }, + ); + it('clears displayed tool calls when completion finalization throws', async () => { const execute = vi.fn().mockResolvedValue({ llmContent: 'alpha output', @@ -4426,13 +5275,26 @@ describe('CoreToolScheduler', () => { await vi.waitFor(() => { expect(onToolCallsUpdate.mock.calls.at(-1)?.[0]).toEqual([]); }); + const toolSpan = toolSpanRecords.findLast( + (record) => + record.name === 'tool.alpha' && + record.attributes['tool.call_id'] === 'call-alpha', + ); + expect(toolSpan?.ended).toBe(true); }); it('applies PostToolBatch stop decisions and preserves additional context', async () => { - const executeA = vi.fn().mockResolvedValue({ - llmContent: 'alpha output', - returnDisplay: 'alpha output', - }); + let resolveAlpha!: (result: { + llmContent: string; + returnDisplay: string; + }) => void; + const alphaResult = new Promise<{ + llmContent: string; + returnDisplay: string; + }>((resolve) => { + resolveAlpha = resolve; + }); + const executeA = vi.fn().mockReturnValue(alphaResult); const executeB = vi.fn().mockResolvedValue({ llmContent: 'beta output', returnDisplay: 'beta output', @@ -4485,7 +5347,7 @@ describe('CoreToolScheduler', () => { onAllToolCallsComplete, }); - await scheduler.schedule( + const schedulePromise = scheduler.schedule( [ { callId: 'call-alpha', @@ -4505,6 +5367,26 @@ describe('CoreToolScheduler', () => { new AbortController().signal, ); + await vi.waitFor(() => { + expect(executeB).toHaveBeenCalled(); + }); + let pendingStoppedToolSpan: ToolSpanRecord | undefined; + await vi.waitFor(() => { + pendingStoppedToolSpan = toolSpanRecords.findLast( + (record) => + record.name === 'tool.beta' && + record.attributes['tool.call_id'] === 'call-beta', + ); + expect(pendingStoppedToolSpan?.spanAttributes['success']).toBe(true); + }); + expect(pendingStoppedToolSpan?.ended).toBe(false); + + resolveAlpha({ + llmContent: 'alpha output', + returnDisplay: 'alpha output', + }); + await schedulePromise; + await vi.waitFor(() => { expect(onAllToolCallsComplete).toHaveBeenCalled(); }); @@ -4518,6 +5400,7 @@ describe('CoreToolScheduler', () => { ); expect(lastCompletedCall?.status).toBe('error'); if (lastCompletedCall?.status === 'error') { + expect(lastCompletedCall.response.executionStatus).toBe('success'); expect(lastCompletedCall.response.errorType).toBe( ToolErrorType.EXECUTION_DENIED, ); @@ -4542,8 +5425,104 @@ describe('CoreToolScheduler', () => { ); expect(batchHookSpan?.hookMetadata?.postBatchStop).toBe(true); expect(batchHookSpan?.hookMetadata?.postBatchStopReason).toBe('halt'); + const stoppedToolSpan = toolSpanRecords.findLast( + (record) => + record.name === 'tool.beta' && + record.attributes['tool.call_id'] === 'call-beta', + ); + expect(stoppedToolSpan?.spanAttributes['success']).toBe(false); + expect(stoppedToolSpan?.spanAttributes['tool.failure_kind']).toBe( + 'post_hook_stopped', + ); + expect(stoppedToolSpan?.statusCalls.at(-1)).toEqual({ + code: SpanStatusCode.ERROR, + message: 'halt', + }); + expect(stoppedToolSpan?.ended).toBe(true); }); + it.each([ + 'not_started', + 'success', + 'error', + 'cancelled', + undefined, + ])( + 'preserves executionStatus=%s when PostToolBatch replaces the last response', + async (executionStatus) => { + const tool = new MockTool({ name: 'alpha', kind: Kind.Read }); + const messageBus = { + request: vi.fn().mockResolvedValue({ + type: MessageBusType.HOOK_EXECUTION_RESPONSE, + correlationId: 'PostToolBatch-hook', + success: true, + output: { + continue: false, + stopReason: 'halt', + hookSpecificOutput: { + hookEventName: 'PostToolBatch', + }, + }, + }), + }; + const onAllToolCallsComplete = vi.fn(); + const { scheduler } = createSchedulerForLegacyToolTests({ + toolsByName: new Map([[tool.name, tool]]), + messageBus, + disableHooks: false, + onAllToolCallsComplete, + }); + const internals = scheduler as unknown as { + toolCalls: ToolCall[]; + postToolBatchEnabledForBatch: boolean; + checkAndNotifyCompletion: () => Promise; + }; + internals.postToolBatchEnabledForBatch = true; + internals.toolCalls = [ + { + status: 'error', + request: { + callId: 'call-alpha', + name: tool.name, + args: {}, + isClientInitiated: false, + prompt_id: 'prompt-batch-status', + }, + tool, + response: { + callId: 'call-alpha', + responseParts: [ + { + functionResponse: { + id: 'call-alpha', + name: tool.name, + response: { error: 'original error' }, + }, + }, + ], + resultDisplay: 'original error', + error: new Error('original error'), + errorType: ToolErrorType.EXECUTION_FAILED, + ...(executionStatus === undefined ? {} : { executionStatus }), + }, + }, + ]; + + await internals.checkAndNotifyCompletion(); + + const completedCalls = onAllToolCallsComplete.mock + .calls[0]?.[0] as CompletedToolCall[]; + expect(completedCalls[0]?.status).toBe('error'); + expect(completedCalls[0]?.response.executionStatus).toBe(executionStatus); + if (executionStatus === undefined) { + expect(completedCalls[0]?.response).not.toHaveProperty( + 'executionStatus', + ); + } + expect(completedCalls[0]?.response.error?.message).toBe('halt'); + }, + ); + it('passes through completed calls when PostToolBatch returns hookError', async () => { const execute = vi.fn().mockResolvedValue({ llmContent: 'alpha output', @@ -4700,6 +5679,9 @@ describe('CoreToolScheduler', () => { const completedCalls = onAllToolCallsComplete.mock .calls[0][0] as ToolCall[]; expect(completedCalls[0].status).toBe('cancelled'); + expect( + (completedCalls[0] as CompletedToolCall).response.executionStatus, + ).toBe('not_started'); }); it('should mark tool call as cancelled when abort happens during confirmation error', async () => { @@ -7200,18 +8182,54 @@ describe('CoreToolScheduler request queueing', () => { 'approved-sibling', ); - expect(hookSystem.firePermissionDeniedEvent).toHaveBeenCalledWith( - ToolNames.SHELL, - { command: "echo '{}' > .qwen/settings.json" }, - 'pending-protected-write', - 'classifier_blocked', - expect.any(AbortSignal), - 'pending-protected-write', - ); - const statuses = onToolCallsUpdate.mock.calls + expect(hookSystem.firePermissionDeniedEvent).toHaveBeenCalledWith( + ToolNames.SHELL, + { command: "echo '{}' > .qwen/settings.json" }, + 'pending-protected-write', + 'classifier_blocked', + expect.any(AbortSignal), + 'pending-protected-write', + ); + const statuses = onToolCallsUpdate.mock.calls + .flatMap((call) => call[0] as ToolCall[]) + .map((call) => call.status); + expect(statuses).toContain('error'); + }); + + it('preserves pending cancellation when AUTO classification resolves after abort', async () => { + const abortController = new AbortController(); + runSideQueryMock + .mockResolvedValueOnce({ shouldBlock: true }) + .mockImplementationOnce(async () => { + abortController.abort(); + return { + shouldBlock: true, + reason: 'protected write', + thinking: 'confirmed', + }; + }); + const { scheduler, onToolCallsUpdate, hookSystem } = + createPendingProtectedWriteHarness({ disableHooks: false }); + + await ( + scheduler as unknown as { + autoApproveCompatiblePendingTools: ( + signal: AbortSignal, + triggeringCallId: string, + ) => Promise; + } + ).autoApproveCompatiblePendingTools( + abortController.signal, + 'approved-sibling', + ); + + const cancelledCall = onToolCallsUpdate.mock.calls .flatMap((call) => call[0] as ToolCall[]) - .map((call) => call.status); - expect(statuses).toContain('error'); + .find((call) => call.status === 'cancelled') as + | CompletedToolCall + | undefined; + expect(cancelledCall?.response.executionStatus).toBe('not_started'); + expect(hookSystem.firePermissionDeniedEvent).not.toHaveBeenCalled(); }); it('continues pending AUTO block handling when PermissionDenied hook fails', async () => { @@ -7816,11 +8834,12 @@ describe('CoreToolScheduler Sequential Execution', () => { expect(onAllToolCallsComplete).toHaveBeenCalled(); }); - // Check that execute was called for all three tools initially - expect(executeFn).toHaveBeenCalledTimes(3); + // The in-flight second call observes cancellation; the third never + // crosses the execution boundary. + expect(executeFn).toHaveBeenCalledTimes(2); expect(executeFn).toHaveBeenCalledWith({ call: 1 }); expect(executeFn).toHaveBeenCalledWith({ call: 2 }); - expect(executeFn).toHaveBeenCalledWith({ call: 3 }); + expect(executeFn).not.toHaveBeenCalledWith({ call: 3 }); const completedCalls = onAllToolCallsComplete.mock .calls[0][0] as ToolCall[]; @@ -7833,6 +8852,12 @@ describe('CoreToolScheduler Sequential Execution', () => { expect(call1?.status).toBe('success'); expect(call2?.status).toBe('cancelled'); expect(call3?.status).toBe('cancelled'); + expect((call2 as CompletedToolCall).response.executionStatus).toBe( + 'cancelled', + ); + expect((call3 as CompletedToolCall).response.executionStatus).toBe( + 'not_started', + ); }); }); @@ -8897,6 +9922,7 @@ describe('CoreToolScheduler Plan shell routing', () => { throw new Error('Expected the guarded tool call to fail'); } expect(deniedCall.response.errorType).toBe(ToolErrorType.EXECUTION_DENIED); + expect(deniedCall.response.executionStatus).toBe('not_started'); }); it('executes once when the host guard allows the final invocation', async () => { @@ -8925,7 +9951,12 @@ describe('CoreToolScheduler Plan shell routing', () => { }); expect(execute).toHaveBeenCalledOnce(); const completed = onAllToolCallsComplete.mock.calls[0][0] as ToolCall[]; - expect(completed[0].status).toBe('success'); + const allowedCall = completed[0]; + expect(allowedCall.status).toBe('success'); + if (allowedCall.status !== 'success') { + throw new Error('Expected the guarded tool call to succeed'); + } + expect(allowedCall.response.executionStatus).toBe('success'); }); it('cancels without execution when aborted while awaiting the host guard', async () => { @@ -8955,7 +9986,12 @@ describe('CoreToolScheduler Plan shell routing', () => { await vi.waitFor(() => expect(onAllToolCallsComplete).toHaveBeenCalled()); expect(execute).not.toHaveBeenCalled(); const completed = onAllToolCallsComplete.mock.calls[0][0] as ToolCall[]; - expect(completed[0].status).toBe('cancelled'); + const cancelledCall = completed[0]; + expect(cancelledCall.status).toBe('cancelled'); + if (cancelledCall.status !== 'cancelled') { + throw new Error('Expected the guarded tool call to be cancelled'); + } + expect(cancelledCall.response.executionStatus).toBe('not_started'); }); it('skips guard evaluation entirely when no guard is configured', async () => { @@ -9496,6 +10532,7 @@ describe('CoreToolScheduler telemetry spans', () => { shouldThrowToolSpanSetAttribute.value = false; shouldThrowToolSpanSetStatus.value = false; mockTelemetrySdkState.initialized = false; + modifyWithEditorOverride.value = undefined; }); function getLastToolSpan(): ToolSpanRecord { @@ -9514,7 +10551,7 @@ describe('CoreToolScheduler telemetry spans', () => { signal?: AbortSignal, updateOutput?: (output: string) => void, ) => Promise; - tools?: MockTool[]; + tools?: AnyDeclarativeTool[]; messageBus?: { request: ReturnType }; disableHooks?: boolean; canUpdateOutput?: boolean; @@ -9524,10 +10561,12 @@ describe('CoreToolScheduler telemetry spans', () => { experimentalZedIntegration?: boolean; includeSensitiveSpanAttributes?: boolean; sensitiveSpanAttributeMaxLength?: number; + onToolCallsUpdate?: ReturnType; }): { scheduler: CoreToolScheduler; onAllToolCallsComplete: ReturnType; onToolCallsUpdate: ReturnType; + ensureTool: ReturnType; } { const tools = options.tools ?? [ new MockTool({ @@ -9544,9 +10583,10 @@ describe('CoreToolScheduler telemetry spans', () => { const toolsByName = new Map(tools.map((t) => [t.name, t])); const lookup = (name?: string) => (name ? toolsByName.get(name) : undefined) ?? tools[0]; + const ensureTool = vi.fn(async (n?: string) => lookup(n)); const mockToolRegistry = { getTool: (n?: string) => lookup(n), - ensureTool: async (n?: string) => lookup(n), + ensureTool, getFunctionDeclarations: () => [], tools: new Map(), discovery: {}, @@ -9598,7 +10638,7 @@ describe('CoreToolScheduler telemetry spans', () => { } as unknown as Config; const onAllToolCallsComplete = vi.fn(); - const onToolCallsUpdate = vi.fn(); + const onToolCallsUpdate = options.onToolCallsUpdate ?? vi.fn(); const scheduler = new CoreToolScheduler({ config: mockConfig, onAllToolCallsComplete, @@ -9606,7 +10646,12 @@ describe('CoreToolScheduler telemetry spans', () => { getPreferredEditor: () => 'vscode', onEditorClose: vi.fn(), }); - return { scheduler, onAllToolCallsComplete, onToolCallsUpdate }; + return { + scheduler, + onAllToolCallsComplete, + onToolCallsUpdate, + ensureTool, + }; } async function runSingleTool( @@ -9625,6 +10670,8 @@ describe('CoreToolScheduler telemetry spans', () => { includeSensitiveSpanAttributes?: boolean; sensitiveSpanAttributeMaxLength?: number; providerCallId?: string; + tools?: AnyDeclarativeTool[]; + toolName?: string; } = {}, ): Promise<{ spanRecord: ToolSpanRecord; @@ -9641,7 +10688,7 @@ describe('CoreToolScheduler telemetry spans', () => { { callId: 'span-call', providerCallId: options.providerCallId, - name: 'mockTool', + name: options.toolName ?? 'mockTool', args: { input: '/secret/path' }, isClientInitiated: false, prompt_id: 'prompt-telemetry', @@ -9819,6 +10866,77 @@ describe('CoreToolScheduler telemetry spans', () => { ); }); + it('does not execute after cancellation settles during PreToolUse', async () => { + toolSpanRecords.length = 0; + const abortController = new AbortController(); + let resolvePreHook: + | ((value: { + type: MessageBusType.HOOK_EXECUTION_RESPONSE; + correlationId: string; + success: true; + output: { decision: 'allow' }; + }) => void) + | undefined; + const preHookPromise = new Promise<{ + type: MessageBusType.HOOK_EXECUTION_RESPONSE; + correlationId: string; + success: true; + output: { decision: 'allow' }; + }>((resolve) => { + resolvePreHook = resolve; + }); + const messageBus = { + request: vi.fn().mockReturnValue(preHookPromise), + }; + const execute = vi.fn().mockResolvedValue({ + llmContent: 'should not execute', + returnDisplay: 'should not execute', + }); + const { scheduler, onAllToolCallsComplete } = buildScheduler({ + execute, + messageBus, + disableHooks: false, + }); + + const schedulePromise = scheduler.schedule( + [ + { + callId: 'pre-hook-cancel', + name: 'mockTool', + args: {}, + isClientInitiated: false, + prompt_id: 'prompt-pre-hook-cancel', + }, + ], + abortController.signal, + ); + + await vi.waitFor(() => expect(messageBus.request).toHaveBeenCalledOnce()); + abortController.abort(); + await new Promise((resolve) => setTimeout(resolve, 0)); + resolvePreHook?.({ + type: MessageBusType.HOOK_EXECUTION_RESPONSE, + correlationId: 'pre-hook-cancel', + success: true, + output: { decision: 'allow' }, + }); + await schedulePromise; + await vi.waitFor(() => expect(onAllToolCallsComplete).toHaveBeenCalled()); + + expect(execute).not.toHaveBeenCalled(); + const completedCalls = onAllToolCallsComplete.mock.calls.at(-1)?.[0] as + | CompletedToolCall[] + | undefined; + expect(completedCalls).toHaveLength(1); + expect(completedCalls?.[0]).toMatchObject({ + status: 'cancelled', + response: { executionStatus: 'not_started' }, + }); + expect( + toolSpanRecords.find((record) => record.name === 'tool.execution'), + ).toBeUndefined(); + }); + it('setToolSpanFailure forwards the truncateSpanError result to the span status (#4321)', async () => { // Lock the integration: if a future change drops the // truncateSpanError(message) call inside setToolSpanFailure, this @@ -10002,6 +11120,108 @@ describe('CoreToolScheduler telemetry spans', () => { } }); + it('preserves successful execution when cancellation arrives during PostToolUse', async () => { + const abortController = new AbortController(); + const messageBus = { + request: vi.fn(async (request: { eventName: string }) => { + if (request.eventName === 'PostToolUse') { + abortController.abort(); + } + return { + type: MessageBusType.HOOK_EXECUTION_RESPONSE, + correlationId: `${request.eventName}-hook`, + success: true, + output: { decision: 'allow' }, + }; + }), + }; + + const { completedCalls } = await runSingleTool({ + abortController, + messageBus, + disableHooks: false, + }); + + const completedCall = completedCalls[0] as CompletedToolCall; + expect(completedCall.status).toBe('cancelled'); + expect(completedCall.response.executionStatus).toBe('success'); + }); + + it.each([ToolErrorType.EXECUTION_FAILED, ToolErrorType.EXECUTION_TIMEOUT])( + 'preserves %s execution when cancellation arrives during failure postprocessing', + async (errorType) => { + const abortController = new AbortController(); + const messageBus = { + request: vi.fn(async (request: { eventName: string }) => { + if (request.eventName === 'PostToolUseFailure') { + abortController.abort(); + } + return { + type: MessageBusType.HOOK_EXECUTION_RESPONSE, + correlationId: `${request.eventName}-hook`, + success: true, + output: + request.eventName === 'PreToolUse' ? { decision: 'allow' } : {}, + }; + }), + }; + + const { completedCalls } = await runSingleTool({ + abortController, + messageBus, + disableHooks: false, + execute: vi.fn().mockResolvedValue({ + llmContent: 'failed', + returnDisplay: 'failed', + error: { + message: 'tool failed', + type: errorType, + }, + }), + }); + + const completedCall = completedCalls[0] as CompletedToolCall; + expect(completedCall.status).toBe('cancelled'); + expect(completedCall.response.executionStatus).toBe('error'); + }, + ); + + it('keeps tool update observers from changing the execution outcome', async () => { + const execute = vi.fn().mockResolvedValue({ + llmContent: 'ok', + returnDisplay: 'ok', + }); + const onToolCallsUpdate = vi.fn((calls: ToolCall[]) => { + if (calls.some((call) => call.status === 'executing')) { + throw new Error('observer failed'); + } + }); + const { scheduler, onAllToolCallsComplete } = buildScheduler({ + execute, + onToolCallsUpdate, + }); + + await scheduler.schedule( + [ + { + callId: 'observer-failure', + name: 'mockTool', + args: {}, + isClientInitiated: false, + prompt_id: 'prompt-observer-failure', + }, + ], + new AbortController().signal, + ); + + expect(execute).toHaveBeenCalledOnce(); + expect(onToolCallsUpdate).toHaveBeenCalled(); + const completedCall = onAllToolCallsComplete.mock + .calls[0][0][0] as CompletedToolCall; + expect(completedCall.status).toBe('success'); + expect(completedCall.response.executionStatus).toBe('success'); + }); + it('preserves PostToolUse artifacts on successful responses', async () => { const messageBus = { request: vi @@ -10318,43 +11538,253 @@ describe('CoreToolScheduler telemetry spans', () => { it('marks successful tool calls with OK status via endToolSpan', async () => { const { spanRecord, completedCalls } = await runSingleTool(); - expect(completedCalls[0].status).toBe('success'); - expect(spanRecord.statusCalls).toEqual([{ code: SpanStatusCode.OK }]); - expect(spanRecord.spanAttributes).not.toHaveProperty('tool.failure_kind'); - expect(spanRecord.ended).toBe(true); + expect(completedCalls[0].status).toBe('success'); + expect(spanRecord.statusCalls).toEqual([{ code: SpanStatusCode.OK }]); + expect(spanRecord.spanAttributes).not.toHaveProperty('tool.failure_kind'); + expect(spanRecord.ended).toBe(true); + }); + + // tool span `success` boolean attribute — must always be present so + // observability backends can filter failures with the same query they + // use for llm_request spans (which carry `success` unconditionally). + + it('tool span: success=true attribute on success', async () => { + const { spanRecord, completedCalls } = await runSingleTool(); + expect(completedCalls[0].status).toBe('success'); + expect( + (completedCalls[0] as CompletedToolCall).response.executionStatus, + ).toBe('success'); + expect(spanRecord.spanAttributes).toHaveProperty('success', true); + }); + + it('tool span: success=false attribute on ToolResult.error', async () => { + const { spanRecord, completedCalls } = await runSingleTool({ + execute: vi.fn().mockResolvedValue({ + llmContent: 'failed', + returnDisplay: 'failed', + error: { + message: 'tool failed', + type: ToolErrorType.EXECUTION_FAILED, + }, + }), + }); + expect(completedCalls[0].status).toBe('error'); + expect( + (completedCalls[0] as CompletedToolCall).response.executionStatus, + ).toBe('error'); + expect(spanRecord.spanAttributes).toHaveProperty('success', false); + }); + + it('tool span: success=false attribute on thrown invocation exception', async () => { + const { spanRecord, completedCalls } = await runSingleTool({ + execute: vi.fn().mockRejectedValue(new Error('boom')), + }); + expect(completedCalls[0].status).toBe('error'); + expect( + (completedCalls[0] as CompletedToolCall).response.executionStatus, + ).toBe('error'); + expect(spanRecord.spanAttributes).toHaveProperty('success', false); + }); + + it('keeps a structured timeout exception ahead of a later parent abort', async () => { + const abortController = new AbortController(); + const { completedCalls } = await runSingleTool({ + abortController, + execute: vi.fn().mockImplementation( + () => + new Promise((_resolve, reject) => { + reject( + new StructuredToolError( + 'MCP request timed out', + ToolErrorType.EXECUTION_TIMEOUT, + ), + ); + abortController.abort(); + }), + ), + }); + + const completedCall = completedCalls[0] as CompletedToolCall; + expect(completedCall.status).toBe('error'); + expect(completedCall.response.executionStatus).toBe('error'); + expect(completedCall.response.errorType).toBe( + ToolErrorType.EXECUTION_TIMEOUT, + ); + expect(getExecutionSpan()?.endMetadata).toMatchObject({ + executionStatus: 'error', + errorType: ToolErrorType.EXECUTION_TIMEOUT, + cancelled: false, + }); + }); + + // The cancellation notice the model sees must match what actually + // happened. Saying "already completed" for a tool interrupted mid-flight + // makes the model skip work that never ran; saying "cancelled" for a tool + // that finished makes it redo work whose side effects already landed. + + it('tells the model a mid-flight cancellation never completed', async () => { + const abortController = new AbortController(); + const { completedCalls } = await runSingleTool({ + abortController, + execute: vi.fn().mockImplementation( + () => + new Promise((_resolve, reject) => { + abortController.abort(); + reject( + Object.assign(new Error('Tool call aborted'), { + name: 'AbortError', + }), + ); + }), + ), + }); + + const completedCall = completedCalls[0] as CompletedToolCall; + expect(completedCall.status).toBe('cancelled'); + expect(completedCall.response.executionStatus).toBe('cancelled'); + const responseText = JSON.stringify(completedCall.response.responseParts); + expect(responseText).toContain('User cancelled tool execution.'); + expect(responseText).not.toContain('had already completed'); + }); + + it('tells the model a post-completion cancellation discarded finished work', async () => { + const abortController = new AbortController(); + const { completedCalls } = await runSingleTool({ + abortController, + execute: vi.fn().mockImplementation(async () => { + abortController.abort(); + return { llmContent: 'done', returnDisplay: 'done' }; + }), + }); + + const completedCall = completedCalls[0] as CompletedToolCall; + expect(completedCall.status).toBe('cancelled'); + expect(completedCall.response.executionStatus).toBe('cancelled'); + const responseText = JSON.stringify(completedCall.response.responseParts); + expect(responseText).toContain('The tool had already completed'); + expect(responseText).not.toContain('User cancelled tool execution.'); + }); + + // A post-execution cancellation drops the model-visible output, but the + // references to files the tool already spilled to disk must survive — + // otherwise nothing points at them and they are orphaned (#8180 review). + + it('keeps persisted output files on a post-completion cancellation', async () => { + const abortController = new AbortController(); + const { completedCalls } = await runSingleTool({ + abortController, + execute: vi.fn().mockImplementation(async () => { + abortController.abort(); + return { + llmContent: 'done', + returnDisplay: 'done', + persistedOutputFiles: ['/tmp/tool-results/span-call.txt'], + }; + }), + }); + + const completedCall = completedCalls[0] as CompletedToolCall; + expect(completedCall.status).toBe('cancelled'); + expect(completedCall.response.executionStatus).toBe('cancelled'); + expect(completedCall.response.persistedOutputFiles).toEqual([ + '/tmp/tool-results/span-call.txt', + ]); + }); + + it('keeps persisted output files when cancellation lands during post-processing', async () => { + const abortController = new AbortController(); + const messageBus = { + request: vi.fn(async (request: { eventName: string }) => { + if (request.eventName === 'PostToolUse') { + abortController.abort(); + } + return { + type: MessageBusType.HOOK_EXECUTION_RESPONSE, + correlationId: `${request.eventName}-hook`, + success: true, + output: { decision: 'allow' }, + }; + }), + }; + const { completedCalls } = await runSingleTool({ + abortController, + messageBus, + disableHooks: false, + execute: vi.fn().mockResolvedValue({ + llmContent: 'done', + returnDisplay: 'done', + persistedOutputFiles: ['/tmp/tool-results/span-call.txt'], + }), + }); + + const completedCall = completedCalls[0] as CompletedToolCall; + expect(completedCall.status).toBe('cancelled'); + expect(completedCall.response.executionStatus).toBe('success'); + expect(completedCall.response.persistedOutputFiles).toEqual([ + '/tmp/tool-results/span-call.txt', + ]); }); - // tool span `success` boolean attribute — must always be present so - // observability backends can filter failures with the same query they - // use for llm_request spans (which carry `success` unconditionally). + it('classifies a thrown MCP invocation as an MCP execution error', async () => { + const mcpTool = new DiscoveredMCPTool( + { + callTool: vi.fn().mockRejectedValue(new Error('MCP transport failed')), + } as unknown as CallableTool, + 'test-server', + 'test-tool', + 'Test MCP tool', + { type: 'object', properties: {} }, + ); - it('tool span: success=true attribute on success', async () => { - const { spanRecord, completedCalls } = await runSingleTool(); - expect(completedCalls[0].status).toBe('success'); - expect(spanRecord.spanAttributes).toHaveProperty('success', true); + const { completedCalls } = await runSingleTool({ + tools: [mcpTool], + toolName: mcpTool.name, + }); + + const completedCall = completedCalls[0] as CompletedToolCall; + expect(completedCall.status).toBe('error'); + expect(completedCall.response.executionStatus).toBe('error'); + expect(completedCall.response.errorType).toBe(ToolErrorType.MCP_TOOL_ERROR); + expect(getExecutionSpan()?.endMetadata?.errorType).toBe( + ToolErrorType.MCP_TOOL_ERROR, + ); }); - it('tool span: success=false attribute on ToolResult.error', async () => { - const { spanRecord, completedCalls } = await runSingleTool({ + it('classifies an untyped MCP soft error as an MCP execution error', async () => { + const mcpTool = new DiscoveredMCPTool( + { + callTool: vi.fn(), + } as unknown as CallableTool, + 'test-server', + 'test-tool', + 'Test MCP tool', + { type: 'object', properties: {} }, + ); + const softErrorInvocation = new MockTool({ + name: mcpTool.name, execute: vi.fn().mockResolvedValue({ - llmContent: 'failed', - returnDisplay: 'failed', + llmContent: 'MCP request failed', + returnDisplay: 'MCP request failed', error: { - message: 'tool failed', - type: ToolErrorType.EXECUTION_FAILED, + message: 'MCP request failed', + type: undefined, }, }), - }); - expect(completedCalls[0].status).toBe('error'); - expect(spanRecord.spanAttributes).toHaveProperty('success', false); - }); + }).build({}); + vi.spyOn(mcpTool, 'build').mockReturnValue(softErrorInvocation); - it('tool span: success=false attribute on thrown invocation exception', async () => { - const { spanRecord, completedCalls } = await runSingleTool({ - execute: vi.fn().mockRejectedValue(new Error('boom')), + const { completedCalls } = await runSingleTool({ + tools: [mcpTool], + toolName: mcpTool.name, }); - expect(completedCalls[0].status).toBe('error'); - expect(spanRecord.spanAttributes).toHaveProperty('success', false); + + const completedCall = completedCalls[0] as CompletedToolCall; + expect(completedCall.status).toBe('error'); + expect(completedCall.response.executionStatus).toBe('error'); + expect(completedCall.response.errorType).toBe(ToolErrorType.MCP_TOOL_ERROR); + expect(getExecutionSpan()?.endMetadata?.errorType).toBe( + ToolErrorType.MCP_TOOL_ERROR, + ); }); it('tool span: success=false attribute on cancellation', async () => { @@ -10367,6 +11797,9 @@ describe('CoreToolScheduler telemetry spans', () => { }), }); expect(completedCalls[0].status).toBe('cancelled'); + expect( + (completedCalls[0] as CompletedToolCall).response.executionStatus, + ).toBe('cancelled'); expect(spanRecord.spanAttributes).toHaveProperty('success', false); }); @@ -10384,9 +11817,17 @@ describe('CoreToolScheduler telemetry spans', () => { const exec = getExecutionSpan(); expect(exec).toBeDefined(); expect(exec!.ended).toBe(true); + expect(exec!.attributes).toMatchObject({ + 'gen_ai.tool.name': 'mockTool', + 'tool.call_id': 'span-call', + }); // cancelled: false because signal is not aborted on the success path // (#4302 review: cancelled flag now propagates through endToolExecutionSpan). - expect(exec!.endMetadata).toEqual({ success: true, cancelled: false }); + expect(exec!.endMetadata).toMatchObject({ + success: true, + cancelled: false, + executionStatus: 'success', + }); }); it('execution sub-span: ended (success: false) when ToolResult.error is set', async () => { @@ -10408,10 +11849,12 @@ describe('CoreToolScheduler telemetry spans', () => { // distinguish a failed-result close from a cancelled one without // cross-referencing the parent tool span. cancelled: false since the // signal isn't aborted (#4302 review). - expect(exec!.endMetadata).toEqual({ + expect(exec!.endMetadata).toMatchObject({ success: false, error: 'Tool execution failed', cancelled: false, + executionStatus: 'error', + errorType: ToolErrorType.EXECUTION_FAILED, }); }); @@ -10440,7 +11883,14 @@ describe('CoreToolScheduler telemetry spans', () => { output: { decision: 'block', reason: 'denied' }, }), }; - await runSingleTool({ messageBus, disableHooks: false }); + const { completedCalls } = await runSingleTool({ + messageBus, + disableHooks: false, + }); + expect(completedCalls[0].status).toBe('error'); + expect( + (completedCalls[0] as CompletedToolCall).response.executionStatus, + ).toBe('not_started'); expect(getExecutionSpan()).toBeUndefined(); }); @@ -11271,9 +12721,10 @@ describe('CoreToolScheduler telemetry spans', () => { const tracing = await import('../telemetry/session-tracing.js'); const runInToolSpanContext = vi.mocked(tracing.runInToolSpanContext); const messageBus = askMessageBus(); - const { scheduler, onToolCallsUpdate } = await scheduleWithAsk({ - messageBus, - }); + const { scheduler, onAllToolCallsComplete, onToolCallsUpdate } = + await scheduleWithAsk({ + messageBus, + }); const waiting = (await waitForStatus( onToolCallsUpdate, 'awaiting_approval', @@ -11283,11 +12734,20 @@ describe('CoreToolScheduler telemetry spans', () => { throw new Error('context failed before callback'); }); - await expect( - waiting.confirmationDetails.onConfirm( - ToolConfirmationOutcome.ProceedOnce, - ), - ).rejects.toThrow('context failed before callback'); + await waiting.confirmationDetails.onConfirm( + ToolConfirmationOutcome.ProceedOnce, + ); + + await vi.waitFor(() => { + expect(onAllToolCallsComplete).toHaveBeenCalledTimes(1); + }); + const completedCalls = onAllToolCallsComplete.mock + .calls[0]?.[0] as CompletedToolCall[]; + expect(completedCalls[0]?.status).toBe('error'); + expect(completedCalls[0]?.response.executionStatus).toBe('not_started'); + expect(completedCalls[0]?.response.error?.message).toBe( + 'context failed before callback', + ); expect( (scheduler as unknown as { bouncedAwaitingApproval: Set }) @@ -11728,6 +13188,36 @@ describe('CoreToolScheduler telemetry spans', () => { } }); + it('records cancellation when abort arrives during exception failure hooks', async () => { + const abortController = new AbortController(); + const messageBus = { + request: vi.fn(async (request: { eventName: string }) => { + if (request.eventName === 'PostToolUseFailure') { + abortController.abort(); + } + return { + type: MessageBusType.HOOK_EXECUTION_RESPONSE, + correlationId: `${request.eventName}-hook`, + success: true, + output: + request.eventName === 'PreToolUse' ? { decision: 'allow' } : {}, + }; + }), + }; + + const { completedCalls } = await runSingleTool({ + abortController, + messageBus, + disableHooks: false, + execute: vi.fn().mockRejectedValue(new Error('real boom')), + }); + + expect(completedCalls[0]).toMatchObject({ + status: 'cancelled', + response: { executionStatus: 'error' }, + }); + }); + it('every span recorded in a successful tool call is ended (#3731 Phase 2)', async () => { // Leak guard: every span we record should be ended by the time // schedule() returns. If a future change forgets to finalize a tool @@ -11762,6 +13252,7 @@ describe('CoreToolScheduler telemetry spans', () => { ): { scheduler: CoreToolScheduler; onToolCallsUpdate: ReturnType; + onAllToolCallsComplete: ReturnType; } { const mockToolRegistry = { getTool: () => tool, @@ -11803,14 +13294,15 @@ describe('CoreToolScheduler telemetry spans', () => { getDisableAllHooks: vi.fn().mockReturnValue(true), } as unknown as Config; const onToolCallsUpdate = vi.fn(); + const onAllToolCallsComplete = vi.fn(); const scheduler = new CoreToolScheduler({ config: mockConfig, - onAllToolCallsComplete: vi.fn(), + onAllToolCallsComplete, onToolCallsUpdate, getPreferredEditor: () => 'vscode', onEditorClose: vi.fn(), }); - return { scheduler, onToolCallsUpdate }; + return { scheduler, onToolCallsUpdate, onAllToolCallsComplete }; } it('keeps the exact runtime through manual approval', async () => { @@ -12053,7 +13545,8 @@ describe('CoreToolScheduler telemetry spans', () => { // both spans must be finalized and the error rethrown — otherwise // operators see a leak until the 30-min TTL. toolSpanRecords.length = 0; - const { scheduler, onToolCallsUpdate } = buildApprovalScheduler({}); + const { scheduler, onToolCallsUpdate, onAllToolCallsComplete } = + buildApprovalScheduler({}); await scheduler.schedule( [ { @@ -12087,6 +13580,17 @@ describe('CoreToolScheduler telemetry spans', () => { ), ).rejects.toBe(boom); + await vi.waitFor(() => { + expect(onAllToolCallsComplete).toHaveBeenCalled(); + }); + const completedCalls = onAllToolCallsComplete.mock.calls.at( + -1, + )?.[0] as ToolCall[]; + expect(completedCalls[0].status).toBe('error'); + expect( + (completedCalls[0] as CompletedToolCall).response.executionStatus, + ).toBe('not_started'); + // Blocked span finalized as 'error' / 'system'. const blockedSpan = toolSpanRecords.find( (r) => r.name === 'tool.blocked_on_user', @@ -12570,32 +14074,46 @@ describe('CoreToolScheduler telemetry spans', () => { expect(blockedSpan).toBeUndefined(); }); - it('prelude throw in _executeToolCallBody transitions tool from scheduled to error (#4321)', async () => { - // _executeToolCallBody's prelude (getMessageBus, - // startToolExecutionSpan, etc.) runs BEFORE the - // `scheduled → executing` transition. If a synchronous throw escapes + it('terminalizes every sequential sibling after an execution prelude throws (#4321)', async () => { + // _executeToolCallBody's prelude (for example getMessageBus) runs BEFORE + // the `scheduled → executing` transition. If a synchronous throw escapes // the prelude, the catch in executeSingleToolCall must finalize the // tool span with failure_kind=tool_exception AND transition the // toolCall to 'error' — otherwise checkAndNotifyCompletion never // sees a terminal state and the scheduler stalls (#4321 review-8 // wenshao Critical refinement of review-7 SF-H2). toolSpanRecords.length = 0; - const mockTool = new MockTool({ + const firstExecute = vi.fn().mockResolvedValue({ + llmContent: 'should not execute', + returnDisplay: 'should not execute', + }); + const firstTool = new MockTool({ name: 'mockTool', - execute: vi.fn().mockResolvedValue({ - llmContent: 'should not execute', - returnDisplay: 'should not execute', - }), + kind: Kind.Edit, + execute: firstExecute, + }); + const secondExecute = vi.fn().mockResolvedValue({ + llmContent: 'should not execute', + returnDisplay: 'should not execute', + }); + const secondTool = new MockTool({ + name: 'secondMockTool', + kind: Kind.Edit, + execute: secondExecute, }); const mockToolRegistry = { - getTool: () => mockTool, - ensureTool: async () => mockTool, + getTool: (name: string) => + name === secondTool.name ? secondTool : firstTool, + ensureTool: async (name: string) => + name === secondTool.name ? secondTool : firstTool, getFunctionDeclarations: () => [], tools: new Map(), discovery: {}, registerTool: () => {}, - getToolByName: () => mockTool, - getToolByDisplayName: () => mockTool, + getToolByName: (name: string) => + name === secondTool.name ? secondTool : firstTool, + getToolByDisplayName: (name: string) => + name === secondTool.name ? secondTool : firstTool, getTools: () => [], discoverTools: async () => {}, getAllTools: () => [], @@ -12637,44 +14155,54 @@ describe('CoreToolScheduler telemetry spans', () => { onEditorClose: vi.fn(), }); - // The prelude throw re-throws out of executeSingleToolCall → - // attemptExecutionOfScheduledCalls → _schedule. That's expected; - // the caller surfaces the error. The critical regression is - // whether the toolCall transitions out of `scheduled` BEFORE the - // throw propagates so checkAndNotifyCompletion sees a terminal - // state — without that transition the scheduler is stuck and - // onAllToolCallsComplete never fires. - await expect( - scheduler.schedule( - [ - { - callId: 'prelude-throw-1', - name: 'mockTool', - args: { input: 'x' }, - isClientInitiated: false, - prompt_id: 'prompt-prelude-throw', - }, - ], - new AbortController().signal, - ), - ).rejects.toThrow('prelude boom'); + await scheduler.schedule( + [ + { + callId: 'prelude-throw-1', + name: firstTool.name, + args: { input: 'x' }, + isClientInitiated: false, + prompt_id: 'prompt-prelude-throw', + }, + { + callId: 'prelude-throw-2', + name: secondTool.name, + args: { input: 'y' }, + isClientInitiated: false, + prompt_id: 'prompt-prelude-throw', + }, + ], + new AbortController().signal, + ); - // onAllToolCallsComplete fired (synchronously dispatched from - // setStatusInternal → checkAndNotifyCompletion) with the call in - // 'error' status — proves the catch transitioned it out of - // 'scheduled' BEFORE re-throwing. + // The first failure must not strand the second unsafe sibling in + // `scheduled`; both become terminal and the batch completes. expect(onAllToolCallsComplete).toHaveBeenCalled(); const completedCalls = onAllToolCallsComplete.mock.calls.at( -1, )?.[0] as ToolCall[]; - expect(completedCalls[0].status).toBe('error'); + expect(completedCalls.map((call) => call.status)).toEqual([ + 'error', + 'error', + ]); + expect( + completedCalls.map( + (call) => (call as CompletedToolCall).response.executionStatus, + ), + ).toEqual(['not_started', 'not_started']); + expect(firstExecute).not.toHaveBeenCalled(); + expect(secondExecute).not.toHaveBeenCalled(); + expect( + toolSpanRecords.some((record) => record.name === 'tool.execution'), + ).toBe(false); - // Tool span finalized with the canonical failure_kind. - const toolSpan = toolSpanRecords.find((r) => r.name === 'tool.mockTool'); - expect(toolSpan?.ended).toBe(true); - expect(toolSpan?.spanAttributes['tool.failure_kind']).toBe( - 'tool_exception', - ); + for (const name of [firstTool.name, secondTool.name]) { + const toolSpan = toolSpanRecords.find((r) => r.name === `tool.${name}`); + expect(toolSpan?.ended).toBe(true); + expect(toolSpan?.spanAttributes['tool.failure_kind']).toBe( + 'tool_exception', + ); + } }); it('signal.abort drains scheduler-local toolSpans + blockedSpans Maps (#4321)', async () => { @@ -12811,27 +14339,70 @@ describe('CoreToolScheduler telemetry spans', () => { ); }); - it('pre-aborted signal: tool span ends without entering execution (#4321)', async () => { - // _schedule line ~1487 early-exit when signal.aborted is true at the - // start of the for-loop. setToolSpanCancelled + finalizeToolSpan - // here are otherwise untested — a regression dropping either would - // leak the span or land it in ERROR rather than UNSET. + it('pre-aborted signal: terminalizes before validation or execution', async () => { toolSpanRecords.length = 0; const execute = vi .fn() .mockResolvedValue({ llmContent: 'ok', returnDisplay: 'ok' }); + const tool = new MockTool({ name: 'mockTool', execute }); + const build = vi.spyOn(tool, 'build'); + const { scheduler, onAllToolCallsComplete, ensureTool } = buildScheduler({ + tools: [tool], + }); const abortController = new AbortController(); abortController.abort(); - await runSingleTool({ execute, abortController }); + await scheduler.schedule( + [ + { + callId: 'pre-aborted-call', + name: 'mockTool', + args: {}, + isClientInitiated: false, + prompt_id: 'prompt-pre-aborted', + }, + ], + abortController.signal, + ); + expect(ensureTool).not.toHaveBeenCalled(); + expect(build).not.toHaveBeenCalled(); expect(execute).not.toHaveBeenCalled(); - const toolSpan = toolSpanRecords.findLast( - (r) => r.name === 'tool.mockTool', - ); - expect(toolSpan?.ended).toBe(true); - // setToolSpanCancelled records UNSET status — distinguishes from - // setToolSpanFailure paths which would land ERROR. - expect(toolSpan?.statusCalls).toEqual([{ code: SpanStatusCode.UNSET }]); + expect(onAllToolCallsComplete).toHaveBeenCalledWith([ + expect.objectContaining({ + status: 'cancelled', + response: expect.objectContaining({ + executionStatus: 'not_started', + }), + }), + ]); + expect( + toolSpanRecords.filter( + (record) => + record.name === 'tool.mockTool' || record.name === 'tool.execution', + ), + ).toEqual([]); + }); + + it('validated pre-execution cancellation keeps the parent span UNSET', async () => { + const abortController = new AbortController(); + const { spanRecord } = await runSingleTool({ + abortController, + tools: [ + new MockTool({ + name: 'mockTool', + getDefaultPermission: async () => { + abortController.abort(); + return 'deny'; + }, + }), + ], + }); + + expect(spanRecord.ended).toBe(true); + expect(spanRecord.statusCalls).toEqual([{ code: SpanStatusCode.UNSET }]); + expect( + toolSpanRecords.find((record) => record.name === 'tool.execution'), + ).toBeUndefined(); }); it('signal.abort during awaiting_approval: blocked span ends with aborted/system (#4321)', async () => { @@ -13016,6 +14587,104 @@ describe('CoreToolScheduler telemetry spans', () => { expect(toolSpan?.ended).toBe(false); }); + it('preserves cancellation when an editor resolves after batch abort', async () => { + const execute = vi.fn(); + const tool = Object.assign( + new MockTool({ + name: 'modifyRaceTool', + kind: Kind.Edit, + params: { + type: 'object', + properties: { value: { type: 'string' } }, + required: ['value'], + additionalProperties: false, + }, + getDefaultPermission: MOCK_TOOL_GET_DEFAULT_PERMISSION, + getConfirmationDetails: async () => ({ + type: 'edit', + title: 'Confirm modifyRaceTool', + fileName: 'test.txt', + filePath: 'test.txt', + fileDiff: 'before', + originalContent: 'old', + newContent: 'new', + onConfirm: async () => {}, + }), + execute, + }), + { + getModifyContext: () => ({ + getFilePath: () => 'test.txt', + getCurrentContent: async () => 'old', + getProposedContent: async () => 'new', + createUpdatedParams: () => ({ unexpected: true }), + }), + }, + ); + const build = tool.build.bind(tool); + const buildSpy = vi.spyOn(tool, 'build').mockImplementation((params) => { + if ('unexpected' in params) { + throw new Error('invalid editor rewrite'); + } + return build(params); + }); + let resolveEditor!: (result: { + updatedParams: Record; + updatedDiff: string; + }) => void; + const editorResult = new Promise<{ + updatedParams: Record; + updatedDiff: string; + }>((resolve) => { + resolveEditor = resolve; + }); + const editorCall = vi.fn(() => editorResult); + modifyWithEditorOverride.value = editorCall; + + const { scheduler, onToolCallsUpdate, onAllToolCallsComplete } = + buildApprovalScheduler({}, tool); + const abortController = new AbortController(); + await scheduler.schedule( + [ + { + callId: 'modify-race', + name: tool.name, + args: { value: 'original' }, + isClientInitiated: false, + prompt_id: 'prompt-modify-race', + }, + ], + abortController.signal, + ); + + const awaitingCall = (await waitForStatus( + onToolCallsUpdate, + 'awaiting_approval', + )) as WaitingToolCall; + const confirmation = awaitingCall.confirmationDetails.onConfirm( + ToolConfirmationOutcome.ModifyWithEditor, + ); + await vi.waitFor(() => expect(editorCall).toHaveBeenCalledOnce()); + + abortController.abort(); + await vi.waitFor(() => + expect(onAllToolCallsComplete).toHaveBeenCalledOnce(), + ); + resolveEditor({ + updatedParams: { unexpected: true }, + updatedDiff: 'after', + }); + await confirmation; + + const completedCalls = onAllToolCallsComplete.mock + .calls[0][0] as CompletedToolCall[]; + expect(completedCalls[0]?.status).toBe('cancelled'); + expect(completedCalls[0]?.response.executionStatus).toBe('not_started'); + expect(buildSpy).toHaveBeenCalledOnce(); + expect(execute).not.toHaveBeenCalled(); + expect(onAllToolCallsComplete).toHaveBeenCalledOnce(); + }); + it('per-batch abort listener removed when batch fully drains synchronously (#4321)', async () => { // Long-running sessions reuse the same AbortSignal across many // _schedule calls. The release-on-finalize hook in diff --git a/packages/core/src/core/coreToolScheduler.ts b/packages/core/src/core/coreToolScheduler.ts index 999d889f0d5..312a303e7ed 100644 --- a/packages/core/src/core/coreToolScheduler.ts +++ b/packages/core/src/core/coreToolScheduler.ts @@ -7,6 +7,7 @@ import type { ToolCallRequestInfo, ToolCallResponseInfo, + ToolExecutionStatus, ToolCallConfirmationDetails, ToolResult, ToolResultDisplay, @@ -135,6 +136,7 @@ import { import * as Diff from 'diff'; import levenshtein from 'fast-levenshtein'; import { ShellToolInvocation } from '../tools/shell.js'; +import { DiscoveredMCPTool } from '../tools/mcp-tool.js'; import { IdeClient } from '../ide/ide-client.js'; import { getPlanRequiredTeammatePreApprovalMessage, @@ -143,6 +145,21 @@ import { shouldUsePlanOnlyReminderInSubagentContext, } from '../agents/runtime/subagent-plan-tool-policy.js'; import { safeSetStatus } from '../telemetry/tracer.js'; +import { + TOOL_FAILURE_KIND_ATTRIBUTE, + TOOL_FAILURE_KIND_BACKGROUND_AGENT_DENIED, + TOOL_FAILURE_KIND_CANCELLED, + TOOL_FAILURE_KIND_INVOCATION_GUARD_DENIED, + TOOL_FAILURE_KIND_NON_INTERACTIVE_DENIED, + TOOL_FAILURE_KIND_PERMISSION_DENIED, + TOOL_FAILURE_KIND_PERMISSION_HOOK_DENIED, + TOOL_FAILURE_KIND_PLAN_MODE_BLOCKED, + TOOL_FAILURE_KIND_POST_HOOK_STOPPED, + TOOL_FAILURE_KIND_PRE_HOOK_BLOCKED, + TOOL_FAILURE_KIND_TIMEOUT, + TOOL_FAILURE_KIND_TOOL_ERROR, + TOOL_FAILURE_KIND_TOOL_EXCEPTION, +} from '../telemetry/constants.js'; import { SpanStatusCode, type Span } from '@opentelemetry/api'; import { startToolSpan, @@ -254,22 +271,6 @@ function extractTextFromPartListUnion(c: PartListUnion): string { return ''; } -const TOOL_FAILURE_KIND_ATTRIBUTE = 'tool.failure_kind'; -const TOOL_FAILURE_KIND_PRE_HOOK_BLOCKED = 'pre_hook_blocked'; -const TOOL_FAILURE_KIND_INVOCATION_GUARD_DENIED = 'invocation_guard_denied'; -const TOOL_FAILURE_KIND_POST_HOOK_STOPPED = 'post_hook_stopped'; -const TOOL_FAILURE_KIND_TOOL_ERROR = 'tool_error'; -const TOOL_FAILURE_KIND_TOOL_EXCEPTION = 'tool_exception'; -const TOOL_FAILURE_KIND_CANCELLED = 'cancelled'; -// Approval-flow failure kinds — distinct from `pre_hook_blocked` (which -// only applies to actual PreToolUse hook denials in `_executeToolCallBody`) -// so dashboards can attribute denies to their real cause (#4321 review). -const TOOL_FAILURE_KIND_PERMISSION_DENIED = 'permission_denied'; -const TOOL_FAILURE_KIND_PERMISSION_HOOK_DENIED = 'permission_hook_denied'; -const TOOL_FAILURE_KIND_PLAN_MODE_BLOCKED = 'plan_mode_blocked'; -const TOOL_FAILURE_KIND_NON_INTERACTIVE_DENIED = 'non_interactive_denied'; -const TOOL_FAILURE_KIND_BACKGROUND_AGENT_DENIED = 'background_agent_denied'; - const TOOL_SPAN_STATUS_PRE_HOOK_BLOCKED = 'Tool execution blocked by hook'; const TOOL_SPAN_STATUS_INVOCATION_GUARD_DENIED = 'Tool execution blocked by host policy'; @@ -288,11 +289,18 @@ const TOOL_SPAN_STATUS_TOOL_ERROR = 'Tool execution failed'; const TOOL_SPAN_STATUS_TOOL_EXCEPTION = 'Tool execution failed with exception'; const TOOL_SPAN_STATUS_TOOL_CANCELLED = 'Tool execution cancelled by user'; -// Timeout-specific observability constants — distinguish timeouts from -// generic tool errors in OTel traces. -const TOOL_FAILURE_KIND_TIMEOUT = 'timeout'; const TOOL_SPAN_STATUS_TOOL_TIMEOUT = 'Tool execution timed out'; +// The cancellation notice handed to the model depends on whether the tool's +// work actually finished. Claiming a tool "already completed" when it was +// interrupted mid-flight makes the model skip work that never happened; the +// converse makes it redo work whose side effects already landed. Both sites +// that can cancel after `execute()` was entered must pick the right one. +const TOOL_CANCELLED_BEFORE_COMPLETION_MESSAGE = + 'User cancelled tool execution.'; +const TOOL_CANCELLED_AFTER_COMPLETION_MESSAGE = + 'The tool had already completed; its output was discarded.'; + /** * Builds the failure ToolResult surfaced when a tool call exceeds the * execution timeout. Reported as a normal tool error so the model can adapt @@ -416,6 +424,10 @@ export type ScheduledToolCall = { outcome?: ToolConfirmationOutcome; }; +type CoreToolCallResponseInfo = ToolCallResponseInfo & { + executionStatus: ToolExecutionStatus; +}; + export type ErroredToolCall = { status: 'error'; request: ToolCallRequestInfo; @@ -467,8 +479,8 @@ export type CancelledToolCall = { status: 'cancelled'; request: ToolCallRequestInfo; response: ToolCallResponseInfo; - tool: AnyDeclarativeTool; - invocation: AnyToolInvocation; + tool?: AnyDeclarativeTool; + invocation?: AnyToolInvocation; durationMs?: number; outcome?: ToolConfirmationOutcome; }; @@ -917,9 +929,10 @@ const createErrorResponse = ( request: ToolCallRequestInfo, error: Error, errorType: ToolErrorType, + executionStatus: ToolExecutionStatus, artifacts?: ToolArtifact[], resultDisplay?: ToolResultDisplay, -): ToolCallResponseInfo => ({ +): CoreToolCallResponseInfo => ({ callId: request.callId, error, responseParts: [ @@ -933,6 +946,7 @@ const createErrorResponse = ( ], resultDisplay: resultDisplay ?? error.message, errorType, + executionStatus, contentLength: error.message.length, ...(artifacts && artifacts.length > 0 ? { artifacts } : {}), }); @@ -940,8 +954,13 @@ const createErrorResponse = ( const createCancelledResponse = ( request: ToolCallRequestInfo, reason: string, + executionStatus: ToolExecutionStatus, artifacts?: ToolArtifact[], -): ToolCallResponseInfo => { + // Disk references and bridge notices survive cancellation: dropping the + // model-visible output must not orphan files already persisted to disk. + persistedOutputFiles?: string[], + visionBridgeNotice?: string, +): CoreToolCallResponseInfo => { const errorMessage = `[Operation Cancelled] Reason: ${reason}`; return { callId: request.callId, @@ -957,8 +976,11 @@ const createCancelledResponse = ( resultDisplay: undefined, error: undefined, errorType: undefined, + executionStatus, contentLength: errorMessage.length, ...(artifacts && artifacts.length > 0 ? { artifacts } : {}), + ...(persistedOutputFiles !== undefined ? { persistedOutputFiles } : {}), + ...(visionBridgeNotice !== undefined ? { visionBridgeNotice } : {}), }; }; @@ -983,6 +1005,7 @@ function serializeToolResponse( result_display: response.resultDisplay, error: response.error?.message, error_type: response.errorType, + execution_status: response.executionStatus, content_length: response.contentLength, ...(response.visionBridgeNotice !== undefined ? { vision_bridge_notice: response.visionBridgeNotice } @@ -1156,15 +1179,24 @@ function withPostToolBatchStop( const calls = [...completedCalls]; const lastCall = calls[calls.length - 1]; + const executionStatus = lastCall.response.executionStatus; + // A batch stop must not invent an outcome the tool never produced: + // when the replaced response had no executionStatus, omit it here too. + const { executionStatus: _es, ...baseResponse } = createErrorResponse( + lastCall.request, + new Error(stopReason), + ToolErrorType.EXECUTION_DENIED, + executionStatus ?? 'not_started', + ); + const response: ToolCallResponseInfo = + executionStatus !== undefined + ? { ...baseResponse, executionStatus } + : baseResponse; calls[calls.length - 1] = { status: 'error', request: lastCall.request, tool: lastCall.tool, - response: createErrorResponse( - lastCall.request, - new Error(stopReason), - ToolErrorType.EXECUTION_DENIED, - ), + response, durationMs: lastCall.durationMs, outcome: undefined, } as ErroredToolCall; @@ -1303,6 +1335,9 @@ export class CoreToolScheduler { private chatRecordingService?: ChatRecordingService; private onToolResultFullTurnModel?: (model: string) => boolean; private isFinalizingToolCalls = false; + private postToolBatchEnabledForBatch = false; + private postToolBatchSpanCallId: string | undefined; + private postToolBatchConfigWarned = false; private isScheduling = false; private validationRetryCounts = new Map(); private autoModeFallbackCallIds = new Set(); @@ -1414,7 +1449,7 @@ export class CoreToolScheduler { private setStatusInternal( targetCallId: string, status: 'success', - response: ToolCallResponseInfo, + response: CoreToolCallResponseInfo, ): void; private setStatusInternal( targetCallId: string, @@ -1424,12 +1459,18 @@ export class CoreToolScheduler { private setStatusInternal( targetCallId: string, status: 'error', - response: ToolCallResponseInfo, + response: CoreToolCallResponseInfo, ): void; private setStatusInternal( targetCallId: string, status: 'cancelled', - reason: string | ToolCallResponseInfo, + reason: string, + executionStatus: ToolExecutionStatus, + ): void; + private setStatusInternal( + targetCallId: string, + status: 'cancelled', + response: CoreToolCallResponseInfo, ): void; private setStatusInternal( targetCallId: string, @@ -1439,6 +1480,7 @@ export class CoreToolScheduler { targetCallId: string, newStatus: Status, auxiliaryData?: unknown, + executionStatus?: ToolExecutionStatus, ): void { this.toolCalls = this.toolCalls.map((currentCall) => { if ( @@ -1469,7 +1511,7 @@ export class CoreToolScheduler { tool: toolInstance, invocation, status: 'success', - response: auxiliaryData as ToolCallResponseInfo, + response: auxiliaryData as CoreToolCallResponseInfo, durationMs, outcome, } as SuccessfulToolCall; @@ -1482,7 +1524,7 @@ export class CoreToolScheduler { request: currentCall.request, status: 'error', tool: toolInstance, - response: auxiliaryData as ToolCallResponseInfo, + response: auxiliaryData as CoreToolCallResponseInfo, durationMs, outcome, } as ErroredToolCall; @@ -1544,9 +1586,15 @@ export class CoreToolScheduler { const preservedResultDisplay = this.compactResultDisplayForInteractiveHistory(resultDisplay); const errorMessage = `[Operation Cancelled] Reason: ${auxiliaryData}`; - const response = isToolCallResponseInfo(auxiliaryData) + const response: CoreToolCallResponseInfo = isToolCallResponseInfo( + auxiliaryData, + ) ? { ...auxiliaryData, + executionStatus: + auxiliaryData.executionStatus ?? + executionStatus ?? + 'not_started', resultDisplay: auxiliaryData.resultDisplay ?? preservedResultDisplay, } @@ -1566,6 +1614,7 @@ export class CoreToolScheduler { resultDisplay: preservedResultDisplay, error: undefined, errorType: undefined, + executionStatus: executionStatus ?? 'not_started', contentLength: errorMessage.length, }; return { @@ -1613,11 +1662,16 @@ export class CoreToolScheduler { }); } - private setArgsInternal(targetCallId: string, args: unknown): void { + private setArgsInternal(targetCallId: string, args: unknown): boolean { + let invocationError: Error | undefined; + let argsUpdated = false; this.toolCalls = this.toolCalls.map((call) => { - // We should never be asked to set args on an ErroredToolCall, but - // we guard for the case anyways. - if (call.request.callId !== targetCallId || call.status === 'error') { + if ( + call.request.callId !== targetCallId || + call.status === 'success' || + call.status === 'error' || + call.status === 'cancelled' + ) { return call; } @@ -1630,10 +1684,12 @@ export class CoreToolScheduler { ), ); if (invocationOrError instanceof Error) { + invocationError = invocationOrError; const response = createErrorResponse( call.request, invocationOrError, ToolErrorType.INVALID_TOOL_PARAMS, + 'not_started', ); return { request: { ...call.request, args: args as Record }, @@ -1643,12 +1699,36 @@ export class CoreToolScheduler { } as ErroredToolCall; } + argsUpdated = true; return { ...call, request: { ...call.request, args: args as Record }, invocation: invocationOrError, }; }); + + if (invocationError) { + this.finalizeBlockedSpan(targetCallId, 'error', 'system'); + const toolSpan = this.toolSpans.get(targetCallId); + if (toolSpan) { + setToolSpanFailure( + toolSpan, + TOOL_FAILURE_KIND_TOOL_EXCEPTION, + invocationError.message, + ); + } + this.finalizeToolSpan(targetCallId); + this.notifyToolCallsUpdate(); + void this.checkAndNotifyCompletion().catch((error: unknown) => { + debugLogger.warn( + `setArgsInternal completion notification failed: ${ + error instanceof Error ? error.message : String(error) + }`, + ); + }); + return false; + } + return argsUpdated; } private isRunning(): boolean { @@ -1661,6 +1741,26 @@ export class CoreToolScheduler { ); } + private cancelPreExecutionIfAborted( + callId: string, + signal: AbortSignal, + toolSpan = this.toolSpans.get(callId), + ): boolean { + if (!signal.aborted) return false; + this.setStatusInternal( + callId, + 'cancelled', + 'Tool call cancelled by user.', + 'not_started', + ); + this.finalizeBlockedSpan(callId, 'aborted', 'system'); + if (toolSpan) { + setToolSpanCancelled(toolSpan); + } + this.finalizeToolSpan(callId); + return true; + } + /** * End the tool span for `callId` (if any) and remove it from the map. * Centralizes terminal-state cleanup so every cancel/error/success path @@ -1670,14 +1770,23 @@ export class CoreToolScheduler { * No `metadata` parameter: every caller pre-sets span status via * `setToolSpan{Failure,Cancelled,Ok}` before this call (#4321 review). */ - private finalizeToolSpan(callId: string): void { + private finalizeToolSpan(callId: string, force = false): void { // Terminal-state cleanup: drop any PreToolUse 'ask' bounce markers so // they never leak past the tool call's lifetime. Done unconditionally // (before the span guard) so a bounced call is cleared even on the // defensive no-span path. this.bouncedAwaitingApproval.delete(callId); this.bouncedToolUseId.delete(callId); + this.autoModeFallbackCallIds.delete(callId); this.runtimeContentGeneratorViews.delete(callId); + // PostToolBatch can replace the response at the last position in request + // order, which is not necessarily the call that settles last. Keep that + // specific span open until the hook has produced the terminal result. + // Known window: if a batch never reaches completion for a non-abort + // reason (e.g. a sibling parked in awaiting_approval when the session + // tears down), this span stays open until process exit. Force-finalize + // on session dispose would close this gap. + if (callId === this.postToolBatchSpanCallId && !force) return; const span = this.toolSpans.get(callId); if (!span) return; this.toolSpans.delete(callId); @@ -1781,6 +1890,7 @@ export class CoreToolScheduler { callId, 'cancelled', 'Tool call cancelled by user.', + 'not_started', ); if (this.blockedSpans.has(callId)) { this.finalizeBlockedSpan(callId, 'aborted', 'system'); @@ -1788,7 +1898,10 @@ export class CoreToolScheduler { const span = this.toolSpans.get(callId); if (span) { setToolSpanCancelled(span); - this.finalizeToolSpan(callId); + // Abort drain is terminal: force-finalize even when this is the + // deferred PostToolBatch parent span, which otherwise stays open + // for a batch hook that will no longer run on an aborted batch. + this.finalizeToolSpan(callId, true); } this.callIdToPostToolBatchSignal.delete(callId); this.autoModeFallbackCallIds.delete(callId); @@ -2072,6 +2185,20 @@ export class CoreToolScheduler { return this._schedule(request, signal, runtimeView); } + private drainRequestQueueIfIdle(): void { + if ( + this.requestQueue.length === 0 || + this.isScheduling || + this.isRunning() + ) { + return; + } + const next = this.requestQueue.shift()!; + this._schedule(next.request, next.signal, next.runtimeView) + .then(next.resolve) + .catch(next.reject); + } + /** * Removes all validation retry counters for the given tool. Keys are * ":", so a plain `Map.delete(toolName)` would not @@ -2182,179 +2309,262 @@ export class CoreToolScheduler { return count; }; for (const [requestIndex, reqInfo] of requestsToProcess.entries()) { - if ( - planModeEntryBoundaryIndex !== undefined && - requestIndex !== planModeEntryBoundaryIndex - ) { + let resolvedTool: AnyDeclarativeTool | undefined; + let resolvedInvocation: AnyToolInvocation | undefined; + const recordPrevalidationCancellation = (): boolean => { + if (!signal.aborted) return false; newToolCalls.push({ - status: 'error', + status: 'cancelled', request: reqInfo, - response: createErrorResponse( + response: createCancelledResponse( reqInfo, - new Error(PLAN_MODE_ENTRY_SIBLING_SKIP_MESSAGE), - ToolErrorType.EXECUTION_DENIED, + 'Tool call cancelled before execution.', + 'not_started', ), + ...(resolvedTool ? { tool: resolvedTool } : {}), + ...(resolvedInvocation ? { invocation: resolvedInvocation } : {}), durationMs: 0, }); - continue; - } - - const canonicalName = canonicalToolName(reqInfo.name); + return true; + }; + try { + if (recordPrevalidationCancellation()) continue; + if ( + planModeEntryBoundaryIndex !== undefined && + requestIndex !== planModeEntryBoundaryIndex + ) { + newToolCalls.push({ + status: 'error', + request: reqInfo, + response: createErrorResponse( + reqInfo, + new Error(PLAN_MODE_ENTRY_SIBLING_SKIP_MESSAGE), + ToolErrorType.EXECUTION_DENIED, + 'not_started', + ), + durationMs: 0, + }); + continue; + } - // Check if the tool is excluded due to permissions/environment restrictions - // This check should happen before registry lookup to provide a clear permission error - const pm = this.config.getPermissionManager?.(); - if (pm && !(await pm.isToolEnabled(canonicalName))) { - const matchingRule = pm.findMatchingDenyRule({ - toolName: canonicalName, - }); - const ruleInfo = matchingRule - ? ` Matching deny rule: "${matchingRule}".` - : ''; - const permissionErrorMessage = `Qwen Code requires permission to use "${reqInfo.name}", but that permission was declined.${ruleInfo}`; - newToolCalls.push({ - status: 'error', - request: reqInfo, - response: createErrorResponse( - reqInfo, - new Error(permissionErrorMessage), - ToolErrorType.EXECUTION_DENIED, - ), - durationMs: 0, - }); - continue; - } + const canonicalName = canonicalToolName(reqInfo.name); + + // Check if the tool is excluded due to permissions/environment restrictions + // This check should happen before registry lookup to provide a clear permission error + const pm = this.config.getPermissionManager?.(); + const permissionEnabled = pm + ? await pm.isToolEnabled(canonicalName) + : true; + if (recordPrevalidationCancellation()) continue; + if (pm && !permissionEnabled) { + const matchingRule = pm.findMatchingDenyRule({ + toolName: canonicalName, + }); + const ruleInfo = matchingRule + ? ` Matching deny rule: "${matchingRule}".` + : ''; + const permissionErrorMessage = `Qwen Code requires permission to use "${reqInfo.name}", but that permission was declined.${ruleInfo}`; + newToolCalls.push({ + status: 'error', + request: reqInfo, + response: createErrorResponse( + reqInfo, + new Error(permissionErrorMessage), + ToolErrorType.EXECUTION_DENIED, + 'not_started', + ), + durationMs: 0, + }); + continue; + } - // Legacy fallback: check getPermissionsDeny() when PM is not available - if (!pm) { - const excludeTools = this.config.getPermissionsDeny?.() ?? undefined; - if (excludeTools && excludeTools.length > 0) { - const normalizedToolName = canonicalName.toLowerCase().trim(); - const excludedMatch = excludeTools.find( - (excludedTool) => - excludedTool.toLowerCase().trim() === normalizedToolName, - ); - if (excludedMatch) { - const permissionErrorMessage = `Qwen Code requires permission to use ${excludedMatch}, but that permission was declined.`; - newToolCalls.push({ - status: 'error', - request: reqInfo, - response: createErrorResponse( - reqInfo, - new Error(permissionErrorMessage), - ToolErrorType.EXECUTION_DENIED, - ), - durationMs: 0, - }); - continue; + // Legacy fallback: check getPermissionsDeny() when PM is not available + if (!pm) { + const excludeTools = + this.config.getPermissionsDeny?.() ?? undefined; + if (excludeTools && excludeTools.length > 0) { + const normalizedToolName = canonicalName.toLowerCase().trim(); + const excludedMatch = excludeTools.find( + (excludedTool) => + excludedTool.toLowerCase().trim() === normalizedToolName, + ); + if (excludedMatch) { + const permissionErrorMessage = `Qwen Code requires permission to use ${excludedMatch}, but that permission was declined.`; + newToolCalls.push({ + status: 'error', + request: reqInfo, + response: createErrorResponse( + reqInfo, + new Error(permissionErrorMessage), + ToolErrorType.EXECUTION_DENIED, + 'not_started', + ), + durationMs: 0, + }); + continue; + } } } - } - const toolInstance = await runInRequestGoalContext(reqInfo, () => - this.toolRegistry.ensureTool(canonicalName), - ); - if (!toolInstance) { - // Tool is not in registry and not excluded - likely hallucinated or typo - const errorMessage = await runInRequestGoalContext(reqInfo, () => - this.getToolNotFoundMessage(reqInfo.name), + const toolInstance = await runInRequestGoalContext(reqInfo, () => + this.toolRegistry.ensureTool(canonicalName), ); - newToolCalls.push({ - status: 'error', - request: reqInfo, - response: createErrorResponse( - reqInfo, - new Error(errorMessage), - ToolErrorType.TOOL_NOT_REGISTERED, - ), - durationMs: 0, - }); - continue; - } + resolvedTool = toolInstance; + if (recordPrevalidationCancellation()) continue; + if (!toolInstance) { + // Tool is not in registry and not excluded - likely hallucinated or typo + const errorMessage = await runInRequestGoalContext(reqInfo, () => + this.getToolNotFoundMessage(reqInfo.name), + ); + if (recordPrevalidationCancellation()) continue; + newToolCalls.push({ + status: 'error', + request: reqInfo, + response: createErrorResponse( + reqInfo, + new Error(errorMessage), + ToolErrorType.TOOL_NOT_REGISTERED, + 'not_started', + ), + durationMs: 0, + }); + continue; + } - // Reject file-modifying calls when truncated to prevent - // writing incomplete content, even if params failed schema validation. - if (reqInfo.wasOutputTruncated && toolInstance.kind === Kind.Edit) { - const count = recordBatchRetryableToolError( - reqInfo.name, - TRUNCATION_EDIT_REJECTION, - ); - const truncationError = new Error( - count >= VALIDATION_RETRY_LOOP_THRESHOLD - ? `${TRUNCATION_EDIT_REJECTION}${TRUNCATION_RETRY_LOOP_DIRECTIVE}` - : TRUNCATION_EDIT_REJECTION, - ); - newToolCalls.push({ - status: 'error', - request: reqInfo, - tool: toolInstance, - response: createErrorResponse( - reqInfo, - truncationError, - ToolErrorType.OUTPUT_TRUNCATED, - ), - durationMs: 0, - }); - continue; - } + // Reject file-modifying calls when truncated to prevent + // writing incomplete content, even if params failed schema validation. + if (reqInfo.wasOutputTruncated && toolInstance.kind === Kind.Edit) { + const count = recordBatchRetryableToolError( + reqInfo.name, + TRUNCATION_EDIT_REJECTION, + ); + const truncationError = new Error( + count >= VALIDATION_RETRY_LOOP_THRESHOLD + ? `${TRUNCATION_EDIT_REJECTION}${TRUNCATION_RETRY_LOOP_DIRECTIVE}` + : TRUNCATION_EDIT_REJECTION, + ); + newToolCalls.push({ + status: 'error', + request: reqInfo, + tool: toolInstance, + response: createErrorResponse( + reqInfo, + truncationError, + ToolErrorType.OUTPUT_TRUNCATED, + 'not_started', + ), + durationMs: 0, + }); + continue; + } - const invocationOrError = runInRequestGoalContext(reqInfo, () => - this.buildInvocation( - toolInstance, - reqInfo.args, - reqInfo.callId, - reqInfo.prompt_id, - ), - ); - if (invocationOrError instanceof Error) { - const displayError = reqInfo.wasOutputTruncated - ? new Error( - `${invocationOrError.message} ${TRUNCATION_PARAM_GUIDANCE}`, - ) - : invocationOrError; - - // Track validation retry for loop detection. Counts accumulate per - // (tool, error message) pair so a different validation mistake on - // the same tool starts fresh rather than tripping the threshold. - const count = recordBatchRetryableToolError( - reqInfo.name, - invocationOrError.message, + const invocationOrError = runInRequestGoalContext(reqInfo, () => + this.buildInvocation( + toolInstance, + reqInfo.args, + reqInfo.callId, + reqInfo.prompt_id, + ), ); - - const finalError = - count >= VALIDATION_RETRY_LOOP_THRESHOLD + if (recordPrevalidationCancellation()) continue; + if (invocationOrError instanceof Error) { + const displayError = reqInfo.wasOutputTruncated ? new Error( - `${invocationOrError.message}${RETRY_LOOP_STOP_DIRECTIVE}`, + `${invocationOrError.message} ${TRUNCATION_PARAM_GUIDANCE}`, ) - : displayError; + : invocationOrError; + + // Track validation retry for loop detection. Counts accumulate per + // (tool, error message) pair so a different validation mistake on + // the same tool starts fresh rather than tripping the threshold. + const count = recordBatchRetryableToolError( + reqInfo.name, + invocationOrError.message, + ); + + const finalError = + count >= VALIDATION_RETRY_LOOP_THRESHOLD + ? new Error( + `${invocationOrError.message}${RETRY_LOOP_STOP_DIRECTIVE}`, + ) + : displayError; + + newToolCalls.push({ + status: 'error', + request: reqInfo, + tool: toolInstance, + response: createErrorResponse( + reqInfo, + finalError, + ToolErrorType.INVALID_TOOL_PARAMS, + 'not_started', + ), + durationMs: 0, + }); + continue; + } + resolvedInvocation = invocationOrError; + + // Reset all validation retry counters for this tool since it passed validation + this.clearRetryCountsForTool(reqInfo.name); newToolCalls.push({ - status: 'error', + status: 'validating', request: reqInfo, tool: toolInstance, + invocation: invocationOrError, + startTime: Date.now(), + }); + } catch (error) { + if (recordPrevalidationCancellation()) continue; + const normalizedError = + error instanceof Error ? error : new Error(String(error)); + newToolCalls.push({ + status: 'error', + request: reqInfo, response: createErrorResponse( reqInfo, - finalError, - ToolErrorType.INVALID_TOOL_PARAMS, + normalizedError, + (error as { errorType?: ToolErrorType } | undefined)?.errorType ?? + ToolErrorType.UNHANDLED_EXCEPTION, + 'not_started', ), + ...(resolvedTool ? { tool: resolvedTool } : {}), durationMs: 0, }); - continue; } - - // Reset all validation retry counters for this tool since it passed validation - this.clearRetryCountsForTool(reqInfo.name); - - newToolCalls.push({ - status: 'validating', - request: reqInfo, - tool: toolInstance, - invocation: invocationOrError, - startTime: Date.now(), - }); } this.toolCalls = this.toolCalls.concat(newToolCalls); + for (const toolCall of newToolCalls) { + this.callIdToPostToolBatchSignal.set(toolCall.request.callId, signal); + } + const postToolBatchParentCallId = newToolCalls.findLast( + (toolCall) => toolCall.status === 'validating', + )?.request.callId; + this.postToolBatchEnabledForBatch = false; + this.postToolBatchSpanCallId = undefined; + try { + this.postToolBatchEnabledForBatch = + !this.config.getDisableAllHooks() && + (this.config.hasHooksForEvent?.('PostToolBatch') ?? false); + if (this.postToolBatchEnabledForBatch) { + this.postToolBatchSpanCallId = postToolBatchParentCallId; + } + } catch (configError) { + // Fail safe: completion will attempt the hook once. Preserve the + // potential parent so a transient configuration failure cannot let + // its span end early. + this.postToolBatchEnabledForBatch = true; + this.postToolBatchSpanCallId = postToolBatchParentCallId; + if (!this.postToolBatchConfigWarned) { + this.postToolBatchConfigWarned = true; + debugLogger.warn( + 'PostToolBatch hook detection failed; deferring span as a precaution:', + configError, + ); + } + } this.notifyToolCallsUpdate(); // Per-batch abort-listener state. Shared by every callId added in @@ -2407,17 +2617,11 @@ export class CoreToolScheduler { this.toolSpans.set(reqInfo.callId, toolSpan); batchState.callIds.add(reqInfo.callId); this.callIdToBatch.set(reqInfo.callId, batchState); - this.callIdToPostToolBatchSignal.set(reqInfo.callId, signal); try { - if (signal.aborted) { - this.setStatusInternal( - reqInfo.callId, - 'cancelled', - 'Tool call cancelled by user.', - ); - setToolSpanCancelled(toolSpan); - this.finalizeToolSpan(reqInfo.callId); + if ( + this.cancelPreExecutionIfAborted(reqInfo.callId, signal, toolSpan) + ) { continue; } @@ -2435,6 +2639,11 @@ export class CoreToolScheduler { toolParams, ), ); + if ( + this.cancelPreExecutionIfAborted(reqInfo.callId, signal, toolSpan) + ) { + continue; + } const { defaultPermission, finalPermission, @@ -2505,6 +2714,7 @@ export class CoreToolScheduler { reqInfo, new Error(message), ToolErrorType.EXECUTION_DENIED, + 'not_started', ), ); setToolSpanFailure( @@ -2525,6 +2735,7 @@ export class CoreToolScheduler { reqInfo, new Error(denyMessage ?? `Tool "${reqInfo.name}" is denied.`), ToolErrorType.EXECUTION_DENIED, + 'not_started', ), ); setToolSpanFailure( @@ -2566,12 +2777,18 @@ export class CoreToolScheduler { }), ) : ({ classification: 'not-applicable' } as const); + if ( + this.cancelPreExecutionIfAborted(reqInfo.callId, signal, toolSpan) + ) { + continue; + } const rejectPlanShell = (message: string) => { this.setStatusInternal(reqInfo.callId, 'error', { ...createErrorResponse( reqInfo, new Error(message), ToolErrorType.EXECUTION_DENIED, + 'not_started', ), resultDisplay: message, }); @@ -2598,6 +2815,11 @@ export class CoreToolScheduler { signal, }), ); + if ( + this.cancelPreExecutionIfAborted(reqInfo.callId, signal, toolSpan) + ) { + continue; + } if (initialPlanShellError) { rejectPlanShell(initialPlanShellError); continue; @@ -2671,6 +2893,11 @@ export class CoreToolScheduler { : undefined, }), ); + if ( + this.cancelPreExecutionIfAborted(reqInfo.callId, signal, toolSpan) + ) { + continue; + } const outcome = applyAutoModeDecision( decision, @@ -2700,6 +2927,11 @@ export class CoreToolScheduler { ); } } + if ( + this.cancelPreExecutionIfAborted(reqInfo.callId, signal, toolSpan) + ) { + continue; + } switch (outcome.kind) { case 'approved': this.setToolCallOutcome( @@ -2720,8 +2952,15 @@ export class CoreToolScheduler { reqInfo, new Error(outcome.errorMessage), ToolErrorType.EXECUTION_DENIED, + 'not_started', ), ); + setToolSpanFailure( + toolSpan, + TOOL_FAILURE_KIND_PERMISSION_DENIED, + outcome.errorMessage, + ); + this.finalizeToolSpan(reqInfo.callId); continue; case 'fallback': // Drop through to the manual-approval flow below. The @@ -2772,6 +3011,11 @@ export class CoreToolScheduler { confirmationDetails = await runInRequestGoalContext(reqInfo, () => invocation.getConfirmationDetails(signal), ); + if ( + this.cancelPreExecutionIfAborted(reqInfo.callId, signal, toolSpan) + ) { + continue; + } if (autoModeFallbackMessage) { confirmationDetails = decorateClassifierUnavailableConfirmation( @@ -2795,6 +3039,15 @@ export class CoreToolScheduler { signal, }), ); + if ( + this.cancelPreExecutionIfAborted( + reqInfo.callId, + signal, + toolSpan, + ) + ) { + continue; + } if (preDisplayPlanShellError) { rejectPlanShell(preDisplayPlanShellError); continue; @@ -2847,6 +3100,7 @@ export class CoreToolScheduler { reqInfo, planModeError, ToolErrorType.EXECUTION_DENIED, + 'not_started', ), resultDisplay: 'Plan mode blocked a non-read-only tool call.', }); @@ -2896,6 +3150,7 @@ export class CoreToolScheduler { reqInfo, new Error(errorMessage), ToolErrorType.EXECUTION_DENIED, + 'not_started', ), ); setToolSpanFailure( @@ -2929,6 +3184,15 @@ export class CoreToolScheduler { signal, ), ); + if ( + this.cancelPreExecutionIfAborted( + reqInfo.callId, + signal, + toolSpan, + ) + ) { + continue; + } if ( hookResult.hasDecision && @@ -2954,6 +3218,15 @@ export class CoreToolScheduler { : undefined, }), ); + if ( + this.cancelPreExecutionIfAborted( + reqInfo.callId, + signal, + toolSpan, + ) + ) { + continue; + } if (approval.outcome === ToolConfirmationOutcome.Cancel) { await runInRequestGoalContext(reqInfo, () => preparedConfirmationDetails.onConfirm( @@ -2961,6 +3234,15 @@ export class CoreToolScheduler { approval.payload, ), ); + if ( + this.cancelPreExecutionIfAborted( + reqInfo.callId, + signal, + toolSpan, + ) + ) { + continue; + } rejectPlanShell( approval.payload?.cancelMessage ?? planShellDecision.noApprovalMessage, @@ -2973,6 +3255,15 @@ export class CoreToolScheduler { approval.payload, ), ); + if ( + this.cancelPreExecutionIfAborted( + reqInfo.callId, + signal, + toolSpan, + ) + ) { + continue; + } this.recordAutoModeFallbackResolution( reqInfo.callId, approval.outcome, @@ -2986,16 +3277,29 @@ export class CoreToolScheduler { hookResult.updatedInput && typeof reqInfo.args === 'object' ) { - this.setArgsInternal( - reqInfo.callId, - hookResult.updatedInput, - ); + if ( + !this.setArgsInternal( + reqInfo.callId, + hookResult.updatedInput, + ) + ) { + continue; + } } await runInRequestGoalContext(reqInfo, () => preparedConfirmationDetails.onConfirm( ToolConfirmationOutcome.ProceedOnce, ), ); + if ( + this.cancelPreExecutionIfAborted( + reqInfo.callId, + signal, + toolSpan, + ) + ) { + continue; + } this.recordAutoModeFallbackResolution( reqInfo.callId, ToolConfirmationOutcome.ProceedOnce, @@ -3016,6 +3320,15 @@ export class CoreToolScheduler { cancelPayload, ), ); + if ( + this.cancelPreExecutionIfAborted( + reqInfo.callId, + signal, + toolSpan, + ) + ) { + continue; + } this.recordAutoModeFallbackResolution( reqInfo.callId, ToolConfirmationOutcome.Cancel, @@ -3034,6 +3347,7 @@ export class CoreToolScheduler { `Permission denied by hook for "${reqInfo.name}"`, ), ToolErrorType.EXECUTION_DENIED, + 'not_started', ), ); setToolSpanFailure( @@ -3065,6 +3379,7 @@ export class CoreToolScheduler { reqInfo, new Error(errorMessage), ToolErrorType.EXECUTION_DENIED, + 'not_started', ), ); setToolSpanFailure( @@ -3085,14 +3400,9 @@ export class CoreToolScheduler { // setTimeout(0)) may have already fired by then, so the new // entries would never be drained (#4321 review-3 wenshao // Critical). - if (signal.aborted) { - this.setStatusInternal( - reqInfo.callId, - 'cancelled', - 'Tool call cancelled by user.', - ); - setToolSpanCancelled(toolSpan); - this.finalizeToolSpan(reqInfo.callId); + if ( + this.cancelPreExecutionIfAborted(reqInfo.callId, signal, toolSpan) + ) { continue; } @@ -3110,6 +3420,15 @@ export class CoreToolScheduler { signal, }), ); + if ( + this.cancelPreExecutionIfAborted( + reqInfo.callId, + signal, + toolSpan, + ) + ) { + continue; + } if (finalPreDisplayPlanShellError) { rejectPlanShell(finalPreDisplayPlanShellError); continue; @@ -3220,6 +3539,7 @@ export class CoreToolScheduler { reqInfo.callId, 'cancelled', 'Tool call cancelled by user.', + 'not_started', ); // If this tool was waiting on the user, end the blocked span // as aborted before the tool span itself. @@ -3245,6 +3565,7 @@ export class CoreToolScheduler { reqInfo, error instanceof Error ? error : new Error(String(error)), explicitErrorType ?? ToolErrorType.UNHANDLED_EXCEPTION, + 'not_started', ), ); // Non-aborted catch is a system error (e.g. getConfirmationDetails @@ -3286,6 +3607,7 @@ export class CoreToolScheduler { } } finally { this.isScheduling = false; + this.drainRequestQueueIfIdle(); } } @@ -3351,10 +3673,8 @@ export class CoreToolScheduler { // path already closed them, these are no-ops. // // attemptExecutionOfScheduledCalls is NOT covered by this catch - // (see below). A sister tool's prelude throw escaping through - // attemptExecutionOfScheduledCalls would otherwise corrupt A's - // span — each executeSingleToolCall handles its own span - // lifecycle via its own catch (#4321 review-9 wenshao Critical). + // (see below). Each sister tool owns and terminalizes failures in + // executeSingleToolCall, so none can be mis-attributed to A's span. // // Branch on signal.aborted so a throw caused by the abort signal // (e.g. ModifyWithEditor child interrupted by Ctrl+C) lands as @@ -3362,6 +3682,25 @@ export class CoreToolScheduler { // in `_schedule:1797` and the dashboard intent of separating // user/system aborts from real exceptions (#4321 review-2 wenshao). const aborted = signal.aborted; + if (aborted) { + this.setStatusInternal( + callId, + 'cancelled', + 'Tool call cancelled by user.', + 'not_started', + ); + } else { + this.setStatusInternal( + callId, + 'error', + createErrorResponse( + toolCall.request, + error instanceof Error ? error : new Error(String(error)), + ToolErrorType.UNHANDLED_EXCEPTION, + 'not_started', + ), + ); + } this.finalizeBlockedSpan(callId, aborted ? 'aborted' : 'error', 'system'); const toolSpan = this.toolSpans.get(callId); if (toolSpan) { @@ -3387,13 +3726,8 @@ export class CoreToolScheduler { throw error; } - // Execution phase runs OUTSIDE the catch above so a sister tool's - // prelude throw (re-thrown by executeSingleToolCall after SF-H2) - // can't be mis-attributed to A's span. Each executeSingleToolCall - // handles its own span lifecycle; failures propagate to the caller - // as-is. (#4321 review-9 wenshao Critical refines review-2 - // pushback which became live after SF-H2 added the prelude - // re-throw.) + // Execution runs outside the confirmation catch so each sister tool's + // executeSingleToolCall owns its own terminal response and span. await this.attemptExecutionOfScheduledCalls(signal); } @@ -3444,7 +3778,7 @@ export class CoreToolScheduler { // Use custom cancel message from payload if provided, otherwise use default const cancelMessage = payload?.cancelMessage || 'User did not allow tool call'; - this.setStatusInternal(callId, 'cancelled', cancelMessage); + this.setStatusInternal(callId, 'cancelled', cancelMessage, 'not_started'); // Tool span is cancelled too — finalize it via setToolSpanCancelled // before pulling it out of the map so the status survives end(). const toolSpan = this.toolSpans.get(callId); @@ -3532,7 +3866,7 @@ export class CoreToolScheduler { signal, this.onEditorClose, ); - this.setArgsInternal(callId, updatedParams); + if (!this.setArgsInternal(callId, updatedParams)) return; this.setStatusInternal(callId, 'awaiting_approval', { ...waitingToolCall.confirmationDetails, fileDiff: updatedDiff, @@ -3542,11 +3876,15 @@ export class CoreToolScheduler { } else { // If the client provided new content, apply it before scheduling. if (payload?.newContent && toolCall) { - await this._applyInlineModify( - toolCall as WaitingToolCall, - payload, - signal, - ); + if ( + !(await this._applyInlineModify( + toolCall as WaitingToolCall, + payload, + signal, + )) + ) { + return; + } } this.setStatusInternal(callId, 'scheduled'); // Proceed: end the blocked span before execution begins. ProceedOnce @@ -3678,14 +4016,14 @@ export class CoreToolScheduler { toolCall: WaitingToolCall, payload: ToolConfirmationPayload, signal: AbortSignal, - ): Promise { + ): Promise { const confirmDetails = toolCall.confirmationDetails; if ( confirmDetails.type !== 'edit' || !isModifiableDeclarativeTool(toolCall.tool) || !payload.newContent ) { - return; + return true; } const currentContent = confirmDetails.originalContent ?? ''; @@ -3704,11 +4042,14 @@ export class CoreToolScheduler { 'Proposed', ); - this.setArgsInternal(toolCall.request.callId, updatedParams); + if (!this.setArgsInternal(toolCall.request.callId, updatedParams)) { + return false; + } this.setStatusInternal(toolCall.request.callId, 'awaiting_approval', { ...confirmDetails, fileDiff: updatedDiff, }); + return true; } private async attemptExecutionOfScheduledCalls( @@ -3852,9 +4193,8 @@ export class CoreToolScheduler { this.bouncedToolUseId.delete(callId); // _executeToolCallBody pre-sets span status (OK / FAILURE / // CANCELLED) only AFTER its main try/catch is entered. Throws - // from the prelude — getMessageBus, - // startToolExecutionSpan, etc. — happen BEFORE the - // `scheduled → executing` transition, so the span would end + // from the prelude — for example getMessageBus — happen BEFORE + // the `scheduled → executing` transition, so the span would end // UNSET with no failure_kind AND the tool call would stay in // `scheduled` forever (checkAndNotifyCompletion never sees a // terminal state). Set failure status + error response here so @@ -3879,9 +4219,12 @@ export class CoreToolScheduler { scheduledCall.request, error instanceof Error ? error : new Error(errorMessage), ToolErrorType.UNHANDLED_EXCEPTION, + this.toolCalls.find((call) => call.request.callId === callId) + ?.status === 'executing' + ? 'error' + : 'not_started', ), ); - throw error; } finally { // A PreToolUse 'ask' hook can bounce this tool back to // awaiting_approval (see bounceToAwaitingApprovalForAsk). The tool @@ -4090,7 +4433,7 @@ export class CoreToolScheduler { hasAdditionalContext: !!r.additionalContext, }, ); - if (!preHookResult.shouldProceed) { + if (!signal.aborted && !preHookResult.shouldProceed) { // A PreToolUse hook returning permissionDecision:'ask' wants the // user to confirm in the TUI before the tool runs. When we can // prompt, bounce the tool into the existing awaiting_approval flow @@ -4124,6 +4467,7 @@ export class CoreToolScheduler { scheduledCall.request, new Error(blockMessage), ToolErrorType.EXECUTION_DENIED, + 'not_started', ); this.setStatusInternal(callId, 'error', errorResponse); setToolSpanFailure( @@ -4150,9 +4494,15 @@ export class CoreToolScheduler { this.setStatusInternal( callId, 'cancelled', - 'Tool call cancelled by user.', + createCancelledResponse( + scheduledCall.request, + 'Tool call cancelled before execution.', + 'not_started', + ), ); - setToolSpanCancelled(span); + if (this.toolSpans.has(callId)) { + setToolSpanCancelled(span); + } return; } if (!guardDecision.allowed) { @@ -4160,6 +4510,7 @@ export class CoreToolScheduler { scheduledCall.request, new Error(guardDecision.reason), ToolErrorType.EXECUTION_DENIED, + 'not_started', ); this.setStatusInternal(callId, 'error', errorResponse); setToolSpanFailure( @@ -4171,7 +4522,31 @@ export class CoreToolScheduler { } } - this.setStatusInternal(callId, 'executing'); + if (signal.aborted) { + const currentCall = this.toolCalls.find( + (call) => call.request.callId === callId, + ); + if ( + currentCall && + currentCall.status !== 'success' && + currentCall.status !== 'error' && + currentCall.status !== 'cancelled' + ) { + this.setStatusInternal( + callId, + 'cancelled', + createCancelledResponse( + scheduledCall.request, + 'Tool call cancelled before execution.', + 'not_started', + ), + ); + if (this.toolSpans.has(callId)) { + setToolSpanCancelled(span); + } + } + return; + } const liveOutputCallback = scheduledCall.tool.canUpdateOutput ? (outputChunk: ToolResultDisplay) => { @@ -4206,23 +4581,14 @@ export class CoreToolScheduler { // things like `onPid` and `onLiveOutput`. This will make the scheduler // agnostic to the invocation type. // - // Start the execution sub-span BEFORE invocation.execute() so its - // synchronous setup (shell command preprocessing, child_process.spawn, - // etc.) is bracketed by the span. We don't manually activate the span - // as OTel context here because the surrounding tool span is already - // active via runInToolSpanContext, and tool implementations don't - // currently emit nested OTel spans of their own — the span boundary - // is purely for timing/attribution. - const execSpan = startToolExecutionSpan(); - // try wraps both invocation.execute() and the await so synchronous - // throws (e.g. shell setup failure) flow into the same catch as async - // rejections — otherwise execSpan leaks unended and failure hooks - // are skipped. const sleepInhibitorHandle = acquireSleepInhibitor( this.config, `Qwen Code is executing tool ${canonicalName}`, ); let removeParentAbortForward: (() => void) | undefined; + let executionStatus: ToolExecutionStatus = 'not_started'; + let executionSettled = false; + let execSpan: Span | undefined; try { let promise: Promise; @@ -4306,27 +4672,43 @@ export class CoreToolScheduler { }; this.safelyAddToolArgumentsAttributes(span, invocation.params); promise = todoWorkChainContext.run(todoWorkChainId, () => - promptIdContext.run(scheduledCall.request.prompt_id, () => - invocation.execute( + promptIdContext.run(scheduledCall.request.prompt_id, () => { + // Keep this transition and execution span at the invocation + // boundary so setup failures remain not_started. + this.setStatusInternal(callId, 'executing'); + execSpan = startToolExecutionSpan({ + toolName: canonicalName, + callId, + }); + executionStatus = 'error'; + return invocation.execute( execSignal, liveOutputCallback, shellExecutionConfig, setPidCallback, setPromoteAbortControllerCallback, canPromoteForegroundShell, - ), - ), + ); + }), ); } else { this.safelyAddToolArgumentsAttributes(span, invocation.params); promise = todoWorkChainContext.run(todoWorkChainId, () => - promptIdContext.run(scheduledCall.request.prompt_id, () => - invocation.execute( + promptIdContext.run(scheduledCall.request.prompt_id, () => { + // Keep this transition and execution span at the invocation + // boundary so setup failures remain not_started. + this.setStatusInternal(callId, 'executing'); + execSpan = startToolExecutionSpan({ + toolName: canonicalName, + callId, + }); + executionStatus = 'error'; + return invocation.execute( execSignal, liveOutputCallback, shellExecutionConfig, - ), - ), + ); + }), ); } @@ -4377,21 +4759,41 @@ export class CoreToolScheduler { const isTimeout = toolResult.error?.type === ToolErrorType.EXECUTION_TIMEOUT && (!schedulerTimeoutResultSelected || schedulerTimeoutWon); - const aborted = signal.aborted && !isTimeout; - endToolExecutionSpan(execSpan, { - success: toolResult.error === undefined && !aborted, - error: aborted - ? TOOL_SPAN_STATUS_TOOL_CANCELLED - : isTimeout - ? TOOL_SPAN_STATUS_TOOL_TIMEOUT - : toolResult.error - ? TOOL_SPAN_STATUS_TOOL_ERROR - : undefined, - cancelled: aborted, - }); + const parentAbortedAtExecutionSettle = signal.aborted; + const aborted = parentAbortedAtExecutionSettle && !isTimeout; + const executionErrorType = toolResult.error + ? (toolResult.error.type ?? + (scheduledCall.tool instanceof DiscoveredMCPTool + ? ToolErrorType.MCP_TOOL_ERROR + : ToolErrorType.UNKNOWN)) + : undefined; + executionStatus = aborted + ? 'cancelled' + : toolResult.error + ? 'error' + : 'success'; + executionSettled = true; + if (execSpan) { + const completedExecSpan = execSpan; + execSpan = undefined; + endToolExecutionSpan(completedExecSpan, { + success: executionStatus === 'success', + error: aborted + ? TOOL_SPAN_STATUS_TOOL_CANCELLED + : isTimeout + ? TOOL_SPAN_STATUS_TOOL_TIMEOUT + : toolResult.error + ? TOOL_SPAN_STATUS_TOOL_ERROR + : undefined, + cancelled: aborted, + executionStatus, + errorType: executionErrorType, + }); + } if (aborted) { // PostToolUseFailure Hook - let cancelMessage = 'User cancelled tool execution.'; + // `execute()` returned a result here, so the tool's work did finish. + let cancelMessage = TOOL_CANCELLED_AFTER_COMPLETION_MESSAGE; let failureHookArtifacts: ToolArtifact[] | undefined; if (hooksEnabled && messageBus) { const failureHookResult = await this.withHookSpan( @@ -4424,18 +4826,45 @@ export class CoreToolScheduler { this.setStatusInternal( callId, 'cancelled', - failureHookArtifacts && failureHookArtifacts.length > 0 - ? createCancelledResponse( - scheduledCall.request, - cancelMessage, - failureHookArtifacts, - ) - : cancelMessage, + createCancelledResponse( + scheduledCall.request, + cancelMessage, + executionStatus, + failureHookArtifacts, + toolResult.persistedOutputFiles, + ), ); setToolSpanCancelled(span); return; // Both code paths should return here } + const cancelAfterPostProcessing = ( + artifacts?: ToolArtifact[], + preserved?: { + persistedOutputFiles?: string[]; + visionBridgeNotice?: string; + }, + ): boolean => { + if (!signal.aborted || (isTimeout && parentAbortedAtExecutionSettle)) { + return false; + } + this.setStatusInternal( + callId, + 'cancelled', + createCancelledResponse( + scheduledCall.request, + // Reached only after `execute()` settled with a result. + TOOL_CANCELLED_AFTER_COMPLETION_MESSAGE, + executionStatus, + artifacts, + preserved?.persistedOutputFiles, + preserved?.visionBridgeNotice, + ), + ); + setToolSpanCancelled(span); + return true; + }; + if (toolResult.error === undefined) { let content = toolResult.llmContent ?? ''; let persistedOutputFiles = toolResult.persistedOutputFiles @@ -4511,13 +4940,24 @@ export class CoreToolScheduler { // Check if hook requested to stop execution if (postHookResult.shouldStop) { + if ( + cancelAfterPostProcessing(postHookResult.artifacts, { + persistedOutputFiles, + }) + ) { + return; + } const stopMessage = postHookResult.stopReason || 'Execution stopped by hook'; const errorResponse = createErrorResponse( scheduledCall.request, new Error(stopMessage), ToolErrorType.EXECUTION_DENIED, + executionStatus, ); + if (persistedOutputFiles !== undefined) { + errorResponse.persistedOutputFiles = persistedOutputFiles; + } this.setStatusInternal(callId, 'error', errorResponse); setToolSpanFailure( span, @@ -4788,7 +5228,7 @@ export class CoreToolScheduler { ...(toolResult.artifacts ?? []), ...(postToolUseArtifacts ?? []), ]; - const successResponse: ToolCallResponseInfo = { + const successResponse: CoreToolCallResponseInfo = { callId, responseParts: response, resultDisplay: this.compactResultDisplayForInteractiveHistory( @@ -4796,6 +5236,7 @@ export class CoreToolScheduler { ), error: undefined, errorType: undefined, + executionStatus, contentLength, ...(persistedOutputFiles !== undefined ? { persistedOutputFiles } @@ -4867,6 +5308,14 @@ export class CoreToolScheduler { ); } } + if ( + cancelAfterPostProcessing(artifacts, { + persistedOutputFiles: successResponse.persistedOutputFiles, + visionBridgeNotice: successResponse.visionBridgeNotice, + }) + ) { + return; + } this.setStatusInternal(callId, 'success', successResponse); safeSetStatus(span, { code: SpanStatusCode.OK }); // Mirrors setToolSpanFailure/setToolSpanCancelled — every tool span @@ -4971,6 +5420,14 @@ export class CoreToolScheduler { ...(timeoutContent.persistedOutputFiles ?? []), ]), ); + if ( + cancelAfterPostProcessing(artifacts, { + persistedOutputFiles: timeoutPersistedOutputFiles, + visionBridgeNotice: processedImages.visionBridgeNotice, + }) + ) { + return; + } this.setStatusInternal(callId, 'error', { callId, responseParts, @@ -4979,6 +5436,7 @@ export class CoreToolScheduler { ), error: new Error(operationalErrorMessage), errorType: ToolErrorType.EXECUTION_TIMEOUT, + executionStatus, contentLength, ...(timeoutPersistedOutputFiles !== undefined ? { persistedOutputFiles: timeoutPersistedOutputFiles } @@ -5025,7 +5483,8 @@ export class CoreToolScheduler { let errorResponse = createErrorResponse( scheduledCall.request, error, - toolResult.error.type ?? ToolErrorType.UNKNOWN, + executionErrorType ?? ToolErrorType.UNKNOWN, + executionStatus, failureHookArtifacts, typeof toolResult.returnDisplay === 'string' ? undefined @@ -5086,6 +5545,17 @@ export class CoreToolScheduler { }; } } + if ( + cancelAfterPostProcessing( + [...(toolResult.artifacts ?? []), ...(failureHookArtifacts ?? [])], + { + persistedOutputFiles: errorResponse.persistedOutputFiles, + visionBridgeNotice: errorResponse.visionBridgeNotice, + }, + ) + ) { + return; + } this.setStatusInternal(callId, 'error', errorResponse); setToolSpanFailure( span, @@ -5103,18 +5573,48 @@ export class CoreToolScheduler { // see false positives. Both are still success: false; only the // sanitized error message and (for cancellation) the UNSET status // differ. - const aborted = signal.aborted; - endToolExecutionSpan(execSpan, { - success: false, - error: aborted - ? TOOL_SPAN_STATUS_TOOL_CANCELLED - : TOOL_SPAN_STATUS_TOOL_EXCEPTION, - cancelled: aborted, - }); + const executionThrew = + !executionSettled && executionStatus !== 'not_started'; + const explicitErrorType = ( + executionError as { errorType?: ToolErrorType } | undefined + )?.errorType; + const executionTimedOut = + executionThrew && explicitErrorType === ToolErrorType.EXECUTION_TIMEOUT; + const aborted = signal.aborted && !executionTimedOut; + if (executionThrew) { + executionStatus = aborted ? 'cancelled' : 'error'; + executionSettled = true; + } + const exceptionErrorType = + explicitErrorType ?? + (executionThrew && scheduledCall.tool instanceof DiscoveredMCPTool + ? ToolErrorType.MCP_TOOL_ERROR + : ToolErrorType.UNHANDLED_EXCEPTION); + if (execSpan) { + const failedExecSpan = execSpan; + execSpan = undefined; + endToolExecutionSpan(failedExecSpan, { + success: false, + error: aborted + ? TOOL_SPAN_STATUS_TOOL_CANCELLED + : executionTimedOut + ? TOOL_SPAN_STATUS_TOOL_TIMEOUT + : TOOL_SPAN_STATUS_TOOL_EXCEPTION, + cancelled: aborted, + executionStatus, + errorType: + executionStatus === 'error' ? exceptionErrorType : undefined, + }); + } if (aborted) { // PostToolUseFailure Hook (user interrupt) - let cancelMessage = 'User cancelled tool execution.'; + // `executionThrew` distinguishes a tool interrupted mid-flight (its + // work did NOT finish) from a throw raised after `execute()` already + // settled — e.g. by a post-processing transform. + let cancelMessage = executionThrew + ? TOOL_CANCELLED_BEFORE_COMPLETION_MESSAGE + : TOOL_CANCELLED_AFTER_COMPLETION_MESSAGE; let failureHookArtifacts: ToolArtifact[] | undefined; if (hooksEnabled && messageBus) { const failureHookResult = await this.withHookSpan( @@ -5147,13 +5647,12 @@ export class CoreToolScheduler { this.setStatusInternal( callId, 'cancelled', - failureHookArtifacts && failureHookArtifacts.length > 0 - ? createCancelledResponse( - scheduledCall.request, - cancelMessage, - failureHookArtifacts, - ) - : cancelMessage, + createCancelledResponse( + scheduledCall.request, + cancelMessage, + executionStatus, + failureHookArtifacts, + ), ); setToolSpanCancelled(span); return; @@ -5189,6 +5688,24 @@ export class CoreToolScheduler { } failureHookArtifacts = failureHookResult.artifacts; } + if (signal.aborted && !executionTimedOut) { + this.setStatusInternal( + callId, + 'cancelled', + createCancelledResponse( + scheduledCall.request, + // The abort landed while the failure hook was running; the + // tool's own outcome is still what `executionThrew` says. + executionThrew + ? TOOL_CANCELLED_BEFORE_COMPLETION_MESSAGE + : TOOL_CANCELLED_AFTER_COMPLETION_MESSAGE, + executionStatus, + failureHookArtifacts, + ), + ); + setToolSpanCancelled(span); + return; + } this.setStatusInternal( callId, 'error', @@ -5197,7 +5714,8 @@ export class CoreToolScheduler { executionError instanceof Error ? new Error(exceptionErrorMessage) : new Error(String(executionError)), - ToolErrorType.UNHANDLED_EXCEPTION, + exceptionErrorType, + executionStatus, failureHookArtifacts, ), ); @@ -5236,10 +5754,7 @@ export class CoreToolScheduler { let messageBus: MessageBus | undefined; try { - const shouldFirePostToolBatch = - !this.config.getDisableAllHooks() && - (this.config.hasHooksForEvent?.('PostToolBatch') ?? false); - messageBus = shouldFirePostToolBatch + messageBus = this.postToolBatchEnabledForBatch ? this.config.getMessageBus() : undefined; } catch (error) { @@ -5258,49 +5773,73 @@ export class CoreToolScheduler { if (messageBus) { const batchToolCalls = completedCalls.map(toPostToolBatchToolCall); const permissionMode = this.config.getApprovalMode(); - const batchHookResult = await this.withHookSpan( - { hookEvent: 'PostToolBatch', toolName: 'batch' }, - () => - firePostToolBatchHook( - messageBus, - batchToolCalls, - permissionMode, - batchSignal, - ), - (r) => - r.hookError - ? { - success: false, - error: r.hookError, - shouldStop: false, - postBatchStop: false, - } - : { - success: true, - shouldStop: r.shouldStop, - hasAdditionalContext: !!r.additionalContext, - hasArtifacts: !!r.artifacts?.length, - blockType: r.shouldStop ? 'stop' : undefined, - postBatchStop: r.shouldStop, - postBatchStopReason: r.shouldStop - ? r.stopReason || 'no reason given' - : undefined, - }, - ); + const fireBatchHook = () => + this.withHookSpan( + { hookEvent: 'PostToolBatch', toolName: 'batch' }, + () => + firePostToolBatchHook( + messageBus, + batchToolCalls, + permissionMode, + batchSignal, + ), + (r) => + r.hookError + ? { + success: false, + error: r.hookError, + shouldStop: false, + postBatchStop: false, + } + : { + success: true, + shouldStop: r.shouldStop, + hasAdditionalContext: !!r.additionalContext, + hasArtifacts: !!r.artifacts?.length, + blockType: r.shouldStop ? 'stop' : undefined, + postBatchStop: r.shouldStop, + postBatchStopReason: r.shouldStop + ? r.stopReason || 'no reason given' + : undefined, + }, + ); + const batchParentSpan = this.postToolBatchSpanCallId + ? this.toolSpans.get(this.postToolBatchSpanCallId) + : undefined; + const batchHookResult = await (batchParentSpan + ? runInToolSpanContext(batchParentSpan, fireBatchHook) + : fireBatchHook()); // Order matters: stop replaces the last response, so append // additionalContext only after the stop decision is applied. if (batchHookResult.shouldStop) { + const stopMessage = + batchHookResult.stopReason || + 'Execution stopped by PostToolBatch hook'; debugLogger.info( `PostToolBatch hook stopped batch (${completedCalls.length} calls): ${ batchHookResult.stopReason || 'no reason given' }`, ); - completedCalls = withPostToolBatchStop( - completedCalls, - batchHookResult.stopReason || - 'Execution stopped by PostToolBatch hook', - ); + completedCalls = withPostToolBatchStop(completedCalls, stopMessage); + const stoppedCall = completedCalls.at(-1); + const stoppedSpan = stoppedCall + ? this.toolSpans.get(stoppedCall.request.callId) + : undefined; + if (stoppedSpan) { + setToolSpanFailure( + stoppedSpan, + TOOL_FAILURE_KIND_POST_HOOK_STOPPED, + stopMessage, + ); + } else { + // Known gap: on a mixed batch the stopped call (last completed) + // can differ from the deferred-span call (last validating), so + // post_hook_stopped has no span to attach to. + debugLogger.debug( + `PostToolBatch stop: no tool span for stopped call ${stoppedCall?.request.callId}; post_hook_stopped not recorded`, + ); + } } completedCalls = withPostToolBatchAdditionalContext( @@ -5317,6 +5856,10 @@ export class CoreToolScheduler { // final invariant again after PostToolBatch. completedCalls = await this.applyBatchOutputBudget(completedCalls); + for (const call of completedCalls) { + this.finalizeToolSpan(call.request.callId, true); + } + for (const call of completedCalls) { this.runtimeContentGeneratorViews.delete(call.request.callId); logToolCall(this.config, new ToolCallEvent(call)); @@ -5329,16 +5872,18 @@ export class CoreToolScheduler { } } finally { try { + // Completion callbacks and output-budget transforms are external + // failure points. Never leave the one span deliberately deferred + // for PostToolBatch open when one of them throws. + for (const call of completedCalls) { + this.finalizeToolSpan(call.request.callId, true); + } + this.postToolBatchEnabledForBatch = false; + this.postToolBatchSpanCallId = undefined; this.notifyToolCallsUpdate(); } finally { this.isFinalizingToolCalls = false; - // Always drain the queue, even if completion callbacks throw. - if (this.requestQueue.length > 0) { - const next = this.requestQueue.shift()!; - this._schedule(next.request, next.signal, next.runtimeView) - .then(next.resolve) - .catch(next.reject); - } + this.drainRequestQueueIfIdle(); } } } @@ -5429,6 +5974,7 @@ export class CoreToolScheduler { const result = { callId: call.request.callId, status: call.status, + executionStatus: call.response.executionStatus, resultDisplay: call.response.resultDisplay, ...(call.response.visionBridgeNotice !== undefined ? { visionBridgeNotice: call.response.visionBridgeNotice } @@ -5462,8 +6008,17 @@ export class CoreToolScheduler { } private notifyToolCallsUpdate(): void { - if (this.onToolCallsUpdate) { + if (!this.onToolCallsUpdate) { + return; + } + try { this.onToolCallsUpdate([...this.toolCalls]); + } catch (error) { + debugLogger.error( + `Tool call update observer failed: ${ + error instanceof Error ? error.message : String(error) + }`, + ); } } @@ -5497,6 +6052,11 @@ export class CoreToolScheduler { for (const pendingTool of pendingTools) { try { + if ( + this.cancelPreExecutionIfAborted(pendingTool.request.callId, signal) + ) { + continue; + } // Re-run L3→L4 to see if the tool can now be auto-approved const toolParams = pendingTool.invocation.params as Record< string, @@ -5512,6 +6072,11 @@ export class CoreToolScheduler { toolParams, ), ); + if ( + this.cancelPreExecutionIfAborted(pendingTool.request.callId, signal) + ) { + continue; + } const { finalPermission, pmForcedAsk, pmCtx, requiresUserInteraction } = flowResult; @@ -5552,6 +6117,11 @@ export class CoreToolScheduler { : undefined, }), ); + if ( + this.cancelPreExecutionIfAborted(pendingTool.request.callId, signal) + ) { + continue; + } const outcome = applyAutoModeDecision( decision, @@ -5581,6 +6151,11 @@ export class CoreToolScheduler { ); } } + if ( + this.cancelPreExecutionIfAborted(pendingTool.request.callId, signal) + ) { + continue; + } switch (outcome.kind) { case 'approved': this.setToolCallOutcome( @@ -5602,6 +6177,7 @@ export class CoreToolScheduler { pendingTool.request, new Error(outcome.errorMessage), ToolErrorType.EXECUTION_DENIED, + 'not_started', ), ); this.finalizeBlockedSpan( @@ -5672,6 +6248,11 @@ export class CoreToolScheduler { ); } } catch (error) { + if ( + this.cancelPreExecutionIfAborted(pendingTool.request.callId, signal) + ) { + continue; + } debugLogger.error( `Error checking confirmation for tool ${pendingTool.request.callId}:`, error, diff --git a/packages/core/src/core/nonInteractiveToolExecutor.test.ts b/packages/core/src/core/nonInteractiveToolExecutor.test.ts index fb860175183..0ef4e7965b0 100644 --- a/packages/core/src/core/nonInteractiveToolExecutor.test.ts +++ b/packages/core/src/core/nonInteractiveToolExecutor.test.ts @@ -107,6 +107,7 @@ describe('executeToolCall', () => { callId: 'call1', error: undefined, errorType: undefined, + executionStatus: 'success', resultDisplay: 'Success!', contentLength: typeof toolResult.llmContent === 'string' @@ -224,6 +225,7 @@ describe('executeToolCall', () => { callId: 'call2', error: new Error(expectedErrorMessage), errorType: ToolErrorType.TOOL_NOT_REGISTERED, + executionStatus: 'not_started', resultDisplay: expectedErrorMessage, contentLength: expectedErrorMessage.length, responseParts: [ @@ -262,6 +264,7 @@ describe('executeToolCall', () => { callId: 'call3', error: new Error('Invalid parameters'), errorType: ToolErrorType.INVALID_TOOL_PARAMS, + executionStatus: 'not_started', responseParts: [ { functionResponse: { @@ -306,6 +309,7 @@ describe('executeToolCall', () => { callId: 'call4', error: new Error('Execution failed'), errorType: ToolErrorType.EXECUTION_FAILED, + executionStatus: 'error', responseParts: [ { functionResponse: { @@ -343,6 +347,7 @@ describe('executeToolCall', () => { callId: 'call5', error: new Error('Something went very wrong'), errorType: ToolErrorType.UNHANDLED_EXCEPTION, + executionStatus: 'error', resultDisplay: 'Something went very wrong', contentLength: 'Something went very wrong'.length, responseParts: [ @@ -385,6 +390,7 @@ describe('executeToolCall', () => { callId: 'call6', error: undefined, errorType: undefined, + executionStatus: 'success', resultDisplay: 'Image processed', contentLength: undefined, responseParts: [ diff --git a/packages/core/src/core/turn.test.ts b/packages/core/src/core/turn.test.ts index a049b4b0d0c..ce7f21d953e 100644 --- a/packages/core/src/core/turn.test.ts +++ b/packages/core/src/core/turn.test.ts @@ -14,6 +14,7 @@ import { CompressionStatus, Turn, GeminiEventType, + createDuplicateProviderToolCallResponse, findRepeatedDuplicateProviderToolCall, } from './turn.js'; import type { @@ -101,6 +102,21 @@ describe('findRepeatedDuplicateProviderToolCall', () => { }); }); +describe('createDuplicateProviderToolCallResponse', () => { + it('marks the synthetic response as not started', () => { + const response = createDuplicateProviderToolCallResponse({ + callId: 'duplicate-response', + providerCallId: 'provider-call', + name: 'read_file', + args: {}, + isClientInitiated: false, + prompt_id: 'prompt-duplicate', + }); + + expect(response.executionStatus).toBe('not_started'); + }); +}); + describe('Turn', () => { let turn: Turn; // Define a type for the mocked Chat instance for clarity diff --git a/packages/core/src/core/turn.ts b/packages/core/src/core/turn.ts index 9c8838a4dd6..4439846df03 100644 --- a/packages/core/src/core/turn.ts +++ b/packages/core/src/core/turn.ts @@ -139,12 +139,19 @@ export interface ToolCallRequestInfo { goalContext?: GoalTurnPermit; } +export type ToolExecutionStatus = + | 'not_started' + | 'success' + | 'error' + | 'cancelled'; + export interface ToolCallResponseInfo { callId: string; responseParts: Part[]; resultDisplay: ToolResultDisplay | undefined; error: Error | undefined; errorType: ToolErrorType | undefined; + executionStatus?: ToolExecutionStatus; contentLength?: number; persistedOutputFiles?: string[]; modelOverride?: string; @@ -233,6 +240,7 @@ export function createDuplicateProviderToolCallResponse( resultDisplay: message, error: new Error(message), errorType: ToolErrorType.EXECUTION_FAILED, + executionStatus: 'not_started', }; } diff --git a/packages/core/src/hooks/types.ts b/packages/core/src/hooks/types.ts index 3c5005233f9..ef9d97aa491 100644 --- a/packages/core/src/hooks/types.ts +++ b/packages/core/src/hooks/types.ts @@ -828,8 +828,8 @@ export interface PostToolBatchToolCall { status: 'success' | 'error' | 'cancelled'; /** * Serialized ToolCallResponseInfo fields for the resolved call: - * response_parts, result_display, error, error_type, content_length, and - * vision_bridge_notice when applicable. + * response_parts, result_display, error, error_type, execution_status, + * content_length, and vision_bridge_notice when applicable. */ tool_response?: Record; } diff --git a/packages/core/src/telemetry/constants.ts b/packages/core/src/telemetry/constants.ts index 108cc00e461..66e89e612f2 100644 --- a/packages/core/src/telemetry/constants.ts +++ b/packages/core/src/telemetry/constants.ts @@ -101,3 +101,24 @@ export const SPAN_HOOK = 'qwen-code.hook'; * (#3731 Phase 3). */ export const SPAN_SUBAGENT = 'qwen-code.subagent'; + +// Tool failure kind span attribute and vocabulary — shared across +// coreToolScheduler, session-tracing, and telemetry docs so the +// write sites and documented values cannot drift. +export const TOOL_FAILURE_KIND_ATTRIBUTE = 'tool.failure_kind'; +export const TOOL_FAILURE_KIND_CANCELLED = 'cancelled'; +export const TOOL_FAILURE_KIND_PRE_HOOK_BLOCKED = 'pre_hook_blocked'; +export const TOOL_FAILURE_KIND_INVOCATION_GUARD_DENIED = + 'invocation_guard_denied'; +export const TOOL_FAILURE_KIND_POST_HOOK_STOPPED = 'post_hook_stopped'; +export const TOOL_FAILURE_KIND_TOOL_ERROR = 'tool_error'; +export const TOOL_FAILURE_KIND_TOOL_EXCEPTION = 'tool_exception'; +export const TOOL_FAILURE_KIND_PERMISSION_DENIED = 'permission_denied'; +export const TOOL_FAILURE_KIND_PERMISSION_HOOK_DENIED = + 'permission_hook_denied'; +export const TOOL_FAILURE_KIND_PLAN_MODE_BLOCKED = 'plan_mode_blocked'; +export const TOOL_FAILURE_KIND_NON_INTERACTIVE_DENIED = + 'non_interactive_denied'; +export const TOOL_FAILURE_KIND_BACKGROUND_AGENT_DENIED = + 'background_agent_denied'; +export const TOOL_FAILURE_KIND_TIMEOUT = 'timeout'; diff --git a/packages/core/src/telemetry/index.ts b/packages/core/src/telemetry/index.ts index 0cea7f1d96f..363982d5e4b 100644 --- a/packages/core/src/telemetry/index.ts +++ b/packages/core/src/telemetry/index.ts @@ -112,6 +112,7 @@ export * from './api-activity-tracker.js'; export { // Core metrics functions recordToolCallMetrics, + recordToolExecutionMetrics, recordTokenUsageMetrics, recordApiResponseMetrics, recordApiErrorMetrics, diff --git a/packages/core/src/telemetry/log-to-span-processor.test.ts b/packages/core/src/telemetry/log-to-span-processor.test.ts index 9cab4964408..4bcdc147a29 100644 --- a/packages/core/src/telemetry/log-to-span-processor.test.ts +++ b/packages/core/src/telemetry/log-to-span-processor.test.ts @@ -595,6 +595,42 @@ describe('LogToSpanProcessor', () => { expect(exportedSpans[0].status.code).toBe(SpanStatusCode.OK); }); + it('keeps cancelled tool calls UNSET even when legacy errors are present', async () => { + const logRecord = { + body: 'tool call cancelled', + hrTime: [1000, 0] as [number, number], + attributes: { + 'event.name': 'qwen-code.tool_call', + status: 'cancelled', + success: false, + error: 'cancelled by user', + error_type: 'unhandled_exception', + }, + } as unknown as ReadableLogRecord; + + processor.onEmit(logRecord); + await processor.forceFlush(); + + expect(exportedSpans[0].status.code).toBe(SpanStatusCode.UNSET); + }); + + it('keeps ERROR for cancelled non-tool events that carry an error', async () => { + const logRecord = { + body: 'auth cancelled with error', + hrTime: [1000, 0] as [number, number], + attributes: { + 'event.name': 'qwen-code.auth', + status: 'cancelled', + error_message: 'auth flow failed', + }, + } as unknown as ReadableLogRecord; + + processor.onEmit(logRecord); + await processor.forceFlush(); + + expect(exportedSpans[0].status.code).toBe(SpanStatusCode.ERROR); + }); + it('does not set ERROR for falsy error attributes', async () => { const logRecord = { body: 'ok event', diff --git a/packages/core/src/telemetry/log-to-span-processor.ts b/packages/core/src/telemetry/log-to-span-processor.ts index 5987e99639b..b4b455b8481 100644 --- a/packages/core/src/telemetry/log-to-span-processor.ts +++ b/packages/core/src/telemetry/log-to-span-processor.ts @@ -22,7 +22,11 @@ import { resourceFromAttributes, } from '@opentelemetry/resources'; -import { EVENT_SUBAGENT_EXECUTION, SERVICE_NAME } from './constants.js'; +import { + EVENT_SUBAGENT_EXECUTION, + EVENT_TOOL_CALL, + SERVICE_NAME, +} from './constants.js'; import { deriveTraceId, randomHexString, @@ -458,6 +462,14 @@ function deriveSpanStatus(attrs: Record | undefined): { message?: string; } { if (!attrs) return { code: SpanStatusCode.OK }; + // Only tool calls freeze a `cancelled` terminal that must stay UNSET; other + // events (e.g. auth) can be cancelled AND carry an error, which stays ERROR. + if ( + attrs['event.name'] === EVENT_TOOL_CALL && + attrs['status'] === 'cancelled' + ) { + return { code: SpanStatusCode.UNSET }; + } if ( !!attrs['error'] || !!attrs['error.message'] || diff --git a/packages/core/src/telemetry/loggers.test.ts b/packages/core/src/telemetry/loggers.test.ts index 89885031219..62221808a35 100644 --- a/packages/core/src/telemetry/loggers.test.ts +++ b/packages/core/src/telemetry/loggers.test.ts @@ -66,6 +66,7 @@ import { logApiRetry, logProtocolTagSanitized, logMemoryRecallDelivery, + normalizeToolCallEvent, } from './loggers.js'; import * as metrics from './metrics.js'; import { apiActivityTracker } from './api-activity-tracker.js'; @@ -988,12 +989,16 @@ describe('loggers', () => { const mockMetrics = { recordToolCallMetrics: vi.fn(), + recordToolExecutionMetrics: vi.fn(), }; beforeEach(() => { vi.spyOn(metrics, 'recordToolCallMetrics').mockImplementation( mockMetrics.recordToolCallMetrics, ); + vi.spyOn(metrics, 'recordToolExecutionMetrics').mockImplementation( + mockMetrics.recordToolExecutionMetrics, + ); vi.spyOn(QwenLogger.prototype, 'logToolCallEvent').mockImplementation( () => undefined, ); @@ -1004,13 +1009,11 @@ describe('loggers', () => { const recordUiTelemetryEvent = vi.fn(); const configWithRecording = { ...mockConfig, - getChatRecordingService: () => ({ - recordUiTelemetryEvent, - }), + getChatRecordingService: () => ({ recordUiTelemetryEvent }), } as unknown as Config; const event = { 'event.name': 'tool_call', - 'event.timestamp': '2024-12-31T23:59:59.000Z', + 'event.timestamp': '2025-01-01T00:00:00.000Z', function_name: ' ', function_args: { value: 1 }, duration_ms: 25, @@ -1028,6 +1031,7 @@ describe('loggers', () => { function_name: 'unknown_tool', status: 'error', success: false, + execution_status: 'unknown', error: 'failed', error_type: ToolErrorType.UNKNOWN, }); @@ -1045,6 +1049,7 @@ describe('loggers', () => { function_name: 'unknown_tool', status: 'error', success: false, + execution_status: 'unknown', error: 'failed', error_type: ToolErrorType.UNKNOWN, 'error.message': 'failed', @@ -1063,17 +1068,65 @@ describe('loggers', () => { tool_type: 'native', }, ); - expect(event).toMatchObject({ - function_name: ' ', + expect(mockMetrics.recordToolExecutionMetrics).toHaveBeenCalledWith( + configWithRecording, + { + execution_status: 'unknown', + tool_type: 'native', + }, + ); + expect(event).not.toHaveProperty('execution_status'); + expect(event.function_name).toBe(' '); + expect(event.success).toBe(true); + expect(event.error_type).toBe(' '); + }); + + it('clears call errors when cancellation is the final outcome', () => { + const event = { + 'event.name': 'tool_call', + 'event.timestamp': '2025-01-01T00:00:00.000Z', + function_name: 'shell', + function_args: {}, + duration_ms: 1, + status: 'cancelled', + execution_status: 'cancelled', success: true, - error_type: ' ', - }); + error: 'cancelled by user', + error_type: ToolErrorType.UNHANDLED_EXCEPTION, + prompt_id: 'prompt-id', + tool_type: 'native', + } as ToolCallEvent; + + const normalized = normalizeToolCallEvent(event); + + expect(normalized.success).toBe(false); + expect(normalized).not.toHaveProperty('error'); + expect(normalized).not.toHaveProperty('error_type'); + expect(event.error).toBe('cancelled by user'); + }); + + it('preserves a nonblank function name byte-for-byte', () => { + const event = { + 'event.name': 'tool_call', + 'event.timestamp': '2025-01-01T00:00:00.000Z', + function_name: ' padded_tool ', + function_args: {}, + duration_ms: 1, + status: 'success', + success: true, + prompt_id: 'prompt-padded', + tool_type: 'native', + } as ToolCallEvent; + + expect(normalizeToolCallEvent(event).function_name).toBe( + ' padded_tool ', + ); }); it('preserves an explicitly classified error type', () => { const event = { 'event.name': 'tool_call', - 'event.timestamp': '2024-12-31T23:59:59.000Z', + 'event.timestamp': '2025-01-01T00:00:00.000Z', function_name: 'test-function', function_args: {}, duration_ms: 10, @@ -1090,6 +1143,7 @@ describe('loggers', () => { expect(QwenLogger.prototype.logToolCallEvent).toHaveBeenCalledWith( expect.objectContaining({ error_type: ToolErrorType.EXECUTION_FAILED, + execution_status: 'unknown', }), ); expect(mockLogger.emit.mock.calls[0][0].attributes).toMatchObject({ @@ -1098,6 +1152,41 @@ describe('loggers', () => { }); }); + it('normalizes a missing execution_status to unknown end-to-end', () => { + const configWithRecording = { + ...mockConfig, + getChatRecordingService: () => ({ recordUiTelemetryEvent: vi.fn() }), + } as unknown as Config; + const event = { + 'event.name': 'tool_call', + 'event.timestamp': '2025-01-01T00:00:00.000Z', + function_name: 'legacy_tool', + function_args: {}, + duration_ms: 42, + status: 'success', + success: true, + prompt_id: 'prompt-legacy', + tool_type: 'native', + } as ToolCallEvent; + + expect(event).not.toHaveProperty('execution_status'); + + logToolCall(configWithRecording, event); + + expect(mockMetrics.recordToolExecutionMetrics).toHaveBeenCalledWith( + configWithRecording, + { + execution_status: 'unknown', + tool_type: 'native', + }, + ); + expect(QwenLogger.prototype.logToolCallEvent).toHaveBeenCalledWith( + expect.objectContaining({ + execution_status: 'unknown', + }), + ); + }); + it.each([ { status: 'success' as const, expectedSuccess: true }, { status: 'cancelled' as const, expectedSuccess: false }, @@ -1106,7 +1195,7 @@ describe('loggers', () => { ({ status, expectedSuccess }) => { const event = { 'event.name': 'tool_call', - 'event.timestamp': '2024-12-31T23:59:59.000Z', + 'event.timestamp': '2025-01-01T00:00:00.000Z', function_name: 'test-function', function_args: {}, duration_ms: 10, @@ -1120,39 +1209,31 @@ describe('loggers', () => { logToolCall(mockConfig, event); - expect(QwenLogger.prototype.logToolCallEvent).toHaveBeenCalledWith( - expect.objectContaining({ - status, - success: expectedSuccess, - error: undefined, - error_type: undefined, - }), - ); - const attributes = mockLogger.emit.mock.calls[0][0].attributes; - expect(attributes).toMatchObject({ + const normalizedEvent = vi.mocked(QwenLogger.prototype.logToolCallEvent) + .mock.calls[0][0]; + expect(normalizedEvent).toMatchObject({ status, success: expectedSuccess, - error: undefined, - error_type: undefined, + execution_status: 'unknown', }); + expect(normalizedEvent).not.toHaveProperty('error'); + expect(normalizedEvent).not.toHaveProperty('error_type'); + const attributes = mockLogger.emit.mock.calls[0][0].attributes; expect(attributes).not.toHaveProperty('error.message'); expect(attributes).not.toHaveProperty('error.type'); expect(mockMetrics.recordToolCallMetrics).toHaveBeenCalledWith( mockConfig, 10, - expect.objectContaining({ - status, - success: expectedSuccess, - }), + expect.objectContaining({ status, success: expectedSuccess }), ); }, ); - it('normalizes UI and QwenLogger events when the OTel SDK is disabled', () => { + it('normalizes non-OTel consumers when the SDK is disabled', () => { vi.spyOn(sdk, 'isTelemetrySdkInitialized').mockReturnValue(false); const event = { 'event.name': 'tool_call', - 'event.timestamp': '2024-12-31T23:59:59.000Z', + 'event.timestamp': '2025-01-01T00:00:00.000Z', function_name: '', function_args: {}, duration_ms: 10, @@ -1168,6 +1249,7 @@ describe('loggers', () => { function_name: 'unknown_tool', status: 'error', success: false, + execution_status: 'unknown', error_type: ToolErrorType.UNKNOWN, }); expect(QwenLogger.prototype.logToolCallEvent).toHaveBeenCalledWith( @@ -1179,6 +1261,67 @@ describe('loggers', () => { ); expect(mockLogger.emit).not.toHaveBeenCalled(); expect(mockMetrics.recordToolCallMetrics).not.toHaveBeenCalled(); + expect(mockMetrics.recordToolExecutionMetrics).not.toHaveBeenCalled(); + }); + + it('isolates every tool-call telemetry sink failure', () => { + const chatSink = vi.fn(() => { + throw new Error('chat sink failed'); + }); + const qwenSink = vi.fn(() => { + throw new Error('qwen sink failed'); + }); + const qwenLoggerSpy = vi + .spyOn(QwenLogger, 'getInstance') + .mockReturnValue({ + logToolCallEvent: qwenSink, + } as unknown as QwenLogger); + mockUiEvent.addEvent.mockImplementationOnce(() => { + throw new Error('ui sink failed'); + }); + mockLogger.emit.mockImplementationOnce(() => { + throw new Error('otel sink failed'); + }); + mockMetrics.recordToolCallMetrics.mockImplementationOnce(() => { + throw new Error('legacy metric sink failed'); + }); + mockMetrics.recordToolExecutionMetrics.mockImplementationOnce(() => { + throw new Error('execution metric sink failed'); + }); + const config = { + ...mockConfig, + getChatRecordingService: () => ({ + recordUiTelemetryEvent: chatSink, + }), + } as unknown as Config; + const event = { + 'event.name': 'tool_call', + 'event.timestamp': '2025-01-01T00:00:00.000Z', + call_id: 'call-id', + function_name: 'read_file', + function_args: {}, + duration_ms: 1, + status: 'success', + execution_status: 'success', + success: true, + prompt_id: 'prompt-id', + tool_type: 'native', + } as ToolCallEvent; + + expect(() => logToolCall(config, event)).not.toThrow(); + expect(mockUiEvent.addEvent).toHaveBeenCalled(); + expect(chatSink).toHaveBeenCalled(); + expect(qwenSink).toHaveBeenCalled(); + expect(mockLogger.emit).toHaveBeenCalled(); + expect(mockMetrics.recordToolCallMetrics).toHaveBeenCalled(); + expect(mockMetrics.recordToolExecutionMetrics).toHaveBeenCalledWith( + config, + { + execution_status: 'success', + tool_type: 'native', + }, + ); + qwenLoggerSpy.mockRestore(); }); it('should log a tool call with all fields', () => { @@ -1217,6 +1360,7 @@ describe('loggers', () => { error: undefined, errorType: undefined, contentLength: 13, + executionStatus: 'success', }, tool, invocation: {} as AnyToolInvocation, @@ -1233,6 +1377,7 @@ describe('loggers', () => { 'session.id': 'test-session-id', 'event.name': EVENT_TOOL_CALL, 'event.timestamp': '2025-01-01T00:00:00.000Z', + call_id: 'test-call-id', function_name: 'test-function', function_args: JSON.stringify( { @@ -1244,13 +1389,11 @@ describe('loggers', () => { ), duration_ms: 100, status: 'success', + execution_status: 'success', success: true, decision: ToolCallDecision.ACCEPT, prompt_id: 'prompt-id-1', tool_type: 'native', - error: undefined, - error_type: undefined, - metadata: { model_added_lines: 1, model_removed_lines: 2, @@ -1262,6 +1405,8 @@ describe('loggers', () => { user_removed_chars: 8, }, content_length: 13, + mcp_server_name: undefined, + response_id: undefined, }, }); @@ -1276,10 +1421,17 @@ describe('loggers', () => { tool_type: 'native', }, ); + expect(mockMetrics.recordToolExecutionMetrics).toHaveBeenCalledWith( + mockConfig, + { + execution_status: 'success', + tool_type: 'native', + }, + ); expect(mockUiEvent.addEvent).toHaveBeenCalledWith( { - ...event, + ...normalizeToolCallEvent(event), 'event.name': EVENT_TOOL_CALL, 'event.timestamp': '2025-01-01T00:00:00.000Z', }, @@ -1306,6 +1458,7 @@ describe('loggers', () => { error: undefined, errorType: undefined, contentLength: undefined, + executionStatus: 'not_started', }, durationMs: 100, outcome: ToolConfirmationOutcome.Cancel, @@ -1320,6 +1473,7 @@ describe('loggers', () => { 'session.id': 'test-session-id', 'event.name': EVENT_TOOL_CALL, 'event.timestamp': '2025-01-01T00:00:00.000Z', + call_id: 'test-call-id', function_name: 'test-function', function_args: JSON.stringify( { @@ -1331,6 +1485,7 @@ describe('loggers', () => { ), duration_ms: 100, status: 'error', + execution_status: 'not_started', success: false, decision: ToolCallDecision.REJECT, prompt_id: 'prompt-id-2', @@ -1338,10 +1493,10 @@ describe('loggers', () => { error: undefined, error_type: ToolErrorType.UNKNOWN, 'error.type': ToolErrorType.UNKNOWN, - mcp_server_name: undefined, - response_id: undefined, metadata: undefined, content_length: undefined, + mcp_server_name: undefined, + response_id: undefined, }, }); @@ -1359,8 +1514,7 @@ describe('loggers', () => { expect(mockUiEvent.addEvent).toHaveBeenCalledWith( { - ...event, - error_type: ToolErrorType.UNKNOWN, + ...normalizeToolCallEvent(event), 'event.name': EVENT_TOOL_CALL, 'event.timestamp': '2025-01-01T00:00:00.000Z', }, @@ -1388,6 +1542,7 @@ describe('loggers', () => { error: undefined, errorType: undefined, contentLength: 13, + executionStatus: 'success', }, outcome: ToolConfirmationOutcome.ModifyWithEditor, tool: new EditTool(mockConfig), @@ -1404,6 +1559,7 @@ describe('loggers', () => { 'session.id': 'test-session-id', 'event.name': EVENT_TOOL_CALL, 'event.timestamp': '2025-01-01T00:00:00.000Z', + call_id: 'test-call-id', function_name: 'test-function', function_args: JSON.stringify( { @@ -1415,14 +1571,15 @@ describe('loggers', () => { ), duration_ms: 100, status: 'success', + execution_status: 'success', success: true, decision: ToolCallDecision.MODIFY, prompt_id: 'prompt-id-3', tool_type: 'native', - error: undefined, - error_type: undefined, metadata: undefined, content_length: 13, + mcp_server_name: undefined, + response_id: undefined, }, }); @@ -1440,7 +1597,7 @@ describe('loggers', () => { expect(mockUiEvent.addEvent).toHaveBeenCalledWith( { - ...event, + ...normalizeToolCallEvent(event), 'event.name': EVENT_TOOL_CALL, 'event.timestamp': '2025-01-01T00:00:00.000Z', }, @@ -1468,6 +1625,7 @@ describe('loggers', () => { error: undefined, errorType: undefined, contentLength: 13, + executionStatus: 'success', }, tool: new EditTool(mockConfig), invocation: {} as AnyToolInvocation, @@ -1483,6 +1641,7 @@ describe('loggers', () => { 'session.id': 'test-session-id', 'event.name': EVENT_TOOL_CALL, 'event.timestamp': '2025-01-01T00:00:00.000Z', + call_id: 'test-call-id', function_name: 'test-function', function_args: JSON.stringify( { @@ -1494,14 +1653,15 @@ describe('loggers', () => { ), duration_ms: 100, status: 'success', + execution_status: 'success', success: true, prompt_id: 'prompt-id-4', tool_type: 'native', decision: undefined, - error: undefined, - error_type: undefined, metadata: undefined, content_length: 13, + mcp_server_name: undefined, + response_id: undefined, }, }); @@ -1519,7 +1679,7 @@ describe('loggers', () => { expect(mockUiEvent.addEvent).toHaveBeenCalledWith( { - ...event, + ...normalizeToolCallEvent(event), 'event.name': EVENT_TOOL_CALL, 'event.timestamp': '2025-01-01T00:00:00.000Z', }, @@ -1548,6 +1708,7 @@ describe('loggers', () => { error: new Error(errorMessage), errorType: ToolErrorType.UNKNOWN, contentLength: errorMessage.length, + executionStatus: 'error', }, durationMs: 100, }; @@ -1561,6 +1722,7 @@ describe('loggers', () => { 'session.id': 'test-session-id', 'event.name': EVENT_TOOL_CALL, 'event.timestamp': '2025-01-01T00:00:00.000Z', + call_id: 'test-call-id', function_name: 'test-function', function_args: JSON.stringify( { @@ -1572,6 +1734,7 @@ describe('loggers', () => { ), duration_ms: 100, status: 'error', + execution_status: 'error', success: false, error: 'test-error', 'error.message': 'test-error', @@ -1582,6 +1745,8 @@ describe('loggers', () => { decision: undefined, metadata: undefined, content_length: errorMessage.length, + mcp_server_name: undefined, + response_id: undefined, }, }); @@ -1599,7 +1764,7 @@ describe('loggers', () => { expect(mockUiEvent.addEvent).toHaveBeenCalledWith( { - ...event, + ...normalizeToolCallEvent(event), 'event.name': EVENT_TOOL_CALL, 'event.timestamp': '2025-01-01T00:00:00.000Z', }, @@ -1638,6 +1803,7 @@ describe('loggers', () => { resultDisplay: undefined, error: undefined, errorType: undefined, + executionStatus: 'success', }, tool: mockMcpTool, invocation: {} as AnyToolInvocation, @@ -1653,6 +1819,7 @@ describe('loggers', () => { 'session.id': 'test-session-id', 'event.name': EVENT_TOOL_CALL, 'event.timestamp': '2025-01-01T00:00:00.000Z', + call_id: 'test-call-id', function_name: 'mock_mcp_tool', function_args: JSON.stringify( { @@ -1664,13 +1831,12 @@ describe('loggers', () => { ), duration_ms: 100, status: 'success', + execution_status: 'success', success: true, prompt_id: 'prompt-id', tool_type: 'mcp', mcp_server_name: 'mock_mcp_server', decision: undefined, - error: undefined, - error_type: undefined, metadata: undefined, content_length: undefined, response_id: undefined, @@ -1704,6 +1870,7 @@ describe('loggers', () => { resultDisplay: undefined, error: undefined, errorType: undefined, + executionStatus: 'success', }, tool: new EditTool(mockConfig), invocation: {} as AnyToolInvocation, diff --git a/packages/core/src/telemetry/loggers.ts b/packages/core/src/telemetry/loggers.ts index 3b960d64c08..acd4a647d02 100644 --- a/packages/core/src/telemetry/loggers.ts +++ b/packages/core/src/telemetry/loggers.ts @@ -10,7 +10,6 @@ import { SemanticAttributes } from '@opentelemetry/semantic-conventions'; import type { Config } from '../config/config.js'; import { isInternalPromptId } from '../utils/internalPromptIds.js'; import { safeJsonStringify } from '../utils/safeJsonStringify.js'; -import { ToolErrorType } from '../tools/tool-error.js'; import { EVENT_API_ERROR, EVENT_API_CANCEL, @@ -72,6 +71,7 @@ import { recordSubagentExecutionMetrics, recordTokenUsageMetrics, recordToolCallMetrics, + recordToolExecutionMetrics, recordArenaSessionStartedMetrics, recordArenaAgentCompletedMetrics, recordArenaSessionEndedMetrics, @@ -138,6 +138,8 @@ import { uiTelemetryService } from './uiTelemetry.js'; import { apiActivityTracker } from './api-activity-tracker.js'; import { recordTokenUsageFromApiResponseBestEffort } from '../services/tokenUsageService.js'; import { isChatRecordingSuppressed } from '../utils/chat-recording-suppression-context.js'; +import { ToolErrorType } from '../tools/tool-error.js'; +import { createDebugLogger } from '../utils/debugLogger.js'; const shouldLogUserPrompts = (config: Config): boolean => config.getTelemetryLogPromptsEnabled(); @@ -155,6 +157,47 @@ function recordUiTelemetryEventToChat(config: Config, uiEvent: UiEvent): void { export { getCommonAttributes }; +type NormalizedToolCallEvent = ToolCallEvent & { + execution_status: NonNullable; +}; + +/** + * Normalizes a tool call event for telemetry sinks. Error fields are + * deleted (not set to undefined) on success so downstream consumers + * see key-absent rather than key-present-with-undefined. + */ +export function normalizeToolCallEvent( + event: ToolCallEvent, +): NormalizedToolCallEvent { + const functionName = event.function_name ?? ''; + const normalized: NormalizedToolCallEvent = { + ...event, + function_name: + functionName.trim().length > 0 ? functionName : 'unknown_tool', + success: event.status === 'success', + execution_status: event.execution_status ?? 'unknown', + }; + + if (event.status === 'error') { + normalized.error_type = event.error_type?.trim() || ToolErrorType.UNKNOWN; + } else { + delete normalized.error; + delete normalized.error_type; + } + + return normalized; +} + +const debugLogger = createDebugLogger('TELEMETRY_SINK'); + +function runToolTelemetrySink(sink: () => void): void { + try { + sink(); + } catch (e) { + debugLogger.debug('Telemetry sink failed (best-effort):', e); + } +} + export function logStartSession( config: Config, event: StartSessionEvent, @@ -245,24 +288,6 @@ export function logUserRetry(config: Config, event: UserRetryEvent): void { logger.emit(logRecord); } -function normalizeToolCallEvent(event: ToolCallEvent): ToolCallEvent { - const isError = event.status === 'error'; - return { - ...event, - function_name: - event.function_name.trim().length > 0 - ? event.function_name - : 'unknown_tool', - success: event.status === 'success', - error: isError ? event.error : undefined, - error_type: isError - ? event.error_type?.trim() - ? event.error_type - : ToolErrorType.UNKNOWN - : undefined, - }; -} - export function logToolCall(config: Config, event: ToolCallEvent): void { const normalizedEvent = normalizeToolCallEvent(event); const uiEvent = { @@ -270,39 +295,55 @@ export function logToolCall(config: Config, event: ToolCallEvent): void { 'event.name': EVENT_TOOL_CALL, 'event.timestamp': new Date().toISOString(), } as UiEvent; - uiTelemetryService.addEvent(uiEvent, config.getSessionId()); - if (!isInternalPromptId(normalizedEvent.prompt_id)) { - recordUiTelemetryEventToChat(config, uiEvent); - } - QwenLogger.getInstance(config)?.logToolCallEvent(normalizedEvent); + runToolTelemetrySink(() => { + uiTelemetryService.addEvent(uiEvent, config.getSessionId()); + }); + runToolTelemetrySink(() => { + if (!isInternalPromptId(normalizedEvent.prompt_id)) { + recordUiTelemetryEventToChat(config, uiEvent); + } + }); + runToolTelemetrySink(() => { + QwenLogger.getInstance(config)?.logToolCallEvent(normalizedEvent); + }); if (!isTelemetrySdkInitialized()) return; - const attributes: LogAttributes = { - ...getCommonAttributes(config), - ...normalizedEvent, - 'event.name': EVENT_TOOL_CALL, - 'event.timestamp': new Date().toISOString(), - function_args: safeJsonStringify(normalizedEvent.function_args, 2), - }; - if (normalizedEvent.error) { - attributes['error.message'] = normalizedEvent.error; - } - if (normalizedEvent.error_type) { - attributes['error.type'] = normalizedEvent.error_type; - } + runToolTelemetrySink(() => { + const attributes: LogAttributes = { + ...getCommonAttributes(config), + ...normalizedEvent, + 'event.name': EVENT_TOOL_CALL, + 'event.timestamp': new Date().toISOString(), + function_args: safeJsonStringify(normalizedEvent.function_args, 2), + }; + if (normalizedEvent.error) { + attributes['error.message'] = normalizedEvent.error; + } + if (normalizedEvent.error_type) { + attributes['error.type'] = normalizedEvent.error_type; + } - const logger = logs.getLogger(SERVICE_NAME); - const logRecord: LogRecord = { - body: `Tool call: ${normalizedEvent.function_name}${normalizedEvent.decision ? `. Decision: ${normalizedEvent.decision}` : ''}. Success: ${normalizedEvent.success}. Duration: ${normalizedEvent.duration_ms}ms.`, - attributes, - }; - logger.emit(logRecord); - recordToolCallMetrics(config, normalizedEvent.duration_ms, { - function_name: normalizedEvent.function_name, - status: normalizedEvent.status, - success: normalizedEvent.success, - decision: normalizedEvent.decision, - tool_type: normalizedEvent.tool_type, + const logger = logs.getLogger(SERVICE_NAME); + const logRecord: LogRecord = { + body: `Tool call: ${normalizedEvent.function_name}${normalizedEvent.decision ? `. Decision: ${normalizedEvent.decision}` : ''}. Success: ${normalizedEvent.success}. Duration: ${normalizedEvent.duration_ms}ms.`, + attributes, + }; + logger.emit(logRecord); + }); + runToolTelemetrySink(() => { + recordToolCallMetrics(config, normalizedEvent.duration_ms, { + function_name: normalizedEvent.function_name, + status: normalizedEvent.status, + success: normalizedEvent.success, + decision: normalizedEvent.decision, + tool_type: normalizedEvent.tool_type, + }); + }); + runToolTelemetrySink(() => { + recordToolExecutionMetrics(config, { + execution_status: normalizedEvent.execution_status, + tool_type: normalizedEvent.tool_type, + }); }); } diff --git a/packages/core/src/telemetry/metrics.test.ts b/packages/core/src/telemetry/metrics.test.ts index e075f6c6b02..df147e6ab4e 100644 --- a/packages/core/src/telemetry/metrics.test.ts +++ b/packages/core/src/telemetry/metrics.test.ts @@ -69,6 +69,7 @@ describe('Telemetry Metrics', () => { let initializeMetricsModule: typeof import('./metrics.js').initializeMetrics; let recordToolCallMetricsModule: typeof import('./metrics.js').recordToolCallMetrics; let recordTokenUsageMetricsModule: typeof import('./metrics.js').recordTokenUsageMetrics; + let recordToolExecutionMetricsModule: typeof import('./metrics.js').recordToolExecutionMetrics; let recordFileOperationMetricModule: typeof import('./metrics.js').recordFileOperationMetric; let recordChatCompressionMetricsModule: typeof import('./metrics.js').recordChatCompressionMetrics; let recordStartupPerformanceModule: typeof import('./metrics.js').recordStartupPerformance; @@ -96,6 +97,8 @@ describe('Telemetry Metrics', () => { initializeMetricsModule = metricsJsModule.initializeMetrics; recordToolCallMetricsModule = metricsJsModule.recordToolCallMetrics; recordTokenUsageMetricsModule = metricsJsModule.recordTokenUsageMetrics; + recordToolExecutionMetricsModule = + metricsJsModule.recordToolExecutionMetrics; recordFileOperationMetricModule = metricsJsModule.recordFileOperationMetric; recordChatCompressionMetricsModule = metricsJsModule.recordChatCompressionMetrics; @@ -277,6 +280,62 @@ describe('Telemetry Metrics', () => { }); }); + describe('recordToolExecutionMetrics', () => { + const mockConfig = { + getSessionId: () => 'test-session-id', + getTelemetryEnabled: () => true, + getTelemetryMetricsIncludeSessionId: () => false, + } as unknown as Config; + + it('does not record before metrics are initialized', () => { + recordToolExecutionMetricsModule(mockConfig, { + execution_status: 'unknown', + tool_type: 'native', + }); + + expect(mockCounterAddFn).not.toHaveBeenCalled(); + }); + + it('uses a dedicated low-cardinality counter', () => { + initializeMetricsModule(mockConfig); + mockCounterAddFn.mockClear(); + + recordToolExecutionMetricsModule(mockConfig, { + execution_status: 'error', + tool_type: 'mcp', + }); + + expect(mockCreateCounterFn).toHaveBeenCalledWith( + 'qwen-code.tool.execution.count', + expect.any(Object), + ); + expect(mockCounterAddFn).toHaveBeenCalledWith(1, { + execution_status: 'error', + tool_type: 'mcp', + }); + }); + + it('merges common attributes when session id is opted in', () => { + const configWithSession = { + ...mockConfig, + getTelemetryMetricsIncludeSessionId: () => true, + } as unknown as Config; + initializeMetricsModule(configWithSession); + mockCounterAddFn.mockClear(); + + recordToolExecutionMetricsModule(configWithSession, { + execution_status: 'success', + tool_type: 'native', + }); + + expect(mockCounterAddFn).toHaveBeenCalledWith(1, { + 'session.id': 'test-session-id', + execution_status: 'success', + tool_type: 'native', + }); + }); + }); + describe('recordFileOperationMetric', () => { const mockConfig = { getSessionId: () => 'test-session-id', diff --git a/packages/core/src/telemetry/metrics.ts b/packages/core/src/telemetry/metrics.ts index a3cbac5f2f4..c8b2676aed5 100644 --- a/packages/core/src/telemetry/metrics.ts +++ b/packages/core/src/telemetry/metrics.ts @@ -15,8 +15,10 @@ import type { MemoryRecallDeliveryPoint, MemoryRecallDiscardReason, } from './types.js'; +import type { ToolExecutionStatus } from '../core/turn.js'; const TOOL_CALL_COUNT = `${SERVICE_NAME}.tool.call.count`; +const TOOL_EXECUTION_COUNT = `${SERVICE_NAME}.tool.execution.count`; const TOOL_CALL_LATENCY = `${SERVICE_NAME}.tool.call.latency`; const API_REQUEST_COUNT = `${SERVICE_NAME}.api.request.count`; const API_REQUEST_LATENCY = `${SERVICE_NAME}.api.request.latency`; @@ -96,6 +98,15 @@ const COUNTER_DEFINITIONS = { tool_type?: 'native' | 'mcp'; }, }, + [TOOL_EXECUTION_COUNT]: { + description: 'Counts tool execution outcomes.', + valueType: ValueType.INT, + assign: (c: Counter) => (toolExecutionCounter = c), + attributes: {} as { + execution_status: ToolExecutionStatus | 'unknown'; + tool_type: 'native' | 'mcp'; + }, + }, [API_REQUEST_COUNT]: { description: 'Counts API requests, tagged by model and status.', valueType: ValueType.INT, @@ -371,6 +382,7 @@ export enum ApiRequestPhase { let cliMeter: Meter | undefined; let toolCallCounter: Counter | undefined; +let toolExecutionCounter: Counter | undefined; let toolCallLatencyHistogram: Histogram | undefined; let apiRequestCounter: Counter | undefined; let apiRequestLatencyHistogram: Histogram | undefined; @@ -602,6 +614,17 @@ export function recordToolCallMetrics( }); } +export function recordToolExecutionMetrics( + config: TelemetryRuntimeConfig, + attributes: MetricDefinitions[typeof TOOL_EXECUTION_COUNT]['attributes'], +): void { + if (!toolExecutionCounter || !isMetricsInitialized) return; + toolExecutionCounter.add(1, { + ...baseMetricDefinition.getCommonAttributes(config), + ...attributes, + }); +} + export function recordTokenUsageMetrics( config: Config, tokenCount: number, diff --git a/packages/core/src/telemetry/qwen-logger/qwen-logger.test.ts b/packages/core/src/telemetry/qwen-logger/qwen-logger.test.ts index 822c5655946..d494f12cf62 100644 --- a/packages/core/src/telemetry/qwen-logger/qwen-logger.test.ts +++ b/packages/core/src/telemetry/qwen-logger/qwen-logger.test.ts @@ -595,6 +595,47 @@ describe('QwenLogger', () => { }); }); + describe('logToolCallEvent outcomes', () => { + it('records terminal and execution outcomes with tool identity', () => { + const logger = QwenLogger.getInstance(mockConfig)!; + const enqueueSpy = vi.spyOn(logger, 'enqueueLogEvent'); + const event = { + function_name: 'mcp_tool', + call_id: 'call-1', + prompt_id: 'prompt-1', + response_id: 'response-1', + status: 'error', + execution_status: 'error', + success: false, + decision: undefined, + duration_ms: 25, + tool_type: 'mcp', + mcp_server_name: 'server-1', + error_type: 'mcp_tool_error', + error: 'failed', + } as ToolCallEvent; + + logger.logToolCallEvent(event); + + expect(enqueueSpy).toHaveBeenCalledWith( + expect.objectContaining({ + event_type: 'action', + type: 'tool', + name: 'tool_call#mcp_tool', + properties: expect.objectContaining({ + call_id: 'call-1', + status: 'error', + execution_status: 'error', + tool_type: 'mcp', + success: 0, + }), + }), + ); + const rumEvent = enqueueSpy.mock.calls[0][0]; + expect(rumEvent.properties).not.toHaveProperty('mcp_server_name'); + }); + }); + describe('logHookCallEvent', () => { it('should log a successful hook call event', () => { const logger = QwenLogger.getInstance(mockConfig)!; @@ -989,8 +1030,8 @@ describe('QwenLogger', () => { }); }); - describe('logToolCallEvent', () => { - it('records terminal status and tool type without MCP server metadata', () => { + describe('logToolCallEvent privacy', () => { + it('records terminal status without forwarding MCP server metadata or function arguments', () => { const logger = QwenLogger.getInstance(mockConfig)!; const enqueueSpy = vi.spyOn(logger, 'enqueueLogEvent'); const event = { @@ -1025,8 +1066,8 @@ describe('QwenLogger', () => { }), ); const rumEvent = enqueueSpy.mock.calls[0][0]; - expect(rumEvent.properties).not.toHaveProperty('mcp_server_name'); expect(rumEvent.properties).not.toHaveProperty('function_args'); + expect(rumEvent.properties).not.toHaveProperty('mcp_server_name'); }); }); }); diff --git a/packages/core/src/telemetry/qwen-logger/qwen-logger.ts b/packages/core/src/telemetry/qwen-logger/qwen-logger.ts index 7d7fcfc8cc0..502d63d43de 100644 --- a/packages/core/src/telemetry/qwen-logger/qwen-logger.ts +++ b/packages/core/src/telemetry/qwen-logger/qwen-logger.ts @@ -538,12 +538,14 @@ export class QwenLogger { `tool_call#${event.function_name}`, { properties: { + call_id: event.call_id, prompt_id: event.prompt_id, response_id: event.response_id, tool_name: event.function_name, - permission: event.decision, status: event.status, + execution_status: event.execution_status, tool_type: event.tool_type, + permission: event.decision, success: event.success ? 1 : 0, duration_ms: event.duration_ms, error_type: event.error_type, diff --git a/packages/core/src/telemetry/session-tracing.test.ts b/packages/core/src/telemetry/session-tracing.test.ts index e55474bc497..5f1559427bd 100644 --- a/packages/core/src/telemetry/session-tracing.test.ts +++ b/packages/core/src/telemetry/session-tracing.test.ts @@ -13,6 +13,7 @@ const mockState = vi.hoisted(() => ({ // try/catch hardening in end*Span helpers (span.end() must still run). throwOnSetAttributes: false, throwOnSetStatus: false, + throwOnStartSpan: false, // When set, `context.active()` returns a context that carries this fake // span and `trace.getSpan()` reports it. Lets tests exercise the // active-OTel-span fallback in resolveParentContext (#4212). @@ -123,7 +124,12 @@ vi.mock('@opentelemetry/api', async () => { name: string, opts?: { kind?: number; attributes?: Record }, parentCtx?: unknown, - ) => createMockSpan(name, opts, parentCtx), + ) => { + if (mockState.throwOnStartSpan) { + throw new Error('startSpan failed'); + } + return createMockSpan(name, opts, parentCtx); + }, }; return { @@ -200,6 +206,7 @@ describe('session-tracing', () => { mockState.sdkInitialized = true; mockState.throwOnSetAttributes = false; mockState.throwOnSetStatus = false; + mockState.throwOnStartSpan = false; mockState.activeOtelSpan = undefined; }); @@ -1082,6 +1089,15 @@ describe('session-tracing', () => { }); describe('tool spans', () => { + it('returns a NOOP span when telemetry start fails', () => { + mockState.throwOnStartSpan = true; + + const span = startToolSpan('ReadFile'); + + expect(span.spanContext().traceId).toBe('0'.repeat(32)); + expect(mockSpans).toHaveLength(0); + }); + it('creates and ends a tool span', () => { const span = startToolSpan('ReadFile', { 'tool.call_id': 'call-1' }); @@ -1128,6 +1144,17 @@ describe('session-tracing', () => { expect(mockSpans[0]!.statuses[0]!.message).toBe('command failed'); }); + it('records cancellation without marking the tool span as an error', () => { + const span = startToolSpan('Bash'); + endToolSpan(span, { success: false, cancelled: true }); + + expect(mockSpans[0]!.attributes).toMatchObject({ + success: false, + 'tool.failure_kind': 'cancelled', + }); + expect(mockSpans[0]!.statuses).toEqual([{ code: SpanStatusCode.UNSET }]); + }); + it('does not set status when no metadata is passed', () => { const span = startToolSpan('Read'); endToolSpan(span); @@ -1483,6 +1510,18 @@ describe('session-tracing', () => { }); describe('tool execution sub-spans', () => { + it('returns a NOOP span when execution telemetry start fails', () => { + mockState.throwOnStartSpan = true; + + const span = startToolExecutionSpan({ + toolName: 'Bash', + callId: 'call-1', + }); + + expect(span.spanContext().traceId).toBe('0'.repeat(32)); + expect(mockSpans).toHaveLength(0); + }); + it('creates a tool execution span as child of tool span via runInToolSpanContext', () => { const toolSpan = startToolSpan('Bash'); @@ -1501,6 +1540,21 @@ describe('session-tracing', () => { expect(mockSpans[1]!.ended).toBe(true); }); + it('records optional tool identity on execution spans', () => { + const execSpan = startToolExecutionSpan({ + toolName: 'Bash', + callId: 'call-1', + }); + + const record = mockSpans.find( + (span) => span.name === 'qwen-code.tool.execution', + ); + expect(record?.attributes['gen_ai.tool.name']).toBe('Bash'); + expect(record?.attributes['tool.call_id']).toBe('call-1'); + + endToolExecutionSpan(execSpan, { success: true }); + }); + it('returns NOOP span when SDK is not initialized', () => { mockState.sdkInitialized = false; startToolSpan('Bash'); @@ -1570,6 +1624,38 @@ describe('session-tracing', () => { expect(record?.statuses[0]!.code).toBe(SpanStatusCode.ERROR); expect(record?.statuses[0]!.message).toBe('Tool execution failed'); }); + + it('records canonical execution outcome and structured error type', () => { + const execSpan = startToolExecutionSpan(); + endToolExecutionSpan(execSpan, { + success: false, + error: 'Tool execution failed', + executionStatus: 'error', + errorType: 'execution_failed', + }); + + const record = mockSpans.find( + (span) => span.name === 'qwen-code.tool.execution', + ); + expect(record?.attributes['execution_status']).toBe('error'); + expect(record?.attributes['error_type']).toBe('execution_failed'); + expect(record?.attributes['error.type']).toBe('execution_failed'); + expect(record?.statuses[0]!.code).toBe(SpanStatusCode.ERROR); + }); + + it('uses execution_status to keep cancellation UNSET', () => { + const execSpan = startToolExecutionSpan(); + endToolExecutionSpan(execSpan, { + success: false, + executionStatus: 'cancelled', + }); + + const record = mockSpans.find( + (span) => span.name === 'qwen-code.tool.execution', + ); + expect(record?.attributes['execution_status']).toBe('cancelled'); + expect(record?.statuses).toHaveLength(0); + }); }); describe('blocked_on_user spans (#3731 Phase 2)', () => { diff --git a/packages/core/src/telemetry/session-tracing.ts b/packages/core/src/telemetry/session-tracing.ts index 652f3b755c3..b7e42e1a7c1 100644 --- a/packages/core/src/telemetry/session-tracing.ts +++ b/packages/core/src/telemetry/session-tracing.ts @@ -25,11 +25,14 @@ import { SPAN_TOOL, SPAN_TOOL_BLOCKED_ON_USER, SPAN_TOOL_EXECUTION, + TOOL_FAILURE_KIND_ATTRIBUTE, + TOOL_FAILURE_KIND_CANCELLED, } from './constants.js'; import { ApiRequestPhase, recordApiRequestBreakdown } from './metrics.js'; import { isTelemetrySdkInitialized } from './sdk.js'; import { getCurrentSessionId, setSessionContext } from './session-context.js'; import { createDebugLogger } from '../utils/debugLogger.js'; +import type { ToolExecutionStatus } from '../core/turn.js'; const debugLogger = createDebugLogger('SESSION_TRACING'); @@ -133,6 +136,7 @@ export interface LLMRequestMetadata { export interface ToolSpanMetadata { success?: boolean; error?: string; + cancelled?: boolean; } interface SpanContext { @@ -880,49 +884,63 @@ export function startToolSpan( return NOOP_SPAN; } - // Prefer subagentContext over interactionContext (see startLLMRequestSpan - // for rationale; wenshao @ #4410). - const parentCtx = subagentContext.getStore() ?? interactionContext.getStore(); - // Same fallback as startLLMRequestSpan: prefer active OTel span for - // tools-inside-tools cases before becoming a trace root. - const ctx = resolveParentContext(parentCtx); - - const sessionId = resolveSessionId(parentCtx); - const userId = resolveGenAiUserId(parentCtx, promptId); - const attributes: Attributes = { - ...(sessionId ? { 'session.id': sessionId } : {}), - ...attrs, - ...(userId ? { 'gen_ai.user.id': userId } : {}), - 'gen_ai.operation.name': 'execute_tool', - 'gen_ai.tool.name': toolName, - 'gen_ai.tool.type': 'function', - ...(description - ? { - 'gen_ai.tool.description': truncateSpanText( - description, - TOOL_DESCRIPTION_MAX_CHARS, - ), - } - : {}), - }; + let span: Span | undefined; + try { + // Prefer subagentContext over interactionContext (see startLLMRequestSpan + // for rationale; wenshao @ #4410). + const parentCtx = + subagentContext.getStore() ?? interactionContext.getStore(); + // Same fallback as startLLMRequestSpan: prefer active OTel span for + // tools-inside-tools cases before becoming a trace root. + const ctx = resolveParentContext(parentCtx); + + const sessionId = resolveSessionId(parentCtx); + const userId = resolveGenAiUserId(parentCtx, promptId); + const attributes: Attributes = { + ...(sessionId ? { 'session.id': sessionId } : {}), + ...attrs, + ...(userId ? { 'gen_ai.user.id': userId } : {}), + 'gen_ai.operation.name': 'execute_tool', + 'gen_ai.tool.name': toolName, + 'gen_ai.tool.type': 'function', + ...(description + ? { + 'gen_ai.tool.description': truncateSpanText( + description, + TOOL_DESCRIPTION_MAX_CHARS, + ), + } + : {}), + }; - const span = getTracer().startSpan( - SPAN_TOOL, - { kind: SpanKind.INTERNAL, attributes }, - ctx, - ); + span = getTracer().startSpan( + SPAN_TOOL, + { kind: SpanKind.INTERNAL, attributes }, + ctx, + ); - const spanId = getSpanId(span); - const spanContextObj: SpanContext = { - span, - startTime: Date.now(), - attributes: attributes as Record, - type: 'tool', - }; - activeSpans.set(spanId, new WeakRef(spanContextObj)); - strongSpans.set(spanId, spanContextObj); + const spanId = getSpanId(span); + const spanContextObj: SpanContext = { + span, + startTime: Date.now(), + attributes: attributes as Record, + type: 'tool', + }; + activeSpans.set(spanId, new WeakRef(spanContextObj)); + strongSpans.set(spanId, spanContextObj); - return span; + return span; + } catch (error) { + try { + span?.end(); + } catch { + // Telemetry is best-effort. + } + debugLogger.warn( + `Failed to start tool span: ${error instanceof Error ? error.message : String(error)}`, + ); + return NOOP_SPAN; + } } /** @@ -966,16 +984,25 @@ export function endToolSpan(span: Span, metadata?: ToolSpanMetadata): void { const endAttributes: Attributes = { duration_ms: duration }; if (metadata) { - if (metadata.success !== undefined) - endAttributes['success'] = metadata.success; + if (metadata.success !== undefined || metadata.cancelled) { + endAttributes['success'] = metadata.cancelled + ? false + : (metadata.success ?? false); + } if (metadata.error !== undefined) endAttributes['error'] = truncateSpanError(metadata.error); + if (metadata.cancelled) { + endAttributes[TOOL_FAILURE_KIND_ATTRIBUTE] = + TOOL_FAILURE_KIND_CANCELLED; + } } spanCtx.span.setAttributes(endAttributes); if (metadata) { - if (metadata.success !== false) { + if (metadata.cancelled) { + spanCtx.span.setStatus({ code: SpanStatusCode.UNSET }); + } else if (metadata.success !== false) { spanCtx.span.setStatus({ code: SpanStatusCode.OK }); } else { spanCtx.span.setStatus({ @@ -1005,63 +1032,90 @@ export function endToolSpan(span: Span, metadata?: ToolSpanMetadata): void { // --- Tool Execution Sub-Spans --- -export function startToolExecutionSpan(): Span { +export interface StartToolExecutionSpanOptions { + toolName?: string; + callId?: string; +} + +export interface EndToolExecutionSpanMetadata { + success?: boolean; + error?: string; + /** + * Mark the execution as user-cancelled: success/error attributes are + * still recorded but status stays UNSET, mirroring setToolSpanCancelled + * on the parent tool span. + */ + cancelled?: boolean; + executionStatus?: ToolExecutionStatus; + errorType?: string; + /** Extra span attributes recorded verbatim alongside the standard set. */ + attributes?: Attributes; +} + +export function startToolExecutionSpan( + options?: StartToolExecutionSpanOptions, +): Span { if (!isTelemetrySdkInitialized()) { return NOOP_SPAN; } - const parentCtx = toolContext.getStore(); - if (!parentCtx) { - debugLogger.warn( - 'startToolExecutionSpan called outside runInToolSpanContext — span will not be parented to tool span', - ); - } - // Without an explicit toolContext parent we still try the active OTel span - // (some tool execution paths run inside a withSpan() block from another - // subsystem) before becoming a trace root. - const ctx = resolveParentContext(parentCtx); + let span: Span | undefined; + try { + const parentCtx = toolContext.getStore(); + if (!parentCtx) { + debugLogger.warn( + 'startToolExecutionSpan called outside runInToolSpanContext — span will not be parented to tool span', + ); + } + // Without an explicit toolContext parent we still try the active OTel span + // (some tool execution paths run inside a withSpan() block from another + // subsystem) before becoming a trace root. + const ctx = resolveParentContext(parentCtx); - const sessionId = resolveSessionId( - parentCtx ?? interactionContext.getStore(), - ); - const span = getTracer().startSpan( - SPAN_TOOL_EXECUTION, - { - kind: SpanKind.INTERNAL, - attributes: sessionId ? { 'session.id': sessionId } : {}, - }, - ctx, - ); + const sessionId = resolveSessionId( + parentCtx ?? interactionContext.getStore(), + ); + const attributes: Attributes = { + ...(sessionId ? { 'session.id': sessionId } : {}), + ...(options?.toolName ? { 'gen_ai.tool.name': options.toolName } : {}), + ...(options?.callId ? { 'tool.call_id': options.callId } : {}), + }; + span = getTracer().startSpan( + SPAN_TOOL_EXECUTION, + { + kind: SpanKind.INTERNAL, + attributes, + }, + ctx, + ); - const spanId = getSpanId(span); - const spanContextObj: SpanContext = { - span, - startTime: Date.now(), - attributes: sessionId ? { 'session.id': sessionId } : {}, - type: 'tool.execution', - }; - activeSpans.set(spanId, new WeakRef(spanContextObj)); - strongSpans.set(spanId, spanContextObj); + const spanId = getSpanId(span); + const spanContextObj: SpanContext = { + span, + startTime: Date.now(), + attributes: attributes as Record, + type: 'tool.execution', + }; + activeSpans.set(spanId, new WeakRef(spanContextObj)); + strongSpans.set(spanId, spanContextObj); - return span; + return span; + } catch (error) { + try { + span?.end(); + } catch { + // Telemetry is best-effort. + } + debugLogger.warn( + `Failed to start tool execution span: ${error instanceof Error ? error.message : String(error)}`, + ); + return NOOP_SPAN; + } } export function endToolExecutionSpan( span: Span, - metadata?: { - success?: boolean; - error?: string; - /** - * Mark the execution as user-cancelled: success/error attributes are - * still recorded but status stays UNSET, mirroring setToolSpanCancelled - * on the parent tool span. Without this, success: false unconditionally - * sets ERROR and trace backends filtering for errors false-positive on - * user cancels. - */ - cancelled?: boolean; - /** Extra span attributes recorded verbatim alongside the standard set. */ - attributes?: Attributes; - }, + metadata?: EndToolExecutionSpanMetadata, ): void { const spanId = getSpanId(span); const spanCtx = activeSpans.get(spanId)?.deref(); @@ -1091,6 +1145,13 @@ export function endToolExecutionSpan( endAttributes['success'] = metadata.success; if (metadata.error !== undefined) endAttributes['error'] = truncateSpanError(metadata.error); + if (metadata.executionStatus !== undefined) { + endAttributes['execution_status'] = metadata.executionStatus; + } + if (metadata.errorType !== undefined) { + endAttributes['error_type'] = metadata.errorType; + endAttributes['error.type'] = metadata.errorType; + } } spanCtx.span.setAttributes(endAttributes); @@ -1099,8 +1160,17 @@ export function endToolExecutionSpan( // status (e.g. via setToolSpanCancelled) and then call this without // metadata get their pre-set status preserved. Cancellation also // preserves UNSET so the child agrees with the cancelled parent. - if (metadata && !metadata.cancelled) { - if (metadata.success !== false) { + const executionStatus = metadata?.executionStatus; + const cancelled = + metadata?.cancelled === true || executionStatus === 'cancelled'; + // The not_started guard is unreachable by construction (the span only + // exists once execution is attempted); kept as defence-in-depth. + if (metadata && !cancelled && executionStatus !== 'not_started') { + const succeeded = + executionStatus === undefined + ? metadata.success !== false + : executionStatus === 'success'; + if (succeeded) { spanCtx.span.setStatus({ code: SpanStatusCode.OK }); } else { spanCtx.span.setStatus({ diff --git a/packages/core/src/telemetry/types.ts b/packages/core/src/telemetry/types.ts index 5d921b0cedc..9222c8b0e4a 100644 --- a/packages/core/src/telemetry/types.ts +++ b/packages/core/src/telemetry/types.ts @@ -11,6 +11,7 @@ import type { CompletedToolCall } from '../core/coreToolScheduler.js'; import { DiscoveredMCPTool } from '../tools/mcp-tool.js'; import type { FileDiff } from '../tools/tools.js'; import type { AuthType } from '../core/contentGenerator.js'; +import type { ToolExecutionStatus } from '../core/turn.js'; import { getDecisionFromOutcome, ToolCallDecision, @@ -175,10 +176,12 @@ export class UserRetryEvent implements BaseTelemetryEvent { export class ToolCallEvent implements BaseTelemetryEvent { 'event.name': 'tool_call'; 'event.timestamp': string; + call_id?: string; function_name: string; function_args: Record; duration_ms: number; status: 'success' | 'error' | 'cancelled'; + execution_status?: ToolExecutionStatus | 'unknown'; success: boolean; // Keep for backward compatibility decision?: ToolCallDecision; error?: string; @@ -194,6 +197,7 @@ export class ToolCallEvent implements BaseTelemetryEvent { constructor(call: CompletedToolCall) { this['event.name'] = 'tool_call'; this['event.timestamp'] = new Date().toISOString(); + this.call_id = call.request.callId; this.function_name = call.request.name; // structured_output args ARE the user's final structured payload (the // command's actual answer, already emitted in stdout `result` / @@ -212,6 +216,7 @@ export class ToolCallEvent implements BaseTelemetryEvent { : call.request.args; this.duration_ms = call.durationMs ?? 0; this.status = call.status; + this.execution_status = call.response.executionStatus; this.success = call.status === 'success'; // Keep for backward compatibility this.decision = call.outcome ? getDecisionFromOutcome(call.outcome) diff --git a/packages/core/src/tools/mcp-tool.test.ts b/packages/core/src/tools/mcp-tool.test.ts index f613b9089f3..d55353585b3 100644 --- a/packages/core/src/tools/mcp-tool.test.ts +++ b/packages/core/src/tools/mcp-tool.test.ts @@ -916,6 +916,41 @@ describe('DiscoveredMCPTool', () => { controller.abort(); expect(controller.signal.aborted).toBe(true); }); + + it('forwards parent abort into the combined signal passed to the direct SDK client', async () => { + let capturedSignal: AbortSignal | undefined; + const mockDirectCallTool = vi.fn( + async (_params, _schema, options) => { + capturedSignal = options?.signal; + return new Promise(() => {}); + }, + ); + const directClient: McpDirectClient = { + callTool: mockDirectCallTool, + }; + + const directTool = new DiscoveredMCPTool( + mockCallableToolInstance, + serverName, + serverToolName, + baseDescription, + inputSchema, + undefined, + undefined, + undefined, + directClient, + ); + const controller = new AbortController(); + const invocation = directTool.build({ param: 'test' }); + const promise = invocation.execute(controller.signal); + + await vi.waitFor(() => expect(mockDirectCallTool).toHaveBeenCalled()); + + controller.abort(); + + expect(capturedSignal?.aborted).toBe(true); + await expect(promise).rejects.toThrow('Tool call aborted'); + }); }); }); @@ -1865,9 +1900,258 @@ describe('DiscoveredMCPTool', () => { expect(discoverToolsForServer).toHaveBeenCalled(); }); + + it('reconnects instead of reporting a timeout when the server is known disconnected', async () => { + const params = { param: 'test' }; + // -32001 with a dead transport means the connection died mid-request, + // not that the tool ran too long. Classifying it as EXECUTION_TIMEOUT + // would strand a call the reconnect path can still recover. + const requestTimeout = Object.assign(new Error('Request timed out'), { + code: -32001, + }); + const mockMcpClient: McpDirectClient = { + callTool: vi.fn().mockRejectedValueOnce(requestTimeout), + }; + const newMockMcpClient: McpDirectClient = { + callTool: vi + .fn() + .mockResolvedValueOnce({ content: [{ type: 'text', text: 'OK' }] }), + }; + const newTool = new DiscoveredMCPTool( + mockCallableToolInstance, + serverName, + serverToolName, + baseDescription, + inputSchema, + undefined, + undefined, + undefined, + newMockMcpClient, + ); + const discoverToolsForServer = vi.fn().mockResolvedValue(undefined); + const mockConfig = { + isTrustedFolder: () => true, + getToolRegistry: () => ({ + discoverToolsForServer, + ensureTool: vi.fn().mockResolvedValue(newTool), + }), + getTruncateToolOutputThreshold: () => 0, + getTruncateToolOutputLines: () => 0, + }; + + updateMCPServerStatus(serverName, MCPServerStatus.DISCONNECTED); + + const reconnectTool = new DiscoveredMCPTool( + mockCallableToolInstance, + serverName, + serverToolName, + baseDescription, + inputSchema, + undefined, + undefined, + mockConfig as any, + mockMcpClient, + ); + + await reconnectTool.build(params).execute(new AbortController().signal); + + expect(discoverToolsForServer).toHaveBeenCalled(); + }); + + it('still reports a timeout when the server is connected', async () => { + const requestTimeout = Object.assign(new Error('Request timed out'), { + code: -32001, + }); + const discoverToolsForServer = vi.fn().mockResolvedValue(undefined); + const mockMcpClient: McpDirectClient = { + callTool: vi.fn().mockRejectedValue(requestTimeout), + }; + const mockConfig = { + getToolRegistry: () => ({ + discoverToolsForServer, + ensureTool: vi.fn(), + }), + }; + + updateMCPServerStatus(serverName, MCPServerStatus.CONNECTED); + + const tool = new DiscoveredMCPTool( + mockCallableToolInstance, + serverName, + serverToolName, + baseDescription, + inputSchema, + undefined, + undefined, + mockConfig as any, + mockMcpClient, + ); + + await expect( + tool.build({ param: 'test' }).execute(new AbortController().signal), + ).rejects.toMatchObject({ + errorType: ToolErrorType.EXECUTION_TIMEOUT, + }); + expect(discoverToolsForServer).not.toHaveBeenCalled(); + }); }); describe('MCP Tool Idle Timeout', () => { + it('classifies an MCP SDK request timeout without parsing its message', async () => { + const requestTimeout = Object.assign( + new Error('localized timeout message'), + { code: -32001 }, + ); + const mockMcpClient: McpDirectClient = { + callTool: vi.fn().mockRejectedValue(requestTimeout), + }; + const tool = new DiscoveredMCPTool( + mockCallableToolInstance, + serverName, + serverToolName, + baseDescription, + inputSchema, + true, + undefined, + undefined, + mockMcpClient, + ); + + const executePromise = tool + .build({ param: 'test' }) + .execute(new AbortController().signal); + + await expect(executePromise).rejects.toMatchObject({ + message: 'localized timeout message', + errorType: ToolErrorType.EXECUTION_TIMEOUT, + }); + }); + + it('does not classify a parent abort wrapped by the MCP SDK as a timeout', async () => { + const requestCancelled = Object.assign(new Error('Request cancelled'), { + code: -32001, + }); + const discoverToolsForServer = vi.fn(); + const mockMcpClient: McpDirectClient = { + callTool: vi.fn().mockImplementation( + (_params, _schema, options) => + new Promise((_resolve, reject) => { + options?.signal?.addEventListener( + 'abort', + () => reject(requestCancelled), + { once: true }, + ); + }), + ), + }; + const mockConfig = { + getToolRegistry: () => ({ + discoverToolsForServer, + ensureTool: vi.fn(), + }), + }; + const tool = new DiscoveredMCPTool( + mockCallableToolInstance, + serverName, + serverToolName, + baseDescription, + inputSchema, + true, + undefined, + mockConfig as any, + mockMcpClient, + ); + const abortController = new AbortController(); + const executePromise = tool + .build({ param: 'test' }) + .execute(abortController.signal); + + updateMCPServerStatus(serverName, MCPServerStatus.DISCONNECTED); + abortController.abort(); + + const rejection = await executePromise.catch((error) => error); + expect(rejection).toMatchObject({ name: 'AbortError' }); + expect(discoverToolsForServer).not.toHaveBeenCalled(); + expect(rejection).not.toMatchObject({ + errorType: ToolErrorType.EXECUTION_TIMEOUT, + }); + }); + + it('does not classify a direct -32001 that races with a parent abort as a timeout', async () => { + // Once the caller has cancelled, a `-32001` is indistinguishable from + // the SDK's own abort rejection, so a timeout that settles the race + // first must not reclassify the cancellation — the abort side wins + // regardless of ordering (#8180 review). + const requestTimeout = Object.assign(new Error('raced timeout'), { + code: -32001, + }); + let rejectRequest: ((reason?: unknown) => void) | undefined; + const mockMcpClient: McpDirectClient = { + callTool: vi.fn().mockReturnValue( + new Promise((_resolve, reject) => { + rejectRequest = reject; + }), + ), + }; + const tool = new DiscoveredMCPTool( + mockCallableToolInstance, + serverName, + serverToolName, + baseDescription, + inputSchema, + true, + undefined, + undefined, + mockMcpClient, + ); + const abortController = new AbortController(); + const executePromise = tool + .build({ param: 'test' }) + .execute(abortController.signal); + + rejectRequest?.(requestTimeout); + abortController.abort(); + + await expect(executePromise).rejects.toBe(requestTimeout); + }); + + it('classifies an MCP SDK request timeout on the callable fallback', async () => { + mockCallTool.mockRejectedValueOnce( + Object.assign(new Error('fallback timeout'), { code: -32001 }), + ); + + const executePromise = tool + .build({ param: 'test' }) + .execute(new AbortController().signal); + + await expect(executePromise).rejects.toMatchObject({ + message: 'fallback timeout', + errorType: ToolErrorType.EXECUTION_TIMEOUT, + }); + }); + + it('does not classify a callable -32001 that races with a parent abort as a timeout', async () => { + const requestTimeout = Object.assign( + new Error('raced fallback timeout'), + { code: -32001 }, + ); + let rejectRequest: ((reason?: unknown) => void) | undefined; + mockCallTool.mockReturnValueOnce( + new Promise((_resolve, reject) => { + rejectRequest = reject; + }), + ); + const abortController = new AbortController(); + const executePromise = tool + .build({ param: 'test' }) + .execute(abortController.signal); + + rejectRequest?.(requestTimeout); + abortController.abort(); + + await expect(executePromise).rejects.toBe(requestTimeout); + }); + it('should abort when MCP server does not respond within idle timeout', async () => { vi.useFakeTimers(); @@ -1913,12 +2197,58 @@ describe('DiscoveredMCPTool', () => { await expect(executePromise).rejects.toThrow( /did not respond within.*idle timeout/, ); + await expect(executePromise).rejects.toMatchObject({ + errorType: ToolErrorType.EXECUTION_TIMEOUT, + }); // The external abort signal should not have been triggered expect(abortController.signal.aborted).toBe(false); vi.useRealTimers(); }); + it('keeps an idle timeout when the parent aborts before rejection settles', async () => { + vi.useFakeTimers(); + try { + const idleTimeoutMs = 1000; + const mockMcpClient: McpDirectClient = { + callTool: vi.fn().mockImplementation( + (_params, _schema, options) => + new Promise((_resolve, reject) => { + options?.signal?.addEventListener('abort', () => { + queueMicrotask(() => reject(options.signal?.reason)); + }); + }), + ), + }; + const tool = new DiscoveredMCPTool( + mockCallableToolInstance, + serverName, + serverToolName, + baseDescription, + inputSchema, + true, + undefined, + undefined, + mockMcpClient, + undefined, + idleTimeoutMs, + ); + const abortController = new AbortController(); + const executePromise = tool + .build({ param: 'test' }) + .execute(abortController.signal); + + vi.advanceTimersByTime(idleTimeoutMs); + abortController.abort(); + + await expect(executePromise).rejects.toMatchObject({ + errorType: ToolErrorType.EXECUTION_TIMEOUT, + }); + } finally { + vi.useRealTimers(); + } + }); + it('should reset idle timeout on progress updates', async () => { vi.useFakeTimers(); diff --git a/packages/core/src/tools/mcp-tool.ts b/packages/core/src/tools/mcp-tool.ts index 19baf76a631..2f8816b1004 100644 --- a/packages/core/src/tools/mcp-tool.ts +++ b/packages/core/src/tools/mcp-tool.ts @@ -23,12 +23,16 @@ import type { Part, PartListUnion, } from '@google/genai'; -import { ToolErrorType } from './tool-error.js'; +import { StructuredToolError, ToolErrorType } from './tool-error.js'; import type { Config } from '../config/config.js'; import { truncateToolOutput } from '../utils/truncation.js'; import { createDebugLogger } from '../utils/debugLogger.js'; import { getErrorMessage, isAbortError } from '../utils/errors.js'; -import { getMCPServerStatus, MCPServerStatus } from './mcp-status.js'; +import { + getAllMCPServerStatuses, + getMCPServerStatus, + MCPServerStatus, +} from './mcp-status.js'; import { getInvocationContext, INVOCATION_CONTEXT_META_KEY, @@ -51,6 +55,106 @@ const MCP_CONNECTION_ERROR_PATTERNS = [ /disconnected/i, /transport closed/i, ]; +// The MCP SDK's generic `RequestTimeout` code. It is emitted for both +// client-configured timeouts (`timeout` / `resetTimeoutOnProgress`) and +// server-side timeouts, so both collapse into a single EXECUTION_TIMEOUT +// classification here. +const MCP_REQUEST_TIMEOUT_CODE = -32001; + +function isMcpRequestTimeout(error: unknown): boolean { + return ( + typeof error === 'object' && + error !== null && + 'code' in error && + (error as { code?: unknown }).code === MCP_REQUEST_TIMEOUT_CODE + ); +} + +/** + * A `-32001` rejection only tells us the request never got a response. That is + * a genuine execution timeout while the transport is believed healthy, but + * when the server is known to be DISCONNECTED the same code just means the + * connection died mid-request — which `handleReconnectOnError` can still + * recover from by reconnecting and retrying. Classifying that as + * EXECUTION_TIMEOUT would turn a recoverable transport failure into a hard + * error the user has to retry by hand. + * + * Deliberately checks for a *recorded* DISCONNECTED rather than + * `getMCPServerStatus(...) !== CONNECTED`: that getter reports DISCONNECTED + * for servers it has never seen, so the simpler comparison would misroute + * every timeout from a server whose status was never registered. Default to + * "timeout" and only divert on positive evidence the transport is dead. + */ +function isExecutionTimeoutFailure( + error: unknown, + serverName: string, + signal: AbortSignal, +): boolean { + // A `-32001` that lands while the parent signal is aborted is the SDK's + // abort rejection (forwarded by createParentAbortRace) or a timeout that + // raced with a cancel. Classifying it as EXECUTION_TIMEOUT would count a + // user cancellation against the timeout SLI, so the abort side wins. + if (signal.aborted) return false; + if (!isMcpRequestTimeout(error)) return false; + const statuses = getAllMCPServerStatuses(); + return !( + statuses.has(serverName) && + statuses.get(serverName) === MCPServerStatus.DISCONNECTED + ); +} + +const PARENT_ABORT_OUTCOME = Symbol('parent_abort_outcome'); + +type ParentAbortOutcome = { + [PARENT_ABORT_OUTCOME]: true; + reason: unknown; +}; + +function createToolCallAbortError(): Error { + return Object.assign(new Error('Tool call aborted'), { name: 'AbortError' }); +} + +function isParentAbortOutcome(value: unknown): value is ParentAbortOutcome { + return ( + typeof value === 'object' && value !== null && PARENT_ABORT_OUTCOME in value + ); +} + +function createParentAbortRace( + signal: AbortSignal, + forwardAbort?: (reason: unknown) => void, +): { + promise: Promise; + dispose: () => void; +} { + let onAbort: (() => void) | undefined; + const promise = new Promise((resolve) => { + onAbort = () => { + const reason = createToolCallAbortError(); + // Freeze the parent outcome before forwarding to the SDK, whose abort + // rejection uses the same -32001 code as a genuine request timeout. + // Ordering-safe: resolve() queues its Promise.race reaction as a + // microtask before forwardAbort triggers the SDK rejection, so the + // parent outcome always wins the race. + resolve({ [PARENT_ABORT_OUTCOME]: true, reason }); + forwardAbort?.(reason); + }; + if (signal.aborted) { + onAbort(); + } else { + signal.addEventListener('abort', onAbort, { once: true }); + } + }); + + return { + promise, + dispose: () => { + if (onAbort) { + signal.removeEventListener('abort', onAbort); + } + }, + }; +} type ToolParams = Record; @@ -259,6 +363,10 @@ class DiscoveredMCPToolInvocation extends BaseToolInvocation< ): Promise { debugLogger.error(`MCP server error '${this.serverName}': ${error}`); + if (signal.aborted) { + throw error; + } + if (!this.shouldAttemptReconnect(error)) { throw error; } @@ -336,13 +444,22 @@ class DiscoveredMCPToolInvocation extends BaseToolInvocation< signal: AbortSignal, updateOutput?: (output: ToolResultDisplay) => void, ): Promise { + if (signal.aborted) { + throw createToolCallAbortError(); + } + // Create an AbortController for idle timeout const idleTimeoutController = new AbortController(); + const parentAbortController = new AbortController(); + const parentAbortRace = createParentAbortRace(signal, (reason) => { + parentAbortController.abort(reason); + }); let idleTimeoutId: ReturnType | undefined; + let idleTimeoutWon = false; // Combine the external signal with our idle timeout controller const combinedSignal = AbortSignal.any([ - signal, + parentAbortController.signal, idleTimeoutController.signal, ]); @@ -352,6 +469,10 @@ class DiscoveredMCPToolInvocation extends BaseToolInvocation< } if (this.mcpToolIdleTimeoutMs && this.mcpToolIdleTimeoutMs > 0) { const timer = setTimeout(() => { + if (signal.aborted) { + return; + } + idleTimeoutWon = true; const error = new Error( `MCP tool '${this.serverToolName}' on server '${this.serverName}' ` + `did not respond within ${this.mcpToolIdleTimeoutMs}ms idle timeout`, @@ -371,7 +492,7 @@ class DiscoveredMCPToolInvocation extends BaseToolInvocation< const invocationContext = this.allowInvocationContext ? getInvocationContext() : undefined; - const callToolResult = await this.mcpClient!.callTool( + const callPromise = this.mcpClient!.callTool( { name: this.serverToolName, arguments: this.params as Record, @@ -403,6 +524,14 @@ class DiscoveredMCPToolInvocation extends BaseToolInvocation< signal: combinedSignal, }, ); + const outcome = await Promise.race([ + callPromise, + parentAbortRace.promise, + ]); + if (isParentAbortOutcome(outcome)) { + throw outcome.reason; + } + const callToolResult = outcome; // Wrap the raw CallToolResult into the Part[] format that the // existing transform/display functions expect. @@ -430,12 +559,24 @@ class DiscoveredMCPToolInvocation extends BaseToolInvocation< persistedOutputFiles: truncated.persistedOutputFiles, }; } catch (error) { + // `idleTimeoutWon` is our own client-side timer firing, so it is an + // execution timeout regardless of what the transport thinks. + if ( + idleTimeoutWon || + isExecutionTimeoutFailure(error, this.serverName, signal) + ) { + throw new StructuredToolError( + getErrorMessage(error), + ToolErrorType.EXECUTION_TIMEOUT, + ); + } return this.handleReconnectOnError(error, signal, updateOutput); } finally { // Clear the idle timeout in all cases if (idleTimeoutId) { clearTimeout(idleTimeoutId); } + parentAbortRace.dispose(); } } @@ -446,44 +587,29 @@ class DiscoveredMCPToolInvocation extends BaseToolInvocation< private async executeWithCallableTool( signal: AbortSignal, ): Promise { + if (signal.aborted) { + throw createToolCallAbortError(); + } + const functionCalls: FunctionCall[] = [ { name: this.serverToolName, args: this.params, }, ]; + const parentAbortRace = createParentAbortRace(signal); // Race MCP tool call with abort signal to respect cancellation try { - const rawResponseParts = await new Promise((resolve, reject) => { - if (signal.aborted) { - const error = new Error('Tool call aborted'); - error.name = 'AbortError'; - reject(error); - return; - } - const onAbort = () => { - cleanup(); - const error = new Error('Tool call aborted'); - error.name = 'AbortError'; - reject(error); - }; - const cleanup = () => { - signal.removeEventListener('abort', onAbort); - }; - signal.addEventListener('abort', onAbort, { once: true }); - - this.mcpTool - .callTool(functionCalls) - .then((res) => { - cleanup(); - resolve(res); - }) - .catch((err) => { - cleanup(); - reject(err); - }); - }); + const callPromise = this.mcpTool.callTool(functionCalls); + const outcome = await Promise.race([ + callPromise, + parentAbortRace.promise, + ]); + if (isParentAbortOutcome(outcome)) { + throw outcome.reason; + } + const rawResponseParts = outcome; if (this.isMCPToolError(rawResponseParts)) { return await this.buildMcpToolError(rawResponseParts, functionCalls[0]); @@ -501,7 +627,15 @@ class DiscoveredMCPToolInvocation extends BaseToolInvocation< persistedOutputFiles: truncated.persistedOutputFiles, }; } catch (error) { + if (isExecutionTimeoutFailure(error, this.serverName, signal)) { + throw new StructuredToolError( + getErrorMessage(error), + ToolErrorType.EXECUTION_TIMEOUT, + ); + } return this.handleReconnectOnError(error, signal); + } finally { + parentAbortRace.dispose(); } } diff --git a/packages/core/src/tools/priorReadEnforcement.ts b/packages/core/src/tools/priorReadEnforcement.ts index 0d2d5eba1c8..baa7cf77c5c 100644 --- a/packages/core/src/tools/priorReadEnforcement.ts +++ b/packages/core/src/tools/priorReadEnforcement.ts @@ -9,34 +9,7 @@ import type { FileReadCache } from '../services/fileReadCache.js'; import { ToolErrorType } from './tool-error.js'; import { ToolNames } from './tool-names.js'; -/** - * Error thrown by `getConfirmationDetails()` when it needs to surface - * a structured `ToolErrorType` to the scheduler instead of letting - * the throw collapse into a generic `UNHANDLED_EXCEPTION`. Originally - * introduced for prior-read enforcement (hence the file location) - * but now also carries other content-derived `calculateEdit` errors - * — `EDIT_NO_OCCURRENCE_FOUND`, `EDIT_EXPECTED_OCCURRENCE_MISMATCH`, - * `EDIT_NO_CHANGE`, `ATTEMPT_TO_CREATE_EXISTING_FILE` — through the - * confirmation path so they keep their proper error code instead of - * being reported as "unhandled exception". - * - * Caught by `coreToolScheduler` via the `errorType` instance field. - * - * Naming note: kept generic (`StructuredToolError`) rather than - * `PriorReadEnforcementError` so the name matches the broader set of - * `ToolErrorType` values it actually carries — an oncall engineer - * seeing this in a log paired with `edit_no_occurrence_found` should - * not have to wonder what prior-read has to do with it. - */ -export class StructuredToolError extends Error { - override readonly name = 'StructuredToolError'; - constructor( - message: string, - readonly errorType: ToolErrorType, - ) { - super(message); - } -} +export { StructuredToolError } from './tool-error.js'; /** * Result of checking whether a tool that mutates an existing file is diff --git a/packages/core/src/tools/tool-error.ts b/packages/core/src/tools/tool-error.ts index 1c3a5b64069..9ae670f0d30 100644 --- a/packages/core/src/tools/tool-error.ts +++ b/packages/core/src/tools/tool-error.ts @@ -140,3 +140,32 @@ export enum ToolErrorType { SEND_MESSAGE_NOT_FOUND = 'send_message_not_found', SEND_MESSAGE_NOT_RUNNING = 'send_message_not_running', } + +/** + * Error thrown by `getConfirmationDetails()` when it needs to surface + * a structured `ToolErrorType` to the scheduler instead of letting + * the throw collapse into a generic `UNHANDLED_EXCEPTION`. Originally + * introduced for prior-read enforcement + * but now also carries other content-derived `calculateEdit` errors + * — `EDIT_NO_OCCURRENCE_FOUND`, `EDIT_EXPECTED_OCCURRENCE_MISMATCH`, + * `EDIT_NO_CHANGE`, `ATTEMPT_TO_CREATE_EXISTING_FILE` — through the + * confirmation path so they keep their proper error code instead of + * being reported as "unhandled exception". + * + * Caught by `coreToolScheduler` via the `errorType` instance field. + * + * Naming note: kept generic (`StructuredToolError`) rather than + * `PriorReadEnforcementError` so the name matches the broader set of + * `ToolErrorType` values it actually carries — an oncall engineer + * seeing this in a log paired with `edit_no_occurrence_found` should + * not have to wonder what prior-read has to do with it. + */ +export class StructuredToolError extends Error { + override readonly name = 'StructuredToolError'; + constructor( + message: string, + readonly errorType: ToolErrorType, + ) { + super(message); + } +} diff --git a/packages/vscode-ide-companion/scripts/generate-notices.js b/packages/vscode-ide-companion/scripts/generate-notices.js index 339d0144371..a03f3a8afc3 100644 --- a/packages/vscode-ide-companion/scripts/generate-notices.js +++ b/packages/vscode-ide-companion/scripts/generate-notices.js @@ -191,7 +191,13 @@ function collectDependencies( const realInfo = packageLock.packages[packageInfo.resolved]; if (realInfo?.dependencies) { for (const depName of Object.keys(realInfo.dependencies)) { - collectDependencies(depName, packageLock, dependenciesMap, resolveFrom, visitedKeys); + collectDependencies( + depName, + packageLock, + dependenciesMap, + resolveFrom, + visitedKeys, + ); } } return; @@ -211,7 +217,13 @@ function collectDependencies( if (packageInfo.dependencies) { for (const depName of Object.keys(packageInfo.dependencies)) { // Resolve transitive deps from THIS package's location - collectDependencies(depName, packageLock, dependenciesMap, resolvedKey, visitedKeys); + collectDependencies( + depName, + packageLock, + dependenciesMap, + resolvedKey, + visitedKeys, + ); } } }