diff --git a/docs/design/tool-call-terminal-telemetry-contract.md b/docs/design/tool-call-terminal-telemetry-contract.md new file mode 100644 index 00000000000..0d9ef0f242d --- /dev/null +++ b/docs/design/tool-call-terminal-telemetry-contract.md @@ -0,0 +1,85 @@ +# Tool Call Terminal Telemetry Contract + +## Problem + +Tool-call terminal events are produced by both the Core scheduler and ACP. +They already expose `status`, `success`, `error`, and `error_type`, but those +fields can disagree or be absent. In particular, a tool can return a soft +error without an error type, and ACP can call the telemetry logger without +constructing a `ToolCallEvent`. + +This leaves logs, usage statistics, metrics, hooks, and chat recording with +different views of the same terminal result. + +## PR1 scope + +PR1 establishes a runtime contract at two boundaries: + +1. The Core scheduler converts an unclassified `ToolResult.error` to + `ToolErrorType.UNKNOWN` before building a completed call. +2. `logToolCall` normalizes every event before sending it to any telemetry + consumer. + +The terminal contract is: + +| `status` | `success` | `error` | `error_type` | +| ----------- | --------- | --------- | --------------------------- | +| `success` | `true` | absent | absent | +| `error` | `false` | preserved | explicit value or `unknown` | +| `cancelled` | `false` | absent | absent | + +`status` is authoritative. A blank `function_name` becomes `unknown_tool`. +Non-empty tool names and non-empty error types are preserved verbatim. The +normalizer returns a copy and is idempotent. + +The Core boundary is intentionally private. Public tool implementations may +continue to omit `ToolResult.error.type`, and `ToolCallResponseInfo.errorType` +remains optional because successful and cancelled calls do not have an error +classification. + +## Consumers + +The normalized event is used by UI telemetry, the chat-recorded UI event, +QwenLogger, OpenTelemetry logs, and tool-call metrics. OpenTelemetry +`error.message` and `error.type` aliases are populated independently. + +The tool-call counter adds the low-cardinality `status` attribute while +retaining `success`. The public `recordToolCallMetrics` input accepts an +optional status for source compatibility; callers that omit it are mapped from +the legacy success boolean. The latency histogram remains keyed only by +`function_name`, and `error_type` is not added to metrics. + +QwenLogger receives `status` and `tool_type`. It does not receive +`mcp_server_name`, function arguments, results, or stack traces as part of this +change. + +## Compatibility and follow-ups + +This change is additive for logs and metrics, but it changes an unclassified +Core error from a missing value to `unknown` in PostToolBatch and Core chat +recording. Historical queries should coalesce missing error types to `unknown`; +no data backfill is required. + +The following remain outside PR1: + +- correcting ACP permission cancellation and other producer-side terminal + status bugs; +- normalizing ACP's separate raw `tool_result` recording; +- adding `error_type` to the PostToolUseFailure hook contract; +- adding error classification to primary tool spans; +- classifying individual built-in and MCP error sites; +- changing legacy UI `totalFail` semantics. + +The new `status` metric must not become the stability SLO source until the ACP +terminal-status fixes land. + +## Rollout checks + +For the new service version, operators should verify that: + +- error tool-call logs never have a blank `error_type`; +- tool-call logs never have a blank `function_name`; +- success and cancelled events do not carry error fields; +- explicitly classified errors retain their previous type; +- the tool-call counter total remains aligned with tool-call log volume; and +- the increase in `unknown` corresponds to the previous missing bucket. diff --git a/docs/developers/development/telemetry.md b/docs/developers/development/telemetry.md index f1441e112a2..60a6212a8ca 100644 --- a/docs/developers/development/telemetry.md +++ b/docs/developers/development/telemetry.md @@ -576,8 +576,8 @@ The following events are logged: #### Tool Events -- `qwen-code.tool_call`: Each function/tool call. - - **Attributes**: `function_name` (string), `function_args` (object), `duration_ms` (int), `status` (string: "success", "error", or "cancelled"), `success` (boolean), `decision` (string: "accept", "reject", "auto_accept", or "modify", optional), `error` (string, optional), `error_type` (string, optional), `prompt_id` (string), `response_id` (string, optional), `content_length` (int, optional), `tool_type` (string: "native" or "mcp"), `mcp_server_name` (string, optional), `metadata` (object, optional — for file-writing tools contains `model_added_lines`, `model_removed_lines`, `user_added_lines`, `user_removed_lines`, `model_added_chars`, `model_removed_chars`, `user_added_chars`, `user_removed_chars`) +- `qwen-code.tool_call`: Each function/tool call. Terminal events are normalized so `status` is authoritative: success and cancelled events omit error fields, while error events always have a non-empty `error_type` (`unknown` when the producer did not classify the error). Blank tool names are emitted as `unknown_tool`. + - **Attributes**: `function_name` (string), `function_args` (object), `duration_ms` (int), `status` (string: "success", "error", or "cancelled"), `success` (boolean), `decision` (string: "accept", "reject", "auto_accept", or "modify", optional), `error` (string, optional), `error_type` (string, present for error events), `prompt_id` (string), `response_id` (string, optional), `content_length` (int, optional), `tool_type` (string: "native" or "mcp"), `mcp_server_name` (string, optional), `metadata` (object, optional — for file-writing tools contains `model_added_lines`, `model_removed_lines`, `user_added_lines`, `user_removed_lines`, `model_added_chars`, `model_removed_chars`, `user_added_chars`, `user_removed_chars`) - `qwen-code.file_operation`: Each file operation. - **Attributes**: `tool_name` (string), `operation` (string: "create", "read", "update"), `lines` (int, optional), `mimetype` (string, optional), `extension` (string, optional), `programming_language` (string, optional) @@ -721,7 +721,7 @@ Metrics are numerical measurements of behavior over time. Metric names use the ` - `qwen-code.session.count` (Counter, Int): Incremented once per CLI startup. - `qwen-code.tool.call.count` (Counter, Int): Counts tool calls. - - **Attributes**: `function_name`, `success` (boolean), `decision` ("accept"/"reject"/"auto_accept"/"modify", optional), `tool_type` ("mcp"/"native", optional) + - **Attributes**: `function_name`, `status` ("success"/"error"/"cancelled"), `success` (boolean, retained for compatibility), `decision` ("accept"/"reject"/"auto_accept"/"modify", optional), `tool_type` ("mcp"/"native", optional) - `qwen-code.tool.call.latency` (Histogram, ms): Measures tool call latency. - **Attributes**: `function_name` (string) diff --git a/docs/users/configuration/settings.md b/docs/users/configuration/settings.md index b59c8206e08..4ca7c4ee080 100644 --- a/docs/users/configuration/settings.md +++ b/docs/users/configuration/settings.md @@ -855,7 +855,7 @@ To help us improve Qwen Code, we collect anonymized usage statistics. This data **What we collect:** -- **Tool Calls:** We log the names of the tools that are called, whether they succeed or fail, and how long they take to execute. We do not collect the arguments passed to the tools or any data returned by them. +- **Tool Calls:** We log the names and categories (native or MCP) of the tools that are called, their terminal status (success, error, or cancelled), and how long they take to execute. We do not collect the arguments passed to the tools or any data returned by them. - **API Requests:** We log the model used for each request, the duration of the request, and whether it was successful. We do not collect the content of the prompts or responses. - **Session Information:** We collect information about the configuration of the CLI, such as the enabled tools and the approval mode. diff --git a/packages/core/src/core/coreToolScheduler.test.ts b/packages/core/src/core/coreToolScheduler.test.ts index 9c21fe4a83c..6706642d064 100644 --- a/packages/core/src/core/coreToolScheduler.test.ts +++ b/packages/core/src/core/coreToolScheduler.test.ts @@ -3982,6 +3982,11 @@ describe('CoreToolScheduler', () => { returnDisplay: 'alpha output', }); const executeB = vi.fn().mockRejectedValue(new Error('beta failed')); + const executeC = vi.fn().mockResolvedValue({ + llmContent: 'gamma failed', + returnDisplay: 'gamma failed', + error: { message: 'gamma failed' }, + }); const toolsByName = new Map([ [ 'alpha', @@ -3999,6 +4004,14 @@ describe('CoreToolScheduler', () => { execute: executeB, }), ], + [ + 'gamma', + new MockTool({ + name: 'gamma', + kind: Kind.Read, + execute: executeC, + }), + ], ]); const messageBus = { request: vi.fn().mockImplementation( @@ -4013,11 +4026,15 @@ describe('CoreToolScheduler', () => { ), }; const onAllToolCallsComplete = vi.fn(); + const recordToolResult = vi.fn(); const { scheduler } = createSchedulerForLegacyToolTests({ toolsByName, messageBus, disableHooks: false, onAllToolCallsComplete, + chatRecordingService: { + recordToolResult, + } as unknown as ChatRecordingService, }); await scheduler.schedule( @@ -4036,6 +4053,13 @@ describe('CoreToolScheduler', () => { isClientInitiated: false, prompt_id: 'prompt-batch-failure', }, + { + callId: 'call-gamma', + name: 'gamma', + args: { value: 'c' }, + isClientInitiated: false, + prompt_id: 'prompt-batch-failure', + }, ], new AbortController().signal, ); @@ -4068,10 +4092,36 @@ describe('CoreToolScheduler', () => { error_type: ToolErrorType.UNHANDLED_EXCEPTION, }), }), + expect.objectContaining({ + tool_name: 'gamma', + status: 'error', + tool_response: expect.objectContaining({ + error: 'gamma failed', + error_type: ToolErrorType.UNKNOWN, + }), + }), ], }, }), ); + const completedCalls = onAllToolCallsComplete.mock + .calls[0][0] as ToolCall[]; + expect( + completedCalls.find((call) => call.request.callId === 'call-gamma'), + ).toMatchObject({ + status: 'error', + response: { + errorType: ToolErrorType.UNKNOWN, + }, + }); + expect( + recordToolResult.mock.calls.find( + ([, metadata]) => metadata?.callId === 'call-gamma', + )?.[1], + ).toMatchObject({ + status: 'error', + errorType: ToolErrorType.UNKNOWN, + }); }); it('queues new tool calls while a PostToolBatch hook is still running', async () => { diff --git a/packages/core/src/core/coreToolScheduler.ts b/packages/core/src/core/coreToolScheduler.ts index 54f53613c18..32b5ce4ff72 100644 --- a/packages/core/src/core/coreToolScheduler.ts +++ b/packages/core/src/core/coreToolScheduler.ts @@ -910,7 +910,7 @@ const TRUNCATION_RETRY_LOOP_DIRECTIVE = const createErrorResponse = ( request: ToolCallRequestInfo, error: Error, - errorType: ToolErrorType | undefined, + errorType: ToolErrorType, artifacts?: ToolArtifact[], resultDisplay?: ToolResultDisplay, ): ToolCallResponseInfo => ({ @@ -4982,7 +4982,7 @@ export class CoreToolScheduler { let errorResponse = createErrorResponse( scheduledCall.request, error, - toolResult.error.type, + toolResult.error.type ?? ToolErrorType.UNKNOWN, failureHookArtifacts, typeof toolResult.returnDisplay === 'string' ? undefined diff --git a/packages/core/src/telemetry/loggers.test.ts b/packages/core/src/telemetry/loggers.test.ts index 7bfdf622345..e584cc6b985 100644 --- a/packages/core/src/telemetry/loggers.test.ts +++ b/packages/core/src/telemetry/loggers.test.ts @@ -992,9 +992,193 @@ describe('loggers', () => { vi.spyOn(metrics, 'recordToolCallMetrics').mockImplementation( mockMetrics.recordToolCallMetrics, ); + vi.spyOn(QwenLogger.prototype, 'logToolCallEvent').mockImplementation( + () => undefined, + ); mockLogger.emit.mockReset(); }); + it('normalizes an unclassified error before every consumer', () => { + const recordUiTelemetryEvent = vi.fn(); + const configWithRecording = { + ...mockConfig, + getChatRecordingService: () => ({ + recordUiTelemetryEvent, + }), + } as unknown as Config; + const event = { + 'event.name': 'tool_call', + 'event.timestamp': '2024-12-31T23:59:59.000Z', + function_name: ' ', + function_args: { value: 1 }, + duration_ms: 25, + status: 'error', + success: true, + error: 'failed', + error_type: ' ', + prompt_id: 'prompt-normalize', + tool_type: 'native', + } as ToolCallEvent; + + logToolCall(configWithRecording, event); + + const normalized = expect.objectContaining({ + function_name: 'unknown_tool', + status: 'error', + success: false, + error: 'failed', + error_type: ToolErrorType.UNKNOWN, + }); + expect(QwenLogger.prototype.logToolCallEvent).toHaveBeenCalledWith( + normalized, + ); + expect(mockUiEvent.addEvent).toHaveBeenCalledWith( + normalized, + 'test-session-id', + ); + expect(recordUiTelemetryEvent).toHaveBeenCalledWith(normalized); + expect(mockLogger.emit).toHaveBeenCalledWith( + expect.objectContaining({ + attributes: expect.objectContaining({ + function_name: 'unknown_tool', + status: 'error', + success: false, + error: 'failed', + error_type: ToolErrorType.UNKNOWN, + 'error.message': 'failed', + 'error.type': ToolErrorType.UNKNOWN, + }), + }), + ); + expect(mockMetrics.recordToolCallMetrics).toHaveBeenCalledWith( + configWithRecording, + 25, + { + function_name: 'unknown_tool', + status: 'error', + success: false, + decision: undefined, + tool_type: 'native', + }, + ); + expect(event).toMatchObject({ + function_name: ' ', + success: true, + error_type: ' ', + }); + }); + + it('preserves an explicitly classified error type', () => { + const event = { + 'event.name': 'tool_call', + 'event.timestamp': '2024-12-31T23:59:59.000Z', + function_name: 'test-function', + function_args: {}, + duration_ms: 10, + status: 'error', + success: false, + error: 'classified failure', + error_type: ToolErrorType.EXECUTION_FAILED, + prompt_id: 'prompt-classified', + tool_type: 'native', + } as ToolCallEvent; + + logToolCall(mockConfig, event); + + expect(QwenLogger.prototype.logToolCallEvent).toHaveBeenCalledWith( + expect.objectContaining({ + error_type: ToolErrorType.EXECUTION_FAILED, + }), + ); + expect(mockLogger.emit.mock.calls[0][0].attributes).toMatchObject({ + error_type: ToolErrorType.EXECUTION_FAILED, + 'error.type': ToolErrorType.EXECUTION_FAILED, + }); + }); + + it.each([ + { status: 'success' as const, expectedSuccess: true }, + { status: 'cancelled' as const, expectedSuccess: false }, + ])( + 'clears stale error fields for $status events', + ({ status, expectedSuccess }) => { + const event = { + 'event.name': 'tool_call', + 'event.timestamp': '2024-12-31T23:59:59.000Z', + function_name: 'test-function', + function_args: {}, + duration_ms: 10, + status, + success: !expectedSuccess, + error: 'stale error', + error_type: ToolErrorType.EXECUTION_FAILED, + prompt_id: 'prompt-terminal', + tool_type: 'native', + } as ToolCallEvent; + + logToolCall(mockConfig, event); + + expect(QwenLogger.prototype.logToolCallEvent).toHaveBeenCalledWith( + expect.objectContaining({ + status, + success: expectedSuccess, + error: undefined, + error_type: undefined, + }), + ); + const attributes = mockLogger.emit.mock.calls[0][0].attributes; + expect(attributes).toMatchObject({ + status, + success: expectedSuccess, + error: undefined, + error_type: undefined, + }); + expect(attributes).not.toHaveProperty('error.message'); + expect(attributes).not.toHaveProperty('error.type'); + expect(mockMetrics.recordToolCallMetrics).toHaveBeenCalledWith( + mockConfig, + 10, + expect.objectContaining({ + status, + success: expectedSuccess, + }), + ); + }, + ); + + it('normalizes UI and QwenLogger events when the OTel SDK is disabled', () => { + vi.spyOn(sdk, 'isTelemetrySdkInitialized').mockReturnValue(false); + const event = { + 'event.name': 'tool_call', + 'event.timestamp': '2024-12-31T23:59:59.000Z', + function_name: '', + function_args: {}, + duration_ms: 10, + status: 'error', + success: true, + prompt_id: 'prompt-no-otel', + tool_type: 'native', + } as ToolCallEvent; + + logToolCall(mockConfig, event); + + const normalized = expect.objectContaining({ + function_name: 'unknown_tool', + status: 'error', + success: false, + error_type: ToolErrorType.UNKNOWN, + }); + expect(QwenLogger.prototype.logToolCallEvent).toHaveBeenCalledWith( + normalized, + ); + expect(mockUiEvent.addEvent).toHaveBeenCalledWith( + normalized, + 'test-session-id', + ); + expect(mockLogger.emit).not.toHaveBeenCalled(); + expect(mockMetrics.recordToolCallMetrics).not.toHaveBeenCalled(); + }); + it('should log a tool call with all fields', () => { const tool = new EditTool(mockConfig); const call: CompletedToolCall = { @@ -1084,6 +1268,7 @@ describe('loggers', () => { 100, { function_name: 'test-function', + status: 'success', success: true, decision: ToolCallDecision.ACCEPT, tool_type: 'native', @@ -1149,7 +1334,10 @@ describe('loggers', () => { prompt_id: 'prompt-id-2', tool_type: 'native', error: undefined, - error_type: undefined, + error_type: ToolErrorType.UNKNOWN, + 'error.type': ToolErrorType.UNKNOWN, + mcp_server_name: undefined, + response_id: undefined, metadata: undefined, content_length: undefined, }, @@ -1160,6 +1348,7 @@ describe('loggers', () => { 100, { function_name: 'test-function', + status: 'error', success: false, decision: ToolCallDecision.REJECT, tool_type: 'native', @@ -1169,6 +1358,7 @@ describe('loggers', () => { expect(mockUiEvent.addEvent).toHaveBeenCalledWith( { ...event, + error_type: ToolErrorType.UNKNOWN, 'event.name': EVENT_TOOL_CALL, 'event.timestamp': '2025-01-01T00:00:00.000Z', }, @@ -1239,6 +1429,7 @@ describe('loggers', () => { 100, { function_name: 'test-function', + status: 'success', success: true, decision: ToolCallDecision.MODIFY, tool_type: 'native', @@ -1317,6 +1508,7 @@ describe('loggers', () => { 100, { function_name: 'test-function', + status: 'success', success: true, decision: undefined, tool_type: 'native', @@ -1396,6 +1588,7 @@ describe('loggers', () => { 100, { function_name: 'test-function', + status: 'error', success: false, decision: undefined, tool_type: 'native', diff --git a/packages/core/src/telemetry/loggers.ts b/packages/core/src/telemetry/loggers.ts index 3019bb5e936..3b960d64c08 100644 --- a/packages/core/src/telemetry/loggers.ts +++ b/packages/core/src/telemetry/loggers.ts @@ -10,6 +10,7 @@ import { SemanticAttributes } from '@opentelemetry/semantic-conventions'; import type { Config } from '../config/config.js'; import { isInternalPromptId } from '../utils/internalPromptIds.js'; import { safeJsonStringify } from '../utils/safeJsonStringify.js'; +import { ToolErrorType } from '../tools/tool-error.js'; import { EVENT_API_ERROR, EVENT_API_CANCEL, @@ -244,44 +245,64 @@ export function logUserRetry(config: Config, event: UserRetryEvent): void { logger.emit(logRecord); } +function normalizeToolCallEvent(event: ToolCallEvent): ToolCallEvent { + const isError = event.status === 'error'; + return { + ...event, + function_name: + event.function_name.trim().length > 0 + ? event.function_name + : 'unknown_tool', + success: event.status === 'success', + error: isError ? event.error : undefined, + error_type: isError + ? event.error_type?.trim() + ? event.error_type + : ToolErrorType.UNKNOWN + : undefined, + }; +} + export function logToolCall(config: Config, event: ToolCallEvent): void { + const normalizedEvent = normalizeToolCallEvent(event); const uiEvent = { - ...event, + ...normalizedEvent, 'event.name': EVENT_TOOL_CALL, 'event.timestamp': new Date().toISOString(), } as UiEvent; uiTelemetryService.addEvent(uiEvent, config.getSessionId()); - if (!isInternalPromptId(event.prompt_id)) { + if (!isInternalPromptId(normalizedEvent.prompt_id)) { recordUiTelemetryEventToChat(config, uiEvent); } - QwenLogger.getInstance(config)?.logToolCallEvent(event); + QwenLogger.getInstance(config)?.logToolCallEvent(normalizedEvent); if (!isTelemetrySdkInitialized()) return; const attributes: LogAttributes = { ...getCommonAttributes(config), - ...event, + ...normalizedEvent, 'event.name': EVENT_TOOL_CALL, 'event.timestamp': new Date().toISOString(), - function_args: safeJsonStringify(event.function_args, 2), + function_args: safeJsonStringify(normalizedEvent.function_args, 2), }; - if (event.error) { - attributes['error.message'] = event.error; - if (event.error_type) { - attributes['error.type'] = event.error_type; - } + if (normalizedEvent.error) { + attributes['error.message'] = normalizedEvent.error; + } + if (normalizedEvent.error_type) { + attributes['error.type'] = normalizedEvent.error_type; } const logger = logs.getLogger(SERVICE_NAME); const logRecord: LogRecord = { - body: `Tool call: ${event.function_name}${event.decision ? `. Decision: ${event.decision}` : ''}. Success: ${event.success}. Duration: ${event.duration_ms}ms.`, + body: `Tool call: ${normalizedEvent.function_name}${normalizedEvent.decision ? `. Decision: ${normalizedEvent.decision}` : ''}. Success: ${normalizedEvent.success}. Duration: ${normalizedEvent.duration_ms}ms.`, attributes, }; logger.emit(logRecord); - recordToolCallMetrics(config, event.duration_ms, { - function_name: event.function_name, - success: event.success, - decision: event.decision, - tool_type: event.tool_type, + recordToolCallMetrics(config, normalizedEvent.duration_ms, { + function_name: normalizedEvent.function_name, + status: normalizedEvent.status, + success: normalizedEvent.success, + decision: normalizedEvent.decision, + tool_type: normalizedEvent.tool_type, }); } diff --git a/packages/core/src/telemetry/metrics.test.ts b/packages/core/src/telemetry/metrics.test.ts index 820fa36fbdf..e075f6c6b02 100644 --- a/packages/core/src/telemetry/metrics.test.ts +++ b/packages/core/src/telemetry/metrics.test.ts @@ -67,6 +67,7 @@ vi.mock('@opentelemetry/api'); describe('Telemetry Metrics', () => { let initializeMetricsModule: typeof import('./metrics.js').initializeMetrics; + let recordToolCallMetricsModule: typeof import('./metrics.js').recordToolCallMetrics; let recordTokenUsageMetricsModule: typeof import('./metrics.js').recordTokenUsageMetrics; let recordFileOperationMetricModule: typeof import('./metrics.js').recordFileOperationMetric; let recordChatCompressionMetricsModule: typeof import('./metrics.js').recordChatCompressionMetrics; @@ -93,6 +94,7 @@ describe('Telemetry Metrics', () => { const metricsJsModule = await import('./metrics.js'); initializeMetricsModule = metricsJsModule.initializeMetrics; + recordToolCallMetricsModule = metricsJsModule.recordToolCallMetrics; recordTokenUsageMetricsModule = metricsJsModule.recordTokenUsageMetrics; recordFileOperationMetricModule = metricsJsModule.recordFileOperationMetric; recordChatCompressionMetricsModule = @@ -127,6 +129,48 @@ describe('Telemetry Metrics', () => { mockCreateHistogramFn.mockReturnValue(mockHistogramInstance); }); + describe('recordToolCallMetrics', () => { + const config = makeFakeConfig({ + sessionId: 'test-session-id', + }); + + it('records an explicit terminal status only on the counter', () => { + initializeMetricsModule(config); + + recordToolCallMetricsModule(config, 25, { + function_name: 'read_file', + success: false, + status: 'cancelled', + tool_type: 'native', + }); + + expect(mockCounterAddFn).toHaveBeenCalledWith(1, { + function_name: 'read_file', + success: false, + status: 'cancelled', + tool_type: 'native', + }); + expect(mockHistogramRecordFn).toHaveBeenCalledWith(25, { + function_name: 'read_file', + }); + }); + + it('derives status from success for legacy callers', () => { + initializeMetricsModule(config); + + recordToolCallMetricsModule(config, 10, { + function_name: 'legacy_tool', + success: false, + }); + + expect(mockCounterAddFn).toHaveBeenCalledWith(1, { + function_name: 'legacy_tool', + success: false, + status: 'error', + }); + }); + }); + describe('recordChatCompressionMetrics', () => { it('does not record metrics if not initialized', () => { const lol = makeFakeConfig({}); diff --git a/packages/core/src/telemetry/metrics.ts b/packages/core/src/telemetry/metrics.ts index 862145a4464..a3cbac5f2f4 100644 --- a/packages/core/src/telemetry/metrics.ts +++ b/packages/core/src/telemetry/metrics.ts @@ -84,12 +84,14 @@ const baseMetricDefinition = { const COUNTER_DEFINITIONS = { [TOOL_CALL_COUNT]: { - description: 'Counts tool calls, tagged by function name and success.', + description: + 'Counts tool calls, tagged by function name and terminal status.', valueType: ValueType.INT, assign: (c: Counter) => (toolCallCounter = c), attributes: {} as { function_name: string; success: boolean; + status?: 'success' | 'error' | 'cancelled'; decision?: 'accept' | 'reject' | 'modify' | 'auto_accept'; tool_type?: 'native' | 'mcp'; }, @@ -591,6 +593,7 @@ export function recordToolCallMetrics( const metricAttributes: Attributes = { ...baseMetricDefinition.getCommonAttributes(config), ...attributes, + status: attributes.status ?? (attributes.success ? 'success' : 'error'), }; toolCallCounter.add(1, metricAttributes); toolCallLatencyHistogram.record(durationMs, { diff --git a/packages/core/src/telemetry/qwen-logger/qwen-logger.test.ts b/packages/core/src/telemetry/qwen-logger/qwen-logger.test.ts index ff3556a1bfb..822c5655946 100644 --- a/packages/core/src/telemetry/qwen-logger/qwen-logger.test.ts +++ b/packages/core/src/telemetry/qwen-logger/qwen-logger.test.ts @@ -27,6 +27,7 @@ import { SkillLaunchEvent, ProtocolTagSanitizedEvent, RipgrepRuntimeRecoveryEvent, + type ToolCallEvent, } from '../types.js'; import type { RumEvent, RumPayload } from './event-types.js'; @@ -987,4 +988,45 @@ describe('QwenLogger', () => { ); }); }); + + describe('logToolCallEvent', () => { + it('records terminal status and tool type without MCP server metadata', () => { + const logger = QwenLogger.getInstance(mockConfig)!; + const enqueueSpy = vi.spyOn(logger, 'enqueueLogEvent'); + const event = { + 'event.name': 'tool_call', + 'event.timestamp': '2025-01-01T12:00:00.000Z', + function_name: 'remote_tool', + function_args: { secret: 'not-forwarded' }, + duration_ms: 42, + status: 'error', + success: false, + error: 'failed', + error_type: 'unknown', + prompt_id: 'prompt-tool', + tool_type: 'mcp', + mcp_server_name: 'private-server', + } as ToolCallEvent; + + logger.logToolCallEvent(event); + + expect(enqueueSpy).toHaveBeenCalledWith( + expect.objectContaining({ + name: 'tool_call#remote_tool', + properties: expect.objectContaining({ + tool_name: 'remote_tool', + status: 'error', + tool_type: 'mcp', + success: 0, + duration_ms: 42, + error_type: 'unknown', + error_message: 'failed', + }), + }), + ); + const rumEvent = enqueueSpy.mock.calls[0][0]; + expect(rumEvent.properties).not.toHaveProperty('mcp_server_name'); + expect(rumEvent.properties).not.toHaveProperty('function_args'); + }); + }); }); diff --git a/packages/core/src/telemetry/qwen-logger/qwen-logger.ts b/packages/core/src/telemetry/qwen-logger/qwen-logger.ts index c1329437e1f..7d7fcfc8cc0 100644 --- a/packages/core/src/telemetry/qwen-logger/qwen-logger.ts +++ b/packages/core/src/telemetry/qwen-logger/qwen-logger.ts @@ -542,6 +542,8 @@ export class QwenLogger { response_id: event.response_id, tool_name: event.function_name, permission: event.decision, + status: event.status, + tool_type: event.tool_type, success: event.success ? 1 : 0, duration_ms: event.duration_ms, error_type: event.error_type, diff --git a/packages/desktop/packages/shared/src/agent/__tests__/qwen-agent-slash-history.test.ts b/packages/desktop/packages/shared/src/agent/__tests__/qwen-agent-slash-history.test.ts index 37428de3ea8..90c4389dc58 100644 --- a/packages/desktop/packages/shared/src/agent/__tests__/qwen-agent-slash-history.test.ts +++ b/packages/desktop/packages/shared/src/agent/__tests__/qwen-agent-slash-history.test.ts @@ -2114,6 +2114,74 @@ describe('QwenAgent slash command history', () => { }); }); + it('restores cancelled Qwen transcript tool telemetry as interrupted', async () => { + const cwd = mkdtempSync(join(tmpdir(), 'qwen-cwd-')); + const runtimeRoot = mkdtempSync(join(tmpdir(), 'qwen-runtime-')); + tempRoots.push(cwd, runtimeRoot); + process.env.QWEN_RUNTIME_DIR = runtimeRoot; + + const sessionId = 'qwen-session'; + const commandArgs = { command: 'sleep 10' }; + writeQwenTranscript(runtimeRoot, cwd, sessionId, [ + { + uuid: 'assistant-1', + sessionId, + timestamp: '2026-05-31T02:15:02.868Z', + type: 'assistant', + message: { + role: 'model', + parts: [ + { + functionCall: { + id: 'call-sleep', + name: 'run_shell_command', + args: commandArgs, + }, + }, + ], + }, + }, + { + uuid: 'tool-telemetry-1', + sessionId, + timestamp: '2026-05-31T02:15:06.203Z', + type: 'system', + subtype: 'ui_telemetry', + systemPayload: { + uiEvent: { + 'event.name': 'qwen-code.tool_call', + function_name: 'run_shell_command', + function_args: commandArgs, + status: 'cancelled', + success: false, + }, + }, + }, + ]); + + const agent = createAgent(cwd); + const internals = agent as unknown as QwenAvailableCommandsInternals; + internals.ensureProcess = async () => {}; + internals.callAcp = async (_method, execute) => + execute({ + extMethod: async () => ({ updates: [] }), + loadSession: async () => ({ models: {}, modes: {} }), + }); + + const result = await agent.loadSessionMessages(sessionId, { cwd }); + agent.destroy(); + + expect(result.messages.filter((message) => message.role === 'tool')).toEqual([ + expect.objectContaining({ + toolUseId: 'call-sleep', + toolName: 'Bash', + toolStatus: 'error', + toolResult: 'Interrupted', + isError: true, + }), + ]); + }); + it('closes dangling Qwen transcript tool calls as terminal errors', async () => { const cwd = mkdtempSync(join(tmpdir(), 'qwen-cwd-')); const runtimeRoot = mkdtempSync(join(tmpdir(), 'qwen-runtime-')); diff --git a/packages/desktop/packages/shared/src/agent/qwen-agent.ts b/packages/desktop/packages/shared/src/agent/qwen-agent.ts index 792cea65b42..37010558276 100644 --- a/packages/desktop/packages/shared/src/agent/qwen-agent.ts +++ b/packages/desktop/packages/shared/src/agent/qwen-agent.ts @@ -4433,14 +4433,21 @@ export class QwenAgent extends BaseAgent { const toolUseId = asString(record.uuid) || `qwen-transcript-tool-${nextId()}`; const input = toRecord(uiEvent.function_args); - const isError = uiEvent.success === false || uiEvent.status === 'error'; + const status = asString(uiEvent.status); + const isInterrupted = isQwenUserInterruptStatus(status); + const isError = + uiEvent.success === false || + isQwenToolFailureStatus(status) || + isInterrupted; const error = asString(uiEvent.error); const contentLength = asNumber(uiEvent.content_length); - const toolResult = isError - ? error || 'Tool failed' - : contentLength != null - ? `Completed (${contentLength} bytes)` - : 'Completed'; + const toolResult = isInterrupted + ? 'Interrupted' + : isError + ? error || 'Tool failed' + : contentLength != null + ? `Completed (${contentLength} bytes)` + : 'Completed'; return { id: nextId(),