Skip to content

feat(core): Align GenAI request telemetry with ARMS #7635

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
doudouOUC merged 1 commit into from
Jul 24, 2026
Merged

feat(core): Align GenAI request telemetry with ARMS #7635

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 39 additions & 10 deletions docs/design/gen-ai-arms-field-alignment.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,14 +20,15 @@ An upgrade to either baseline requires regenerating and reviewing this matrix.

## Field contract

| Span | Standard attributes emitted in this phase | Source and omission rule |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| LLM | `gen_ai.operation.name`, `gen_ai.provider.name`, `gen_ai.conversation.id`, `gen_ai.request.model` | Written at span creation. Conversation ID is the existing session ID. |
| LLM response | `gen_ai.response.id`, `gen_ai.response.model`, `gen_ai.response.finish_reasons` | Provider response data only. Missing response model is omitted rather than replaced with the request model. All candidate finish reasons are ordered by candidate index. |
| LLM output | `gen_ai.output.type` | Gemini and Vertex AI only, and only when an explicit response MIME type or one unambiguous response modality is sent on the wire. |
| LLM usage | `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.cache_read.input_tokens`, `gen_ai.usage.cache_creation.input_tokens` | Only provider-reported non-negative safe integers. Explicit zero is retained. When only a total is reported, input/output are omitted instead of estimated. |
| Tool | `gen_ai.operation.name=execute_tool`, `gen_ai.tool.name`, `gen_ai.tool.type=function`, `gen_ai.tool.call.id` | Tool call ID prefers the provider/model ID and falls back to Qwen Code's internal ID. |
| Agent | `gen_ai.operation.name=invoke_agent`, `gen_ai.agent.name`, `gen_ai.agent.description`, `gen_ai.conversation.id`, optional `gen_ai.request.model` | Description uses the existing 1024-UTF-16-code-unit truncation threshold and never splits surrogate pairs. Internal invocation IDs remain private. |
| Span | Standard attributes emitted in this phase | Source and omission rule |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| LLM | `gen_ai.operation.name`, `gen_ai.provider.name`, `gen_ai.conversation.id`, `gen_ai.request.model` | Written at span creation. Conversation ID is the existing session ID. |
| LLM request | `gen_ai.request.choice.count`, `gen_ai.request.max_tokens`, `gen_ai.request.temperature`, `gen_ai.request.top_p`, `gen_ai.request.frequency_penalty`, `gen_ai.request.presence_penalty`, `gen_ai.request.stop_sequences` | Read from the first provider-final SDK request object. Invalid or unavailable values are omitted; no SDK or server defaults are inferred. |
| LLM response | `gen_ai.response.id`, `gen_ai.response.model`, `gen_ai.response.finish_reasons` | Provider response data only. Missing response model is omitted rather than replaced with the request model. All candidate finish reasons are ordered by candidate index. |
| LLM output | `gen_ai.output.type` | Gemini and Vertex AI only, and only when an explicit response MIME type or one unambiguous response modality is sent on the wire. |
| LLM usage | `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.cache_read.input_tokens`, `gen_ai.usage.cache_creation.input_tokens` | Only provider-reported non-negative safe integers. Explicit zero is retained. When only a total is reported, input/output are omitted instead of estimated. |
| Tool | `gen_ai.operation.name=execute_tool`, `gen_ai.tool.name`, `gen_ai.tool.type=function`, `gen_ai.tool.call.id` | Tool call ID prefers the provider/model ID and falls back to Qwen Code's internal ID. |
| Agent | `gen_ai.operation.name=invoke_agent`, `gen_ai.agent.name`, `gen_ai.agent.description`, `gen_ai.conversation.id`, optional `gen_ai.request.model` | Description uses the existing 1024-UTF-16-code-unit truncation threshold and never splits surrogate pairs. Internal invocation IDs remain private. |

Private attributes without an exact standard equivalent remain available for
compatibility. Exact-equivalent private aliases and invalid GenAI aliases are
Expand Down Expand Up @@ -65,6 +66,36 @@ from the model name.
OpenAI-compatible, Anthropic, and Qwen OAuth requests use operation `chat`.
Gemini and Vertex AI requests use `generate_content`.

## Request parameters

Request attributes are collected after provider adapters have applied defaults,
overrides, unsupported-field removal, and output-window clamps, immediately
before calling the provider SDK. This is the final SDK request object visible
to Qwen Code, not the original logical configuration or the serialized HTTP
body. A logical LLM span records only its first such request snapshot.

| Standard attribute | OpenAI-compatible and Qwen OAuth | Anthropic | Gemini and Vertex AI |
| ---------------------------------- | ---------------------------------------------------------- | ------------------ | ------------------------- |
| `gen_ai.request.choice.count` | `n` | Not applicable | `config.candidateCount` |
| `gen_ai.request.max_tokens` | `max_tokens`, `max_completion_tokens`, or `max_new_tokens` | `max_tokens` | `config.maxOutputTokens` |
| `gen_ai.request.temperature` | `temperature` | `temperature` | `config.temperature` |
| `gen_ai.request.top_p` | `top_p` | `top_p` | `config.topP` |
| `gen_ai.request.frequency_penalty` | `frequency_penalty` | Not currently sent | `config.frequencyPenalty` |
| `gen_ai.request.presence_penalty` | `presence_penalty` | Not currently sent | `config.presencePenalty` |
| `gen_ai.request.stop_sequences` | `stop` | `stop_sequences` | `config.stopSequences` |

Finite numbers and safe integers are preserved exactly, including zero and
negative values on failed provider requests. Choice count is omitted when it is
one. Stop sequences must be a complete string array; OpenAI's single-string
form is normalized to a one-element array. Empty arrays are retained and mixed
arrays are omitted rather than filtered. Explicit adapter defaults are
recorded, while implicit SDK or server defaults are not inferred.

When multiple OpenAI-compatible output-budget aliases are present, the standard
maximum is emitted only if all present values are valid safe integers and
equal. Conflicting values are omitted because compatible endpoints do not have
a common precedence rule.

## Response and usage provenance

Provider converters attach internal provenance to normalized Gemini usage
Expand Down Expand Up @@ -104,8 +135,6 @@ Qwen Code does not inject that vendor-specific resource attribute or

## Deferred work

- Request sampling, choice, maximum-output, and stop fields need hooks after
provider adapters finalize the wire request.
- `seed` and `top_k` have incompatible ARMS and GenAI types in the baselines.
- Messages, instructions, tool definitions, arguments, and results require a
standard JSON schema, privacy controls, and payload caps.
Expand Down
5 changes: 3 additions & 2 deletions docs/developers/development/telemetry.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -822,9 +822,10 @@ Distributed tracing spans form a tree rooted at `qwen-code.interaction`. Each in
- **Attributes**: `session.id`, `qwen-code.prompt_id`, `qwen-code.message_type`, `qwen-code.model`, `qwen-code.approval_mode`, `interaction.sequence`, `interaction.duration_ms`, `qwen-code.turn_status` ("ok"/"error"/"cancelled")

- `qwen-code.llm_request`: Wraps a single LLM API call.
- **GenAI attributes**: `gen_ai.operation.name`, `gen_ai.provider.name`, `gen_ai.conversation.id`, `gen_ai.request.model`, optional `gen_ai.output.type`, `gen_ai.response.id`, `gen_ai.response.model`, `gen_ai.response.finish_reasons`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.cache_read.input_tokens`, `gen_ai.usage.cache_creation.input_tokens`
- **GenAI attributes**: `gen_ai.operation.name`, `gen_ai.provider.name`, `gen_ai.conversation.id`, `gen_ai.request.model`, `gen_ai.request.choice.count`, `gen_ai.request.max_tokens`, `gen_ai.request.temperature`, `gen_ai.request.top_p`, `gen_ai.request.frequency_penalty`, `gen_ai.request.presence_penalty`, `gen_ai.request.stop_sequences`, optional `gen_ai.output.type`, `gen_ai.response.id`, `gen_ai.response.model`, `gen_ai.response.finish_reasons`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.cache_read.input_tokens`, `gen_ai.usage.cache_creation.input_tokens`
- **Compatibility attributes**: `session.id`, `qwen-code.prompt_id`, `llm_request.context` ("subagent"/"interaction"/"standalone"), `duration_ms`, `ttft_ms`, `request_setup_ms`, `attempt`, `retry_total_delay_ms`, `sampling_ms`, `output_tokens_per_second`, `success`, `error`, `finish_reason`, `thoughts_token_count`, `subagent_name`, `error_type`, `error_status_code`
- Standard response fields come from the provider response. Standard token fields are emitted only for provider-reported non-negative safe integers. If the provider reports only a total token count, input/output usage is omitted rather than estimated.
- Standard request-parameter fields come from the first provider-final SDK request object after adapter defaults, overrides, unsupported-field removal, and output-window clamps. Qwen Code does not infer SDK or server defaults.

- `qwen-code.tool`: Wraps the full tool lifecycle (approval wait + execution).
- **Attributes**: `session.id`, `gen_ai.operation.name` (`execute_tool`), `gen_ai.tool.name`, `gen_ai.tool.type` (`function`), `gen_ai.tool.call.id`, `tool.call_id`, `duration_ms`, `success`, `error`
Expand All @@ -843,7 +844,7 @@ Distributed tracing spans form a tree rooted at `qwen-code.interaction`. Each in

#### GenAI field migration and ARMS recognition

LLM spans now use `gen_ai.request.model`, `gen_ai.response.id`, and `gen_ai.usage.*` without the exact-equivalent private `qwen-code.model`, `response_id`, `input_tokens`, `output_tokens`, and `cached_input_tokens` aliases. Tool spans similarly use `gen_ai.tool.name` without `tool.name`; blocked-on-user and hook spans keep `tool.name` because they are not GenAI Tool spans. The invalid aliases `gen_ai.usage.cached_tokens`, `gen_ai.server.time_to_first_token`, and `gen_ai.usage.reasoning_tokens` are no longer emitted. Use `gen_ai.usage.cache_read.input_tokens` for provider-reported cache reads; continue using the private `ttft_ms` and `thoughts_token_count` fields where no GenAI/ARMS-common replacement exists. The full version-pinned contract and deferred fields are documented in [GenAI and ARMS field alignment](../../design/gen-ai-arms-field-alignment.md).
LLM spans now use standard `gen_ai.request.*`, `gen_ai.response.*`, and `gen_ai.usage.*` fields without exact-equivalent private aliases. Request sampling attributes are written only under their standard names; no bare `temperature`, `top_p`, `max_tokens`, penalty, choice-count, or stop-sequence aliases are emitted. Tool spans similarly use `gen_ai.tool.name` without `tool.name`; blocked-on-user and hook spans keep `tool.name` because they are not GenAI Tool spans. The invalid aliases `gen_ai.usage.cached_tokens`, `gen_ai.server.time_to_first_token`, and `gen_ai.usage.reasoning_tokens` are no longer emitted. Use `gen_ai.usage.cache_read.input_tokens` for provider-reported cache reads; continue using the private `ttft_ms` and `thoughts_token_count` fields where no GenAI/ARMS-common replacement exists. The full version-pinned contract and deferred fields are documented in [GenAI and ARMS field alignment](../../design/gen-ai-arms-field-alignment.md).

To make ARMS recognize exported spans as a GenAI application, configure its resource feature explicitly:

Expand Down
115 changes: 114 additions & 1 deletion integration-tests/cli/gen-ai-telemetry.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,20 @@ describeLocal('GenAI telemetry fields', () => {
rig.setup('gen-ai-telemetry', {
settings: {
security: { auth: { selectedType: 'openai' } },
model: { name: 'request-model' },
model: {
name: 'request-model',
generationConfig: {
samplingParams: {
n: 2,
max_tokens: 128,
temperature: 0,
top_p: 0.8,
frequency_penalty: -0.1,
presence_penalty: 0.2,
stop: ['END', 'DONE'],
},
},
},
ui: { enableFollowupSuggestions: false },
},
});
Expand Down Expand Up @@ -151,6 +164,13 @@ describeLocal('GenAI telemetry fields', () => {
'gen_ai.operation.name': 'chat',
'gen_ai.provider.name': 'openai',
'gen_ai.request.model': 'request-model',
'gen_ai.request.choice.count': 2,
'gen_ai.request.max_tokens': 128,
'gen_ai.request.temperature': 0,
'gen_ai.request.top_p': 0.8,
'gen_ai.request.frequency_penalty': -0.1,
'gen_ai.request.presence_penalty': 0.2,
'gen_ai.request.stop_sequences': ['END', 'DONE'],
'gen_ai.response.model': 'provider-model-tool',
'gen_ai.response.finish_reasons': ['STOP'],
'gen_ai.usage.input_tokens': 20,
Expand All @@ -160,6 +180,13 @@ describeLocal('GenAI telemetry fields', () => {
expect(secondLlm).toMatchObject({
'gen_ai.operation.name': 'chat',
'gen_ai.provider.name': 'openai',
'gen_ai.request.choice.count': 2,
'gen_ai.request.max_tokens': 128,
'gen_ai.request.temperature': 0,
'gen_ai.request.top_p': 0.8,
'gen_ai.request.frequency_penalty': -0.1,
'gen_ai.request.presence_penalty': 0.2,
'gen_ai.request.stop_sequences': ['END', 'DONE'],
'gen_ai.response.model': 'provider-model-final',
'gen_ai.response.finish_reasons': ['STOP'],
'gen_ai.usage.input_tokens': 30,
Expand All @@ -181,6 +208,26 @@ describeLocal('GenAI telemetry fields', () => {
'gen_ai.server.time_to_first_token',
);
expect(attributes).not.toHaveProperty('gen_ai.usage.reasoning_tokens');
expect(attributes).not.toHaveProperty('choice_count');
expect(attributes).not.toHaveProperty('max_tokens');
expect(attributes).not.toHaveProperty('temperature');
expect(attributes).not.toHaveProperty('top_p');
expect(attributes).not.toHaveProperty('frequency_penalty');
expect(attributes).not.toHaveProperty('presence_penalty');
expect(attributes).not.toHaveProperty('stop_sequences');
}

expect(server.requests).toHaveLength(2);
for (const { body } of server.requests) {
expect(body).toMatchObject({
n: 2,
max_tokens: 128,
temperature: 0,
top_p: 0.8,
frequency_penalty: -0.1,
presence_penalty: 0.2,
stop: ['END', 'DONE'],
});
}

const toolSpan = records.find(
Expand All @@ -197,4 +244,70 @@ describeLocal('GenAI telemetry fields', () => {
});
expect(toolSpan?.attributes).not.toHaveProperty('tool.name');
});

it('omits the default choice count from the exported span', async () => {
server = await startFakeOpenAIServer(() => ({
model: 'provider-model',
content: 'Done.',
usage: {
prompt_tokens: 10,
completion_tokens: 2,
total_tokens: 12,
},
}));

rig = new TestRig();
rig.setup('gen-ai-default-choice-count', {
settings: {
security: { auth: { selectedType: 'openai' } },
model: {
name: 'request-model',
generationConfig: {
samplingParams: { n: 1 },
},
},
ui: { enableFollowupSuggestions: false },
},
});

const restoreEnvironment = setEnvironment({
HOME: rig.testDir!,
QWEN_HOME: join(rig.testDir!, '.qwen'),
OPENAI_API_KEY: 'fake-key',
OPENAI_BASE_URL: server.baseUrl,
OPENAI_MODEL: 'request-model',
QWEN_MODEL: 'request-model',
NO_PROXY: '127.0.0.1,localhost',
no_proxy: '127.0.0.1,localhost',
HTTP_PROXY: undefined,
HTTPS_PROXY: undefined,
ALL_PROXY: undefined,
http_proxy: undefined,
https_proxy: undefined,
all_proxy: undefined,
DASHSCOPE_PROXY_BASE_URL: undefined,
});

try {
await rig.run('Reply with done.', '--output-format', 'json');
} finally {
restoreEnvironment();
}

expect(server.requests.length).toBeGreaterThan(0);
for (const { body } of server.requests) {
expect(body).toMatchObject({ n: 1 });
}

const records = parseTelemetry(rig.readFile('telemetry.log'));
const llmSpans = records.filter(
(record) => record.name === 'qwen-code.llm_request',
);
expect(llmSpans).toHaveLength(server.requests.length);
for (const llmSpan of llmSpans) {
expect(llmSpan.attributes).not.toHaveProperty(
'gen_ai.request.choice.count',
);
}
});
});
Loading
Loading