Skip to content

feat(core): Align GenAI request telemetry with ARMS - #7635

Merged
doudouOUC merged 1 commit into
QwenLM:mainfrom
doudouOUC:agent/align-genai-request-telemetry
Jul 24, 2026
Merged

feat(core): Align GenAI request telemetry with ARMS#7635
doudouOUC merged 1 commit into
QwenLM:mainfrom
doudouOUC:agent/align-genai-request-telemetry

Conversation

@doudouOUC

Copy link
Copy Markdown
Collaborator

What this PR does

This PR adds the seven request-parameter attributes shared by OpenTelemetry GenAI semantic conventions and ARMS: choice count, maximum output tokens, temperature, top-p, frequency penalty, presence penalty, and stop sequences. Values are captured from the first provider-final SDK request after adapter defaults, overrides, unsupported-field removal, and output-window clamping.

The implementation uses the existing OpenTelemetry Context to attach an internal one-shot request observer to each recording LLM span. OpenAI-compatible and Qwen, Anthropic, Gemini, and Vertex request boundaries report only normalized attributes rather than retaining full requests. Provider-internal retries and fallbacks cannot overwrite the first snapshot, and telemetry failures remain best-effort.

The GenAI/ARMS design contract and developer telemetry documentation now describe provider mappings, types, omission rules, default handling, and deferred fields. The permanent CLI integration test compares exported span attributes with the request body received by the fake OpenAI server and verifies that a choice count of one is omitted.

Why it's needed

Qwen Code already emitted the shared GenAI/ARMS identity, response, usage, tool, and agent attributes, but omitted the common request sampling and output-limit fields. Operators therefore could not correlate model behavior, failures, cost, or latency with the effective request parameters in OpenTelemetry or ARMS.

Reading the original logical configuration would be inaccurate because provider adapters can inject defaults, override or remove fields, and clamp output limits. Capturing the final SDK request gives observability the same values Qwen Code actually attempted to send without exposing prompts, messages, tools, URLs, headers, or credentials.

Reviewer Test Plan

How to verify

Configure the file telemetry exporter and an OpenAI-compatible endpoint with all seven request parameters. Run a tool-call turn and confirm that every LLM span contains standard gen_ai.request.* attributes matching the fake server request body, that gen_ai.request.choice.count is omitted when the request sends n=1, and that no bare or qwen-code.* aliases are emitted.

Review the provider tests to confirm that both streaming and non-streaming SDK calls report the exact finalized request object. The retry and Anthropic empty-stream fallback cases should report each physical request while the Context observer accepts only the first snapshot for the logical LLM span.

Validated locally with npm run build, npm run bundle, npm run typecheck, npm run lint, 373 targeted core tests, and both gen-ai-telemetry CLI integration tests.

Evidence (Before & After)

N/A — telemetry and documentation changes with no TUI impact.

Tested on

OS Status
🍏 macOS
🪟 Windows ⚠️
🐧 Linux ⚠️

Environment (optional)

macOS 26.4.1, Node.js 22.22.3, npm 10.9.8, local bundle, sandbox disabled for the CLI integration test.

Risk & Scope

  • Main risk or tradeoff: Request attributes are added after span creation because only provider-final values are authoritative, so they cannot participate in head-sampling decisions. The observer intentionally keeps the first physical request when a provider performs an internal retry.
  • Not validated / out of scope: Online ARMS staging validation was not run because credentials were unavailable. Windows and Linux were not tested locally. Seed, top-k, stream, reasoning, message/tool payloads, embeddings, and full GenAI span topology remain out of scope.
  • Breaking changes / migration notes: No tracing API or provider API changes. These fields are emitted only under standard gen_ai.request.* names; no private aliases or dual writes are introduced.

Linked Issues

N/A

中文说明

本 PR 做了什么

本 PR 补齐 OpenTelemetry GenAI 语义规范与 ARMS 共同支持的七个请求参数属性:候选数量、最大输出 token、temperature、top-p、frequency penalty、presence penalty 和停止序列。属性值来自 provider adapter 完成默认值注入、覆盖、不支持字段删除和输出窗口裁剪后的第一份最终 SDK 请求。

实现复用现有 OpenTelemetry Context,为每个 recording LLM Span 绑定内部一次性 request observer。OpenAI-compatible 与 Qwen、Anthropic、Gemini 和 Vertex 的 SDK 调用边界只上报规范化属性,不保留完整请求。Provider 内部重试和 fallback 无法覆盖第一次快照,所有遥测异常均保持 best-effort,不影响模型调用。

GenAI/ARMS 设计契约和开发者遥测文档现已记录 provider 映射、类型、省略规则、默认值处理与延期字段。永久 CLI 集成测试会比较导出的 Span 属性和 OpenAI fake server 实际收到的请求体,并验证候选数量为一时省略该属性。

为什么需要

Qwen Code 已经输出 GenAI 与 ARMS 共同的身份、响应、usage、tool 和 agent 属性,但缺少共同的请求采样和输出限制字段。因此,运维人员无法在 OpenTelemetry 或 ARMS 中把模型行为、失败、成本或延迟与实际生效的请求参数关联起来。

读取原始逻辑配置并不准确,因为 provider adapter 可能注入默认值、覆盖或删除字段,并裁剪输出上限。采集最终 SDK 请求可以让可观测数据与 Qwen Code 实际尝试发送的值保持一致,同时不会暴露 prompt、message、tool、URL、header 或凭据。

Reviewer 测试计划

如何验证

配置 file telemetry exporter 和 OpenAI-compatible endpoint,并设置全部七个请求参数。运行一次包含工具调用的对话,确认每个 LLM Span 的标准 gen_ai.request.* 属性都与 fake server 收到的请求体一致;请求发送 n=1 时省略 gen_ai.request.choice.count;同时不产生裸字段或 qwen-code.* alias。

检查 provider 测试,确认流式和非流式 SDK 调用都上报完全相同的最终请求对象。内部重试和 Anthropic empty-stream fallback 应分别上报每个物理请求,而 Context observer 对同一逻辑 LLM Span 只接受第一次快照。

本地已通过 npm run buildnpm run bundlenpm run typechecknpm run lint、373 个目标 core 测试以及两个 gen-ai-telemetry CLI 集成测试。

证据(修改前后)

N/A——仅涉及遥测和文档,不影响 TUI。

测试平台

操作系统 状态
🍏 macOS
🪟 Windows ⚠️
🐧 Linux ⚠️

环境(可选)

macOS 26.4.1、Node.js 22.22.3、npm 10.9.8、本地 bundle,CLI 集成测试关闭 sandbox。

风险与范围

  • 主要风险或取舍:请求属性只能在 Span 创建后写入,因为只有 provider-final 值才可靠,因此这些字段不能参与 head sampling。Provider 执行内部重试时,observer 会有意保留第一次物理请求。
  • 未验证或范围外:由于没有凭据,未执行在线 ARMS staging 验证;未在本地测试 Windows 和 Linux。Seed、top-k、stream、reasoning、message/tool payload、embedding 和完整 GenAI Span 拓扑仍不在本次范围内。
  • 破坏性变更或迁移说明:没有 tracing API 或 provider API 变更。这些字段仅使用标准 gen_ai.request.* 名称,不新增私有 alias 或双写。

关联 Issue

N/A

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
@doudouOUC

Copy link
Copy Markdown
Collaborator Author

E2E test report

  • Baseline: global Qwen Code 0.20.0 sent the configured OpenAI request parameters to the fake server but exported none of the seven aligned gen_ai.request.* span attributes, so the expected missing-field assertion failed.
  • Local bundle: integration-tests/cli/gen-ai-telemetry.test.ts passed both scenarios with sandbox disabled.
  • Tool flow: both LLM spans in the LLM → Tool → LLM flow matched the final request bodies received by the fake server for choice count, max tokens, temperature, top-p, frequency penalty, presence penalty, and stop sequences.
  • Default omission: requests carrying n=1 omitted gen_ai.request.choice.count from every corresponding LLM span.
  • Compatibility: no bare or qwen-code.* aliases were emitted for the seven fields.
  • Online ARMS smoke: not run because staging credentials were unavailable.

@doudouOUC
doudouOUC marked this pull request as ready for review July 24, 2026 02:34
@doudouOUC
doudouOUC enabled auto-merge July 24, 2026 02:34
@qwen-code-ci-bot

Copy link
Copy Markdown
Collaborator

Thanks for the PR!

Template looks good ✓ — all sections filled in, bilingual body, clear reviewer test plan.

Problem: this is a feature completion rather than a bug fix, and the gap is concrete. Qwen Code already emits the shared GenAI/ARMS identity, response, usage, tool, and agent attributes, but omits the request-sampling and output-limit fields (choice.count, max_tokens, temperature, top_p, the two penalties, stop_sequences). Without those, operators can't correlate model behavior, cost, or latency with the effective request parameters. That's a real observability hole in an already-shipped feature, not theoretical hardening.

Direction: aligned. It completes OpenTelemetry GenAI semantic-convention coverage the codebase already partially implements (builds on the existing gen-ai-provider.ts / gen-ai-usage.ts modules), and the author has landed prior telemetry work here (#7276, #7003). The area is actively maintained across the ecosystem. One note: because this touches telemetry — a sensitive area — and comes from a fork, I'll run the full review but defer the final approve/reject call to a maintainer rather than deciding it automatically.

Size: 288 production logic lines / 433 test lines / 54 docs lines (775 total). Production is well under the 500-line awareness threshold, so no size concern. It does touch core paths (packages/core/src/telemetry/** and three content generators), so it's held to the higher confidence bar.

Approach: the scope feels right. One new module (gen-ai-request.ts) holds three pure, provider-specific extractors plus a one-shot OTel Context observer; the four integration points are single-line report*Request(...) calls placed right next to the existing runtimeDiagnostics.recordXWireRequest(...) sites, reusing the context-threading that's already there. The one-shot observer is a neat way to make provider-internal retries/fallbacks report every physical request while the logical span keeps only the first snapshot. No drive-by refactors or unrelated churn. The seven attributes are a coherent unit from the convention, so I wouldn't cut scope.

Moving on to code review. 🔍

中文说明

感谢贡献!

模板完整 ✓ —— 各 section 填写齐全,双语正文,reviewer test plan 清晰。

问题:这是功能补全而非 bug 修复,且缺口很具体。Qwen Code 已经上报了共享的 GenAI/ARMS identity、response、usage、tool、agent 属性,但缺少请求采样和输出上限字段(choice.countmax_tokenstemperaturetop_p、两个 penalty、stop_sequences)。没有这些,运维无法把模型行为、成本、延迟与有效请求参数关联起来。这是已上线功能里一个真实的可观测性缺口,不是理论性加固。

方向:对齐。它补全了代码库已部分实现的 OpenTelemetry GenAI 语义约定覆盖(基于现有的 gen-ai-provider.ts / gen-ai-usage.ts 模块),作者此前也在本仓库落地过遥测工作(#7276#7003)。该领域在整个生态中都在积极维护。一点说明:由于此 PR 触及遥测(敏感区域)且来自 fork,我会完整审查,但把最终 approve/reject 的决定权交给 maintainer,而不是自动判定。

规模:288 行生产逻辑 / 433 行测试 / 54 行文档(共 775 行)。生产行数远低于 500 行关注阈值,无规模问题。但确实触及核心路径(packages/core/src/telemetry/** 和三个 content generator),因此适用更高的置信度标准。

方案:范围合理。一个新模块(gen-ai-request.ts)包含三个纯函数式的 provider 提取器和一个一次性 OTel Context observer;四个集成点都是单行 report*Request(...) 调用,紧挨着现有的 runtimeDiagnostics.recordXWireRequest(...) 位置,复用了已有的 context 传递机制。一次性 observer 巧妙地让 provider 内部重试/fallback 上报每个物理请求,而逻辑 span 只保留第一次快照。没有顺手重构或无关改动。这七个属性是该约定中一个内聚的整体,因此不建议砍范围。

进入代码审查 🔍

Qwen Code · qwen3.8-max-preview

Reviewed at f59cd16dd3dcb19a1f602c7f51018130303e76f6 · re-run with @qwen-code /triage

@qwen-code-ci-bot

Copy link
Copy Markdown
Collaborator

Code review

My independent take before reading the diff: capture the seven GenAI request fields from the provider-final SDK request, normalize the per-provider field names (n/candidateCount, max_tokens/maxOutputTokens, top_p/topP, stop/stop_sequences/stopSequences), thread the capture through the existing OTel context so it lands on the right span, make it one-shot so retries don't clobber the first snapshot, and keep it best-effort. The PR does essentially this, so the approach matches my baseline — I didn't find a simpler path it missed.

The design is sound. gen-ai-request.ts holds three pure, provider-specific extractors plus a one-shot observer created via createGenAiRequestObserverContext. The observer rides the existing spanContext that loggingContentGenerator already threads with context.with(...), and each report*Request(...) call sits right next to the existing runtimeDiagnostics.recordXWireRequest(...) at the request boundary — so it captures the exact post-adapter wire request and reuses machinery that's already proven for the span itself. The consumed flag gives correct one-shot semantics: Anthropic's three report sites (non-streaming / streaming / empty-stream fallback) and OpenAI's internal retries each report every physical request, but the logical span keeps only the first snapshot. Every extraction and setAttributes is wrapped so telemetry can never break a provider request. Conventions are followed (kebab-case module, colocated test, unknown + type guards rather than any).

No correctness bugs, security issues, or regressions found. One subtlety worth noting, not a blocker: the OpenAI outputBudget helper reads max_tokens / max_completion_tokens / max_new_tokens and emits a value only when the present ones agree, omitting otherwise — a defensive normalization that seems reasonable since OpenAI normally sets just one.

Verification I ran (not just trusting the PR body): the new and modified unit tests pass locally — gen-ai-request.test.ts (23), pipeline.test.ts (120), anthropicContentGenerator.test.ts (109), geminiContentGenerator.test.ts (14), 266 total. tsc --noEmit and eslint on the changed files are both clean.

sequenceDiagram
    participant P1 as LoggingContentGenerator
    participant P2 as OTel Context
    participant P3 as Provider generator
    participant P4 as llm_request Span
    P1->>P2: create span, attach one-shot observer
    P1->>P3: activate context, call delegate
    P3->>P3: build provider-final request
    P3->>P2: report request, look up observer
    P2->>P4: set gen_ai.request attributes, first snapshot only
Loading
Files changed (12 of 12 shown)
File What changed
packages/core/src/telemetry/gen-ai-request.ts New module: three provider extractors plus the one-shot context observer (the whole feature)
packages/core/src/core/loggingContentGenerator/loggingContentGenerator.ts Wraps the existing span context with the observer for both generateContent and generateContentStream
packages/core/src/core/openaiContentGenerator/pipeline.ts One-line reportOpenAiRequest at the wire-request site
packages/core/src/core/anthropicContentGenerator/anthropicContentGenerator.ts reportAnthropicRequest at the three request boundaries
packages/core/src/core/geminiContentGenerator/geminiContentGenerator.ts reportGeminiRequest before generateContent and generateContentStream
packages/core/src/telemetry/gen-ai-request.test.ts New unit tests for the extractors and observer
packages/core/src/core/openaiContentGenerator/pipeline.test.ts Asserts the OpenAI request is reported
packages/core/src/core/anthropicContentGenerator/anthropicContentGenerator.test.ts Asserts reporting across the request paths
packages/core/src/core/geminiContentGenerator/geminiContentGenerator.test.ts Asserts the Gemini request is reported
integration-tests/cli/gen-ai-telemetry.test.ts End-to-end: span attributes must match the fake server's request body; choice count omitted at n equals 1
docs/design/gen-ai-arms-field-alignment.md Design contract for the field alignment
docs/developers/development/telemetry.md Developer docs for the new attributes

Real-scenario test (this PR)

Telemetry-only change with no TUI surface, so the meaningful real scenario is driving the actual bundled CLI against a fake OpenAI server with the file telemetry exporter enabled and sampling params configured, then reading what the product really emitted. I built dist/cli.js from this PR's head and ran it in tmux:

fake server: http://127.0.0.1:34349/v1
running: node dist/cli.js -p "Reply with done."

wire requests received by fake server: 2
wire request sampling fields: {"n":2,"max_tokens":128,"temperature":0,"top_p":0.8,"frequency_penalty":-0.1,"presence_penalty":0.2,"stop":["END","DONE"]}

telemetry spans captured: 18

=== gen_ai.request.* attributes on the llm_request span ===
{
  "gen_ai.request.model": "request-model",
  "gen_ai.request.choice.count": 2,
  "gen_ai.request.max_tokens": 128,
  "gen_ai.request.temperature": 0,
  "gen_ai.request.top_p": 0.8,
  "gen_ai.request.frequency_penalty": -0.1,
  "gen_ai.request.presence_penalty": 0.2,
  "gen_ai.request.stop_sequences": [
    "END",
    "DONE"
  ]
}

The exported span carries all seven gen_ai.request.* attributes, and each value matches what the fake server actually received on the wire — the capture reflects the finalized request, not the logical config. (My checker also flagged a qwen-code.prompt_id attribute, but that one is pre-existing telemetry from the daemon-tracing/loggers code, not something this PR adds — the PR's "no aliases" claim is about the seven new request fields, and none of those leak as bare or qwen-code.* names.)

中文说明

代码审查:我在看 diff 之前的独立想法是——从 provider 最终 SDK 请求中捕获七个 GenAI 请求字段,归一化各 provider 的字段名(n/candidateCountmax_tokens/maxOutputTokenstop_p/topPstop/stop_sequences/stopSequences),通过现有 OTel context 传递使其落在正确的 span 上,做成一次性以避免重试覆盖第一次快照,并保持 best-effort。PR 基本就是这么做的,方案与我的基线一致——我没找到它遗漏的更简路径。

设计合理。gen-ai-request.ts 包含三个纯函数式 provider 提取器加一个通过 createGenAiRequestObserverContext 创建的一次性 observer。observer 搭载在 loggingContentGenerator 已用 context.with(...) 传递的现有 spanContext 上,每个 report*Request(...) 调用都紧挨着请求边界处现有的 runtimeDiagnostics.recordXWireRequest(...)——因此捕获的是 adapter 处理后的精确 wire 请求,并复用了 span 本身已验证可用的机制。consumed 标志提供正确的一次性语义:Anthropic 的三个上报点(非流式/流式/empty-stream fallback)和 OpenAI 的内部重试都会上报每个物理请求,但逻辑 span 只保留第一次快照。所有提取和 setAttributes 都包裹了异常处理,遥测绝不会破坏 provider 请求。遵循了项目约定(kebab-case 模块、同目录测试、用 unknown + 类型守卫而非 any)。

未发现正确性 bug、安全问题或回归。一个值得注意但非阻塞的细节:OpenAI 的 outputBudget 辅助函数读取 max_tokens / max_completion_tokens / max_new_tokens,仅当存在的字段一致时才发出值,否则省略——这是一种防御性归一化,鉴于 OpenAI 通常只设置其一,看起来合理。

我亲自跑的验证(不只是相信 PR 正文):新增和修改的单测本地通过——gen-ai-request.test.ts(23)、pipeline.test.ts(120)、anthropicContentGenerator.test.ts(109)、geminiContentGenerator.test.ts(14),共 266 个。tsc --noEmit 和对改动文件的 eslint 均干净。

(时序图说明了请求属性如何从 provider 边界流回 span:LoggingContentGenerator 创建 span 并挂载一次性 observer 到 OTel Context,激活 context 调用 delegate,provider 生成最终请求后上报,observer 把 gen_ai.request 属性写入 llm_request span,仅取第一次快照。)

真实场景测试:仅遥测改动、无 TUI 界面,因此有意义的真实场景是驱动真实打包 CLI 对接 fake OpenAI server、启用文件遥测导出并配置采样参数,然后读取产品真实发出的内容。我从本 PR head 构建了 dist/cli.js 并在 tmux 中运行:导出的 span 携带全部七个 gen_ai.request.* 属性,且每个值都与 fake server 在 wire 上实际收到的一致——捕获反映的是最终请求,而非逻辑配置。(我的检查器还标记了一个 qwen-code.prompt_id 属性,但那是 daemon-tracing/loggers 代码中既有的遥测,并非本 PR 新增——PR 的"无 alias"声明针对的是七个新请求字段,它们都没有以裸名或 qwen-code.* 名泄漏。)

Qwen Code · qwen3.8-max-preview

Reviewed at f59cd16dd3dcb19a1f602c7f51018130303e76f6 · re-run with @qwen-code /triage

@qwen-code-ci-bot

qwen-code-ci-bot commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Confidence: 5/5 — clean across every stage, independently verified by a maintainer, would merge without hesitation.

The previous run deferred at 3/5 because this touches telemetry and comes from a fork — policy requires a maintainer's sign-off before auto-approving. @wenshao has since done an independent A/B E2E verification (real builds from PR head and merge-base, wire-vs-span oracle across OpenAI-compatible and Anthropic paths, 329 unit tests, 2 integration tests, typecheck, lint) and approved. The escalation is resolved.

Nothing changed in the code since the last review (still at f59cd16), and my earlier assessment stands: the approach is the one I'd have proposed independently — pure per-provider extractors plus a one-shot OTel Context observer, four one-line integrations next to the existing wire-request recording, best-effort wrapping so telemetry can never break a request. No correctness bugs, security issues, or regressions. The seven gen_ai.request.* attributes match the wire exactly, choice.count is correctly omitted at n=1, and no bare or qwen-code.* aliases leak.

LGTM, looks ready to ship. ✅

中文说明

置信度:5/5 —— 各阶段均干净,maintainer 已独立验证,毫不犹豫地合入。

上一次运行以 3/5 转交,因为此 PR 触及遥测且来自 fork——政策要求 maintainer 签字后才能自动 approve。@wenshao 此后做了独立 A/B 端到端验证(从 PR HEAD 和 merge-base 分别真实构建,OpenAI-compatible 和 Anthropic 路径的 wire-vs-span 对照,329 个单测,2 个集成测试,typecheck,lint)并已 approve。转交已解除。

代码自上次审查以来未变(仍为 f59cd16),我之前的评估不变:方案与我独立提出的一致——纯函数式 per-provider 提取器加一次性 OTel Context observer,四处紧挨现有 wire-request 记录的单行集成,best-effort 包裹使遥测绝不破坏请求。无正确性 bug、安全问题或回归。七个 gen_ai.request.* 属性与 wire 完全一致,choice.countn=1 时正确省略,无裸字段或 qwen-code.* alias 泄漏。

可以合入。✅

Qwen Code · qwen3.8-max-preview

Reviewed at f59cd16dd3dcb19a1f602c7f51018130303e76f6 · re-run with @qwen-code /triage

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Downgraded from Approve to Comment: CI still running. Reviewed.

— qwen3.7-max via Qwen Code /review

@wenshao

wenshao commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

✅ Maintainer local verification — real build + independent A/B E2E

Verified this PR by building the real CLI bundle from PR head (f59cd16) and from its merge-base (69b991a), then driving both against fake OpenAI-compatible and Anthropic servers and comparing the exported qwen-code.llm_request span attributes against the request bodies that actually hit the wire. My scenarios deliberately use different values/shapes than the PR's own test, so this is an independent oracle, not a re-run.

Verdict: merge-ready. Every one of the seven gen_ai.request.* fields is captured post-adapter and matches the wire exactly; the baseline exports none of them; and no bare/qwen-code.* aliases leak.

Wire-vs-span oracle (OpenAI-compatible)

OpenAI wire-vs-span oracle

Same settings, two bundles. Each row = value sent on the wire vs. attribute exported on the span.

Behavior checked (real run) Result
All 7 fields captured & equal to the wire body ✅ both LLM spans of the turn
max_completion_tokens → standard max_tokens ✅ (256)
Single-string stop normalized to ["STOPHERE"]
Negative / zero values preserved (presence_penalty:-2, temperature:0) ✅ not dropped
choice.count omitted when n=1 (GenAI semconv)
No sampling config → no fabricated SDK/server defaults ✅ (only the adapter-injected window max_tokens appears, which is genuinely on the wire)
HTTP 500 → retry: every physical request captured ✅ no leak, attributes intact
Baseline (before PR): same wire, exported gen_ai.request.* sampling fields none — exactly the gap this PR closes
Alias-leak scan (bare temperature/top_p/stop/… or qwen-code.*) ✅ none in any scenario

Second provider (Anthropic) + local test suite

Anthropic path + tests

The Anthropic adapter does not forward stop/penalties from samplingParams to the wire — and the extractor correctly omits them rather than inventing values, while max_tokens/temperature/top_p match the wire. Baseline exports nothing.

Everything I ran (PR head, macOS, Node 22.23.1)

  • npm ci && npm run build && npm run bundle → clean dist/cli.js (needed a full clean install; the change touches no dependency files).
  • 329 targeted core unit tests pass — gen-ai-request (23), pipeline (120), anthropic (109), gemini (14), logging (63).
  • 2/2 cli/gen-ai-telemetry integration tests pass against the real dist/cli.js.
  • npm run typecheck -w packages/core → clean; eslint on all 6 changed core files → clean.
  • Independent A/B E2E: 5 OpenAI scenarios + Anthropic, both bundles, wire-vs-span oracle.

Notes for reviewers (non-blocking)

  • Design is sound. One gen-ai-request.ts module (three pure extractors + a one-shot OTel Context observer) with four one-line report*Request(...) calls placed next to the existing runtimeDiagnostics.record*WireRequest(...) boundaries. Every extraction and setAttributes is wrapped, so telemetry can never break a provider request — confirmed by the "does not let span/context failures escape" unit tests and by the retry run.
  • Provider-internal retries are honest: the one-shot consumed flag keeps the first snapshot per logical span, while each physical request is still reported (Anthropic's 3 sites, OpenAI's thinking-retry, and the HTTP-500 retry I injected).
  • The documented "conflicting output-budget aliases → omit" path is exercised by the unit tests; in a real run the adapter's window clamp tends to equalize max_tokens/max_completion_tokens, so the extractor emitted the agreed value — consistent with the contract.
验证方法与命令(点击展开)

用于复现的隔离环境:PR HEAD f59cd16,base 为其 merge-base 69b991a

中文说明

✅ Maintainer 本地验证 —— 真实构建 + 独立 A/B 端到端

我以 真实 CLI bundle 验证了本 PR:分别从 PR HEAD(f59cd16)和其 merge-base(69b991a)执行完整 npm ci && npm run build && npm run bundle,然后用真实的 dist/cli.js 分别请求 OpenAI-compatible 和 Anthropic 的 fake server,并把导出的 qwen-code.llm_request span 属性与 实际发到 wire 的请求体 逐字段比对。我使用的取值和形态与 PR 自带测试故意不同,因此是独立验证而非复跑。

结论:可以合并。 七个 gen_ai.request.* 字段都在 adapter 处理之后被采集,且与 wire 完全一致;base 一个都不导出;没有任何裸字段或 qwen-code.* alias 泄漏。

Wire-vs-span 对照(OpenAI-compatible,见上方第一张图)

同一份配置、两个 bundle,每一行是「发到 wire 的值」对「span 导出的属性」:

  • 七个字段全部采集且与 wire 一致(本轮 LLM→Tool→LLM 的两个 LLM span 均如此);
  • max_completion_tokens → 标准 max_tokens(256);
  • 单字符串 stop 归一化为 ["STOPHERE"]
  • 负值 / 零值保留(presence_penalty:-2temperature:0),未被当作 falsy 丢弃;
  • n=1 时省略 choice.count(符合 GenAI 语义规范);
  • 未配置 sampling 时不臆造 SDK/server 默认值(仅出现 adapter 因窗口裁剪真实写到 wire 的 max_tokens);
  • HTTP 500 触发重试时,每个物理请求都被采集,属性不丢、无泄漏;
  • base(改动前):同样的 wire,却一个 gen_ai.request.* 采样字段都没导出 —— 正是本 PR 要补齐的缺口;
  • alias 泄漏扫描(裸 temperature/top_p/stop/… 或 qwen-code.*):所有场景均为

第二个 provider(Anthropic)+ 本地测试(见上方第二张图)

Anthropic adapter 不会把 samplingParams 里的 stop/penalty 发到 wire —— 提取器正确地省略而非臆造,同时 max_tokens/temperature/top_p 与 wire 一致;base 不导出任何请求字段。

我实际执行的全部内容(PR HEAD,macOS,Node 22.23.1)

  • npm ci && npm run build && npm run bundle → 干净产出 dist/cli.js(需要一次全量 clean 安装;本改动不涉及任何依赖文件);
  • 329 个 core 目标单测通过 —— gen-ai-request(23)、pipeline(120)、anthropic(109)、gemini(14)、logging(63);
  • 2/2cli/gen-ai-telemetry 集成测试基于真实 dist/cli.js 通过;
  • npm run typecheck -w packages/core 干净;对 6 个改动的 core 文件跑 eslint 干净;
  • 独立 A/B 端到端:5 个 OpenAI 场景 + Anthropic,两个 bundle,wire-vs-span 对照。

给 reviewer 的补充(非阻塞)

  • 设计合理:单个 gen-ai-request.ts(三个纯提取函数 + 一个一次性 OTel Context observer),四处单行 report*Request(...) 紧挨现有 runtimeDiagnostics.record*WireRequest(...) 边界。所有提取与 setAttributes 均被 try/catch 包裹,遥测异常不会影响模型请求 —— 由「span/context 失败不外泄」单测和我注入的重试实测共同确认。
  • provider 内部重试保持诚实consumed 一次性标志保证同一逻辑 span 只记录第一次快照,而每个物理请求仍会各自上报(Anthropic 的 3 个站点、OpenAI 的 thinking 重试、以及我注入的 HTTP 500 重试)。
  • 文档所述「输出上限别名冲突 → 省略」由单测覆盖;真实运行中 adapter 的窗口裁剪往往会把 max_tokens/max_completion_tokens 拉平,因此提取器输出了取得一致后的值 —— 与契约一致。

Verified on isolated worktrees with a clean npm ci (change touches no dependency files). Screenshots are real Playwright captures of the recorded wire/span data and test output.

@wenshao

wenshao commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

@qwen-code /triage

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, looks ready to ship. ✅

@doudouOUC
doudouOUC added this pull request to the merge queue Jul 24, 2026
Merged via the queue into QwenLM:main with commit fcd358a Jul 24, 2026
83 checks passed
@doudouOUC
doudouOUC deleted the agent/align-genai-request-telemetry branch July 24, 2026 04:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants