feat(core): Align GenAI request telemetry with ARMS - #7635
Conversation
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
E2E test report
|
|
Thanks for the PR! Template looks good ✓ — all sections filled in, bilingual body, clear reviewer test plan. Problem: this is a feature completion rather than a bug fix, and the gap is concrete. Qwen Code already emits the shared GenAI/ARMS identity, response, usage, tool, and agent attributes, but omits the request-sampling and output-limit fields ( Direction: aligned. It completes OpenTelemetry GenAI semantic-convention coverage the codebase already partially implements (builds on the existing Size: 288 production logic lines / 433 test lines / 54 docs lines (775 total). Production is well under the 500-line awareness threshold, so no size concern. It does touch core paths ( Approach: the scope feels right. One new module ( Moving on to code review. 🔍 中文说明感谢贡献! 模板完整 ✓ —— 各 section 填写齐全,双语正文,reviewer test plan 清晰。 问题:这是功能补全而非 bug 修复,且缺口很具体。Qwen Code 已经上报了共享的 GenAI/ARMS identity、response、usage、tool、agent 属性,但缺少请求采样和输出上限字段( 方向:对齐。它补全了代码库已部分实现的 OpenTelemetry GenAI 语义约定覆盖(基于现有的 规模:288 行生产逻辑 / 433 行测试 / 54 行文档(共 775 行)。生产行数远低于 500 行关注阈值,无规模问题。但确实触及核心路径( 方案:范围合理。一个新模块( 进入代码审查 🔍 — Qwen Code · qwen3.8-max-preview Reviewed at |
Code reviewMy independent take before reading the diff: capture the seven GenAI request fields from the provider-final SDK request, normalize the per-provider field names ( The design is sound. No correctness bugs, security issues, or regressions found. One subtlety worth noting, not a blocker: the OpenAI Verification I ran (not just trusting the PR body): the new and modified unit tests pass locally — sequenceDiagram
participant P1 as LoggingContentGenerator
participant P2 as OTel Context
participant P3 as Provider generator
participant P4 as llm_request Span
P1->>P2: create span, attach one-shot observer
P1->>P3: activate context, call delegate
P3->>P3: build provider-final request
P3->>P2: report request, look up observer
P2->>P4: set gen_ai.request attributes, first snapshot only
Files changed (12 of 12 shown)
Real-scenario test (this PR)Telemetry-only change with no TUI surface, so the meaningful real scenario is driving the actual bundled CLI against a fake OpenAI server with the file telemetry exporter enabled and sampling params configured, then reading what the product really emitted. I built The exported span carries all seven 中文说明代码审查:我在看 diff 之前的独立想法是——从 provider 最终 SDK 请求中捕获七个 GenAI 请求字段,归一化各 provider 的字段名( 设计合理。 未发现正确性 bug、安全问题或回归。一个值得注意但非阻塞的细节:OpenAI 的 我亲自跑的验证(不只是相信 PR 正文):新增和修改的单测本地通过—— (时序图说明了请求属性如何从 provider 边界流回 span:LoggingContentGenerator 创建 span 并挂载一次性 observer 到 OTel Context,激活 context 调用 delegate,provider 生成最终请求后上报,observer 把 gen_ai.request 属性写入 llm_request span,仅取第一次快照。) 真实场景测试:仅遥测改动、无 TUI 界面,因此有意义的真实场景是驱动真实打包 CLI 对接 fake OpenAI server、启用文件遥测导出并配置采样参数,然后读取产品真实发出的内容。我从本 PR head 构建了 — Qwen Code · qwen3.8-max-preview Reviewed at |
|
Confidence: 5/5 — clean across every stage, independently verified by a maintainer, would merge without hesitation. The previous run deferred at 3/5 because this touches telemetry and comes from a fork — policy requires a maintainer's sign-off before auto-approving. @wenshao has since done an independent A/B E2E verification (real builds from PR head and merge-base, wire-vs-span oracle across OpenAI-compatible and Anthropic paths, 329 unit tests, 2 integration tests, typecheck, lint) and approved. The escalation is resolved. Nothing changed in the code since the last review (still at LGTM, looks ready to ship. ✅ 中文说明置信度:5/5 —— 各阶段均干净,maintainer 已独立验证,毫不犹豫地合入。 上一次运行以 3/5 转交,因为此 PR 触及遥测且来自 fork——政策要求 maintainer 签字后才能自动 approve。@wenshao 此后做了独立 A/B 端到端验证(从 PR HEAD 和 merge-base 分别真实构建,OpenAI-compatible 和 Anthropic 路径的 wire-vs-span 对照,329 个单测,2 个集成测试,typecheck,lint)并已 approve。转交已解除。 代码自上次审查以来未变(仍为 可以合入。✅ — Qwen Code · qwen3.8-max-preview Reviewed at |
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
— qwen3.7-max via Qwen Code /review
✅ Maintainer local verification — real build + independent A/B E2EVerified this PR by building the real CLI bundle from PR head ( Verdict: merge-ready. Every one of the seven Wire-vs-span oracle (OpenAI-compatible)Same settings, two bundles. Each row = value sent on the wire vs. attribute exported on the span.
Second provider (Anthropic) + local test suiteThe Anthropic adapter does not forward Everything I ran (PR head, macOS, Node 22.23.1)
Notes for reviewers (non-blocking)
验证方法与命令(点击展开)用于复现的隔离环境:PR HEAD 中文说明✅ Maintainer 本地验证 —— 真实构建 + 独立 A/B 端到端我以 真实 CLI bundle 验证了本 PR:分别从 PR HEAD( 结论:可以合并。 七个 Wire-vs-span 对照(OpenAI-compatible,见上方第一张图)同一份配置、两个 bundle,每一行是「发到 wire 的值」对「span 导出的属性」:
第二个 provider(Anthropic)+ 本地测试(见上方第二张图)Anthropic adapter 不会把 我实际执行的全部内容(PR HEAD,macOS,Node 22.23.1)
给 reviewer 的补充(非阻塞)
Verified on isolated worktrees with a clean |
|
@qwen-code /triage |
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship. ✅


What this PR does
This PR adds the seven request-parameter attributes shared by OpenTelemetry GenAI semantic conventions and ARMS: choice count, maximum output tokens, temperature, top-p, frequency penalty, presence penalty, and stop sequences. Values are captured from the first provider-final SDK request after adapter defaults, overrides, unsupported-field removal, and output-window clamping.
The implementation uses the existing OpenTelemetry Context to attach an internal one-shot request observer to each recording LLM span. OpenAI-compatible and Qwen, Anthropic, Gemini, and Vertex request boundaries report only normalized attributes rather than retaining full requests. Provider-internal retries and fallbacks cannot overwrite the first snapshot, and telemetry failures remain best-effort.
The GenAI/ARMS design contract and developer telemetry documentation now describe provider mappings, types, omission rules, default handling, and deferred fields. The permanent CLI integration test compares exported span attributes with the request body received by the fake OpenAI server and verifies that a choice count of one is omitted.
Why it's needed
Qwen Code already emitted the shared GenAI/ARMS identity, response, usage, tool, and agent attributes, but omitted the common request sampling and output-limit fields. Operators therefore could not correlate model behavior, failures, cost, or latency with the effective request parameters in OpenTelemetry or ARMS.
Reading the original logical configuration would be inaccurate because provider adapters can inject defaults, override or remove fields, and clamp output limits. Capturing the final SDK request gives observability the same values Qwen Code actually attempted to send without exposing prompts, messages, tools, URLs, headers, or credentials.
Reviewer Test Plan
How to verify
Configure the file telemetry exporter and an OpenAI-compatible endpoint with all seven request parameters. Run a tool-call turn and confirm that every LLM span contains standard
gen_ai.request.*attributes matching the fake server request body, thatgen_ai.request.choice.countis omitted when the request sendsn=1, and that no bare orqwen-code.*aliases are emitted.Review the provider tests to confirm that both streaming and non-streaming SDK calls report the exact finalized request object. The retry and Anthropic empty-stream fallback cases should report each physical request while the Context observer accepts only the first snapshot for the logical LLM span.
Validated locally with
npm run build,npm run bundle,npm run typecheck,npm run lint, 373 targeted core tests, and bothgen-ai-telemetryCLI integration tests.Evidence (Before & After)
N/A — telemetry and documentation changes with no TUI impact.
Tested on
Environment (optional)
macOS 26.4.1, Node.js 22.22.3, npm 10.9.8, local bundle, sandbox disabled for the CLI integration test.
Risk & Scope
gen_ai.request.*names; no private aliases or dual writes are introduced.Linked Issues
N/A
中文说明
本 PR 做了什么
本 PR 补齐 OpenTelemetry GenAI 语义规范与 ARMS 共同支持的七个请求参数属性:候选数量、最大输出 token、temperature、top-p、frequency penalty、presence penalty 和停止序列。属性值来自 provider adapter 完成默认值注入、覆盖、不支持字段删除和输出窗口裁剪后的第一份最终 SDK 请求。
实现复用现有 OpenTelemetry Context,为每个 recording LLM Span 绑定内部一次性 request observer。OpenAI-compatible 与 Qwen、Anthropic、Gemini 和 Vertex 的 SDK 调用边界只上报规范化属性,不保留完整请求。Provider 内部重试和 fallback 无法覆盖第一次快照,所有遥测异常均保持 best-effort,不影响模型调用。
GenAI/ARMS 设计契约和开发者遥测文档现已记录 provider 映射、类型、省略规则、默认值处理与延期字段。永久 CLI 集成测试会比较导出的 Span 属性和 OpenAI fake server 实际收到的请求体,并验证候选数量为一时省略该属性。
为什么需要
Qwen Code 已经输出 GenAI 与 ARMS 共同的身份、响应、usage、tool 和 agent 属性,但缺少共同的请求采样和输出限制字段。因此,运维人员无法在 OpenTelemetry 或 ARMS 中把模型行为、失败、成本或延迟与实际生效的请求参数关联起来。
读取原始逻辑配置并不准确,因为 provider adapter 可能注入默认值、覆盖或删除字段,并裁剪输出上限。采集最终 SDK 请求可以让可观测数据与 Qwen Code 实际尝试发送的值保持一致,同时不会暴露 prompt、message、tool、URL、header 或凭据。
Reviewer 测试计划
如何验证
配置 file telemetry exporter 和 OpenAI-compatible endpoint,并设置全部七个请求参数。运行一次包含工具调用的对话,确认每个 LLM Span 的标准
gen_ai.request.*属性都与 fake server 收到的请求体一致;请求发送n=1时省略gen_ai.request.choice.count;同时不产生裸字段或qwen-code.*alias。检查 provider 测试,确认流式和非流式 SDK 调用都上报完全相同的最终请求对象。内部重试和 Anthropic empty-stream fallback 应分别上报每个物理请求,而 Context observer 对同一逻辑 LLM Span 只接受第一次快照。
本地已通过
npm run build、npm run bundle、npm run typecheck、npm run lint、373 个目标 core 测试以及两个gen-ai-telemetryCLI 集成测试。证据(修改前后)
N/A——仅涉及遥测和文档,不影响 TUI。
测试平台
环境(可选)
macOS 26.4.1、Node.js 22.22.3、npm 10.9.8、本地 bundle,CLI 集成测试关闭 sandbox。
风险与范围
gen_ai.request.*名称,不新增私有 alias 或双写。关联 Issue
N/A