perf(telemetry): lazy-load the SDK and split OTLP exporter chains by protocol - #7276
Conversation
E2E Test & Benchmark ReportEnvironment: 2C4G Linux (Alibaba Cloud ECS), bundled CLI ( Stage 1 — lazy SDK facade (control = pre-change main)Default configuration (telemetry disabled), cold start, P50 over 30 pairs:
Stage 2 — protocol split (control = stage 1 build, telemetry enabled with outfile)
The cold-start gain exceeds the channel-initialize gain because on 2 cores the ~2 MiB protocol-chain module load previously contended with config construction and bootstrap for CPU; removing the load also halves config-construction time. Functional checks (same run, telemetry enabled builds)
Static guards & unit tests (macOS, this branch)
|
🩺 serve daemon A/BBuilt the PR base vs this PR head ✅ No response changes against the PR base across 4 scenario(s). — Qwen Code · serve A/B |
|
Thanks for the PR! This is a well-researched performance optimization with solid benchmarks. Template looks good ✓ Problem: Observed and well-documented. The metafile audit in the design doc identifies the 2.16 MiB telemetry cluster as the single largest coherent block in the ACP child's eager static closure. Paired benchmarks on 2C4G (30 pairs per scenario) show −144 ms P50 for the default-disabled path and −51 ms P50 for the telemetry-enabled path. This is a real, measured cost — not theoretical hardening. Linked to #4748 and #7264. Direction: Aligned. Cold start latency is a tracked priority (#4748). The two-phase lazy loading strategy is the natural next step after #7182 (TUI module removal). CHANGELOG shows performance improvements are regularly shipped in this area. However, this touches telemetry infrastructure — a sensitive area where a regression silently degrades observability. Flagging for maintainer awareness per the Stage 1c escalation rule. Size: Touches core paths (
Approach: Scope feels right. The three-way file split (facade → impl → per-protocol modules) is the minimal decomposition that achieves the stated goal. The new commit ( Moving on to code review. 🔍 中文说明感谢贡献!这是一个经过充分研究的性能优化,有可靠的基准数据。 模板完整 ✓ 问题:已观测且有充分文档。 设计文档中的 metafile 审计将 2.16 MiB 的 telemetry 集群识别为 ACP 子进程静态闭包中最大的连贯块。2C4G 上的成对基准(每场景 30 对)显示默认关闭路径 P50 降低 144 ms,开启路径 P50 降低 51 ms。这是真实可测量的成本——不是理论性加固。关联 #4748 和 #7264。 方向:对齐。 冷启动延迟是跟踪中的优先级(#4748)。两阶段懒加载策略是 #7182(TUI 模块移除)后的自然下一步。然而,此改动触及 telemetry 基础设施——一个静默回归会损害可观测性的敏感区域。按 Stage 1c 升级规则标记给维护者关注。 规模:触及核心路径(
方案:范围合理。 三层文件拆分(facade → impl → 按协议模块)是实现目标所需的最小分解。新提交( 进入代码审查 🔍 — Qwen Code · qwen3.7-max Reviewed at |
Code ReviewI read the diff against an independent proposal formed from the PR title and "Why it's needed" section alone. My independent proposal for the lazy-loading goal arrived at the same architecture: a light facade keeping New commit review ( The commit addresses three real problems introduced by the lazy-loading split:
esbuild stub Proxy fix: the
Bundle guard extension: adding explicit HTTP exporter packages and Tests: the 4 new lifecycle tests are well-targeted — single-flight init, retry after failure, shutdown/init race regression guard, and stale shutdown promise cleanup. All 63 telemetry SDK tests pass, all 6 stub tests pass, all 25 bundle guard tests pass. No critical issues. No AGENTS.md violations. The changes are minimal, well-tested, and directly address the stated problems. Real-Scenario TestingDefault path (telemetry disabled), dev build from PR head CLI starts and responds correctly. The QWEN_HOME warnings are CI environment noise (no settings.json at the CI home), unrelated to this PR. No telemetry errors, no SDK load failures. The default-disabled path works as expected. Unit test results at PR head:
中文说明代码审查我在阅读 diff 之前,仅根据 PR 标题和"为什么需要"部分形成了独立方案。我的独立方案得出了相同的架构:保持 新提交审查( 该提交解决了懒加载拆分引入的三个真实问题:
esbuild stub Proxy 修复:
Bundle 守卫扩展: 将显式 HTTP exporter 包和 测试: 4 个新生命周期测试目标明确——单飞 init、失败后重试、shutdown/init 竞态回归守卫、陈旧 shutdown promise 清理。63 个 telemetry SDK 测试全部通过,6 个 stub 测试全部通过,25 个 bundle 守卫测试全部通过。 无关键问题。无 AGENTS.md 违规。改动最小化、测试充分、直接解决所述问题。 真实场景测试默认路径(telemetry 关闭),PR head 单元测试结果:63/63、6/6、25/25 全部通过。 — Qwen Code · qwen3.7-max Reviewed at |
|
Confidence: 3/5 — clean review across every stage including the new race-fix commit, but 1565 production lines in core telemetry infrastructure and the Stage 1c telemetry escalation rule require a maintainer's sign-off before merge. Stepping back: this PR has now been through two review passes. The original lazy-loading split (facade/impl/per-protocol) was already well-engineered. The new commit ( The evidence remains strong: paired benchmarks show −144 ms P50 for the default-disabled path and −51 ms P50 for telemetry-enabled. All 63 telemetry SDK tests pass (including 4 new lifecycle race tests), all 6 stub resolve tests pass, all 25 bundle guard tests pass. The tmux headless test confirms the default path works with no telemetry errors. The new commit specifically:
If I had to maintain this in six months, I'd thank the author: the facade/impl boundary is crisp, the race semantics are well-documented in comments, the bundle guards prevent silent regression, and the test coverage is comprehensive. Why defer instead of approve: the PR touches telemetry infrastructure (Stage 1c escalation area) and totals 1565 production logic lines in core paths — well above the 500-line threshold that flags for maintainer awareness and the 1000-line large-PR advisory. These are policy-driven deferrals, not genuine concerns about the code quality or correctness. A human maintainer should confirm the architectural direction for telemetry loading and sign off on the env-var exporter stubbing behavior change (loud throw instead of silent localhost export — documented, but worth a human eye). ⏸️ Deferring to the maintainer for sign-off. The code is ready — needs a human call on the telemetry infrastructure change scope. 中文说明置信度:3/5 — 包括新竞态修复提交在内,所有阶段审查均无问题,但核心 telemetry 基础设施中 1565 行生产代码及 Stage 1c telemetry 升级规则要求维护者签字后才能合并。 回顾全局:此 PR 已经历两轮审查。原始的懒加载拆分(facade/impl/按协议)已经工程精良。新提交( 证据依然有力:成对基准显示默认关闭路径 P50 降低 144 ms,开启路径 P50 降低 51 ms。63 个 telemetry SDK 测试全部通过(含 4 个新生命周期竞态测试),6 个 stub 解析测试全部通过,25 个 bundle 守卫测试全部通过。tmux 无头测试确认默认路径正常工作且无 telemetry 错误。 新提交具体地:
如果六个月后需要维护,我会感谢作者:facade/impl 边界清晰,竞态语义在注释中有充分文档,bundle 守卫防止静默回归,测试覆盖全面。 为什么转交而非批准: PR 触及 telemetry 基础设施(Stage 1c 升级区域),核心路径共 1565 行生产逻辑——远超需维护者关注的 500 行阈值及 1000 行大 PR 建议。这是策略驱动的转交,不是对代码质量或正确性的真实担忧。人类维护者应确认 telemetry 加载的架构方向,并对环境变量 exporter 打桩的行为变更签字。 ⏸️ 转交维护者签字。代码已就绪——需要人工对 telemetry 基础设施变更范围做决定。 — Qwen Code · qwen3.7-max Reviewed at |
|
Qwen Code review paused — model quota exhausted. Qwen review stopped: the model API quota is exhausted (reset at 07-20 07:32:00 UTC.). Transient errors auto-retry, but a quota reset is too far out to wait on a runner. Re-run once it resets by commenting |
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Reviewed. 1 Suggestion-level finding(s) could not be anchored to a changed line and were dropped; nothing further to act on here. Not reviewed: chunk 1, chunk 7, chunk 4, chunk 3, chunk 5, chunk 6, chunk 8, chunk 9, chunk 2 — launched with a prompt that is not the one the CLI built. Not reviewed: Agent 0: Issue fidelity & root-cause ownership, Test coverage matrix (whole-diff), Agent 1b: Removed-behavior audit, Agent 1c: Cross-file tracer, Agent 7: Build & test verification, Invariant agent A: state, timers, collections — packages/core/src/telemetry/sdk.ts, Invariant agent B: counters, return values, error taxonomies — packages/core/src/telemetry/sdk.ts, Invariant agent C: config fields, early returns — packages/core/src/telemetry/sdk.ts — its prompt was built, but no agent on record was launched with it. Not reviewed: reverse audit — an auditor ran and opened its brief, but no agent was launched with the prompt the CLI built — the launch was written by hand, and what the agent was actually asked is not what this skill certifies.
— qwen3.7-max via Qwen Code /review
ReviewReviewed at Three things I'd like addressed before merge, then some smaller notes. 1.
|
doudouOUC
left a comment
There was a problem hiding this comment.
Reviewed. Suggestions are inline. 1 Suggestion-level finding(s) could not be anchored to a changed line and were dropped; nothing further to act on here. Not reviewed: reverse audit — an auditor ran and opened its brief, but no agent was launched with the prompt the CLI built. Not reviewed: coverage — the plan could not be used (ENOENT: no such file or directory, open '.qwen/tmp/qwen-review-pr-7276-fetch.json'), so this run cannot show that any of the diff was read. Not reviewed: verification — could not check that Step 4 and Step 5 ran (ENOENT: no such file or directory, open '.qwen/tmp/qwen-review-pr-7276-fetch.json').
— qwen3.7-max via Qwen Code /review
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Reviewed — no blockers. 2 Suggestion-level finding(s) could not be anchored to a changed line and were dropped; nothing further to act on here.
— qwen3.7-max via Qwen Code /review
…ure non-fatal Addresses PR QwenLM#7276 review feedback: shutdown now awaits an in-flight init before tearing down (was racing past the sync flag and leaking a started SDK whose buffered spans/logs never flushed); the dynamic imports now sit inside init's try so a chunk-load failure degrades telemetry instead of aborting daemon runtime startup. Also breaks the sdk<->sdk-impl import cycle via a leaf otlp-urls module, hardens the sdk-node exporter stub for thenable/interop probes with a unit-tested separator-independent resolve, lists the HTTP exporter packages explicitly in the bundle guard, and adds lazy-init lifecycle tests.
|
[qwen] Thanks for the careful review — all three pre-merge items are addressed in 1 — shutdown/init race (highest value). 2 — try/catch coverage. Both 3 — behavior change on non-deferred paths. Documented explicitly in the Phase 1 design doc: on the ACP child and headless Smaller notes: the Independent confirmation of the stub safety analysis matches mine — thank you for verifying the |
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Reviewed — no blockers. Suggestions are inline.
— qwen3.7-max via Qwen Code /review
| : logToSpanProcessor | ||
| ? [logToSpanProcessor] | ||
| : [], | ||
| ...(metricReader && { metricReader }), |
There was a problem hiding this comment.
[Suggestion] The metrics path does not follow the same empty-array defense documented for traces and logs on lines 414–421. When metricReader is undefined, the conditional spread ...(metricReader && { metricReader }) omits the key entirely from the NodeSDK config. NodeSDK then falls back to env-driven metrics exporter selection (configureMetricProviderFromEnv()), which reads OTEL_METRICS_EXPORTER and constructs exporters. In the bundled CLI, the esbuild stub makes those constructors throw — aborting sdk.start() and disabling ALL telemetry (traces, logs, metrics) for the process lifetime.
Failure scenario: a user has OTEL_METRICS_EXPORTER=otlp in their environment from an unrelated project, enables qwen-code telemetry but provides no metrics endpoint. NodeSDK reaches the env fallback, hits the stub, throws, and every subsequent isTelemetrySdkInitialized() call returns false.
| ...(metricReader && { metricReader }), | |
| metricReaders: metricReader ? [metricReader] : [], |
This makes the env fallback unreachable for metrics (same as traces and logs), leaving the stub purely a bundle-size mechanism.
— qwen3.7-max via Qwen Code /review
| await core.initializeTelemetry( | ||
| createDaemonTelemetryRuntimeConfig( |
There was a problem hiding this comment.
[Suggestion] No test verifies that initializeDaemonMetrics() runs only after initializeTelemetry resolves. The await here is the sole guard against a silent metrics blackout — if a future edit drops it (a one-word deletion), initializeDaemonMetrics() runs before the SDK's MeterProvider is registered, and metrics.getMeter() caches a noop meter permanently for the process lifetime.
The existing test at run-qwen-serve.test.ts only inspects initializeTelemetry's arguments — it never observes initializeDaemonMetrics or its ordering.
Failure scenario: one-word deletion → daemon Status metrics silently produce no data, undetected by the test suite.
Suggested fix: add a test that spies on both core.initializeTelemetry and core.initializeDaemonMetrics, makes initializeTelemetry return a deferred promise, and asserts that initializeDaemonMetrics is not called until after the telemetry promise resolves.
— qwen3.7-max via Qwen Code /review
wenshao
left a comment
There was a problem hiding this comment.
No issues found. LGTM! ✅
— qwen3.8-max-preview via Qwen Code /review
|
@qwen-code /triage |
|
⏸️ Deferring to @wenshao — this PR touches core telemetry infrastructure (Stage 1c escalation area) and totals 1565 production logic lines in core paths, well above the 500-line maintainer-awareness threshold. The code review is clean across both passes (original split + new race-fix commit 中文说明⏸️ 转交 @wenshao — 此 PR 触及核心 telemetry 基础设施(Stage 1c 升级区域),核心路径共 1565 行生产逻辑,远超 500 行维护者关注阈值。两轮代码审查(原始拆分 + 新竞态修复提交 — Qwen Code · qwen3.7-max |
|
Triage re-run completed without a new review. The stage comments above were updated with the latest result. View workflow run. |
ReviewOverviewTwo-stage lazy loading of the telemetry stack: (1) What I verified beyond the diff
Findings (all non-blocking)
Code quality
VerdictLGTM. All findings are non-blocking; #1 is the one I'd like to see addressed (even just a docs note or a 🤖 Generated with Claude Code — Claude Fable 5 |
yiliang114
left a comment
There was a problem hiding this comment.
LGTM. Clean two-stage lazy load with proper single-flight guard and shutdown/init race handling. Bundle guards lock the split against regression. Benchmarks are convincing.
|
Released in v0.20.1. |
|
[API Error: Connection error. (cause: fetch failed)] ❌ failed |
What this PR does
Makes telemetry SDK loading lazy, in two stages. First, the telemetry entry point is split into a light facade and a heavy implementation half: processes that never enable telemetry (the default) no longer parse and compile the OpenTelemetry NodeSDK, instrumentations, or any exporter — the heavy half is loaded on demand, behind a single-flight dynamic import, only when telemetry is actually enabled. Second, the OTLP exporter chains are split by protocol: the gRPC chain (including the gRPC transport stack and protobuf runtime) and the HTTP chain (including the shared OTLP serialization layer) each live in their own dynamically imported module, so a process loads at most the one chain its configuration needs — file-based telemetry output loads neither, and a misconfigured gRPC setup without an endpoint loads nothing before it skips.
Two supporting changes keep the split honest in the bundled CLI. The bundler now stubs the exporter packages that the OpenTelemetry NodeSDK eagerly requires for env-var-based auto-configuration (an unsupported configuration surface here), because those eager requires would otherwise drag both protocol chains back into the static closure; the stubs fail loudly if ever constructed. And the existing bundle-closure guard gains a third check that fails CI if either protocol chain ever becomes statically reachable from the implementation half again.
Why it's needed
Daemon cold start (#4748) paid roughly 2.1 MiB of telemetry module parse/compile cost in every ACP child process, including the default case where telemetry is disabled. On a 2C4G reference machine this was worth about 144 ms P50 (-7.5%) of process-to-first-session latency for default configurations. The protocol split then targets the remaining cost for users who do enable telemetry (#7264): both protocol chains loaded even though a configuration uses at most one, and on small machines that extra module loading contended with config construction and bootstrap on the CPU, adding back ~50 ms. With the split, telemetry-enabled cold start improved by a further ~51 ms P50 on the same reference machine, with config-construction time cut in half.
Reviewer Test Plan
How to verify
npm run build && cross-env DEV=true npm run bundle && node scripts/check-serve-fast-path-bundle.jspasses three closure checks.cd packages/core && npx vitest run src/telemetry(671 tests) andnpm run test:scripts(guard tests including six new boundary cases).OTEL_METRICS_EXPORTER=otlp(env-based exporter auto-configuration, never a supported configuration surface here) now fails loudly inside SDK start — caught and logged by the existing error handling — instead of silently exporting to a default localhost endpoint.Evidence (Before & After)
N/A (no UI change). Paired benchmark on a 2C4G Linux host, 30 pairs per scenario, P50:
Tested on
Environment (optional)
macOS: unit tests, typecheck, lint, bundle guards, and runtime smoke tests against the bundled CLI. Linux (2C4G): paired cold/preheated benchmarks against the bundled CLI.
Risk & Scope
Linked Issues
Part of #4748. Implements the first candidate (per-protocol exporter split) from #7264.
中文说明
本 PR 做了什么
将 telemetry SDK 的加载改为两级懒加载。第一级:把 telemetry 入口拆分为轻量 facade 和重量实现两半——默认不开启 telemetry 的进程(绝大多数)不再解析和编译 OpenTelemetry NodeSDK、instrumentation 及任何 exporter;重的那一半只在 telemetry 真正开启时通过单飞动态 import 按需加载。第二级:把 OTLP exporter 链按协议拆分——gRPC 链(含 gRPC 传输栈和 protobuf 运行时)与 HTTP 链(含共享的 OTLP 序列化层)各自独立成动态导入模块,进程最多只加载配置所需的那一条链:文件输出模式两条链都不加载;gRPC 缺少 endpoint 的错误配置在加载任何协议模块之前就跳过。
两项配套改动保证拆分在打包后的 CLI 中真实生效。打包器现在会对 NodeSDK 为环境变量自动配置而急切 require 的 exporter 包打桩(这从来不是本项目支持的配置面),否则这些急切 require 会把两条协议链重新拖回静态闭包;桩一旦被构造会响亮报错。同时现有的 bundle 闭包守卫新增第三项检查:若任一协议链重新静态可达实现半,CI 直接失败。
为什么需要
Daemon 冷启动(#4748)中每个 ACP 子进程都要支付约 2.1 MiB 的 telemetry 模块解析/编译成本,包括默认关闭 telemetry 的场景。在 2C4G 参考机器上,默认配置的进程到首 session 延迟因此高出约 144 ms P50(-7.5%)。协议拆分进一步针对开启 telemetry 的用户(#7264):配置最多用到一条协议链却加载了两条,且在小机器上这些额外的模块加载与 config 构建、bootstrap 抢占 CPU,额外增加约 50 ms。拆分后,同一参考机器上开启 telemetry 的冷启动再降约 51 ms P50,config 构建耗时减半。
评审验证计划
如何验证
npm run build && cross-env DEV=true npm run bundle && node scripts/check-serve-fast-path-bundle.js三项闭包检查全过。cd packages/core && npx vitest run src/telemetry(671 个)以及npm run test:scripts(守卫测试含六个新边界用例)。OTEL_METRICS_EXPORTER=otlp(基于环境变量的 exporter 自动配置,从来不是支持的配置面)现在会在 SDK 启动时响亮失败——被现有错误处理捕获并记录——而不是静默导出到默认 localhost 端点。证据(前后对比)
N/A(无 UI 变化)。2C4G Linux 机器成对基准,每场景 30 对,P50:
测试平台
macOS:单元测试、typecheck、lint、bundle 守卫、针对打包 CLI 的运行时冒烟。Linux(2C4G):针对打包 CLI 的成对冷启动/预热基准。
风险与范围
关联 Issue
属于 #4748 的一部分。实现了 #7264 中的第一个候选项(按协议拆分 exporter)。