From d2fdeb8b52f561cda979c68ce1330e7a28115d2e Mon Sep 17 00:00:00 2001 From: jinye Date: Sat, 23 May 2026 08:04:24 +0800 Subject: [PATCH 01/97] =?UTF-8?q?perf(core):=20F2=20cleanup=20PR=20A=20?= =?UTF-8?q?=E2=80=94=20R9/W11/W12/R10=20(post-merge=20follow-ups)=20(#4411?= =?UTF-8?q?)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * refactor(core): F2 PR A R9 — McpClientManager options-object ctor R9 (filed as F2 follow-up from #4336 review): 7 positional ctor args collapse to (config, toolRegistry, options?: McpClientManagerOptions). The trailing 5 (eventEmitter, sendSdkMcpMessage, healthConfig, budgetConfig, pool) become named fields on `McpClientManagerOptions`. Test factory `mkManager(overrides?)` introduced at the top of `mcp-client-manager.test.ts` so each of the prior 80 inline constructions becomes a single line naming only the field(s) the test overrides; the 4 `undefined` sentinels each test threaded through to reach the trailing `pool` arg are gone. Net: 113 LOC removed (test) + 35 LOC added (src exposes interface + mkManager factory + tool-registry call site update). Behavior unchanged — same field assignments, same downgrade-enforce-without- budget breadcrumb, same budget event wiring. Filed bucket: F2 perf / cleanup PR A (R9 + W11 + W12 + R10/R23 T7), see issue #4175 item 7 "F2 post-merge cleanup PRs". This is the first of the 4 fixes in PR A; W11/W12/R10 follow as separate commits. Test sweep: 84/84 mcp-client-manager.test.ts pass; typecheck clean. * refactor(core): F2 PR A W11 — extract attachPooledSession + rollbackReservationOnSpawnFailure W11 (filed as F2 follow-up from #4336 review): two private helpers on `McpTransportPool` to eliminate inline duplication in `acquire()`: - `attachPooledSession(entry, id, serverName, cfg, sessionId, toolReg, promptReg)`: builds `SessionMcpView` + `entry.attach` with the standard pool release callback. Used by both the fast-path attach (existing entry) and the post-spawn attach (after `await inFlight`). NOT used by `createUnpooledConnection` — its release callback runs `entry.forceShutdown('manual')` + `indexDetach` directly (no pool refcount accounting since unpooled entries are per-session). - `rollbackReservationOnSpawnFailure(reservationResult, serverName)`: R24 T17 contract — only release the budget slot if THIS acquire actually reserved a new slot (`'reserved'`); `'already_held'` skips because the sibling owns it. Used by both the unpooled catch and the pooled spawn-in-flight catch. Race-window invariants (W10 / W77 / W90 / W111 / W125 / R24 T17) stay at the call sites because they describe the SURROUNDING ordering, not the helpers themselves. Helpers are documented to defer those decisions back to callers. Behavior unchanged. Filed bucket: F2 perf cleanup PR A (R9 done / W11 this commit / W12 + R10 to follow). Test sweep: 28/28 mcp-transport-pool.test.ts pass; typecheck clean. * refactor(core): F2 PR A W12 — SessionMcpView precompute filter Sets W12 (filed as F2 follow-up from #4336 review): `applyTools` / `applyPrompts` precompute `excludeSet` + `includeSet` once per pass instead of scanning `cfg.includeTools` / `cfg.excludeTools` arrays inside every per-tool iteration. Pre-fix the per-tool predicate (`passesSessionFilter`) walked both arrays for every snapshot entry → O(M × N) per `applyTools` call. With M tools × N filter entries, typical M=5-20 / N=2-5 case finishes in microseconds either way; the win is data-structure correctness and code clarity, not perceived perf. `passesSessionFilter` / `passesSessionPromptFilter` (the array- based predicates) stay exported and unchanged for unit tests + any caller wanting to test a single name without paying Set construction. The bulk path uses two new private helpers `compileNameFilter` + `compiledFilterAccepts` whose Sets live on the `applyTools` / `applyPrompts` stack frame. Same semantics: `excludeTools` is direct-equality match (no parens strip — pre-F2 behavior preserved); `includeTools` strips the first `(...)` suffix so `toolName(args)` matches `toolName`. Filed bucket: F2 perf cleanup PR A (R9 + W11 done / W12 this commit / R10 to follow). Test sweep: 13/13 session-mcp-view.test.ts pass; typecheck clean. * perf(core): F2 PR A R10 / R23 T7 — pid-descendants ps snapshot + pgrep fallback R10 / R23 T7 (filed as F2 follow-up from #4336 review): the Linux / macOS pid-descendant enumeration moves from per-pid `pgrep -P ` BFS (one subprocess fork per node visited) to a single `ps -A -o pid=,ppid=` snapshot followed by an in-memory tree walk over `Map`. Windows analog: single `Get-CimInstance Win32_Process | ConvertTo-Csv` snapshot of all `(ProcessId, ParentProcessId)` rows replaces per-pid `Get-CimInstance -Filter "ParentProcessId=$p"` BFS. Two motivations: 1. **Fork count**: typical `npx → tool` / `uvx → tool` wrapper trees are 2-3 levels deep with B=1-3 children per node → pre-fix BFS forked ~5-10 subprocesses per pool-shutdown call. Post-fix: exactly 1 fork regardless of tree depth. 2. **Snapshot consistency**: pre-fix BFS walked the table level by level; a child that forked between two adjacent BFS levels could be missed (we'd see the child but query its descendants AFTER the new fork). The snapshot path captures the table at one instant; new descendants forked after the snapshot are tolerated by the existing ESRCH-tolerant SIGTERM loop. Caveats: - `ps -A -o pid=,ppid=` is POSIX standard (macOS / Linux / *BSD), but BusyBox `ps` ` add. `root` seeded into `visited` so a malformed snapshot listing root as a descendant of its own child doesn't re-enqueue root either. PR-A-R2 #2 (session-mcp-view.ts:117 — predicate dedup): After W12, the exported `passesSessionFilter` / `passesSessionPromptFilter` still called `passesNameFilter` (the pre-W12 array-based implementation), while `applyTools` / `applyPrompts` used `compiledFilterAccepts(compileNameFilter(...))`. Two parallel implementations of the same predicate — future change to one without the other would silently diverge: - the exported function's tests (passesSessionFilter unit tests) would still pass - the production filter path in applyTools/applyPrompts would behave differently Reviewer also noted `passesSessionPromptFilter` had zero callers in production code or tests after W12 — `applyPrompts` no longer references it. Kept the export rather than deleting it (matches the `passesSessionFilter` shape for symmetry + the F3 audit-path comment block earmarks both as the replay predicates), but routed both through `compiledFilterAccepts(compileNameFilter(...))` so there is a single source of truth. Set construction is per-call for these exports (negligible for unit-test / one-off probes); the bulk paths in `applyTools` / `applyPrompts` still construct ONE filter per pass via the original W12 code path. `passesNameFilter` (the standalone array-based helper) deleted — its only callers were the two exports, which now use the compiled path. Public-API surface unchanged: the two exported functions keep their signatures and semantics. Test sweep: 19/19 pid-descendants + session-mcp-view tests pass; typecheck + ESLint clean. Continues commit chain: f05917071 (R9) → 20d2f1b90 (W11) → 6cf18f641 (W12) → 2a41c6fae (R10) → this (R2 followups). * fix(core): F2 PR A R3 T3 — Windows CSV delimiter locale fix `ConvertTo-Csv -NoTypeInformation` honors the system locale's list separator on PowerShell 5.1. On German / French / Dutch / Italian / ... locales the separator is `;` not `,`, so the regex `^"(\d+)","(\d+)"$` in `snapshotProcessTreeWin` never matched → `parsedRows === 0` → snapshot threw → fell back to the per-pid CIM filter path with ~0.5-1s extra PowerShell startup latency per descendant on every pool shutdown. Fix: 1-LOC `-Delimiter ","` on `ConvertTo-Csv`. Forces comma regardless of locale or PowerShell version. PowerShell 7+ defaults to comma already; 5.1 (the Windows-bundled version most users have without explicit upgrade) honored locale. The explicit delimiter makes both consistent. Skipped wenshao's companion Suggestion T4 (test coverage for walkDescendants MAX_DESCENDANTS / MAX_DEPTH caps) as F2 hardening follow-up — the caps are simple 2-line guards exercisable by inspection; ~50 LOC of mock infrastructure isn't commensurate with the regression risk on currently-stable defensive code, and (per the issue #4175 follow-up bucket) we keep dedicated test-coverage work out of perf-cleanup PRs. Continues commit chain: f05917071 (R9) → 20d2f1b90 (W11) → 6cf18f641 (W12) → 2a41c6fae (R10) → ced5d62b0 (R2) → this (R3 T3). Test sweep: 6/6 pid-descendants tests pass; typecheck + ESLint clean. --- .gitattributes | 3 - .github/pull_request_template.md | 93 +- .github/release.yml | 6 - .github/workflows/release.yml | 27 +- .github/workflows/sync-release-to-oss.yml | 238 - .gitignore | 3 - .qwen/commands/qc/create-pr.md | 13 +- .qwen/skills/memory-leak-debug/SKILL.md | 161 - ...act-reconciler-performance-measure-leak.md | 65 - .../scripts/find-leaf-node.sh | 16 - AGENTS.md | 30 +- .../2026-05-15-async-memory-recall-design.md | 206 - .../auto-compaction-threshold-redesign.md | 436 -- docs/design/f2-mcp-transport-pool.md | 1457 ++++++ .../telemetry-llm-request-timing-design.md | 538 -- .../telemetry-outbound-propagation-design.md | 878 ---- .../telemetry-resource-attributes-design.md | 762 --- docs/design/worktree.md | 129 +- docs/developers/daemon-client-adapters/tui.md | 96 - .../daemon-client-adapters/web-ui.md | 118 + docs/developers/development/telemetry.md | 230 - docs/developers/qwen-serve-protocol.md | 79 +- docs/developers/sdk-java.md | 2 +- docs/developers/sdk-typescript.md | 2 +- docs/developers/tools/_meta.ts | 1 - docs/developers/tools/file-system.md | 75 +- docs/developers/tools/introduction.md | 1 - docs/developers/tools/monitor.md | 154 - ...2026-05-18-qwen-memory-benchmark-report.md | 286 -- .../2026-05-19-oom-reproduction-report.md | 437 -- ...en-runtime-diagnostics-benchmark-report.md | 904 ---- ...-0.15.11-default-heap-oom-stress-report.md | 338 -- docs/e2e-tests/worktree-phase-c.md | 594 --- ...5-14-auto-compaction-threshold-redesign.md | 1752 ------- ...05-18-qwen-runtime-memory-investigation.md | 240 - .../plans/2026-05-15-worktree-phase-c.md | 1232 ----- docs/users/configuration/settings.md | 86 +- docs/users/features/_meta.ts | 1 - docs/users/features/approval-mode.md | 121 +- docs/users/features/auto-mode.md | 263 - docs/users/features/commands.md | 21 +- docs/users/features/headless.md | 32 - docs/users/features/memory.md | 27 +- docs/users/features/skills.md | 3 - docs/users/features/sub-agents.md | 52 +- .../abort-controller-refactor/README.md | 120 - .../automated-results.md | 139 - .../listener-accumulation-repro.mjs | 107 - .../migration-completeness.txt | 28 - .../scripts/02-lite.sh | 23 - .../scripts/06-headless-sigint.sh | 18 - .../abort-controller-refactor/smoke-boot.log | 2 - .../test-summary.txt | 10 - esbuild.config.js | 5 - eslint.config.js | 9 +- integration-tests/cli/notebook-edit.test.ts | 239 - .../sdk-typescript/system-control.test.ts | 12 +- package-lock.json | 1749 +++++-- package.json | 6 +- packages/acp-bridge/README.md | 99 +- packages/acp-bridge/package.json | 20 +- packages/acp-bridge/src/bridge.ts | 3757 ++++++++++++++ packages/acp-bridge/src/bridgeClient.test.ts | 446 ++ packages/acp-bridge/src/bridgeClient.ts | 937 ++++ packages/acp-bridge/src/bridgeErrors.ts | 163 + packages/acp-bridge/src/bridgeFileSystem.ts | 97 + packages/acp-bridge/src/bridgeOptions.ts | 81 + packages/acp-bridge/src/bridgeTypes.ts | 51 +- packages/acp-bridge/src/channel.ts | 13 +- packages/acp-bridge/src/eventBus.test.ts | 145 +- packages/acp-bridge/src/eventBus.ts | 44 + packages/acp-bridge/src/index.ts | 5 + .../acp-bridge/src/internal/stderrLine.ts | 30 + packages/acp-bridge/src/permission.ts | 32 +- .../acp-bridge/src/permissionMediator.test.ts | 1215 +++++ packages/acp-bridge/src/permissionMediator.ts | 1291 +++++ packages/acp-bridge/src/spawnChannel.test.ts | 154 + packages/acp-bridge/src/spawnChannel.ts | 344 ++ packages/acp-bridge/src/status.ts | 32 + packages/channels/base/package.json | 2 +- packages/channels/dingtalk/package.json | 2 +- packages/channels/plugin-example/package.json | 2 +- packages/channels/telegram/package.json | 2 +- packages/channels/weixin/package.json | 2 +- packages/channels/weixin/src/media.ts | 4 +- packages/channels/weixin/src/send.test.ts | 50 +- packages/channels/weixin/src/send.ts | 35 +- packages/cli/package.json | 4 +- .../cli/src/acp-integration/acpAgent.test.ts | 33 + packages/cli/src/acp-integration/acpAgent.ts | 791 ++- .../acp-integration/acpAgent.worktree.test.ts | 394 -- .../session/HistoryReplayer.test.ts | 7 + .../src/acp-integration/session/Session.ts | 123 +- .../session/Session.worktree.test.ts | 260 - .../session/emitters/ToolCallEmitter.test.ts | 148 +- .../session/emitters/ToolCallEmitter.ts | 61 + .../cli/src/acp-integration/session/types.ts | 7 +- packages/cli/src/auth/allProviders.ts | 102 + .../install/applyProviderInstallPlan.test.ts | 370 ++ .../auth/install/applyProviderInstallPlan.ts | 180 + packages/cli/src/auth/providerConfig.test.ts | 489 ++ packages/cli/src/auth/providerConfig.ts | 450 ++ .../providers/alibaba/alibabaStandard.test.ts | 124 + .../auth/providers/alibaba/alibabaStandard.ts | 62 + .../auth/providers/alibaba/codingPlan.test.ts | 79 + .../src/auth/providers/alibaba/codingPlan.ts | 97 + .../auth/providers/alibaba/tokenPlan.test.ts | 80 + .../src/auth/providers/alibaba/tokenPlan.ts | 48 + .../providers/custom/customProvider.test.ts | 118 + .../auth/providers/custom/customProvider.ts | 45 + .../auth/providers/oauth/openrouter.test.ts | 83 + .../src/auth/providers/oauth/openrouter.ts | 52 + .../providers/oauth/openrouterOAuth.test.ts | 774 +++ .../auth/providers/oauth/openrouterOAuth.ts | 734 +++ .../providers/thirdParty/deepseek.test.ts | 56 + .../src/auth/providers/thirdParty/deepseek.ts | 31 + .../auth/providers/thirdParty/idealab.test.ts | 73 + .../src/auth/providers/thirdParty/idealab.ts | 48 + .../auth/providers/thirdParty/minimax.test.ts | 43 + .../src/auth/providers/thirdParty/minimax.ts | 40 + .../providers/thirdParty/modelscope.test.ts | 60 + .../auth/providers/thirdParty/modelscope.ts | 40 + .../src/auth/providers/thirdParty/zai.test.ts | 65 + .../cli/src/auth/providers/thirdParty/zai.ts | 39 + packages/cli/src/auth/types.ts | 64 + .../cli/src/commands/extensions/utils.test.ts | 37 +- packages/cli/src/commands/extensions/utils.ts | 8 +- packages/cli/src/commands/serve.ts | 38 +- packages/cli/src/config/config.test.ts | 6 +- packages/cli/src/config/config.ts | 113 +- .../src/config/loadedSettingsAdapter.test.ts | 186 - .../cli/src/config/loadedSettingsAdapter.ts | 108 - .../src/config/normalizeDisabledTools.test.ts | 119 + .../cli/src/config/normalizeDisabledTools.ts | 63 + packages/cli/src/config/settings.test.ts | 116 - packages/cli/src/config/settings.ts | 16 +- packages/cli/src/config/settingsSchema.ts | 220 +- packages/cli/src/gemini.test.tsx | 14 +- packages/cli/src/gemini.tsx | 34 +- packages/cli/src/i18n/locales/ca.js | 9 +- packages/cli/src/i18n/locales/de.js | 9 +- packages/cli/src/i18n/locales/en.js | 11 +- packages/cli/src/i18n/locales/fr.js | 9 +- packages/cli/src/i18n/locales/ja.js | 9 +- packages/cli/src/i18n/locales/pt.js | 9 +- packages/cli/src/i18n/locales/ru.js | 9 +- packages/cli/src/i18n/locales/zh-TW.js | 9 +- packages/cli/src/i18n/locales/zh.js | 9 +- .../nonInteractive/control/ControlContext.ts | 2 - .../controllers/permissionController.test.ts | 186 - .../controllers/permissionController.ts | 6 +- .../controllers/systemController.test.ts | 207 - .../control/controllers/systemController.ts | 18 - .../io/StreamJsonOutputAdapter.test.ts | 82 - .../io/StreamJsonOutputAdapter.ts | 20 - packages/cli/src/nonInteractive/types.ts | 14 +- packages/cli/src/nonInteractiveCli.test.ts | 188 - packages/cli/src/nonInteractiveCli.ts | 196 +- .../cli/src/nonInteractiveCliCommands.test.ts | 92 - .../cli/src/serve/auth/deviceFlow.test.ts | 353 +- packages/cli/src/serve/auth/deviceFlow.ts | 306 +- .../serve/auth/qwenDeviceFlowProvider.test.ts | 120 + .../src/serve/auth/qwenDeviceFlowProvider.ts | 29 +- .../src/serve/bridgeFileSystemAdapter.test.ts | 503 ++ .../cli/src/serve/bridgeFileSystemAdapter.ts | 147 + packages/cli/src/serve/capabilities.ts | 48 +- .../src/serve/fs/workspaceFileSystem.test.ts | 203 + .../cli/src/serve/fs/workspaceFileSystem.ts | 196 +- packages/cli/src/serve/httpAcpBridge.test.ts | 784 ++- packages/cli/src/serve/httpAcpBridge.ts | 4343 +---------------- packages/cli/src/serve/index.ts | 11 + .../cli/src/serve/permissionAudit.test.ts | 231 + packages/cli/src/serve/permissionAudit.ts | 262 + packages/cli/src/serve/runQwenServe.test.ts | 175 + packages/cli/src/serve/runQwenServe.ts | 317 +- packages/cli/src/serve/server.test.ts | 561 ++- packages/cli/src/serve/server.ts | 537 +- packages/cli/src/serve/types.ts | 37 + .../cli/src/services/BuiltinCommandLoader.ts | 2 + .../src/services/BundledSkillLoader.test.ts | 14 - .../src/services/SkillCommandLoader.test.ts | 15 - packages/cli/src/services/tips/index.ts | 1 + .../cli/src/services/tips/tipRegistry.test.ts | 92 - packages/cli/src/services/tips/tipRegistry.ts | 36 +- packages/cli/src/ui/AppContainer.test.tsx | 102 +- packages/cli/src/ui/AppContainer.tsx | 267 +- packages/cli/src/ui/auth/AuthDialog.test.tsx | 120 +- packages/cli/src/ui/auth/AuthDialog.tsx | 94 +- .../src/ui/auth/ProviderSetupSteps.test.tsx | 1 - .../cli/src/ui/auth/ProviderSetupSteps.tsx | 9 +- packages/cli/src/ui/auth/useAuth.test.ts | 503 +- packages/cli/src/ui/auth/useAuth.ts | 365 +- .../cli/src/ui/auth/useProviderSetupFlow.ts | 60 +- .../cli/src/ui/commands/aboutCommand.test.ts | 56 - packages/cli/src/ui/commands/aboutCommand.ts | 23 - .../src/ui/commands/approvalModeCommand.ts | 25 +- .../cli/src/ui/commands/authCommand.test.ts | 5 +- packages/cli/src/ui/commands/authCommand.ts | 4 +- .../src/ui/commands/contextCommand.test.ts | 172 +- .../cli/src/ui/commands/contextCommand.ts | 95 +- .../cli/src/ui/commands/diffCommand.test.ts | 74 +- packages/cli/src/ui/commands/diffCommand.ts | 69 +- .../src/ui/commands/directoryCommand.test.tsx | 120 +- .../cli/src/ui/commands/directoryCommand.tsx | 9 +- .../cli/src/ui/commands/doctorCommand.test.ts | 24 +- .../src/ui/commands/extensionsCommand.test.ts | 45 - .../cli/src/ui/commands/extensionsCommand.ts | 10 +- .../cli/src/ui/commands/goalCommand.test.ts | 18 +- packages/cli/src/ui/commands/goalCommand.ts | 5 +- .../ui/commands/manageModelsCommand.test.ts | 38 + .../src/ui/commands/manageModelsCommand.ts | 24 + packages/cli/src/ui/commands/recapCommand.ts | 4 +- .../cli/src/ui/commands/skillsCommand.test.ts | 66 - packages/cli/src/ui/commands/skillsCommand.ts | 15 +- packages/cli/src/ui/commands/types.ts | 6 +- packages/cli/src/ui/components/AppHeader.tsx | 8 +- .../src/ui/components/AutoAcceptIndicator.tsx | 5 - packages/cli/src/ui/components/Composer.tsx | 20 +- .../cli/src/ui/components/DialogManager.tsx | 36 +- packages/cli/src/ui/components/DiffDialog.tsx | 732 --- .../components/ExternalAuthProgress.test.tsx | 4 +- packages/cli/src/ui/components/Footer.tsx | 21 - packages/cli/src/ui/components/Help.test.tsx | 38 - .../src/ui/components/InputPrompt.test.tsx | 208 - .../cli/src/ui/components/InputPrompt.tsx | 124 +- .../src/ui/components/MainContent.test.tsx | 1 + .../ui/components/ManageModelsDialog.test.tsx | 263 + .../src/ui/components/ManageModelsDialog.tsx | 656 +++ .../src/ui/components/SuggestionsDisplay.tsx | 2 - packages/cli/src/ui/components/Tips.test.ts | 19 +- .../ui/components/WorktreeExitDialog.test.tsx | 52 - .../src/ui/components/WorktreeExitDialog.tsx | 256 - .../components/agent-view/AgentComposer.tsx | 3 - .../agent-view/AgentTabBar.test.tsx | 6 +- .../ui/components/agent-view/AgentTabBar.tsx | 7 +- .../background-view/BackgroundTasksDialog.tsx | 16 +- .../background-view/LiveAgentPanel.test.tsx | 22 +- .../background-view/LiveAgentPanel.tsx | 84 +- .../extensions/steps/ExtensionDetailStep.tsx | 7 +- .../InlineParallelAgentsDisplay.test.tsx | 263 - .../messages/InlineParallelAgentsDisplay.tsx | 322 -- .../messages/ToolGroupMessage.test.tsx | 9 - .../components/messages/ToolGroupMessage.tsx | 31 +- .../ui/components/shared/TextInput.test.tsx | 2 - .../src/ui/components/shared/TextInput.tsx | 10 +- .../text-buffer-external-editor.test.ts | 890 ---- .../src/ui/components/shared/text-buffer.ts | 422 +- .../ui/components/views/ContextUsage.test.tsx | 135 - .../src/ui/components/views/ContextUsage.tsx | 113 +- .../ui/contexts/BackgroundTaskViewContext.tsx | 55 +- .../cli/src/ui/contexts/UIActionsContext.tsx | 11 +- .../cli/src/ui/contexts/UIStateContext.tsx | 19 +- .../src/ui/daemon/DaemonTuiAdapter.test.ts | 969 ---- .../cli/src/ui/daemon/DaemonTuiAdapter.ts | 905 ---- .../ui/hooks/slashCommandProcessor.test.ts | 1 + .../cli/src/ui/hooks/slashCommandProcessor.ts | 8 +- .../cli/src/ui/hooks/useAtCompletion.test.ts | 9 - packages/cli/src/ui/hooks/useAtCompletion.ts | 4 - .../ui/hooks/useAutoAcceptIndicator.test.ts | 19 +- .../src/ui/hooks/useAutoAcceptIndicator.ts | 93 +- .../cli/src/ui/hooks/useAwaySummary.test.ts | 10 +- packages/cli/src/ui/hooks/useAwaySummary.ts | 2 +- .../src/ui/hooks/useCommandCompletion.test.ts | 115 - .../cli/src/ui/hooks/useCommandCompletion.tsx | 5 +- .../cli/src/ui/hooks/useContextualTips.ts | 7 +- packages/cli/src/ui/hooks/useDialogClose.ts | 45 +- packages/cli/src/ui/hooks/useDiffData.ts | 87 - .../cli/src/ui/hooks/useFeedbackDialog.ts | 17 +- .../cli/src/ui/hooks/useGeminiStream.test.tsx | 664 --- packages/cli/src/ui/hooks/useGeminiStream.ts | 88 +- .../ui/hooks/useManageModelsCommand.test.ts | 40 + .../src/ui/hooks/useManageModelsCommand.ts | 32 + .../cli/src/ui/hooks/usePreferredEditor.ts | 36 - .../src/ui/hooks/useProviderUpdates.test.ts | 10 +- .../cli/src/ui/hooks/useProviderUpdates.ts | 27 +- .../src/ui/hooks/useSlashCompletion.test.ts | 35 - .../cli/src/ui/hooks/useSlashCompletion.ts | 1 - packages/cli/src/ui/hooks/useStatusLine.ts | 32 +- .../cli/src/ui/hooks/useTurnDiffs.test.ts | 194 - packages/cli/src/ui/hooks/useTurnDiffs.ts | 157 - .../src/ui/hooks/useWorktreeSession.test.tsx | 113 - .../cli/src/ui/hooks/useWorktreeSession.ts | 99 - .../src/ui/manageModels/manageModels.test.ts | 140 + .../cli/src/ui/manageModels/manageModels.ts | 211 + packages/cli/src/ui/types.ts | 25 - packages/cli/src/ui/utils/historyMapping.ts | 10 +- packages/cli/src/ui/utils/osc8.test.ts | 29 +- packages/cli/src/ui/utils/osc8.ts | 12 +- packages/cli/src/ui/utils/textUtils.test.ts | 53 - packages/cli/src/ui/utils/textUtils.ts | 47 - packages/cli/src/utils/apiPreconnect.test.ts | 17 +- packages/cli/src/utils/apiPreconnect.ts | 14 +- packages/cli/src/utils/commentJson.test.ts | 64 - packages/cli/src/utils/commentJson.ts | 50 +- packages/cli/src/utils/doctorChecks.test.ts | 11 +- packages/cli/src/utils/doctorChecks.ts | 6 +- packages/cli/src/utils/errors.ts | 30 - .../src/utils/headlessSafetyWarnings.test.ts | 96 - .../cli/src/utils/headlessSafetyWarnings.ts | 48 - packages/cli/src/utils/runBudget.test.ts | 227 - packages/cli/src/utils/runBudget.ts | 305 -- packages/cli/src/utils/sessionPaths.test.ts | 316 -- packages/cli/src/utils/sessionPaths.ts | 189 - packages/cli/src/utils/settingsUtils.test.ts | 50 - packages/cli/src/utils/settingsUtils.ts | 35 +- packages/cli/src/utils/systemInfoFields.ts | 6 +- packages/cli/src/utils/warningHandler.test.ts | 253 - packages/cli/src/utils/warningHandler.ts | 107 - packages/core/package.json | 3 +- .../agents/background-agent-resume.test.ts | 138 - .../src/agents/background-agent-resume.ts | 60 +- .../core/src/agents/background-tasks.test.ts | 101 - packages/core/src/agents/background-tasks.ts | 71 - .../core/src/agents/runtime/agent-core.ts | 476 +- .../core/src/agents/runtime/agent-headless.ts | 216 +- .../src/agents/runtime/agent-interactive.ts | 25 +- packages/core/src/config/config.test.ts | 354 +- packages/core/src/config/config.ts | 390 +- .../core/src/confirmation-bus/message-bus.ts | 2 +- .../core/__snapshots__/prompts.test.ts.snap | 95 +- .../anthropicContentGenerator.test.ts | 2 - .../anthropicContentGenerator.ts | 3 - packages/core/src/core/baseLlmClient.test.ts | 70 +- packages/core/src/core/baseLlmClient.ts | 44 +- packages/core/src/core/client.test.ts | 592 +-- packages/core/src/core/client.ts | 446 +- .../core/src/core/contentGenerator.test.ts | 4 - .../core/src/core/coreToolScheduler.test.ts | 1862 +------ packages/core/src/core/coreToolScheduler.ts | 890 +--- packages/core/src/core/geminiChat.test.ts | 3793 ++------------ packages/core/src/core/geminiChat.ts | 1539 +----- .../core/geminiContentGenerator/index.test.ts | 2 - .../loggingContentGenerator.test.ts | 152 +- .../loggingContentGenerator.ts | 34 +- .../streamContentDetection.test.ts | 131 - .../streamContentDetection.ts | 57 - .../src/core/openaiContentGenerator/index.ts | 6 - .../openaiContentGenerator/pipeline.test.ts | 85 - .../core/openaiContentGenerator/pipeline.ts | 29 +- .../provider/dashscope.test.ts | 14 - .../provider/dashscope.ts | 14 +- .../provider/deepseek.ts | 28 +- .../provider/default.test.ts | 81 - .../provider/default.ts | 36 - .../openaiContentGenerator/provider/index.ts | 1 - .../provider/mimo.test.ts | 184 - .../openaiContentGenerator/provider/mimo.ts | 69 - .../openaiContentGenerator/provider/types.ts | 5 - .../openaiContentGenerator/provider/utils.ts | 29 - .../core/src/core/permission-helpers.test.ts | 15 - packages/core/src/core/permission-helpers.ts | 9 +- packages/core/src/core/prompts.ts | 11 +- .../core/src/core/toolHookTriggers.test.ts | 104 +- packages/core/src/core/toolHookTriggers.ts | 79 +- packages/core/src/core/turn.ts | 13 - .../src/extension/claude-converter.test.ts | 157 +- .../core/src/extension/claude-converter.ts | 55 +- .../core/src/extension/extensionManager.ts | 10 +- packages/core/src/extension/github.test.ts | 102 - packages/core/src/extension/github.ts | 35 +- packages/core/src/extension/index.ts | 1 - packages/core/src/extension/marketplace.ts | 3 +- packages/core/src/extension/npm.test.ts | 57 - packages/core/src/extension/npm.ts | 15 +- packages/core/src/extension/redaction.test.ts | 77 - packages/core/src/extension/redaction.ts | 22 - packages/core/src/followup/speculation.ts | 333 +- .../src/followup/speculationToolGate.test.ts | 20 - .../core/src/followup/speculationToolGate.ts | 1 - .../src/followup/suggestionGenerator.test.ts | 112 +- .../core/src/followup/suggestionGenerator.ts | 4 +- .../src/hooks/combinedAbortSignal.test.ts | 111 + .../core/src/hooks/combinedAbortSignal.ts | 57 + packages/core/src/hooks/functionHookRunner.ts | 2 +- .../core/src/hooks/hookEventHandler.test.ts | 233 - packages/core/src/hooks/hookEventHandler.ts | 20 +- packages/core/src/hooks/hookPlanner.test.ts | 65 +- packages/core/src/hooks/hookPlanner.ts | 142 +- packages/core/src/hooks/hookRunner.ts | 2 +- packages/core/src/hooks/httpHookRunner.ts | 6 +- packages/core/src/hooks/promptHookRunner.ts | 23 +- packages/core/src/hooks/types.ts | 1 - packages/core/src/index.ts | 43 +- packages/core/src/memory/const.ts | 37 +- packages/core/src/memory/recall.ts | 27 +- packages/core/src/memory/relevanceSelector.ts | 12 +- .../memory/skillReviewAgentPlanner.test.ts | 353 -- .../src/memory/skillReviewAgentPlanner.ts | 103 +- .../src/models/modelConfigResolver.test.ts | 108 - .../core/src/models/modelConfigResolver.ts | 20 +- packages/core/src/models/modelsConfig.test.ts | 121 +- packages/core/src/models/modelsConfig.ts | 63 +- .../core/src/permissions/autoMode.test.ts | 406 -- packages/core/src/permissions/autoMode.ts | 356 -- .../classifier-prompts/system-prompt.test.ts | 182 - .../classifier-prompts/system-prompt.ts | 172 - .../permissions/classifier-transcript.test.ts | 369 -- .../src/permissions/classifier-transcript.ts | 206 - .../core/src/permissions/classifier.test.ts | 268 - packages/core/src/permissions/classifier.ts | 345 -- .../src/permissions/dangerousRules.test.ts | 370 -- .../core/src/permissions/dangerousRules.ts | 253 - .../src/permissions/denialTracking.test.ts | 207 - .../core/src/permissions/denialTracking.ts | 166 - packages/core/src/permissions/index.ts | 26 - .../permissions/permission-manager.test.ts | 194 +- .../src/permissions/permission-manager.ts | 153 - packages/core/src/permissions/rule-parser.ts | 15 +- .../src/providers/__tests__/install.test.ts | 551 --- .../__tests__/presets/openrouter.test.ts | 64 - packages/core/src/providers/index.ts | 79 - packages/core/src/providers/install.ts | 290 -- .../src/providers/presets/custom-provider.ts | 123 - .../core/src/providers/presets/openrouter.ts | 42 - packages/core/src/providers/types.ts | 198 - .../services/chatCompressionService.test.ts | 805 +-- .../src/services/chatCompressionService.ts | 281 +- .../core/src/services/chatRecordingService.ts | 7 +- .../src/services/compactionInputSlimming.ts | 8 +- .../src/services/fileHistoryService.test.ts | 320 +- .../core/src/services/fileHistoryService.ts | 501 +- .../core/src/services/fileReadCache.test.ts | 11 - packages/core/src/services/fileReadCache.ts | 25 +- .../gitWorktreeService.hooks.integ.test.ts | 100 - .../core/src/services/gitWorktreeService.ts | 95 - packages/core/src/services/sessionRecap.ts | 14 +- .../core/src/services/sessionService.test.ts | 51 - packages/core/src/services/sessionService.ts | 43 +- packages/core/src/services/sessionTitle.ts | 4 +- .../core/src/services/tokenEstimation.test.ts | 91 - packages/core/src/services/tokenEstimation.ts | 78 - .../core/src/services/toolUseSummary.test.ts | 19 + packages/core/src/services/toolUseSummary.ts | 15 +- .../services/worktreeSessionService.test.ts | 226 - .../src/services/worktreeSessionService.ts | 250 - .../core/src/skills/bundled/review/SKILL.md | 15 +- packages/core/src/skills/index.ts | 3 - packages/core/src/skills/skill-load.test.ts | 157 - packages/core/src/skills/skill-load.ts | 72 - .../core/src/skills/skill-manager.test.ts | 151 - packages/core/src/skills/skill-manager.ts | 41 +- packages/core/src/skills/types.ts | 12 +- .../src/subagents/model-selection.test.ts | 77 + .../core/src/subagents/model-selection.ts | 69 + .../subagent-manager-override.test.ts | 2 +- .../src/subagents/subagent-manager.test.ts | 72 +- .../core/src/subagents/subagent-manager.ts | 62 +- packages/core/src/subagents/types.ts | 4 +- packages/core/src/subagents/validation.ts | 4 +- packages/core/src/telemetry/config.test.ts | 184 - packages/core/src/telemetry/config.ts | 54 - packages/core/src/telemetry/constants.ts | 4 - packages/core/src/telemetry/index.ts | 10 - packages/core/src/telemetry/metrics.test.ts | 97 +- packages/core/src/telemetry/metrics.ts | 16 +- .../src/telemetry/resource-attributes.test.ts | 272 -- .../core/src/telemetry/resource-attributes.ts | 175 - packages/core/src/telemetry/sdk.test.ts | 630 --- packages/core/src/telemetry/sdk.ts | 249 +- .../src/telemetry/session-tracing.test.ts | 543 --- .../core/src/telemetry/session-tracing.ts | 469 +- .../src/tools/agent/agent-override.test.ts | 17 +- packages/core/src/tools/agent/agent.test.ts | 121 +- packages/core/src/tools/agent/agent.ts | 296 +- packages/core/src/tools/cron-create.ts | 31 - packages/core/src/tools/edit.test.ts | 1 - packages/core/src/tools/edit.ts | 23 - .../enter-worktree.session.integ.test.ts | 103 - packages/core/src/tools/enter-worktree.ts | 40 - .../tools/exit-worktree.session.integ.test.ts | 156 - packages/core/src/tools/exit-worktree.test.ts | 5 - packages/core/src/tools/exit-worktree.ts | 39 - .../core/src/tools/mcp-client-manager.test.ts | 1306 +++-- packages/core/src/tools/mcp-client-manager.ts | 505 +- packages/core/src/tools/mcp-client.test.ts | 383 +- packages/core/src/tools/mcp-client.ts | 194 +- .../src/tools/mcp-discovery-timeout.test.ts | 139 + .../core/src/tools/mcp-discovery-timeout.ts | 90 + packages/core/src/tools/mcp-pool-entry.ts | 1172 +++++ packages/core/src/tools/mcp-pool-events.ts | 149 + packages/core/src/tools/mcp-pool-key.test.ts | 293 ++ packages/core/src/tools/mcp-pool-key.ts | 182 + packages/core/src/tools/mcp-tool.ts | 35 + .../core/src/tools/mcp-transport-pool.test.ts | 1068 ++++ packages/core/src/tools/mcp-transport-pool.ts | 1419 ++++++ .../src/tools/mcp-workspace-budget.test.ts | 241 + .../core/src/tools/mcp-workspace-budget.ts | 355 ++ packages/core/src/tools/monitor.ts | 15 - packages/core/src/tools/notebook-edit.test.ts | 870 ---- packages/core/src/tools/notebook-edit.ts | 931 ---- .../core/src/tools/pid-descendants.test.ts | 98 + packages/core/src/tools/pid-descendants.ts | 394 ++ .../core/src/tools/priorReadEnforcement.ts | 7 +- packages/core/src/tools/read-file.test.ts | 57 +- packages/core/src/tools/read-file.ts | 34 +- packages/core/src/tools/send-message.ts | 26 - .../core/src/tools/session-mcp-view.test.ts | 314 ++ packages/core/src/tools/session-mcp-view.ts | 293 ++ packages/core/src/tools/shell.ts | 10 - packages/core/src/tools/skill.test.ts | 45 +- packages/core/src/tools/skill.ts | 17 - .../src/tools/toAutoClassifierInput.test.ts | 198 - packages/core/src/tools/tool-error.ts | 25 +- packages/core/src/tools/tool-names.ts | 2 - packages/core/src/tools/tool-registry.ts | 15 +- packages/core/src/tools/tools.ts | 28 - packages/core/src/tools/web-fetch.ts | 7 - packages/core/src/tools/write-file.test.ts | 1 - packages/core/src/tools/write-file.ts | 16 - .../core/src/utils/abortController.test.ts | 340 -- packages/core/src/utils/abortController.ts | 165 - packages/core/src/utils/editor.test.ts | 149 - packages/core/src/utils/editor.ts | 63 +- packages/core/src/utils/errors.ts | 12 - packages/core/src/utils/fileUtils.ts | 9 +- .../core/src/utils/forkedAgent.agent.test.ts | 119 +- .../core/src/utils/forkedAgent.cache.test.ts | 264 - packages/core/src/utils/forkedAgent.ts | 208 +- packages/core/src/utils/gitUtils.test.ts | 151 - packages/core/src/utils/gitUtils.ts | 76 - .../core/src/utils/memoryDiagnostics.test.ts | 76 +- packages/core/src/utils/memoryDiagnostics.ts | 112 +- .../core/src/utils/memoryDiscovery.test.ts | 324 -- packages/core/src/utils/memoryDiscovery.ts | 98 +- .../core/src/utils/memoryImportProcessor.ts | 34 +- packages/core/src/utils/modelId.test.ts | 118 - packages/core/src/utils/modelId.ts | 49 +- .../core/src/utils/nextSpeakerChecker.test.ts | 42 +- packages/core/src/utils/nextSpeakerChecker.ts | 21 +- packages/core/src/utils/notebook.test.ts | 267 +- packages/core/src/utils/notebook.ts | 256 +- packages/core/src/utils/openaiLogger.test.ts | 62 +- packages/core/src/utils/openaiLogger.ts | 83 +- packages/core/src/utils/projectRoot.test.ts | 86 - packages/core/src/utils/projectRoot.ts | 74 - packages/core/src/utils/readManyFiles.test.ts | 29 - packages/core/src/utils/readManyFiles.ts | 6 +- .../core/src/utils/runtimeDiagnostics.test.ts | 237 - packages/core/src/utils/runtimeDiagnostics.ts | 557 --- .../src/utils/runtimeFetchOptions.test.ts | 37 - .../core/src/utils/runtimeFetchOptions.ts | 17 +- .../core/src/utils/safeJsonStringify.test.ts | 111 - packages/core/src/utils/safeJsonStringify.ts | 26 +- packages/core/src/utils/sideQuery.ts | 5 +- packages/sdk-java/qwencode/QWEN.md | 2 +- packages/sdk-java/qwencode/README.md | 2 +- packages/sdk-typescript/README.md | 2 +- packages/sdk-typescript/package.json | 5 + packages/sdk-typescript/scripts/build.js | 93 +- .../scripts/bundle-cli-from-npm.js | 35 +- packages/sdk-typescript/scripts/bundle-cli.js | 31 +- .../sdk-typescript/src/daemon/DaemonClient.ts | 56 +- packages/sdk-typescript/src/daemon/events.ts | 546 ++- packages/sdk-typescript/src/daemon/index.ts | 67 + packages/sdk-typescript/src/daemon/types.ts | 1 - .../sdk-typescript/src/daemon/ui/index.ts | 59 + .../src/daemon/ui/normalizer.ts | 475 ++ .../sdk-typescript/src/daemon/ui/store.ts | 110 + .../sdk-typescript/src/daemon/ui/terminal.ts | 113 + .../src/daemon/ui/toolPreview.ts | 131 + .../src/daemon/ui/transcript.ts | 569 +++ .../sdk-typescript/src/daemon/ui/types.ts | 265 + .../sdk-typescript/src/daemon/ui/utils.ts | 167 + packages/sdk-typescript/src/index.ts | 9 + packages/sdk-typescript/src/query/Query.ts | 3 - packages/sdk-typescript/src/types/protocol.ts | 5 +- .../src/types/queryOptionsSchema.ts | 4 +- packages/sdk-typescript/src/types/types.ts | 11 +- .../test/unit/DaemonClient.test.ts | 131 + .../sdk-typescript/test/unit/Query.test.ts | 24 - .../test/unit/daemon-public-surface.test.ts | 7 + .../test/unit/daemonEvents.test.ts | 805 ++- .../sdk-typescript/test/unit/daemonUi.test.ts | 1437 ++++++ packages/vscode-ide-companion/package.json | 2 +- .../schemas/settings.schema.json | 111 +- .../scripts/prepackage.js | 10 +- .../vscode-ide-companion/src/diff-manager.ts | 2 +- .../src/extension.test.ts | 16 +- .../vscode-ide-companion/src/extension.ts | 2 +- .../src/ide-server.test.ts | 21 +- .../vscode-ide-companion/src/ide-server.ts | 4 +- .../src/open-files-manager.ts | 5 +- .../open-files-manager/notebook-handler.ts | 2 +- .../open-files-manager/text-handler.ts | 2 +- .../src/services/open-files-manager/utils.ts | 2 +- .../src/services/qwenAgentManager.ts | 17 +- .../src/services/qwenSessionManager.ts | 2 +- .../src/services/qwenSessionReader.ts | 3 +- .../src/services/settingsWriter.test.ts | 263 +- .../src/services/settingsWriter.ts | 414 +- .../subscriptionPlanDefinitions.test.ts | 37 - .../services/subscriptionPlanDefinitions.ts | 16 +- .../src/types/acpTypes.ts | 3 +- .../src/types/approvalModeTypes.ts | 8 - .../src/types/approvalModeValueTypes.ts | 7 +- .../src/utils/acpModelInfo.ts | 3 +- .../src/utils/editorGroupUtils.ts | 4 +- .../src/utils/imageSupport.test.ts | 2 +- .../handlers/AuthMessageHandler.test.ts | 263 +- .../webview/handlers/AuthMessageHandler.ts | 480 +- .../handlers/FileMessageHandler.test.ts | 27 +- .../webview/handlers/FileMessageHandler.ts | 6 +- .../src/webview/handlers/MessageRouter.ts | 8 +- .../src/webview/providers/MessageHandler.ts | 8 +- .../webview/providers/WebViewProvider.test.ts | 225 - .../src/webview/providers/WebViewProvider.ts | 129 +- packages/web-templates/package.json | 2 +- packages/webui/package.json | 3 +- .../components/toolcalls/GenericToolCall.tsx | 16 +- .../components/toolcalls/ShellToolCall.tsx | 5 +- .../components/toolcalls/ThinkToolCall.tsx | 6 +- .../toolcalls/UpdatedPlanToolCall.tsx | 1 + .../src/components/toolcalls/shared/types.ts | 7 +- .../src/components/toolcalls/shared/utils.ts | 3 + .../src/daemon/DaemonSessionProvider.test.tsx | 1086 +++++ .../src/daemon/DaemonSessionProvider.tsx | 580 +++ packages/webui/src/daemon/index.ts | 21 + .../src/daemon/transcriptAdapter.test.ts | 318 ++ .../webui/src/daemon/transcriptAdapter.ts | 427 ++ packages/webui/src/index.ts | 19 + packages/webui/src/types/toolCall.ts | 7 +- packages/webui/tsconfig.json | 7 +- packages/webui/vite.config.ts | 29 +- scripts/build-hosted-installation-assets.js | 350 -- scripts/build.js | 10 +- scripts/build_package.js | 10 +- scripts/installation/INSTALLATION_GUIDE.md | 139 +- .../installation/install-qwen-standalone.bat | 1323 ----- .../installation/install-qwen-standalone.ps1 | 415 -- .../installation/install-qwen-standalone.sh | 1467 ------ .../installation/install-qwen-with-source.bat | 29 +- .../uninstall-qwen-standalone.ps1 | 372 -- .../installation/uninstall-qwen-standalone.sh | 312 -- scripts/release-script-utils.js | 124 - scripts/tests/install-script.test.js | 3065 +----------- .../tests/upload-aliyun-oss-assets.test.js | 175 - scripts/tests/vitest.config.ts | 7 - scripts/upload-aliyun-oss-assets.js | 164 - scripts/verify-installation-release.js | 304 -- scripts/version.js | 5 +- 640 files changed, 44933 insertions(+), 70507 deletions(-) delete mode 100644 .github/release.yml delete mode 100644 .github/workflows/sync-release-to-oss.yml delete mode 100644 .qwen/skills/memory-leak-debug/SKILL.md delete mode 100644 .qwen/skills/memory-leak-debug/examples/react-reconciler-performance-measure-leak.md delete mode 100755 .qwen/skills/memory-leak-debug/scripts/find-leaf-node.sh delete mode 100644 docs/design/2026-05-15-async-memory-recall-design.md delete mode 100644 docs/design/auto-compaction-threshold-redesign.md create mode 100644 docs/design/f2-mcp-transport-pool.md delete mode 100644 docs/design/telemetry-llm-request-timing-design.md delete mode 100644 docs/design/telemetry-outbound-propagation-design.md delete mode 100644 docs/design/telemetry-resource-attributes-design.md delete mode 100644 docs/developers/daemon-client-adapters/tui.md create mode 100644 docs/developers/daemon-client-adapters/web-ui.md delete mode 100644 docs/developers/tools/monitor.md delete mode 100644 docs/e2e-tests/2026-05-18-qwen-memory-benchmark-report.md delete mode 100644 docs/e2e-tests/2026-05-19-oom-reproduction-report.md delete mode 100644 docs/e2e-tests/2026-05-19-qwen-runtime-diagnostics-benchmark-report.md delete mode 100644 docs/e2e-tests/2026-05-21-qwen-0.15.11-default-heap-oom-stress-report.md delete mode 100644 docs/e2e-tests/worktree-phase-c.md delete mode 100644 docs/plans/2026-05-14-auto-compaction-threshold-redesign.md delete mode 100644 docs/plans/2026-05-18-qwen-runtime-memory-investigation.md delete mode 100644 docs/superpowers/plans/2026-05-15-worktree-phase-c.md delete mode 100644 docs/users/features/auto-mode.md delete mode 100644 docs/verification/abort-controller-refactor/README.md delete mode 100644 docs/verification/abort-controller-refactor/automated-results.md delete mode 100644 docs/verification/abort-controller-refactor/listener-accumulation-repro.mjs delete mode 100644 docs/verification/abort-controller-refactor/migration-completeness.txt delete mode 100755 docs/verification/abort-controller-refactor/scripts/02-lite.sh delete mode 100755 docs/verification/abort-controller-refactor/scripts/06-headless-sigint.sh delete mode 100644 docs/verification/abort-controller-refactor/smoke-boot.log delete mode 100644 docs/verification/abort-controller-refactor/test-summary.txt delete mode 100644 integration-tests/cli/notebook-edit.test.ts create mode 100644 packages/acp-bridge/src/bridge.ts create mode 100644 packages/acp-bridge/src/bridgeClient.test.ts create mode 100644 packages/acp-bridge/src/bridgeClient.ts create mode 100644 packages/acp-bridge/src/bridgeFileSystem.ts create mode 100644 packages/acp-bridge/src/internal/stderrLine.ts create mode 100644 packages/acp-bridge/src/permissionMediator.test.ts create mode 100644 packages/acp-bridge/src/permissionMediator.ts create mode 100644 packages/acp-bridge/src/spawnChannel.test.ts create mode 100644 packages/acp-bridge/src/spawnChannel.ts delete mode 100644 packages/cli/src/acp-integration/acpAgent.worktree.test.ts delete mode 100644 packages/cli/src/acp-integration/session/Session.worktree.test.ts create mode 100644 packages/cli/src/auth/allProviders.ts create mode 100644 packages/cli/src/auth/install/applyProviderInstallPlan.test.ts create mode 100644 packages/cli/src/auth/install/applyProviderInstallPlan.ts create mode 100644 packages/cli/src/auth/providerConfig.test.ts create mode 100644 packages/cli/src/auth/providerConfig.ts create mode 100644 packages/cli/src/auth/providers/alibaba/alibabaStandard.test.ts create mode 100644 packages/cli/src/auth/providers/alibaba/alibabaStandard.ts create mode 100644 packages/cli/src/auth/providers/alibaba/codingPlan.test.ts create mode 100644 packages/cli/src/auth/providers/alibaba/codingPlan.ts create mode 100644 packages/cli/src/auth/providers/alibaba/tokenPlan.test.ts create mode 100644 packages/cli/src/auth/providers/alibaba/tokenPlan.ts create mode 100644 packages/cli/src/auth/providers/custom/customProvider.test.ts create mode 100644 packages/cli/src/auth/providers/custom/customProvider.ts create mode 100644 packages/cli/src/auth/providers/oauth/openrouter.test.ts create mode 100644 packages/cli/src/auth/providers/oauth/openrouter.ts create mode 100644 packages/cli/src/auth/providers/oauth/openrouterOAuth.test.ts create mode 100644 packages/cli/src/auth/providers/oauth/openrouterOAuth.ts create mode 100644 packages/cli/src/auth/providers/thirdParty/deepseek.test.ts create mode 100644 packages/cli/src/auth/providers/thirdParty/deepseek.ts create mode 100644 packages/cli/src/auth/providers/thirdParty/idealab.test.ts create mode 100644 packages/cli/src/auth/providers/thirdParty/idealab.ts create mode 100644 packages/cli/src/auth/providers/thirdParty/minimax.test.ts create mode 100644 packages/cli/src/auth/providers/thirdParty/minimax.ts create mode 100644 packages/cli/src/auth/providers/thirdParty/modelscope.test.ts create mode 100644 packages/cli/src/auth/providers/thirdParty/modelscope.ts create mode 100644 packages/cli/src/auth/providers/thirdParty/zai.test.ts create mode 100644 packages/cli/src/auth/providers/thirdParty/zai.ts create mode 100644 packages/cli/src/auth/types.ts delete mode 100644 packages/cli/src/config/loadedSettingsAdapter.test.ts delete mode 100644 packages/cli/src/config/loadedSettingsAdapter.ts create mode 100644 packages/cli/src/config/normalizeDisabledTools.test.ts create mode 100644 packages/cli/src/config/normalizeDisabledTools.ts delete mode 100644 packages/cli/src/nonInteractive/control/controllers/permissionController.test.ts delete mode 100644 packages/cli/src/nonInteractive/control/controllers/systemController.test.ts create mode 100644 packages/cli/src/serve/bridgeFileSystemAdapter.test.ts create mode 100644 packages/cli/src/serve/bridgeFileSystemAdapter.ts create mode 100644 packages/cli/src/serve/permissionAudit.test.ts create mode 100644 packages/cli/src/serve/permissionAudit.ts create mode 100644 packages/cli/src/serve/runQwenServe.test.ts delete mode 100644 packages/cli/src/services/tips/tipRegistry.test.ts create mode 100644 packages/cli/src/ui/commands/manageModelsCommand.test.ts create mode 100644 packages/cli/src/ui/commands/manageModelsCommand.ts delete mode 100644 packages/cli/src/ui/commands/skillsCommand.test.ts delete mode 100644 packages/cli/src/ui/components/DiffDialog.tsx create mode 100644 packages/cli/src/ui/components/ManageModelsDialog.test.tsx create mode 100644 packages/cli/src/ui/components/ManageModelsDialog.tsx delete mode 100644 packages/cli/src/ui/components/WorktreeExitDialog.test.tsx delete mode 100644 packages/cli/src/ui/components/WorktreeExitDialog.tsx delete mode 100644 packages/cli/src/ui/components/messages/InlineParallelAgentsDisplay.test.tsx delete mode 100644 packages/cli/src/ui/components/messages/InlineParallelAgentsDisplay.tsx delete mode 100644 packages/cli/src/ui/components/shared/text-buffer-external-editor.test.ts delete mode 100644 packages/cli/src/ui/components/views/ContextUsage.test.tsx delete mode 100644 packages/cli/src/ui/daemon/DaemonTuiAdapter.test.ts delete mode 100644 packages/cli/src/ui/daemon/DaemonTuiAdapter.ts delete mode 100644 packages/cli/src/ui/hooks/useDiffData.ts create mode 100644 packages/cli/src/ui/hooks/useManageModelsCommand.test.ts create mode 100644 packages/cli/src/ui/hooks/useManageModelsCommand.ts delete mode 100644 packages/cli/src/ui/hooks/usePreferredEditor.ts delete mode 100644 packages/cli/src/ui/hooks/useTurnDiffs.test.ts delete mode 100644 packages/cli/src/ui/hooks/useTurnDiffs.ts delete mode 100644 packages/cli/src/ui/hooks/useWorktreeSession.test.tsx delete mode 100644 packages/cli/src/ui/hooks/useWorktreeSession.ts create mode 100644 packages/cli/src/ui/manageModels/manageModels.test.ts create mode 100644 packages/cli/src/ui/manageModels/manageModels.ts delete mode 100644 packages/cli/src/utils/headlessSafetyWarnings.test.ts delete mode 100644 packages/cli/src/utils/headlessSafetyWarnings.ts delete mode 100644 packages/cli/src/utils/runBudget.test.ts delete mode 100644 packages/cli/src/utils/runBudget.ts delete mode 100644 packages/cli/src/utils/sessionPaths.test.ts delete mode 100644 packages/cli/src/utils/sessionPaths.ts delete mode 100644 packages/cli/src/utils/warningHandler.test.ts delete mode 100644 packages/cli/src/utils/warningHandler.ts delete mode 100644 packages/core/src/core/loggingContentGenerator/streamContentDetection.test.ts delete mode 100644 packages/core/src/core/loggingContentGenerator/streamContentDetection.ts delete mode 100644 packages/core/src/core/openaiContentGenerator/provider/mimo.test.ts delete mode 100644 packages/core/src/core/openaiContentGenerator/provider/mimo.ts delete mode 100644 packages/core/src/core/openaiContentGenerator/provider/utils.ts delete mode 100644 packages/core/src/extension/redaction.test.ts delete mode 100644 packages/core/src/extension/redaction.ts create mode 100644 packages/core/src/hooks/combinedAbortSignal.test.ts create mode 100644 packages/core/src/hooks/combinedAbortSignal.ts delete mode 100644 packages/core/src/memory/skillReviewAgentPlanner.test.ts delete mode 100644 packages/core/src/permissions/autoMode.test.ts delete mode 100644 packages/core/src/permissions/autoMode.ts delete mode 100644 packages/core/src/permissions/classifier-prompts/system-prompt.test.ts delete mode 100644 packages/core/src/permissions/classifier-prompts/system-prompt.ts delete mode 100644 packages/core/src/permissions/classifier-transcript.test.ts delete mode 100644 packages/core/src/permissions/classifier-transcript.ts delete mode 100644 packages/core/src/permissions/classifier.test.ts delete mode 100644 packages/core/src/permissions/classifier.ts delete mode 100644 packages/core/src/permissions/dangerousRules.test.ts delete mode 100644 packages/core/src/permissions/dangerousRules.ts delete mode 100644 packages/core/src/permissions/denialTracking.test.ts delete mode 100644 packages/core/src/permissions/denialTracking.ts delete mode 100644 packages/core/src/providers/__tests__/install.test.ts delete mode 100644 packages/core/src/providers/__tests__/presets/openrouter.test.ts delete mode 100644 packages/core/src/providers/index.ts delete mode 100644 packages/core/src/providers/install.ts delete mode 100644 packages/core/src/providers/presets/custom-provider.ts delete mode 100644 packages/core/src/providers/presets/openrouter.ts delete mode 100644 packages/core/src/providers/types.ts delete mode 100644 packages/core/src/services/gitWorktreeService.hooks.integ.test.ts delete mode 100644 packages/core/src/services/tokenEstimation.test.ts delete mode 100644 packages/core/src/services/tokenEstimation.ts delete mode 100644 packages/core/src/services/worktreeSessionService.test.ts delete mode 100644 packages/core/src/services/worktreeSessionService.ts create mode 100644 packages/core/src/subagents/model-selection.test.ts create mode 100644 packages/core/src/subagents/model-selection.ts delete mode 100644 packages/core/src/telemetry/resource-attributes.test.ts delete mode 100644 packages/core/src/telemetry/resource-attributes.ts delete mode 100644 packages/core/src/tools/enter-worktree.session.integ.test.ts delete mode 100644 packages/core/src/tools/exit-worktree.session.integ.test.ts create mode 100644 packages/core/src/tools/mcp-discovery-timeout.test.ts create mode 100644 packages/core/src/tools/mcp-discovery-timeout.ts create mode 100644 packages/core/src/tools/mcp-pool-entry.ts create mode 100644 packages/core/src/tools/mcp-pool-events.ts create mode 100644 packages/core/src/tools/mcp-pool-key.test.ts create mode 100644 packages/core/src/tools/mcp-pool-key.ts create mode 100644 packages/core/src/tools/mcp-transport-pool.test.ts create mode 100644 packages/core/src/tools/mcp-transport-pool.ts create mode 100644 packages/core/src/tools/mcp-workspace-budget.test.ts create mode 100644 packages/core/src/tools/mcp-workspace-budget.ts delete mode 100644 packages/core/src/tools/notebook-edit.test.ts delete mode 100644 packages/core/src/tools/notebook-edit.ts create mode 100644 packages/core/src/tools/pid-descendants.test.ts create mode 100644 packages/core/src/tools/pid-descendants.ts create mode 100644 packages/core/src/tools/session-mcp-view.test.ts create mode 100644 packages/core/src/tools/session-mcp-view.ts delete mode 100644 packages/core/src/tools/toAutoClassifierInput.test.ts delete mode 100644 packages/core/src/utils/abortController.test.ts delete mode 100644 packages/core/src/utils/abortController.ts delete mode 100644 packages/core/src/utils/gitUtils.test.ts delete mode 100644 packages/core/src/utils/projectRoot.test.ts delete mode 100644 packages/core/src/utils/projectRoot.ts delete mode 100644 packages/core/src/utils/runtimeDiagnostics.test.ts delete mode 100644 packages/core/src/utils/runtimeDiagnostics.ts create mode 100644 packages/sdk-typescript/src/daemon/ui/index.ts create mode 100644 packages/sdk-typescript/src/daemon/ui/normalizer.ts create mode 100644 packages/sdk-typescript/src/daemon/ui/store.ts create mode 100644 packages/sdk-typescript/src/daemon/ui/terminal.ts create mode 100644 packages/sdk-typescript/src/daemon/ui/toolPreview.ts create mode 100644 packages/sdk-typescript/src/daemon/ui/transcript.ts create mode 100644 packages/sdk-typescript/src/daemon/ui/types.ts create mode 100644 packages/sdk-typescript/src/daemon/ui/utils.ts create mode 100644 packages/sdk-typescript/test/unit/daemonUi.test.ts delete mode 100644 packages/vscode-ide-companion/src/services/subscriptionPlanDefinitions.test.ts create mode 100644 packages/webui/src/daemon/DaemonSessionProvider.test.tsx create mode 100644 packages/webui/src/daemon/DaemonSessionProvider.tsx create mode 100644 packages/webui/src/daemon/index.ts create mode 100644 packages/webui/src/daemon/transcriptAdapter.test.ts create mode 100644 packages/webui/src/daemon/transcriptAdapter.ts delete mode 100644 scripts/build-hosted-installation-assets.js delete mode 100644 scripts/installation/install-qwen-standalone.bat delete mode 100644 scripts/installation/install-qwen-standalone.ps1 delete mode 100755 scripts/installation/install-qwen-standalone.sh delete mode 100644 scripts/installation/uninstall-qwen-standalone.ps1 delete mode 100755 scripts/installation/uninstall-qwen-standalone.sh delete mode 100644 scripts/release-script-utils.js delete mode 100644 scripts/tests/upload-aliyun-oss-assets.test.js delete mode 100644 scripts/upload-aliyun-oss-assets.js delete mode 100644 scripts/verify-installation-release.js diff --git a/.gitattributes b/.gitattributes index 3c21e997137..deab5ae88bd 100644 --- a/.gitattributes +++ b/.gitattributes @@ -9,9 +9,6 @@ *.bash eol=lf Makefile eol=lf -# Windows cmd.exe expects batch installers to be checked out with CRLF. -scripts/installation/install-qwen-standalone.bat text eol=crlf - # Explicitly declare binary file types to prevent Git from attempting to # normalize their line endings. *.png binary diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index efc61bf39cd..d11835d66bf 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,66 +1,73 @@ -## What this PR does +## Summary - +- What changed: +- Why it changed: +- Reviewer focus: -## Why it's needed - - - -## Reviewer Test Plan +## Validation - -### How to verify - - - -### Evidence (Before & After) - - +Be concrete. Do not write only "tested locally". +Include the exact commands, prompts, outputs, logs, screenshots, or videos that prove the change was actually run and observed. -### Tested on +For user-visible changes, bug fixes, CLI / TUI behavior changes, or interaction changes, include key screenshots or a short video. +When possible, show before/after behavior. -| OS | Status | -| :--------: | :----: | -| 🍏 macOS | | -| 🪟 Windows | | -| 🐧 Linux | | - - - -### Environment (optional) +If helpful, use the `e2e-testing` skill to gather stronger end-to-end validation evidence. +--> - +- Commands run: + ```bash + # paste commands here + ``` +- Prompts / inputs used: +- Expected result: +- Observed result: +- Quickest reviewer verification path: +- Evidence (output, logs, screenshots, video, JSON, before/after, etc.): -## Risk & Scope +## Scope / Risk - Main risk or tradeoff: -- Not validated / out of scope: +- Not covered / not validated: - Breaking changes / migration notes: -## Linked Issues +## Testing Matrix -
-中文说明 +| | 🍏 | 🪟 | 🐧 | +| -------- | --- | --- | --- | +| npm run | ⚠️ | ⚠️ | ⚠️ | +| npx | ⚠️ | ⚠️ | ⚠️ | +| Docker | ⚠️ | ⚠️ | ⚠️ | +| Podman | ⚠️ | N/A | N/A | +| Seatbelt | ⚠️ | N/A | N/A | + +Testing matrix notes: + +- + +## Linked Issues / Bugs +If this PR fully resolves an issue, use one of: +- Closes # +- Fixes # +- Resolves # -
+Otherwise reference related issues without a closing keyword. +--> diff --git a/.github/release.yml b/.github/release.yml deleted file mode 100644 index 05c7dc55e67..00000000000 --- a/.github/release.yml +++ /dev/null @@ -1,6 +0,0 @@ -# Configuration for GitHub's automatic release notes generation -# PRs with 'skip-changelog' label will be excluded from release notes -changelog: - exclude: - labels: - - 'skip-changelog' diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index a0b0365ac94..59e7dac8361 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -57,7 +57,7 @@ jobs: steps: - name: 'Checkout' - uses: 'actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd' # v6.0.2 + uses: 'actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd' # v6.0.2 with: ref: '${{ github.event.inputs.ref || github.sha }}' fetch-depth: 0 @@ -89,7 +89,7 @@ jobs: echo "is_dry_run=${is_dry_run}" >> "${GITHUB_OUTPUT}" - name: 'Setup Node.js' - uses: 'actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e' # v6.4.0 + uses: 'actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e' # v6.4.0 with: node-version-file: '.nvmrc' cache: 'npm' @@ -153,13 +153,13 @@ jobs: steps: - name: 'Checkout' - uses: 'actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd' # v6.0.2 + uses: 'actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd' # v6.0.2 with: ref: '${{ github.event.inputs.ref || github.sha }}' fetch-depth: 0 - name: 'Setup Node.js' - uses: 'actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e' # v6.4.0 + uses: 'actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e' # v6.4.0 with: node-version-file: '.nvmrc' cache: 'npm' @@ -206,13 +206,13 @@ jobs: steps: - name: 'Checkout' - uses: 'actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd' # v6.0.2 + uses: 'actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd' # v6.0.2 with: ref: '${{ github.event.inputs.ref || github.sha }}' fetch-depth: 0 - name: 'Setup Node.js' - uses: 'actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e' # v6.4.0 + uses: 'actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e' # v6.4.0 with: node-version-file: '.nvmrc' cache: 'npm' @@ -247,13 +247,13 @@ jobs: steps: - name: 'Checkout' - uses: 'actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd' # v6.0.2 + uses: 'actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd' # v6.0.2 with: ref: '${{ github.event.inputs.ref || github.sha }}' fetch-depth: 0 - name: 'Setup Node.js' - uses: 'actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e' # v6.4.0 + uses: 'actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e' # v6.4.0 with: node-version-file: '.nvmrc' cache: 'npm' @@ -317,13 +317,13 @@ jobs: steps: - name: 'Checkout' - uses: 'actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd' # v6.0.2 + uses: 'actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd' # v6.0.2 with: ref: '${{ github.event.inputs.ref || github.sha }}' fetch-depth: 0 - name: 'Setup Node.js' - uses: 'actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e' # v6.4.0 + uses: 'actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e' # v6.4.0 with: node-version-file: '.nvmrc' cache: 'npm' @@ -400,16 +400,11 @@ jobs: env: NODE_AUTH_TOKEN: '${{ secrets.NPM_TOKEN }}' - - name: 'Verify Standalone Archives' - run: |- - npm run verify:installation-release -- --dir dist/standalone - - name: 'Create GitHub Release and Tag' if: |- ${{ needs.prepare.outputs.is_dry_run == 'false' }} env: - # CI_BOT_PAT required: GITHUB_TOKEN events cannot trigger downstream workflows (sync-release-to-oss.yml). - GITHUB_TOKEN: '${{ secrets.CI_BOT_PAT }}' + GITHUB_TOKEN: '${{ secrets.GITHUB_TOKEN }}' RELEASE_BRANCH: '${{ steps.release_branch.outputs.BRANCH_NAME }}' RELEASE_TAG: '${{ needs.prepare.outputs.release_tag }}' PREVIOUS_RELEASE_TAG: '${{ needs.prepare.outputs.previous_release_tag }}' diff --git a/.github/workflows/sync-release-to-oss.yml b/.github/workflows/sync-release-to-oss.yml deleted file mode 100644 index c2eee4c5fb4..00000000000 --- a/.github/workflows/sync-release-to-oss.yml +++ /dev/null @@ -1,238 +0,0 @@ -name: 'Sync Release to Aliyun OSS' - -on: - release: - types: ['published'] - workflow_dispatch: - inputs: - tag: - description: 'The release tag to sync (e.g., v0.1.11).' - required: true - type: 'string' - -concurrency: - group: 'sync-release-to-oss' - cancel-in-progress: false - -jobs: - sync: - name: 'Sync Release Assets to Aliyun OSS' - runs-on: 'ubuntu-latest' - if: |- - ${{ github.repository == 'QwenLM/qwen-code' }} - environment: - name: 'production-release' - permissions: - contents: 'read' - - env: - RELEASE_TAG: '${{ github.event.release.tag_name || inputs.tag }}' - - steps: - - name: 'Checkout' - uses: 'actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd' # v6.0.2 - with: - ref: '${{ env.RELEASE_TAG }}' - - - name: 'Determine release type' - id: 'meta' - env: - TAG: '${{ env.RELEASE_TAG }}' - run: |- - is_nightly="false" - is_preview="false" - if [[ "${TAG}" == *"nightly"* ]]; then - is_nightly="true" - elif [[ "${TAG}" == *"preview"* ]]; then - is_preview="true" - fi - echo "is_nightly=${is_nightly}" >> "${GITHUB_OUTPUT}" - echo "is_preview=${is_preview}" >> "${GITHUB_OUTPUT}" - echo "is_stable=$([[ ${is_nightly} == 'false' && ${is_preview} == 'false' ]] && echo true || echo false)" >> "${GITHUB_OUTPUT}" - - - name: 'Setup Node.js' - uses: 'actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e' # v6.4.0 - with: - node-version-file: '.nvmrc' - cache: 'npm' - cache-dependency-path: 'package-lock.json' - - - name: 'Install Dependencies' - env: - NPM_CONFIG_PREFER_OFFLINE: 'true' - run: |- - npm ci --no-audit --progress=false - - - name: 'Download Release Assets from GitHub' - env: - GITHUB_TOKEN: '${{ secrets.GITHUB_TOKEN }}' - RELEASE_TAG: '${{ env.RELEASE_TAG }}' - run: |- - set -euo pipefail - - mkdir -p dist/standalone - gh release download "${RELEASE_TAG}" --dir dist/standalone --pattern '*.tar.gz' --pattern '*.zip' --pattern 'SHA256SUMS' - - - name: 'Verify Downloaded Release Assets' - run: |- - npm run verify:installation-release -- --dir dist/standalone - - - name: 'Package Hosted Installation Assets' - if: |- - ${{ steps.meta.outputs.is_stable == 'true' }} - env: - RELEASE_TAG: '${{ env.RELEASE_TAG }}' - run: |- - RELEASE_VERSION="${RELEASE_TAG#v}" - npm run package:hosted-installation -- --out-dir dist/installation --version "${RELEASE_VERSION}" - - - name: 'Install ossutil' - env: - OSSUTIL_URL: "${{ vars.OSSUTIL_URL || 'https://gosspublic.alicdn.com/ossutil/1.7.19/ossutil-v1.7.19-linux-amd64.zip' }}" - OSSUTIL_SHA256: "${{ vars.OSSUTIL_SHA256 || 'dcc512e4a893e16bbee63bc769339d8e56b21744fd83c8212a9d8baf28767343' }}" - run: |- - set -euo pipefail - - tmp_dir="$(mktemp -d)" - curl -fsSL --connect-timeout 15 --max-time 300 "${OSSUTIL_URL}" -o "${tmp_dir}/ossutil.zip" - echo "${OSSUTIL_SHA256} ${tmp_dir}/ossutil.zip" | sha256sum -c - - unzip -q "${tmp_dir}/ossutil.zip" -d "${tmp_dir}" - - ossutil_path="$(find "${tmp_dir}" -type f \( -name 'ossutil' -o -name 'ossutil64' \) -print -quit)" - if [[ -z "${ossutil_path}" ]]; then - echo "::error::ossutil binary not found in downloaded archive" - exit 1 - fi - - chmod +x "${ossutil_path}" - mkdir -p "${HOME}/.local/bin" - install -m 0755 "${ossutil_path}" "${HOME}/.local/bin/ossutil" - echo "${HOME}/.local/bin" >> "${GITHUB_PATH}" - rm -rf "${tmp_dir}" - "${HOME}/.local/bin/ossutil" >/dev/null - - - name: 'Configure Aliyun OSS Credentials' - env: - ALIYUN_OSS_ACCESS_KEY_ID: '${{ secrets.ALIYUN_OSS_ACCESS_KEY_ID }}' - ALIYUN_OSS_ACCESS_KEY_SECRET: '${{ secrets.ALIYUN_OSS_ACCESS_KEY_SECRET }}' - ALIYUN_OSS_ENDPOINT: "${{ vars.ALIYUN_OSS_ENDPOINT || 'https://oss-cn-hangzhou.aliyuncs.com' }}" - run: |- - set -euo pipefail - - if [[ -z "${ALIYUN_OSS_ACCESS_KEY_ID}" || -z "${ALIYUN_OSS_ACCESS_KEY_SECRET}" ]]; then - echo "::error::Missing Aliyun OSS credentials. Set ALIYUN_OSS_ACCESS_KEY_ID and ALIYUN_OSS_ACCESS_KEY_SECRET in the production-release environment secrets." - exit 1 - fi - - ossutil config \ - -e "${ALIYUN_OSS_ENDPOINT}" \ - -i "${ALIYUN_OSS_ACCESS_KEY_ID}" \ - -k "${ALIYUN_OSS_ACCESS_KEY_SECRET}" \ - -L EN \ - -c "${RUNNER_TEMP}/.ossutilconfig" - - - name: 'Sync Release Assets to Aliyun OSS' - env: - ALIYUN_OSS_BUCKET: "${{ vars.ALIYUN_OSS_BUCKET || 'qwen-code-assets' }}" - RELEASE_TAG: '${{ env.RELEASE_TAG }}' - run: |- - set -euo pipefail - - mapfile -t release_assets < <(node scripts/verify-installation-release.js --dir dist/standalone --list-release-asset-paths) - node scripts/upload-aliyun-oss-assets.js \ - --bucket "${ALIYUN_OSS_BUCKET}" \ - --config "${RUNNER_TEMP}/.ossutilconfig" \ - --prefix "releases/qwen-code/${RELEASE_TAG}" \ - "${release_assets[@]}" - - - name: 'Verify Aliyun OSS Release Assets' - env: - ALIYUN_OSS_PUBLIC_BASE_URL: "${{ vars.ALIYUN_OSS_PUBLIC_BASE_URL || 'https://qwen-code-assets.oss-cn-hangzhou.aliyuncs.com' }}" - RELEASE_TAG: '${{ env.RELEASE_TAG }}' - run: |- - set -euo pipefail - - npm run verify:installation-release -- --base-url "${ALIYUN_OSS_PUBLIC_BASE_URL}/releases/qwen-code/${RELEASE_TAG}" - - - name: 'Sync Hosted Installation Assets to Aliyun OSS' - if: |- - ${{ steps.meta.outputs.is_stable == 'true' }} - env: - ALIYUN_OSS_BUCKET: "${{ vars.ALIYUN_OSS_BUCKET || 'qwen-code-assets' }}" - RELEASE_TAG: '${{ env.RELEASE_TAG }}' - run: |- - set -euo pipefail - - hosted_assets=( - dist/installation/install-qwen-standalone.sh - dist/installation/install-qwen-standalone.ps1 - dist/installation/install-qwen-standalone.bat - dist/installation/uninstall-qwen-standalone.sh - dist/installation/uninstall-qwen-standalone.ps1 - dist/installation/SHA256SUMS - ) - node scripts/upload-aliyun-oss-assets.js \ - --bucket "${ALIYUN_OSS_BUCKET}" \ - --config "${RUNNER_TEMP}/.ossutilconfig" \ - --prefix "installation/${RELEASE_TAG}" \ - "${hosted_assets[@]}" - node scripts/upload-aliyun-oss-assets.js \ - --bucket "${ALIYUN_OSS_BUCKET}" \ - --config "${RUNNER_TEMP}/.ossutilconfig" \ - --prefix "installation" \ - "${hosted_assets[@]}" - - - name: 'Verify Aliyun OSS Hosted Installation Assets' - if: |- - ${{ steps.meta.outputs.is_stable == 'true' }} - env: - ALIYUN_OSS_PUBLIC_BASE_URL: "${{ vars.ALIYUN_OSS_PUBLIC_BASE_URL || 'https://qwen-code-assets.oss-cn-hangzhou.aliyuncs.com' }}" - RELEASE_TAG: '${{ env.RELEASE_TAG }}' - run: |- - set -euo pipefail - - hosted_tmp_dir="$(mktemp -d)" - trap 'rm -rf "${hosted_tmp_dir}"' EXIT - mkdir -p "${hosted_tmp_dir}/versioned" "${hosted_tmp_dir}/global" - for asset in install-qwen-standalone.sh install-qwen-standalone.ps1 install-qwen-standalone.bat uninstall-qwen-standalone.sh uninstall-qwen-standalone.ps1 SHA256SUMS; do - url="${ALIYUN_OSS_PUBLIC_BASE_URL}/installation/${RELEASE_TAG}/${asset}" - global_url="${ALIYUN_OSS_PUBLIC_BASE_URL}/installation/${asset}" - curl -fsSL --connect-timeout 15 --max-time 300 "${url}" -o "${hosted_tmp_dir}/versioned/${asset}" - curl -fsSL --connect-timeout 15 --max-time 300 "${global_url}" -o "${hosted_tmp_dir}/global/${asset}" - done - cmp -s "dist/installation/SHA256SUMS" "${hosted_tmp_dir}/versioned/SHA256SUMS" || { - echo "::error::Hosted installation SHA256SUMS does not match local dist/installation/SHA256SUMS" - diff -u "dist/installation/SHA256SUMS" "${hosted_tmp_dir}/versioned/SHA256SUMS" || true - exit 1 - } - cmp -s "dist/installation/SHA256SUMS" "${hosted_tmp_dir}/global/SHA256SUMS" || { - echo "::error::Global hosted installation SHA256SUMS does not match local dist/installation/SHA256SUMS" - diff -u "dist/installation/SHA256SUMS" "${hosted_tmp_dir}/global/SHA256SUMS" || true - exit 1 - } - (cd "${hosted_tmp_dir}/versioned" && sha256sum -c SHA256SUMS) - (cd "${hosted_tmp_dir}/global" && sha256sum -c SHA256SUMS) - - - name: 'Publish Aliyun OSS Latest VERSION' - if: |- - ${{ steps.meta.outputs.is_stable == 'true' }} - env: - ALIYUN_OSS_BUCKET: "${{ vars.ALIYUN_OSS_BUCKET || 'qwen-code-assets' }}" - ALIYUN_OSS_PUBLIC_BASE_URL: "${{ vars.ALIYUN_OSS_PUBLIC_BASE_URL || 'https://qwen-code-assets.oss-cn-hangzhou.aliyuncs.com' }}" - RELEASE_TAG: '${{ env.RELEASE_TAG }}' - run: |- - set -euo pipefail - - printf '%s\n' "${RELEASE_TAG}" > "${RUNNER_TEMP}/qwen-code-latest-version" - ossutil cp "${RUNNER_TEMP}/qwen-code-latest-version" "oss://${ALIYUN_OSS_BUCKET}/releases/qwen-code/latest/VERSION" -c "${RUNNER_TEMP}/.ossutilconfig" -f --acl public-read - - latest_version="$(curl -fsSL --connect-timeout 15 --max-time 300 "${ALIYUN_OSS_PUBLIC_BASE_URL}/releases/qwen-code/latest/VERSION" | tr -d '[:space:]')" - if [[ "${latest_version}" != "${RELEASE_TAG}" ]]; then - echo "::error::Aliyun latest VERSION points to ${latest_version}, expected ${RELEASE_TAG}" - exit 1 - fi - - - name: 'Cleanup Aliyun OSS Credentials' - if: '${{ always() }}' - run: |- - rm -f "${RUNNER_TEMP}/.ossutilconfig" diff --git a/.gitignore b/.gitignore index 9734c670574..6ff1d950be2 100644 --- a/.gitignore +++ b/.gitignore @@ -56,9 +56,6 @@ bundle junit.xml packages/*/coverage/ -# PR body draft -pr_body.md - # Generated files packages/cli/src/generated/ packages/core/src/generated/ diff --git a/.qwen/commands/qc/create-pr.md b/.qwen/commands/qc/create-pr.md index a1a72c39652..208193cceb0 100644 --- a/.qwen/commands/qc/create-pr.md +++ b/.qwen/commands/qc/create-pr.md @@ -23,10 +23,15 @@ Create a well-structured pull request with proper description and title. 3. **Write PR description** -- Fill in the PR template below — each section's HTML comment explains what - to write. PR title stays in English. -- Append at the end of the PR body, with a line separator: "🤖 Generated - with [Qwen Code](https://github.com/QwenLM/qwen-code)" +- Use PR Template below +- Summarize changes clearly +- Include context and motivation +- List any breaking changes +- Link related issues if provided, or use "No linked issues" +- Leave the "Screenshots / Video Demo" section empty for the author to fill in + manually +- Add this line at the end of PR body: "🤖 Generated with [Qwen + Code](https://github.com/QwenLM/qwen-code)", with a line separator 4. **Set up PR** diff --git a/.qwen/skills/memory-leak-debug/SKILL.md b/.qwen/skills/memory-leak-debug/SKILL.md deleted file mode 100644 index a9d045bece7..00000000000 --- a/.qwen/skills/memory-leak-debug/SKILL.md +++ /dev/null @@ -1,161 +0,0 @@ ---- -name: memory-leak-debug -description: Diagnose memory leaks in the Qwen Code CLI using heap snapshots and - the chrome-devtools CLI. Use when investigating high memory usage, unbounded - growth, or suspected object retention issues. ---- - -# Memory Leak Debugging - -Diagnose memory leaks in the Qwen Code Node.js CLI by capturing heap snapshots -and analyzing retained object sizes via `chrome-devtools` CLI tooling. - -## Prerequisites - -- `chrome-devtools` CLI (from `chrome-devtools-mcp` package). If not found, - install with: `npm i chrome-devtools-mcp@latest -g` after user confirmation. - See https://github.com/ChromeDevTools/chrome-devtools-mcp/blob/main/docs/cli.md -- Node.js 22+ (for `--heapsnapshot-signal` support) - -## Step 1: Start the CLI with Snapshot Signal - -Use tmux so you can interact with the TUI and trigger snapshots from another -pane. Use the tmux-real-user-testing helper script: - -```bash -HELPER=.qwen/skills/tmux-real-user-testing/scripts/tmux-real-user-log.sh -eval "$(bash "$HELPER" start memleak . \ - env QWEN_CODE_NO_RELAUNCH=true NODE_OPTIONS=--heapsnapshot-signal=SIGUSR2 \ - npm run dev)" -echo "SESSION=$SESSION OUTDIR=$OUTDIR" -``` - -The `eval` exports `SESSION` and `OUTDIR`. Note: shell environment does not -persist across separate tool calls — save the session name from the output and -use it explicitly in subsequent commands. - -Notes: - -- `npm run dev` runs from TypeScript source via tsx — no build step needed and - changes to core/cli are reflected immediately. -- `QWEN_CODE_NO_RELAUNCH=true` prevents the CLI from spawning a child process, - so PID management is simpler. -- `NODE_OPTIONS` propagates the flag through npm → tsx → node. - -Get the PID of the actual node process. With `npm run dev`, there's a process -chain (npm → node scripts/dev.js → tsx → node CLI), so walk the tree to the -innermost node child: - -```bash -NODE_PID=$(bash .qwen/skills/memory-leak-debug/scripts/find-leaf-node.sh "") -``` - -To profile the production bundle instead (e.g., verifying tree-shaking): -`npm run bundle` first, then use -`env QWEN_CODE_NO_RELAUNCH=true node --heapsnapshot-signal=SIGUSR2 dist/cli.js` -as the command. Since node is the direct pane process, PID discovery is simpler: - -```bash -NODE_PID=$(tmux list-panes -t "" -F '#{pane_pid}') -``` - -## Step 2: Exercise the Suspected Leak - -Drive the TUI via tmux (see tmux-real-user-testing skill for patterns). Take -snapshots at intervals to compare: - -```bash -kill -USR2 $NODE_PID # snapshot 1 (baseline) -# ... use the CLI via tmux send-keys ... -kill -USR2 $NODE_PID # snapshot 2 (after activity) -# ... more activity ... -kill -USR2 $NODE_PID # snapshot 3 (confirm growth trend) -``` - -Snapshots are written to the CLI's working directory as -`Heap....heapsnapshot`. - -## Step 3: Start chrome-devtools Daemon - -```bash -chrome-devtools start --experimentalMemory --headless --no-usage-statistics -``` - -This starts the daemon in file-analysis mode — no browser or live Node -connection is needed. The memory tools work entirely on `.heapsnapshot` files. - -## Step 4: Identify the Leak - -### Load and summarize - -```bash -chrome-devtools load_memory_snapshot /abs/path/to/snapshot.heapsnapshot -``` - -Returns total heap size, V8 heap breakdown, node count. - -### Get class-level aggregates with retained sizes - -```bash -chrome-devtools get_memory_snapshot_details /abs/path/to/snapshot.heapsnapshot -``` - -Output is CSV: `uid, className, count, selfSize, maxRetainedSize`. - -Compare across snapshots to find classes whose count or retained size grows -unboundedly. - -### Inspect instances of a leaking class - -```bash -chrome-devtools get_nodes_by_class /abs/path/to/snapshot.heapsnapshot -``` - -Where `` is from the `get_memory_snapshot_details` output. Returns -individual instances with their `id`, `retainedSize`, and `nodeIndex`. - -### Trace retainer chains - -```bash -chrome-devtools get_node_retainers /abs/path/to/snapshot.heapsnapshot -``` - -Where `` is the `id` field from `get_nodes_by_class`. Shows what holds -the object alive — follow the chain to find the root retention path. - -## Step 5: Identify Root Cause - -Common patterns: - -- **Unbounded buffer/array**: An array that accumulates entries without eviction - (e.g., `performance.measure()` → `measureEntryBuffer`). -- **Event listener leak**: Listeners registered on long-lived emitters without - cleanup. -- **Closure capture**: A closure inadvertently captures a large object that - outlives its intended scope. -- **Module-level cache**: A Map/Set at module scope that grows with usage. - -The retainer chain tells you _what_ holds the object; the class aggregate -growth rate tells you _how fast_ it leaks. - -## Step 6: Verify Fix - -After applying the fix: - -1. Rebuild: `npm run bundle` -2. Repeat Steps 1-4 with the same workload. -3. Confirm the leaking class count stabilizes (no longer grows with activity). - -## Cleanup - -```bash -HELPER=.qwen/skills/tmux-real-user-testing/scripts/tmux-real-user-log.sh -bash "$HELPER" finish "" "" -chrome-devtools stop -rm *.heapsnapshot # if no longer needed -``` - -## Worked Example - -See `examples/react-reconciler-performance-measure-leak.md` for the ink 7 -upgrade leak that caused ~143 MB retention from `PerformanceMeasure` objects. diff --git a/.qwen/skills/memory-leak-debug/examples/react-reconciler-performance-measure-leak.md b/.qwen/skills/memory-leak-debug/examples/react-reconciler-performance-measure-leak.md deleted file mode 100644 index f5db329af57..00000000000 --- a/.qwen/skills/memory-leak-debug/examples/react-reconciler-performance-measure-leak.md +++ /dev/null @@ -1,65 +0,0 @@ -# React Reconciler PerformanceMeasure Leak - -## Symptom - -After the ink 6→7 upgrade (v0.15.11), moderate CLI usage caused heap to grow -to 300+ MB. RSS climbed steadily and never stabilized. - -## Diagnosis - -### Snapshot comparison - -Took 5 snapshots over ~25 minutes of normal usage. - -Snapshot #1 (baseline): - -``` -PerformanceMeasure: count=184, retainedSize=184 kB -``` - -Snapshot #5 (after activity): - -``` -PerformanceMeasure: count=150,716, retainedSize=146,798 kB (~143 MB) -``` - -Growth: ~800x over the session. Linear with number of React renders. - -### Retainer chain - -``` -chrome-devtools get_node_retainers 1003471 -``` - -Showed `PerformanceMeasure` instances retained by `(object elements)` → `Array` -— the global `measureEntryBuffer` that Node.js maintains for -`performance.measure()` calls. - -### Source identification - -`react-reconciler` ≥0.33 (pulled in by ink 7) calls `performance.measure()` on -every component render in its **development build**. The dev/prod build is -selected at runtime via `process.env.NODE_ENV`. Since the esbuild config never -set `NODE_ENV` to `"production"`, the bundle shipped both builds and selected -dev at runtime. - -## Fix - -Set `process.env.NODE_ENV` to `"production"` in esbuild's `define` map so the -conditional require resolves statically and the entire 15K-line dev build is -tree-shaken: - -```js -// esbuild.config.js -define: { - 'process.env.NODE_ENV': JSON.stringify('production'), -} -``` - -Bundle shrank by ~700 KB / 15,800 lines. PerformanceMeasure objects no longer -accumulate. - -## Commit - -`dbdc94be9` — fix(build): tree-shake React reconciler dev build to prevent -PerformanceMeasure leak diff --git a/.qwen/skills/memory-leak-debug/scripts/find-leaf-node.sh b/.qwen/skills/memory-leak-debug/scripts/find-leaf-node.sh deleted file mode 100755 index 7a5ffd77c0a..00000000000 --- a/.qwen/skills/memory-leak-debug/scripts/find-leaf-node.sh +++ /dev/null @@ -1,16 +0,0 @@ -#!/usr/bin/env bash -# Find the innermost node child process in a tmux session. -# Usage: find-leaf-node.sh -set -euo pipefail - -session=${1:?Usage: find-leaf-node.sh } - -pid=$(tmux list-panes -t "$session" -F '#{pane_pid}' | head -1) - -while true; do - child=$(pgrep -P "$pid" node 2>/dev/null | head -1 || true) - [ -z "$child" ] && break - pid=$child -done - -echo "$pid" diff --git a/AGENTS.md b/AGENTS.md index c0cd3825a5e..f7bfd45037d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -3,24 +3,6 @@ This file provides guidance to Qwen Code when working with code in this repository. -## Working Principles - -### Simplicity First - -**Minimum code that solves the problem. Nothing speculative.** -**(This is the principle we care about most.)** - -- No features beyond what was asked. -- No abstractions for single-use code. -- No "flexibility" or "configurability" that wasn't requested. -- No error handling for impossible scenarios. -- If you write 200 lines and it could be 50, rewrite it. - -Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, -simplify. - -_Adapted from Andrej Karpathy's [CLAUDE.md](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/CLAUDE.md)._ - ## Common Commands ### Building @@ -119,9 +101,6 @@ npm run preflight # Full check: clean → install → format → lint → build between packages - **Tests**: Collocated with source (`file.test.ts` next to `file.ts`), vitest framework -- **File naming**: `PascalCase.tsx` for React components, `kebab-case.ts` for - new non-component files. Leave existing `camelCase` files alone — renaming breaks `git blame` and imports. -- **Comments**: Default to none. Add only when _why_ is non-obvious; don't delete existing ones as cleanup. - **Commits**: Conventional Commits (e.g., `feat(cli): Add --json flag`) - **Node.js**: Development and production both require `>=22` (Ink 7 + React 19.2 requirement) @@ -179,13 +158,8 @@ applicable. - **PR description**: explain the motivation and changes in prose. Avoid referencing file names or function names. -- **Reviewer Test Plan** (template section): describe behaviors a reviewer - should verify and what to expect, not scripted test commands. Use **How to - verify** for reproduction steps; Before/After for TUI evidence when - applicable. -- **Line wrapping**: do not hard-wrap the PR body at a fixed column width. - GitHub renders single newlines as `
`, so a wrapped description displays - as a narrow column. Write each paragraph or list item as one long line. +- **Reviewer Test Plan**: describe behaviors a reviewer should verify and what + to expect, not scripted test commands. ## Project Directories diff --git a/docs/design/2026-05-15-async-memory-recall-design.md b/docs/design/2026-05-15-async-memory-recall-design.md deleted file mode 100644 index f11b2ac5d23..00000000000 --- a/docs/design/2026-05-15-async-memory-recall-design.md +++ /dev/null @@ -1,206 +0,0 @@ -# Async Memory Recall — Design Spec - -**Date:** 2026-05-15 -**Status:** Approved -**Related issues:** #3761, #3759 -**Related PRs:** #3814, #3866 - ---- - -## Problem - -`relevanceSelector.ts` uses `AbortSignal.timeout(1_000)` (introduced by #3866). On first-session cold starts, qwen3.5-flash averages ~908 ms — consistently hitting the 1 s threshold. The outer 2.5 s deadline in `resolveAutoMemoryWithDeadline` means every UserQuery can block for up to 2.5 s even when recall always fails. - -Root cause: the main-agent request path `await`s the recall result before sending to the model. Any slowness in the recall side-query directly adds to user-visible latency. - ---- - -## Design - -### Core idea - -Fire recall on UserQuery and never await it. Consume the result at two opportunistic points — whichever fires first: - -1. **UserQuery consume point** — synchronous `settledAt !== null` check just before `turn.run()`. Zero-wait: if already settled, use it; if not, skip. -2. **ToolResult inject point** — same check on every ToolResult turn. Injects memory as a `system-reminder` **appended after** the functionResponse parts in `requestToSend`, giving the model memory context before its next response. (Append, not prepend: the Qwen API requires the functionResponse to immediately follow the model's functionCall — see the existing `hasPendingToolCall` IDE-context skip for the same constraint.) - -This matches the pattern used by Claude Code upstream (`startRelevantMemoryPrefetch` / `settledAt` polling in `query.ts`). - ---- - -## Data structures - -### New type `MemoryPrefetchHandle` (in `client.ts`) - -```typescript -type MemoryPrefetchHandle = { - promise: Promise; - /** Set by promise.finally(). null until the promise settles. */ - settledAt: number | null; - /** True after memory has been injected — prevents double-inject. */ - consumed: boolean; - controller: AbortController; -}; -``` - -### Field change on `GeminiClient` - -| Remove | Add | -| ------------------------------------------------------------ | ---------------------------------------------------------- | -| `pendingRecallAbortController: AbortController \| undefined` | `pendingMemoryPrefetch: MemoryPrefetchHandle \| undefined` | - ---- - -## Changes - -### 1. `client.ts` — remove `resolveAutoMemoryWithDeadline` - -Delete the function entirely. It is replaced by the `settledAt` flag mechanism. - -### 2. `client.ts` — UserQuery fire path - -Replace the `resolveAutoMemoryWithDeadline` call with: - -```typescript -// Abort any in-flight prefetch from a previous UserQuery before installing -// the new handle (prevents orphan side-queries when the user types again -// before recall settles). -this.pendingMemoryPrefetch?.controller.abort(); -this.pendingMemoryPrefetch = undefined; - -const controller = new AbortController(); -// Bridge the caller's signal into the prefetch controller so a user abort -// (Ctrl-C / Esc) on the parent turn also terminates the recall side-query. -const onParentAbort = () => controller.abort(); -if (signal.aborted) { - controller.abort(); -} else { - signal.addEventListener('abort', onParentAbort, { once: true }); -} - -const promise = this.config - .getMemoryManager() - .recall(projectRoot, partToString(request), { - config: this.config, - excludedFilePaths: this.surfacedRelevantAutoMemoryPaths, - abortSignal: controller.signal, - }) - .catch((error: unknown) => { - if (!(error instanceof DOMException && error.name === 'AbortError')) { - debugLogger.warn('Managed auto-memory recall prefetch failed.', error); - } - return EMPTY_RELEVANT_AUTO_MEMORY_RESULT; - }); - -const handle: MemoryPrefetchHandle = { - promise, - settledAt: null, - consumed: false, - controller, -}; -void promise.finally(() => { - handle.settledAt = Date.now(); - signal.removeEventListener('abort', onParentAbort); -}); -this.pendingMemoryPrefetch = handle; -// no await — continue immediately -``` - -### 3. `client.ts` — UserQuery consume point (replaces `await relevantAutoMemoryPromise`) - -```typescript -const prefetchHandle = this.pendingMemoryPrefetch; -if ( - prefetchHandle && - prefetchHandle.settledAt !== null && - !prefetchHandle.consumed -) { - prefetchHandle.consumed = true; - this.pendingMemoryPrefetch = undefined; - const result = await prefetchHandle.promise; // already settled, returns immediately - if (result.prompt) { - // unshift, not push: keep memory at the front of systemReminders so - // it leads the system-reminder block on UserQuery turns. (ToolResult - // turns instead append to requestToSend to preserve functionCall / - // functionResponse pairing — see below.) - systemReminders.unshift(result.prompt); - for (const doc of result.selectedDocs) { - this.surfacedRelevantAutoMemoryPaths.add(doc.filePath); - } - } -} -``` - -### 4. `client.ts` — ToolResult inject point (new) - -After `requestToSend` is assembled, before `turn.run()`, add: - -```typescript -if (messageType === SendMessageType.ToolResult) { - const prefetchHandle = this.pendingMemoryPrefetch; - if ( - prefetchHandle && - prefetchHandle.settledAt !== null && - !prefetchHandle.consumed - ) { - prefetchHandle.consumed = true; - this.pendingMemoryPrefetch = undefined; - const result = await prefetchHandle.promise; - if (result.prompt) { - // Append (not prepend) so functionResponse parts stay first - // and the model's functionCall/functionResponse pairing - // isn't broken on the native Gemini path. - requestToSend = [...requestToSend, result.prompt]; - for (const doc of result.selectedDocs) { - this.surfacedRelevantAutoMemoryPaths.add(doc.filePath); - } - } - } -} -``` - -### 5. `client.ts` — cleanup paths - -The handle is released by two distinct mechanisms: - -**5 abort-and-clear sites** (the prefetch is still pending, abort the controller before dropping the reference). Replace `pendingRecallAbortController?.abort()` + `= undefined` with: - -```typescript -this.pendingMemoryPrefetch?.controller.abort(); -this.pendingMemoryPrefetch = undefined; -``` - -Sites: `resetChat()`, `MaxSessionTurns` early-return, `boundedTurns=0` early-return, `SessionTokenLimitExceeded` early-return, Arena control-signal early-return. The fire path itself also performs this abort-then-replace when a new UserQuery arrives while the previous prefetch is still in flight. - -**2 clear-only sites** (the prefetch has already settled and we're consuming it — no controller to abort, just drop the reference): - -```typescript -prefetchHandle.consumed = true; -this.pendingMemoryPrefetch = undefined; -``` - -Sites: UserQuery consume point, ToolResult inject point. - -### 6. `relevanceSelector.ts` — remove `AbortSignal.timeout(1_000)` - -Remove the combined `AbortSignal.any([AbortSignal.timeout(1_000), callerAbortSignal])` and pass `callerAbortSignal` directly. - ---- - -## Behaviour comparison - -| Scenario | Before | After | -| -------------------------------------------- | ------------------------------ | ------------------------------------------------------ | -| recall completes before model prep | inject on UserQuery, ~0 wait | inject on UserQuery, ~0 wait | -| recall slow (cold start) | block up to 2.5 s | skip UserQuery, inject on first ToolResult | -| recall times out (1 s) | abort, empty result, no memory | no hard timeout; inject whenever settled | -| no tool calls, recall slow | block up to 2.5 s, then skip | skip UserQuery, no ToolResult opportunity — miss | -| user sends 2nd message before recall settles | 2nd recall races 1st handle | 1st handle aborted when 2nd UserQuery fires new handle | - ---- - -## Out of scope - -- Changing the memory injection format from `system-reminder` to `tool-result` attachment (CC style) -- Per-session byte budget skip gate -- Single-word prompt skip gate diff --git a/docs/design/auto-compaction-threshold-redesign.md b/docs/design/auto-compaction-threshold-redesign.md deleted file mode 100644 index 544f5baecd9..00000000000 --- a/docs/design/auto-compaction-threshold-redesign.md +++ /dev/null @@ -1,436 +0,0 @@ -# Auto-Compaction Threshold Redesign - -**Status:** Draft · 2026-05-14 - -## 背景 - -> 本节描述本 PR 落地**之前**的状态（pre-redesign behavior）。下文出现的 `COMPRESSION_TOKEN_THRESHOLD`、`thinkingConfig.includeThoughts = true`、`hasFailedCompressionAttempt`、以及具体的 file:line 引用都对应 PR #4345 合入前的代码——合入后这些符号 / 行号会不再有效。 - -当前 qwen-code 的自动压缩仅使用单一比例阈值 `COMPRESSION_TOKEN_THRESHOLD = 0.7`（`chatCompressionService.ts:33`），所有窗口大小共用同一比例。对比 claude-code 的「绝对 token 梯子」（autoCompact.ts:62-65），qwen-code 存在三个具体问题： - -1. **大窗口下预留过多**：1M 模型 70% 阈值在 700K 触发，剩余 300K 远超摘要 + 输出实际所需的 ~33K -2. **失败 1 次永久锁**：`hasFailedCompressionAttempt = true` 之后整个 session 不再尝试 auto-compact（geminiChat.ts:504），比 claude-code 的「连续 3 次熔断」更严苛 -3. **tip 系统与 auto 阈值脱钩**：`tipRegistry.ts` 里的三条 `context-*` tip 使用固定的 50/80/95 百分比，与 auto-compact 阈值（70%）完全独立。这意味着在「auto 正常工作」的主路径上 80% / 95% tip 极少触发，而在「auto 失败 / 反应式兜底」的边缘路径上又缺乏与阈值对齐的语义 -4. **压缩调用本身没有输出预算控制**：[chatCompressionService.ts:374-376](packages/core/src/services/chatCompressionService.ts:374) 显式开启 `thinkingConfig.includeThoughts = true`（注释：「Compression quality drives every subsequent main turn」），同时 sideQuery 调用未设 `maxOutputTokens` 上限。代码注释（[:436-437](packages/core/src/services/chatCompressionService.ts:436)）也承认 `compressionOutputTokenCount may include non-persisted tokens (thoughts)`。在压缩接近窗口顶时，总输出可能膨胀，使 buffer 预留缺乏可预测上限。

更糟糕的是跨 provider 行为不一致：Anthropic 的 thinking budget 与 max_tokens 完全独立；OpenAI 的 reasoning tokens 不受 max_completion_tokens 限制；Gemini 的行为又因模型版本而异。这意味着「单靠加 maxOutputTokens 就能控制总输出」在 qwen-code 这种多 provider 项目里不成立 - -5. **阈值判断使用的 `lastPromptTokenCount` 系统性下偏。** [geminiChat.ts:1217-1232](packages/core/src/core/geminiChat.ts:1217) 表明这个数来自上一轮 API response 的 `usageMetadata.totalTokenCount`。两个 gap：(a) 不包含本轮即将加入的 user message，每次 cheap-gate 判断都比真实 prompt 小一段；(b) 首轮初始值是 0，`--continue` 恢复巨大 session / sub-agent 继承大量历史时第一次 send 永远绕过所有阈值。对比 claude-code 的 `tokenCountWithEstimation`（[query.ts:638](src/query.ts:638)）走「最后一条 assistant API usage + 之后新增 message 估算」的双轨制能闭合这两个 gap - -## 设计目标 - -- 引入「比例 + 绝对」混合阈值，让大窗口模型由绝对值接管，小窗口仍走比例兜底 -- 新增 warn / hard 两层（auto 保留为主触发点），形成三层梯子 -- 把 tip 系统重写为跟随新阈值的触发条件 -- 失败处理从「1 次永久锁」升级为「3 次熔断 + 自动恢复」 -- **压缩调用关闭 thinking 并加 `maxOutputTokens` 上限**：与 claude-code 对齐，让总输出受单一参数约束、buffer 预算可预测；接受压缩质量可能下降的代价 -- **加 token 估算补偿**：消除 `lastPromptTokenCount` 的「滞后一轮」和「首轮为 0」两个系统性下偏，让阈值判断更贴近真实 prompt 大小 -- 删除 settings 里的 `contextPercentageThreshold` 配置入口（内部 PCT 常量保留） -- **不引入** env 覆盖通道、**不**新增显式 enabled 开关 - -## 三层阈值梯子 - -``` - window (raw context window) - │ - │ ← SUMMARY_RESERVE = 20K - ▼ - effectiveWindow - │ - │ ← HARD_BUFFER = 3K - ▼ - hard_threshold = effectiveWindow - 3K - │ - │ ← (AUTOCOMPACT_BUFFER - HARD_BUFFER) = 10K - ▼ -auto_threshold = max(PCT * window, effectiveWindow - AUTOCOMPACT_BUFFER) - │ - │ ← WARN_BUFFER = 20K - ▼ -warn_threshold = max((PCT - WARN_OFFSET) * window, auto_threshold - WARN_BUFFER) - │ - ▼ - 0 -``` - -### 三层语义 - -| 层 | 触发条件 | 行为 | -| -------- | ------------------------------ | -------------------------------------------------------- | -| **warn** | `tokenCount >= warn_threshold` | UI 提示「距自动压缩还剩 X tokens」，不改变 send 行为 | -| **auto** | `tokenCount >= auto_threshold` | 在 send 前 `tryCompress(force=false)`，正常压缩流程 | -| **hard** | `tokenCount >= hard_threshold` | 在 send 前 `tryCompress(force=true)`，重置失败锁强制压缩 | - -`hard` 层等同于把现有 reactive overflow（geminiChat.ts:711）的兜底逻辑提前到 send 前，避免一次失败的 oversized request round-trip。 - -## 内部常量 - -```ts -// chatCompressionService.ts -const DEFAULT_PCT = 0.7; // auto 比例兜底 -const WARN_PCT_OFFSET = 0.1; // warn 比例 = PCT - WARN_OFFSET = 0.6 -const COMPACT_MAX_OUTPUT_TOKENS = 20_000; // 压缩 sideQuery 输出硬上限（thinking + summary 合计） -const SUMMARY_RESERVE = 20_000; // 阈值梯子从窗口顶减去的输出预留 = maxOutput -const AUTOCOMPACT_BUFFER = 13_000; // auto 与 effectiveWindow 间距 -const WARN_BUFFER = 20_000; // warn 与 auto 间距 -const HARD_BUFFER = 3_000; // hard 与 effectiveWindow 间距 -const MAX_CONSECUTIVE_FAILURES = 3; // 失败熔断阈值 -``` - -数值来源：全部沿用 claude-code 的实测值（[autoCompact.ts:30,62-65](src/services/compact/autoCompact.ts:30)）。 - -`SUMMARY_RESERVE = COMPACT_MAX_OUTPUT_TOKENS` 是关键关系：模型受 `maxOutputTokens` 硬限制约束，输出不可能超出 20K，因此 reserve 不需要额外 safety margin。注意：本设计关闭 thinking 后该等式成立（output budget 全部给 summary）；若保留 thinking，`thinking + summary` 共享预算（Gemini SDK / 多数 provider 的 `maxOutputTokens` 语义），模型自行在两者间分配，此时 summary 的实际可用空间小于 20K（见「风险与注意事项」第 1、2 条）。 - -## 计算函数 - -```ts -export interface CompactionThresholds { - warn: number; - auto: number; - hard: number; // 当 hard < auto 时等于 auto（小窗口退化） - effectiveWindow: number; -} - -export function computeThresholds(window: number): CompactionThresholds { - const effectiveWindow = window - SUMMARY_RESERVE; - - const absAuto = effectiveWindow - AUTOCOMPACT_BUFFER; - const auto = Math.max(DEFAULT_PCT * window, absAuto); - - const absWarn = auto - WARN_BUFFER; - const warn = Math.max((DEFAULT_PCT - WARN_PCT_OFFSET) * window, absWarn); - - const rawHard = effectiveWindow - HARD_BUFFER; - const hard = Math.max(rawHard, auto); // 小窗口下退化为 auto - - return { warn, auto, hard, effectiveWindow }; -} -``` - -### 实测数据 - -| 窗口 | warn | auto | hard | 备注 | -| ---- | ----------- | ----------- | ------------ | ------------------------------- | -| 32K | 19.2K (pct) | 22.4K (pct) | 22.4K (退化) | 比例兜底 | -| 64K | 38.4K (pct) | 44.8K (pct) | 44.8K (退化) | 比例兜底 | -| 128K | 76.8K (pct) | 95K (abs) | 105K (abs) | 混合（warn=pct, auto/hard=abs） | -| 200K | 147K (abs) | 167K (abs) | 177K (abs) | 绝对接管 | -| 256K | 203K (abs) | 223K (abs) | 233K (abs) | 绝对接管 | -| 1M | 947K (abs) | 967K (abs) | 977K (abs) | 全绝对 | - -`(pct)` 表示该层由比例公式决定，`(abs)` 表示由绝对值公式决定。 - -## 用户配置 - -### ChatCompressionSettings 变更 - -```ts -// packages/core/src/config/config.ts:217 -export interface ChatCompressionSettings { - /** 保留（与本设计无关，由 compactionInputSlimming 使用） */ - imageTokenEstimate?: number; -} -``` - -**删除：** `contextPercentageThreshold` 字段。理由： - -1. 新公式下，对主流窗口（>= 128K）该字段几乎无影响——绝对值接管 -2. 小窗口下用户配置反而可能让阈值"更早"压缩，与节省 token 直觉相反 -3. claude-code 没有暴露此字段，无类似的用户面配置先例 - -### Breaking change 处理 - -**用户面：** 启动时 `Config` 加载发现 `chatCompression.contextPercentageThreshold` 存在： - -- 写入 stderr 一行警告：`"chatCompression.contextPercentageThreshold has been removed and is now controlled by built-in thresholds."` -- **不**报错、**不**阻塞启动 -- 字段值被忽略 - -**SDK 面（R5.4）：** `CompressOptions` 的 `hasFailedCompressionAttempt: boolean` 字段重命名为 `consecutiveFailures: number`。两点差异： - -| | 旧字段 | 新字段 | -| ---- | ------------------------------ | -------------------------------------------------------------------- | -| 名称 | `hasFailedCompressionAttempt` | `consecutiveFailures` | -| 类型 | `boolean` | `number` | -| 语义 | `true` = 永久禁用 auto-compact | `>= MAX_CONSECUTIVE_FAILURES`（默认 3）= 暂时禁用直到 force 成功重置 | - -仓库内只有 `GeminiChat.tryCompress` 一个内部消费方，所以内部 migration 风险低；但 `@qwen-code/qwen-code-core` 是 published package、`CompressOptions` 在 d.ts 里可见，下游 SDK 直接调 `service.compress({ ..., hasFailedCompressionAttempt: true })` 的代码会拿到 TS 编译错误。**迁移指引：** 把 `true` 改为 `MAX_CONSECUTIVE_FAILURES`（或任意 >= 3 的整数），`false` 改为 `0`。如果调用方维护自己的失败计数，直接传入即可。 - -## Token 估算补偿 - -qwen-code 的 `lastPromptTokenCount` 来自上一轮 API response 的 `usageMetadata.totalTokenCount`（[geminiChat.ts:1217-1232](packages/core/src/core/geminiChat.ts:1217)）。这导致： - -1. **滞后一轮**：cheap-gate 用 `lastPromptTokenCount` 判断，但本次 send 实际 prompt = 它 + 本轮 user message。少算的部分可能让阈值判断 false-negative -2. **首轮为 0**：初始值是 0，第一次 send 时无论历史多大都不会触发任何阈值（含 `--continue` 恢复 / sub-agent 继承场景） - -引入轻量本地估算函数 `estimatePromptTokens`，在 send 前 cheap-gate / hard 判断时补足这两段缺失： - -```ts -// chatCompressionService.ts（或新文件 packages/core/src/services/tokenEstimation.ts） - -const BYTES_PER_TOKEN = 4; // 通用 char/4 估算（claude-code 同此） -const BYTES_PER_TOKEN_JSON = 2; // JSON / tool_call input 更密集 - -/** - * 估算一组 Content 的 token 数，用于补偿 API usage metadata 的滞后。 - * 对 image / document 复用现有 imageTokenEstimate（默认 1600）。 - */ -export function estimateContentTokens( - contents: Content[], - imageTokenEstimate = DEFAULT_IMAGE_TOKEN_ESTIMATE, -): number { - // 复用 estimateContentChars（compactionInputSlimming.ts），再除以 bytesPerToken - // 内部对 functionCall / functionResponse 用 BYTES_PER_TOKEN_JSON - // ... -} - -/** - * cheap-gate 与 hard 判断的统一入口。 - * 主路径：lastPromptTokenCount 准 + 本轮 user message 估算 - * 首轮路径：full history 估算 - */ -export function estimatePromptTokens( - history: Content[], - userMessage: Content, - lastPromptTokenCount: number, -): number { - if (lastPromptTokenCount > 0) { - return lastPromptTokenCount + estimateContentTokens([userMessage]); - } - return estimateContentTokens([...history, userMessage]); -} -``` - -应用位置： - -- `chatCompressionService.compress()` 的 cheap-gate：把 `originalTokenCount` 来源换成 `estimatePromptTokens(history, userMessage, lastPromptTokenCount)` -- `geminiChat.sendMessageStream` 入口的 hard 判断（见下一节） - -**估算只用于提前触发，不用于「跳过触发」。** 因为 char/4 是粗略下界估计，作为 false-positive 一侧是安全的（宁可早一点压），作为 false-negative 则不可靠。 - -## 触发链路改动 - -### chatCompressionService.ts - -1. **导出 `computeThresholds`**，供 cheap-gate / UI / 命令复用 -2. **`compress()` cheap-gate** (line 221-249)： - ```ts - if (consecutiveFailures >= MAX_CONSECUTIVE_FAILURES && !force) { - return NOOP; - } - const { auto } = computeThresholds(contextLimit); - const effectiveTokens = estimatePromptTokens( - curatedHistory, - userMessage, - originalTokenCount, - ); - if (!force && effectiveTokens < auto) return NOOP; - ``` -3. **`compress()` 的 runSideQuery 调用** (line 356-380)：关闭 thinking + 加 `maxOutputTokens`： - - ```ts - const summaryResult = await runSideQuery(config, { - // ... - config: { - thinkingConfig: { includeThoughts: false }, // 关闭 thinking（与 claude-code 一致） - maxOutputTokens: COMPACT_MAX_OUTPUT_TOKENS, // 硬上限 20K - }, - // ... - }); - ``` - - 或者直接删掉 `thinkingConfig` 让 `runSideQuery` 默认值（[sideQuery.ts:118](packages/core/src/utils/sideQuery.ts:118) 默认 `includeThoughts: false`）接管。 - - 关 thinking 后，`maxOutputTokens` 直接约束总输出（不存在 thinking 单独 budget 的问题），`SUMMARY_RESERVE = maxOutput = 20K` 是干净的硬关系。 - - 同时更新 [chatCompressionService.ts:374-376](packages/core/src/services/chatCompressionService.ts:374) 的注释，从「Compression quality drives every subsequent main turn — keep reasoning on」改为说明「为保证跨 provider 可预测的输出上限，与 claude-code 设计对齐」。 - - token math 一段（[:436-437](packages/core/src/services/chatCompressionService.ts:436)）的 "may include non-persisted tokens (thoughts)" 注释也可以同步清理 - -### geminiChat.ts: `sendMessageStream` 入口（line 562） - -```ts -// 替换前：tryCompress(force=false) -// 替换后：用估算 token 判断是否触发 hard，决定 force 标志 - -const { hard } = computeThresholds(contextLimit); -const effectiveTokens = estimatePromptTokens( - this.getHistory(true), - createUserContent(params.message), - this.lastPromptTokenCount, -); -const shouldForceFromHard = effectiveTokens >= hard; - -if (shouldForceFromHard) { - // 重置熔断器，等同 force compress - this.consecutiveFailures = 0; -} - -compressionInfo = await this.tryCompress( - prompt_id, - model, - shouldForceFromHard, - params.config?.abortSignal, -); -``` - -### 失败处理升级 (`geminiChat.ts:504-510`) - -```ts -// 替换前 -hasFailedCompressionAttempt: boolean; - -// 替换后 -consecutiveFailures: number; // 默认 0 - -// 失败分支 -} else if (isCompressionFailureStatus(info.compressionStatus)) { - if (!force) { - this.consecutiveFailures += 1; - } -} - -// 成功分支 -this.consecutiveFailures = 0; -``` - -`force=true` 调用失败不计入计数（保持现有 reactive / manual 不"占额"的语义）。 - -## UI 改动 - -### tipRegistry.ts 重写三条 context-\* tip - -三层阈值正好与三条 tip 一一对应。映射关系（按 token 数从低到高）： - -| Tip ID | 当前条件 | 新条件 | 文案变化 | -| ------------------ | --------------------------------------------- | ------------------------------------------------------------------- | ----------------------------------------------------------------- | -| `compress-intro` | `pct >= 50 && < 80 && sessionPromptCount > 5` | `tokenCount >= warn && tokenCount < auto && sessionPromptCount > 5` | 保持不变 | -| `context-high` | `pct >= 80 && < 95` | `tokenCount >= auto && tokenCount < hard` | 保持不变 | -| `context-critical` | `pct >= 95` | `tokenCount >= hard` | 加一句「Auto-compact will force on next send.」反映新 hard 层行为 | - -**对触发频率的影响：** - -- 主路径（auto 正常工作）：`tokenCount` 跨越 auto 后立即触发压缩，下一轮 tokenCount 回落，所以 `context-high` 仅在「触发到压缩生效之间」短暂可见 -- 边缘路径（auto 失败 / 熔断 / reactive 来不及）：`tokenCount` 持续上涨，会依次穿过 warn → auto → hard 触发三条 tip，跟用户视角的"上下文越来越紧"一致 -- `context-critical` 触发时 hard 层已经在 send 前 force compress（spec 触发链路改动一节），所以这条 tip 实际上是「post-rescue 告知」而非「pre-rescue 警告」，文案补一句说明 - -`TipContext` 接口增加： - -```ts -export interface TipContext { - lastPromptTokenCount: number; - contextWindowSize: number; - sessionPromptCount: number; - sessionCount: number; - platform: string; - // 新增：让 isRelevant 函数能拿到阈值。 - // computeThresholds 在调用方算好后注入，避免 tipRegistry 直接依赖 core。 - thresholds?: CompactionThresholds; -} -``` - -`AppContainer.tsx:1150` 构造 `TipContext` 时同步注入。 - -### /context 命令同步 (`contextCommand.ts:177-183`) - -```ts -// 替换硬编码 (1 - threshold) * contextWindowSize -const { warn, auto, hard, effectiveWindow } = - computeThresholds(contextWindowSize); - -// 显示四行： -// Effective window: 180K (window − 20K reserve) -// Warn threshold: 147K (...) -// Auto threshold: 167K ← 当前位置 -// Hard threshold: 177K -// 标记当前 token count 落在哪个 tier -``` - -### Footer 持续提示（可选 follow-up） - -本 spec 不强制实现 footer 持续提示，理由： - -- 现有 tip 系统已经能在 history 里给出提示 -- Footer 持续提示需要改 ink 渲染、增加重绘频率 -- 可作为本 spec 后置 follow-up（独立 PR） - -如果后续要做，建议触发条件 `tokenCount >= warn && tokenCount < auto`，超过 auto 后隐藏（压缩已开始）。 - -## 测试覆盖 - -### 单元测试（chatCompressionService.test.ts） - -- `computeThresholds(32K)` → 比例兜底分支（warn/auto 均 pct，hard 退化） -- `computeThresholds(128K)` → 混合分支（warn=pct，auto=abs，hard=abs） -- `computeThresholds(200K)` → 绝对接管分支（warn/auto/hard 均 abs） -- `computeThresholds(1M)` → 全绝对分支 -- `computeThresholds(window=10K)` → 极小窗口（绝对值全负），公式不崩 -- 三层阈值始终满足 `warn <= auto <= hard` -- max() 公式在边界点（pct \* window == abs）稳定 - -### 单元测试（tokenEstimation.test.ts） - -- `estimateContentTokens` 对纯文本 / json / functionCall / functionResponse / image / document 分别走对应 bytesPerToken -- `estimatePromptTokens` 在 `lastPromptTokenCount > 0` 时走「主路径」，等于 0 时走「首轮路径」 -- 大 user message 在 cheap-gate 阶段被加上去后能跨越 auto 阈值 -- 估算与真实 API usage 的偏差在 ±30% 以内（用真实历史样本回归） - -### 集成测试（geminiChat.test.ts / chatCompressionService.test.ts） - -- 3 次连续失败后 cheap-gate NOOP；下一次 force 后恢复 -- 单次失败不再永久锁 -- 估算 token 跨越 hard 后 send 自动 force compress -- 压缩 sideQuery 调用 `maxOutputTokens = COMPACT_MAX_OUTPUT_TOKENS` 正确透传到 `runSideQuery`，`thinkingConfig.includeThoughts` 为 `false`（或被 sideQuery 默认值接管） -- **首轮覆盖**：构造一个 `lastPromptTokenCount = 0` 但 history 巨大的 chat（模拟 `--continue` 恢复），首次 send 时 auto 阈值能被估算路径触发 - -### 兼容性测试 - -- 设置 `contextPercentageThreshold = 0.5` 启动 → stderr 警告 + 字段被忽略，行为以内部 PCT 常量为准 - -### Tip 系统测试（tipRegistry.test.ts） - -- 三条 context-\* tip 在跨越 warn/auto/hard 时正确触发，且区间不重叠 -- 主路径下 auto 阈值触发压缩后 `context-high` 不持续可见 -- 边缘路径（熔断 + token 继续涨）下三条 tip 依次触发 -- TipContext 缺 `thresholds` 时（fallback）行为合理 - -## 实施分阶段 - -| Phase | 内容 | 独立性 | -| ----- | -------------------------------------------------------------------------------------------- | ------------------ | -| 1 | 内部常量 + `computeThresholds` + cheap-gate 改动（不含估算补偿） | 可独立合并 | -| 2 | 失败处理升级（1 → 3 熔断） | 可独立合并 | -| 3 | hard 层 force compress 提前 | 依赖 P1 + P7 | -| 4 | 配置面变更 + breaking change 警告 | 依赖 P1 | -| 5 | UI（tip 重写 + /context） | 依赖 P1 | -| 6 | 压缩 sideQuery 关 thinking + 加 `maxOutputTokens` 上限 | 独立可先于 P1 落地 | -| 7 | Token 估算补偿（`estimateContentTokens` + `estimatePromptTokens`，应用到 cheap-gate / hard） | 独立可与 P1 并行 | - -每个 Phase 可独立 PR。建议合并顺序 **P6 → P7 → P1 → P2 → P4 → P3 → P5**：先给压缩调用打上 `maxOutputTokens` 上限（让 buffer 假设可信）；再加估算补偿（让 token 数判断更可靠）；再把阈值基础设施落地；再做失败熔断、配置面变更；最后才打开 hard 层主动救场（这时已有可靠的 token 数 + 熔断器）。每个 PR 都能独立验证、独立回滚。 - -## 风险与注意事项 - -1. **关 thinking 可能影响摘要质量。** 原作者注释 "Compression quality drives every subsequent main turn — keep reasoning on" 表达过对此的担忧。本 spec 的判断是「可预测的 token 上限」优先于「最大化质量」，但落地后需要观察 telemetry 里 `compression_input_token_count` / `compression_output_token_count` 的分布，以及主对话在压缩后的质量变化（用户反馈、`COMPRESSION_FAILED_*` 状态率）。如果质量下降明显，再考虑回退到 thinking 开启 + provider-specific thinkingBudget 控制。 - -2. **`maxOutputTokens` 触顶可能导致 summary 被截断。** 关 thinking 后，20K 直接限制 summary 主体；claude-code 实测 p99.99 ≈ 17K，留 ~3K 安全冗余。但 qwen-code 的压缩 prompt 与 claude-code 不同，分布需要观测。建议在压缩失败分支（[chatCompressionService.ts:464-491](packages/core/src/services/chatCompressionService.ts:464)）追加「检测到 finish_reason = MAX_TOKENS」的 NOOP 路径，避免持久化半截 summary。 - -3. **跨 provider 的 maxOutputTokens 映射差异。** OpenAI compat (dashscope) → `max_tokens`、Anthropic → `max_tokens`、Gemini SDK → `maxOutputTokens`。当前 qwen-code 已有这层映射（[contentGenerator.ts:94](packages/core/src/core/contentGenerator.ts:94) 等），需要在 P6 实现时验证 sideQuery 路径上 `maxOutputTokens` 字段确实贯穿到所有 provider 的请求体。 - -4. **Token 估算是粗略下界，不应反向用作"跳过触发"的依据。** `char/4` 与各 provider 真实 tokenizer 偏差可能 ±30%。本 spec 只用估算来「让阈值更早触发」（false-positive 方向，宁可早压不可晚压）。所有「降低 token 计数 / 跳过压缩」的代码路径仍应使用 `lastPromptTokenCount`（API 权威值）。 - -5. **估算函数与现有 `estimateContentChars` 的关系。** [compactionInputSlimming.ts](packages/core/src/services/compactionInputSlimming.ts) 已经有 `estimateContentChars`（用于压缩 split point 计算），新增的 `estimateContentTokens` 应复用它（除以 bytesPerToken）而非新写一套，避免两套估算口径出现分歧。 - -## 不在本 spec 范围 - -- Env 变量覆盖通道（D 方案）：维持「配置面最小」原则 -- Footer 常驻可视化：留作 follow-up -- 摘要 prompt 改进、`MIN_COMPRESSION_FRACTION` 调整：与阈值设计正交 - -## 开放问题（等 review） - -1. **breaking change 强度**：警告 + 忽略字段 vs 启动报错。当前选警告，需要确认对企业部署/团队配置是否够友好 - -## 已结案 - -2. **小窗口（≤ ~76.7K）下 hard 与 auto 退化为同一值** — 决定**不在 `/context` 明示**。理由： - - 塌缩范围不只是 32K，所有 `effectiveWindow - HARD_BUFFER ≤ 0.7 × window` 的窗口都塌缩（包括 64K） - - 用户行为不变：塌缩窗口上 `currentTier` 跳过 `'auto'` 直接报 `'hard'`（`contextCommand.ts:43-44` 先判 `>= hard`），`context-high` band（`auto ≤ t < hard`）变成空带，少一档提示在小窗口上是合理的——窗口本身就小，用户大概率手动管理上下文 - - 如果未来有真实用户报告"小窗口看不到中间档提示"，再决定加 UI 标注或调整 `context-high` 触发条件（这是 UI 工作，不是 spec 工作）。当前选不增加 UI 复杂度 diff --git a/docs/design/f2-mcp-transport-pool.md b/docs/design/f2-mcp-transport-pool.md new file mode 100644 index 00000000000..4da25cf8a0f --- /dev/null +++ b/docs/design/f2-mcp-transport-pool.md @@ -0,0 +1,1457 @@ +# F2: Shared MCP Transport Pool — Design v2.2 + +> Targets `daemon_mode_b_main` (per #4175 branching strategy). Replaces #4175 Wave 5 PR 23. +> **Single-PR delivery** per maintainer's feature-cohesive batch guidance (2026-05-19). +> Author: doudouOUC. Date: 2026-05-20. Revised: 2026-05-20 (v2.2 — implementation review fold-ins). + +--- + +## 0. Changelog + +### v2.2 (2026-05-20) — PR #4336 implementation + 32 review fold-ins + +PR #4336 shipped F2 as 6 atomic commits + 6 fix commits over ~4 hours. Wenshao reviewed cumulatively in 3 batches; each batch produced inline + critical fixes that were folded back. The table below records what changed vs. v2.1, organized by review batch. + +#### v2.1 → first-review batch (commits 1-4, wenshao C1-C7 + S1-S4) + +| # | Site | What was wrong | Fold-in commit | +| --- | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | +| C1 | `acpAgent.ts:269` — IDE-close path | Pool drain only ran in SIGTERM handler; IDE-initiated normal close leaked entries until OS reaped. Mirror SIGTERM's pool drain on `await connection.closed` | `ae0b296c4` | +| C2 | `mcp-pool-entry.ts:cancelDrainTimer` | `cancelDrainTimer` reset `maxIdleTimer` on every flap, defeating the §6.3 hard cap. Now only clears `drainTimer`; max-idle survives entire entry lifetime | `ae0b296c4` | +| C3 | `mcp-pool-entry.ts:doRestart` | Reconnect failure left entry in zombie state (`localStatus=CONNECTED`, `state='active'`, stale snapshot). Try/catch + transition to `'failed'` on failure | `ae0b296c4` | +| C4 | `mcp-pool-entry.ts:forceShutdown` | `state='closed'` set AFTER awaits, so concurrent `acquire` could observe `'active'` and hand out stale connection. Set synchronously at top | `ae0b296c4` | +| C5 | `mcp-transport-pool.ts:drainAll` | Concurrent `acquire` could spawn fresh entry mid-drain. Added `draining` mutex flag + `await Promise.allSettled(spawnInFlight)` before clearing | `ae0b296c4` | +| C6 | `mcp-pool-entry.ts:statusChangeListener` | Listener wasn't filtered by `serverName`; every entry got every server's status notifications + entry's own `markActive` write echoed back | `ae0b296c4` | +| C7 | `mcp-client-manager.ts:discoverAllMcpToolsIncremental` | Pool-mode gate added to `discoverAllMcpTools` but missed `Incremental` — `/mcp refresh` bypassed pool, spawned per-session client | `ae0b296c4` | +| S1 | `session-mcp-view.ts:passesSessionFilter` | Doc didn't call out that `excludeTools` uses direct equality (no parens-form support); divergence vs. `mcp-client.ts:isEnabled` | `ae0b296c4` | +| S2 | `pid-descendants.ts` docstring | Claimed Windows-specific `taskkill /F` branch that didn't exist — Node polyfills `process.kill('SIGTERM')` to `TerminateProcess` | `ae0b296c4` | +| S3 | `session-mcp-view.ts:applyTools` debug log | String contained literal `"N"` instead of interpolation — operators saw `applied 12 tools (filtered to N registered)` | `ae0b296c4` | +| S4 | `mcp-transport-pool.ts:createUnpooledConnection` status cb | Hardcoded to `() => CONNECTED` so `aggregateStatusByName` lied after disconnect. Now `() => client.getStatus()` | `ae0b296c4` | + +#### Commit-5 self-review batch (R1-R3 small) + +| # | Site | What was wrong | Fold-in commit | +| --- | ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | +| R1 | `server.test.ts:918` `/capabilities` envelope | Test asserted `getAdvertisedServeFeatures()` (no toggles) but server.ts passes `mcpPoolActive: opts.mcpPoolActive !== false` (default-on). Anchor toggle | `3e68c00bc` | +| R2 | `server.test.ts` capability default-on coverage | No test booted with default options to verify pool tags advertise. Added explicit `mcpPoolActive: false` test | `3e68c00bc` | +| R3 | `events.ts:DaemonMcpServerRestartRefusedData` | Doc said pre-PR SDKs would "see new value as unknown and surface generically" — actually `MCP_RESTART_REFUSED_REASONS.has(...)` rejects → silent drop | `3e68c00bc` | + +#### Second-review batch (commits 1-5, wenshao R1-R10) + +| # | Site | What was wrong | Fold-in commit | +| --- | --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | +| WR1 | `mcp-pool-entry.ts:maxIdleTimer` | C2 fix correctly preserved `maxIdleTimer` across flap, but fire-action force-closed regardless of `refs.size`. Active session with re-attach inside grace would lose tools after 5min | `72399f109` | +| WR2 | `mcp-client-manager.ts:discoverAllMcpToolsViaPool` | `releaseAllPooledConnections` + re-acquire ALL on every pass left brief window with zero MCP tools registered AND bounced every drain timer. Diff against desired `(name, fingerprint)` | `72399f109` | +| WR3 | `mcp-pool-entry.ts:doRestart` snapshot fan-out | Restart updated `toolsSnapshot`/`promptsSnapshot` and emitted typed events — but no `SessionMcpView` instance subscribed to that stream. Iterate `subscribers` directly post-snapshot | `72399f109` | +| WR4 | `mcp-transport-pool.ts:getSnapshot subprocessCount` | Counted websocket toward `subprocessCount` — websocket dials remote, no local child. Restricted to `'stdio'` only | `72399f109` | +| WR5 | `pid-descendants.ts` PowerShell `-Filter` | Interpolated `${pid}` directly into `-Filter` string. Entry-point `Number.isInteger` guard prevents injection today; bind to `$p` for defense-in-depth against future guard relaxations | `72399f109` | +| WR6 | `mcp-pool-entry.ts` ctor `cfg` field | `readonly cfg: MCPServerConfig` was implicitly public, exposing env API keys / header auth / OAuth fields. Made `private`; new `transportKind` getter for the only external reader | `72399f109` | +| WR7 | `mcp-pool-events.ts` premature exports | 5 PoolEvent type guards + `Prompt` re-export + `PoolEntryConnectionStatus` had zero callers. Removed; kept `MCPCallInterruptedError` (design §13.4 mandate) | `72399f109` | +| WR8 | `acpAgent.ts:269,300` pool drain duplication | SIGTERM + IDE-close had identical `if (agentInstance) { try { await shutdownMcpPool(8_000) } catch... }` blocks. Extracted `drainPoolBeforeExit(label)` helper | `72399f109` | + +#### Commit-6 self-review batch (R1-R3 critical race) + +| # | Site | What was wrong | Fold-in commit | +| --- | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | +| 6R1 | `mcp-transport-pool.ts:onClosed` | Slot-release race: A finishes spawn, B (different fingerprint, same name) starts spawn, A drains. Close-cb checked only `entries` (B not yet registered) → premature release | `0e58a098f` | +| 6R2 | `events.ts:mcpBudgetWarningCount` JSDoc | Workspace-scoped events fan to N sessions → N reducer increments; consumers aggregating across sessions double-count. Docstring updated to call out the multiplier | `0e58a098f` | +| 6R3 | `acpAgent.ts:broadcastBudgetEvent` | Iterated `this.sessions.keys()` directly during async fan-out; concurrent `killSession` could corrupt iterator. Snapshot via `Array.from(...)` | `0e58a098f` | + +#### Third-review batch (commits 1-6, wenshao W1-W15) + +| # | Site | What was wrong | Fold-in commit | +| --- | -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | +| W1 | `mcp-transport-pool.ts:spawnEntry` catch | Spawn failure leaked `statusChangeListener` permanently — only `forceShutdown` removes it. Added `entry.forceShutdown('manual')` to catch | `4a3c5cd90` | +| W2 | `mcp-pool-entry.ts:statusChangeListener` cross-check | Module-level `serverStatuses` map shared across multi-fingerprint entries. A's transport error wrote DISCONNECTED, B's listener corrupted B's `localStatus`. Added `client.getStatus()` check | `4a3c5cd90` | +| W3 | `mcp-pool-entry.ts:doRestart` pid sweep | Restart skipped `listDescendantPids` + `sigtermPids` — every restart of `npx`/`uvx`-wrapped stdio orphaned the actual MCP grandchild. Added sweep before disconnect | `4a3c5cd90` | +| W4 | `mcp-pool-entry.ts:doRestart` drain timer race | Drain timer could fire mid-restart yield → `forceShutdown` removes entry → `client.connect` spawns orphan. Added `cancelDrainTimer` + `state→active` at top of `doRestart` | `4a3c5cd90` | +| W5 | `mcp-client-manager.ts:pooledConnections` dead handles | When entry transitioned to `'failed'`, manager held dead `PooledConnection` forever. Subscribe to entry events; evict on `'failed'` (idempotent via `get(name) === conn` guard) | `4a3c5cd90` | +| W6 | `mcp-client-manager.ts:discoverAllMcpToolsViaPool` re-entrancy | Two passes interleaving could both `set(name, conn)` → first conn leaked. Added `discoveryInFlight` mutex; second caller awaits same promise. New regression test | `4a3c5cd90` | +| W9 | `acpAgent.ts:parsePoolDrainMs` strictness | `Number.parseInt` accepted `'30000ms'` / `'30000abc'`. Strict `^\d+$` regex; reject with stderr warning + default fallback | `4a3c5cd90` | +| W10 | `mcp-transport-pool.ts:acquire` indexAttach order | `indexAttach` mutated `sessionToEntries` BEFORE `entry.attach()`. If `attach` threw, stale reverse-index mapping. Moved `indexAttach` after `attach` succeeds (both fast + in-flight paths) | `4a3c5cd90` | +| W13 | `mcp-transport-pool.ts:subprocessCount` JSDoc | Doc still claimed `stdio + websocket` after WR4 restricted to stdio. Updated | `4a3c5cd90` | +| W14 | `mcp-transport-pool.ts:createUnpooledConnection` catch | Same `statusChangeListener` leak as W1 in the unpooled path. Same mirror: `forceShutdown` before disconnect | `4a3c5cd90` | +| W15 | `bridge.ts:restartMcpServer` response | `as PoolEntries` cast was unsound — untyped JSON from ACP child. `Array.isArray` check + per-entry shape guard; malformed entries skipped with stderr breadcrumb | `4a3c5cd90` | + +#### Declined-with-reply (filed as F2 follow-ups) + +| # | Site | Reason for declining | +| --- | --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| W7 | Test coverage gaps (4 untested critical paths) | 1/4 added (W6 regression test); rest deferred to focused test-coverage PR after F2 series merges | +| W8 | `maxReconnectAttempts` / `reconnectStrategy` unused | Forward-compat placeholders for the deferred health-monitor-driven reconnect (design §6.6); removing + re-adding churns the public type | +| W11 | Duplicate fast-path / in-flight-path attach blocks | ✅ Done in PR A: `attachPooledSession` + `rollbackReservationOnSpawnFailure` private helpers (commit `2d546efca`) | +| W12 | `passesSessionFilter` O(M×N) per `applyTools` | ✅ Done in PR A: `applyTools` / `applyPrompts` precompute filter `Set`s once per pass; predicate becomes O(1) per tool (commit `a4a855ab3`) | +| R9 | `McpClientManager` ctor 7-positional sentinels | ✅ Done in PR A: options-object ctor + `mkManager` test factory (commit `0cb1eaa27`) | +| R10 | `pgrep -P ` per-PID-per-level cost | ✅ Done in PR A: single `ps -A -o pid=,ppid=` snapshot + in-memory BFS walk; pgrep BFS retained as fallback for BusyBox >` reverse index in pool (§6) | `releaseSession` O(N entries) → O(refs of session); needed for 1000-session scale | +| V21-3 | `?fingerprint=` query param on restart route (§13.1) | Operator may want to restart only one entry when same name has multiple fingerprints; near-zero cost to add now | +| V21-4 | Spawn-failure path explicitly releases reserved slot (§6.1, §6.5) | Otherwise slot leaks until next health-monitor pass; subtle real bug | +| V21-5 | New §13.4: in-flight tool call during reconnect semantics | `MCPCallInterruptedError`; pool does NOT auto-replay (writes unsafe) | +| V21-6 | New §10.4: `/mcp disable X` triggers `SessionMcpView` re-apply | Otherwise mid-session disable doesn't drop already-registered tools | +| V21-7 | Status route exposes `entryIndex` not raw fingerprint (§8.3) | Avoids side-channel exposure of OAuth token rotation via fingerprint change | +| V21-8 | Reconnect backoff spec'd: stdio fixed 5s × 3, HTTP/SSE exponential 1/2/4/8/16s × 5 (§6.6) | v2 didn't say; HTTP needs longer retry budget for network flap | +| V21-9 | `canonicalOAuth(o)` normalizes `{enabled: false}` ≡ `undefined` ≡ `null` (§5.1) | Otherwise functionally equivalent configs produce distinct entries | +| V21-10 | Renamed pool fallback helper from "legacy in-process acquire" to `createUnpooledConnection` (§5.3, §6.1) | SDK MCP bypass is permanent, not legacy | +| V21-11 | `drainAll(opts?)` returns `Promise` with `timeoutMs` wall-clock budget (§17) | Caller needs to know when drain finishes for shutdown ordering | +| V21-12 | Locked SDK reducer field names (Q1 resolved): keep `mcpBudgetWarningCount` etc. with scope semantics in JSDoc | No public-API rename mid-PR | +| V21-13 | Locked Q3 (default pool-on, `--no-mcp-pool` kill switch), Q4 (HTTP/SSE opt-in), Q6 (eager construction) | Single-PR delivery; no flag gating needed | +| V21-14 | Added R9/R10/R11 single-PR risks (§23) | Review fatigue, daemon_mode_b_main merge conflict, CI time | +| V21-15 | Extension uninstall orphan entry handling deferred to `MAX_IDLE_MS` natural reap (§16.3) | No explicit `invalidateByExtension`; keeps model uniform | + +### v2 (2026-05-20) — initial review fold-ins from v1 sketch + +| # | What | Why | +| --- | ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | +| C1 | Pool fans out **Tools + Prompts** (was: tools only) | `McpClient` ctor takes both registries; prompts otherwise silently lost in pool mode | +| C2 | New section on **global state coexistence** (`serverStatuses` / `mcpServerRequiresOAuth` module Maps) | Cross-session sharing already exists today; pool inherits + formalizes | +| C3 | `connectToMcpServer` factory path **unified** with `McpClient` class in F2-1 | v1 only refactored the class; would leave a parallel non-pooled path | +| C4 | Snapshot replay on attach (earlyEvents-style) added to `PoolEntry.attach()` | New race: session-B attaches → server emits `tools/list_changed` before subscription wired | +| C5 | `spawnInFlight: Map>` for concurrent-acquire dedupe | v1 mentioned in test matrix but missed in implementation contract | +| C6 | Cross-platform descendant-pid sweep (Linux/macOS pgrep, Windows wmic/PowerShell) | v1 said "copy opencode's `pgrep -P`" — that's Unix-only | +| C7 | `trust` field per-session **copy** of tool object | trust lives on `DiscoveredMCPTool`; shared instance would mix per-session trust | +| C8 | HTTP/SSE transports **opt-in** to pooling (default: stdio + websocket only) | Some MCP HTTP servers maintain per-transport session state; sharing risks state-bleed | +| C9 | SDK MCP server (`isSdkMcpServerConfig`) explicit bypass | `sendSdkMcpMessage` is per-session by design | +| C10 | OAuth path explicitly **deferred to F3** | OAuth flow needs PermissionMediator-style routing; not F2 scope | +| C11 | Restart route semantics spec'd (name → all matching entries) | PR 17's `POST /workspace/mcp/:server/restart` previously unambiguous (1 entry); now 1..N | +| C12 | Status route refactor section (new path: `QwenAgent.getMcpPoolAccounting()`) | `httpAcpBridge.ts:733-770` currently reads bootstrap session's manager — must change | +| C13 | Generation counter on `PoolEntry` for stale `tools/list_changed` handler guard | Opencode pattern: `if (s.clients[name] !== client) return` | +| C14 | Sub-PR breakdown 4 → **6** | v1 underestimated; A2/B1/B3/C6 each add real work | +| C15 | Lazy pool construction (only when N≥2 sessions seen) — optional | `qwen serve --foreground` single-session won't benefit; saves init cost | + +--- + +## 1. Goals / Non-goals + +**Goals** + +- N sessions in 1 workspace sharing 1 process per unique-server-config — fingerprint-keyed +- Per-session `ToolRegistry` / `PromptRegistry` views preserved (filtering, trust) +- Refcount + grace-drain lifecycle resilient to reattach +- Cross-platform descendant-pid cleanup +- Budget guardrails graduate from per-session to per-workspace (PR 14 promised this) +- Backward compat with non-daemon standalone qwen (pool not constructed there) + +**Non-goals (F2 scope)** + +- Cross-workspace pooling (1 daemon = 1 workspace invariant from PR #4113 stands) +- Cross-daemon pooling (out of scope — multi-process orchestrator territory) +- OAuth routing rework (F3 with `PermissionMediator`) +- Pool persistence across daemon restart (in-memory only) +- Auto-detection of "pool-safe" HTTP servers (opt-in flag only) +- Live `MCPServerConfig` diff to in-place mutate entries (config change → new entry, old drains) + +--- + +## 2. Current State (replacement target) + +``` +acpAgent.newSession(sessionId) + → newSessionConfig(cwd, mcpServers) // acpAgent.ts:1771 + → loadCliConfig → new Config → config.initialize() + → ToolRegistry ctor → new McpClientManager(config, ...) // tool-registry.ts:199 + → for (name, cfg) in config.getMcpServers(): + new McpClient(name, cfg, toolRegistry, promptRegistry, workspaceContext, ...) + → client.connect() → client.discover(config) +``` + +**Coupling map (what must be broken or threaded through):** + +| Coupling | Location | Action in F2 | +| -------------------------------------------------------------------------------- | ------------------------------------------------- | ----------------------------------------------------------------------------------- | +| `McpClient` ctor binds 1 ToolRegistry + 1 PromptRegistry | mcp-client.ts:106-119 | Pool owns transport; `SessionMcpView` (per session) owns the per-session registries | +| `McpClient.discover()` calls `toolRegistry.registerTool()` inline | mcp-client.ts:178-198 | Split: `discoverAndReturn()` returns snapshot; view registers | +| `ListRootsRequestSchema` handler closes over `workspaceContext.getDirectories()` | mcp-client.ts:142-153 + connectToMcpServer.ts:893 | Pool's single workspace-bound context | +| `workspaceContext.onDirectoriesChanged` listener registered per connect | mcp-client.ts:907 | Pool registers once per entry | +| `McpClientManager` `new`'d inside ToolRegistry | tool-registry.ts:199 | Add optional `pool?` ctor param; injection from Config | +| Budget enforcement per-session | mcp-client-manager.ts:91-95 comment | Move state machine into pool | +| `serverDiscoveryPromises` dedupe in-flight per server | mcp-client-manager.ts:350 | Pool has `spawnInFlight: Map>` | +| `setMcpBudgetEventCallback` per-session registration | acpAgent.ts:1851-1899 | Pool emits → `QwenAgent` broadcasts to all sessions | + +**Already-shared state (pool inherits, does not introduce):** + +| State | Location | Note | +| ---------------------------------------------- | -------------------------------- | ----------------------------------------------------------------- | +| `serverStatuses: Map` | mcp-client.ts:292 (module-level) | Process-wide today; pool key still by name → "any-CONNECTED-wins" | +| `mcpServerRequiresOAuth: Map` | mcp-client.ts:302 (module-level) | Same | +| `MCPOAuthTokenStorage` on-disk tokens | `~/.qwen/mcp-oauth/.json` | Daemon-host shared; pool just exploits more efficiently | + +--- + +## 3. Reference Findings + +| Project | Pool? | Key | Lifecycle | Patterns to steal | +| --------------- | ------------------ | --------------------------------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| **claude-code** | No, per-process | `name + JSON.stringify(cfg)` (lodash.memoize) | `clearServerCache` + remote backoff×5; stdio crash → `failed` | Sorted-key SHA-256 `hashMcpConfig` for invalidation/keying | +| **opencode** | Yes, per workspace | server **name only** (no config hash) | No refcount / no eviction / no restart; Effect finalizer + `pgrep -P` recursive SIGTERM | Descendant-pid sweep, stale-handler guard (`if (s.clients[name] !== client) return`), `tools/list_changed` fan-out via event bus | + +**What F2 inherits from each:** config-hash from claude-code (handles per-session env/auth divergence opencode doesn't), descendant-pid sweep from opencode (npx/uvx wrappers leak). What we add: refcount + drain (multi-client daemon), auto-restart (long-running daemon), prompt fan-out, generation guard. + +--- + +## 4. Architecture + +### 4.1 Process layout + +``` +HTTP daemon (packages/cli/src/serve, qwen serve) + │ spawns + ▼ +ACP child (qwen --acp, single process per workspace) + │ + QwenAgent (acpAgent.ts) + ├── McpTransportPool ◄── new, workspace-scoped, 1 instance + │ ├── entries: Map + │ ├── spawnInFlight: Map> + │ ├── workspaceContext (bound to daemon workspace) + │ └── budget guardrails (PR 14 state machine, graduated to workspace) + │ + └── sessions: Map + └── Session.Config → ToolRegistry → McpClientManager(pool?) + │ + ┌────────┴────────┐ + │ pool injected │ + ▼ ▼ + pool.acquire(name,cfg,sid) legacy in-process + → SessionMcpView (standalone qwen) + .applyTools/Prompts + (filter + register into + session's own registries) +``` + +**Pool lives in the ACP child**, not the HTTP daemon. The HTTP daemon queries pool state via the existing `bridge.client` extMethod surface (`getMcpPoolAccounting`, `restartMcpServer`). F2 code lives in **`packages/core/src/tools/`** (peer of `mcp-client-manager.ts`), not `packages/acp-bridge/`. + +### 4.2 Class diagram + +``` +McpTransportPool + ├─ acquire(name, cfg, sid) → PooledConnection + ├─ release(connectionId, sid) → void + ├─ releaseSession(sid) → void (bulk release for session teardown) + ├─ restartByName(name) → RestartResult[] + ├─ getAccounting() → McpClientAccounting (workspace-scope) + ├─ getBudgetMode/Budget() + ├─ drainAll() → Promise (shutdown) + └─ onBudgetEvent: (event) => void (set by QwenAgent) + +PoolEntry (internal) + ├─ refs: Set + ├─ client: McpClient + ├─ toolsSnapshot: DiscoveredMCPTool[] + ├─ promptsSnapshot: Prompt[] + ├─ generation: number (++ on reconnect; stale-event guard) + ├─ state: 'spawning' | 'active' | 'draining' | 'closed' | 'failed' + ├─ drainTimer?: NodeJS.Timeout + ├─ healthMonitor: { intervalTimer, consecutiveFailures, isReconnecting } + ├─ subscribers: Map + ├─ attach(sid, view) → PooledConnection + └─ detach(sid) → void + +PooledConnection (handle returned to caller) + ├─ id: ConnectionId + ├─ on('toolsChanged' | 'promptsChanged' | 'disconnected' | 'reconnected' | 'failed', cb) + ├─ callTool(name, args, { sessionId }) → CallToolResult + ├─ readResource(uri, { sessionId, signal }) + └─ release() + +SessionMcpView (per session, per server) + ├─ ctor(toolRegistry, promptRegistry, sessionId, serverName, cfg) + ├─ applyTools(snapshot) → void (filters by include/exclude, decorates trust) + ├─ applyPrompts(snapshot) → void + └─ teardown() → void (removes its registrations) +``` + +--- + +## 5. Pool Key (Fingerprint) + +### 5.1 Hashed canonical fields + +```ts +type PoolKey = string; // sha256 hex, first 16 chars sufficient (collision-free for realistic N) +type ConnectionId = `${serverName}::${PoolKey}`; + +function fingerprint(cfg: MCPServerConfig): PoolKey { + const canonical = { + transport: mcpTransportOf(cfg), + command: cfg.command ?? null, + args: cfg.args ?? [], + cwd: cfg.cwd ?? null, + env: sortedEntries(cfg.env ?? {}), // [[k,v],...] sorted by k + url: cfg.url ?? null, + httpUrl: cfg.httpUrl ?? null, + headers: sortedEntries(cfg.headers ?? {}), + timeout: cfg.timeout ?? null, + oauth: canonicalOAuth(cfg.oauth), + }; + return sha256(JSON.stringify(canonical)).slice(0, 16); +} + +/** + * V21-9: normalize functionally-equivalent OAuth configs so they + * collapse to the same fingerprint. `{enabled: false}`, `undefined`, + * `null`, and `{}` all mean "no OAuth" → all return `null`. + */ +function canonicalOAuth(o?: OAuthConfig | null): OAuthConfig | null { + if (!o || !o.enabled) return null; + return { + enabled: true, + clientId: o.clientId ?? null, + scopes: o.scopes ? [...o.scopes].sort() : null, + authorizationUrl: o.authorizationUrl ?? null, + tokenUrl: o.tokenUrl ?? null, + }; +} + +// Excluded fields (per-session filters, NOT transport-level): +// includeTools, excludeTools, trust, description, extensionName +``` + +### 5.2 Transport-class gating + +```ts +const POOLED_TRANSPORTS_DEFAULT = new Set(['stdio', 'websocket']); + +function isPoolable(cfg: MCPServerConfig, opts: PoolOptions): boolean { + if (isSdkMcpServerConfig(cfg)) return false; + const transport = mcpTransportOf(cfg); + return opts.pooledTransports.has(transport); +} +``` + +**Default `pooledTransports = {stdio, websocket}`**. Operators opt HTTP/SSE in via: + +- CLI: `--mcp-pool-transports=stdio,websocket,http,sse` +- Env: `QWEN_SERVE_MCP_POOL_TRANSPORTS=stdio,websocket,http` + +**Why default exclude HTTP/SSE**: some MCP HTTP server implementations bind state (auth context, conversation memory) to the TCP/SSE stream; multiple ACP sessions sharing it would bleed state. stdio + websocket are true OS processes whose state is observable and isolatable. + +### 5.3 SDK MCP bypass + +`isSdkMcpServerConfig(cfg)` true → pool returns a thin `PooledConnection` wrapper via `createUnpooledConnection(name, cfg, sid)` that constructs an `McpClient` immediately, no sharing, no entry stored in pool. Reason: `sendSdkMcpMessage` is per-session by design (routes through ACP control plane back to the originating session). Same path used for HTTP/SSE when transport not in `pooledTransports` (§10.3). + +V21-10: name is `createUnpooledConnection`, not `legacyInProcessAcquire` — SDK MCP and HTTP-opt-out are permanent design choices, not legacy code. + +--- + +## 6. Lifecycle + +### 6.1 acquire / release + +```ts +class McpTransportPool { + private entries = new Map(); + private spawnInFlight = new Map>(); + + /** V21-2: reverse index, O(refs) releaseSession instead of O(entries). */ + private sessionToEntries = new Map>(); + + async acquire( + name: string, + cfg: MCPServerConfig, + sid: string, + ): Promise { + if (!isPoolable(cfg, this.opts)) { + return this.createUnpooledConnection(name, cfg, sid); + } + const id: ConnectionId = `${name}::${fingerprint(cfg)}`; + + if (this.entries.has(id)) { + this.indexAttach(sid, id); + return this.entries.get(id)!.attach(sid); + } + let inFlight = this.spawnInFlight.get(id); + if (!inFlight) { + const slot = this.tryReserveSlot(name); + if (slot === 'refused') { + throw new BudgetExhaustedError( + name, + this.clientBudget!, + this.reservedSlots.size, + ); + } + inFlight = this.spawnEntry(name, cfg, id) + .catch((err) => { + // V21-4: release reserved slot on spawn failure. Without + // this, slot leaks until health monitor's release path + // runs (which it doesn't, because there's no entry to monitor). + if (slot === 'reserved') this.releaseSlotName(name); + throw err; + }) + .finally(() => this.spawnInFlight.delete(id)); + this.spawnInFlight.set(id, inFlight); + } + const entry = await inFlight; + this.indexAttach(sid, id); + return entry.attach(sid); + } + + release(id: ConnectionId, sid: string): void { + const entry = this.entries.get(id); + if (!entry) return; + entry.detach(sid); + this.indexDetach(sid, id); + if (entry.refs.size === 0) entry.startDrainTimer(this.opts.drainDelayMs); + } + + /** V21-2: O(refs of this session), not O(all entries). */ + releaseSession(sid: string): void { + const ids = this.sessionToEntries.get(sid); + if (!ids) return; + for (const id of ids) { + const entry = this.entries.get(id); + if (!entry) continue; + entry.detach(sid); + if (entry.refs.size === 0) entry.startDrainTimer(this.opts.drainDelayMs); + } + this.sessionToEntries.delete(sid); + } + + private indexAttach(sid: string, id: ConnectionId): void { + let ids = this.sessionToEntries.get(sid); + if (!ids) { + ids = new Set(); + this.sessionToEntries.set(sid, ids); + } + ids.add(id); + } + + private indexDetach(sid: string, id: ConnectionId): void { + const ids = this.sessionToEntries.get(sid); + if (!ids) return; + ids.delete(id); + if (ids.size === 0) this.sessionToEntries.delete(sid); + } +} +``` + +### 6.2 Concurrent-acquire dedupe (`spawnInFlight`) + +Mirrors `McpClientManager.serverDiscoveryPromises` (mcp-client-manager.ts:350). Without it, 5 sessions spawning at boot all see `entries.has(id) === false` and race to spawn 5 child processes. + +### 6.3 Drain grace + idle cap + +```ts +const DRAIN_DELAY_MS_DEFAULT = 30_000; // grace after last release +const MAX_IDLE_MS_DEFAULT = 5 * 60_000; // hard cap (defense against drain cancellation loop) +``` + +State machine in `PoolEntry`: + +``` +spawning ──spawn ok──► active ──last detach──► draining ──timeout──► closed + │ │ │ + │ │ └──attach──► active (cancel timer) + spawn fail───────────►failed + │ + └──manual restart──► spawning +``` + +Hard idle cap: drain timer can be cancelled+restarted indefinitely (acquire/release flap). `MAX_IDLE_MS` is a separate timer started **at first idle** and never reset; when it fires, force-close even if drain is currently in active grace. Prevents zombie pool entries from buggy clients that thrash acquire/release. + +### 6.4 Cross-platform descendant-pid sweep + +**R10 / R23 T7 / PR A update (2026-05-22)**: switched from per-pid BFS (one `pgrep -P ` / `Get-CimInstance -Filter` subprocess per node) to a single process-table snapshot followed by in-memory tree walk. Two motivations: (1) one fork instead of B^D forks on the hot pool-shutdown path; (2) snapshot consistency — pre-fix BFS could miss descendants that forked between adjacent BFS levels. Per-pid path retained as fallback for BusyBox `ps` { + if (!Number.isInteger(rootPid) || rootPid <= 0) return []; + try { + if (process.platform === 'win32') + return await listDescendantPidsWin(rootPid); + return await listDescendantPidsUnix(rootPid); + } catch { + return []; // OS reaps orphans; pool shutdown still proceeds. + } +} + +async function listDescendantPidsUnix(root: number): Promise { + let tree: Map | undefined; + try { + tree = await snapshotProcessTreeUnix(); // ps -A -o pid=,ppid= + } catch { + /* fall through to fallback */ + } + if (tree) return walkDescendants(tree, root); // O(descendants), 1 fork + return await listDescendantPidsUnixPgrepFallback(root); // legacy BFS +} + +async function snapshotProcessTreeUnix(): Promise> { + // -A: all processes (POSIX, equivalent to -e but unambiguous on BSD). + // -o pid=,ppid=: pid + ppid columns, trailing `=` suppresses headers. + const { stdout } = await execFile('ps', ['-A', '-o', 'pid=,ppid='], { + timeout: 2000, + maxBuffer: 8 * 1024 * 1024, // covers >250k-process pathological hosts + }); + const childrenByPpid = new Map(); + for (const line of stdout.split('\n')) { + const m = line.trim().match(/^(\d+)\s+(\d+)$/); + if (!m) continue; + /* parse, push into childrenByPpid */ + } + return childrenByPpid; +} + +// Windows: single Get-CimInstance Win32_Process | ConvertTo-Csv snapshot +// of all (ProcessId, ParentProcessId) rows + in-memory walk; per-pid +// `Get-CimInstance -Filter "ParentProcessId=$p"` retained as fallback. +``` + +Called from `PoolEntry.shutdown()` before `client.disconnect()`. Handles `npx @modelcontextprotocol/server-X`, `uvx ...`, `pnpm dlx ...` wrapper leaks. MAX_DESCENDANTS=256 / MAX_DEPTH=8 caps preserved. + +### 6.5 Spawn failure handling + +If `spawnEntry` rejects after multiple subscribers attached (via `spawnInFlight`): + +- All awaiters get the rejection +- `tryReserveSlot` released **via explicit `.catch` arm in `acquire`** (V21-4); without this fix the slot leaked until next health-monitor pass, which never ran because no entry existed to monitor. +- Failed entry NOT stored in `entries` +- Subscribers' code paths handle as if `acquire` originally failed (existing per-session `discoverMcpToolsForServer` catch logic remains valid) + +### 6.6 Reconnect backoff (V21-8) + +When a `PoolEntry` enters reconnect after transport drop: + +| Transport family | Strategy | Cap | +| ---------------- | -------------------------------------------- | ---------------------------------------------------------------- | +| stdio | Fixed 5s × 3 attempts | Per existing `DEFAULT_HEALTH_CONFIG.reconnectDelayMs` | +| websocket | Fixed 5s × 3 attempts | Same as stdio | +| http (opt-in) | Exponential 1s, 2s, 4s, 8s, 16s × 5 attempts | Remote endpoints flap on transient network issues; longer budget | +| sse (opt-in) | Exponential 1s, 2s, 4s, 8s, 16s × 5 attempts | Same as http | + +After cap exhaustion: entry transitions to `failed` state; subscribers receive `failed` event; new `acquire` for same `ConnectionId` retries spawn once, then throws. Operator restart (§13) resets state. + +--- + +## 7. Discovery / SessionMcpView + +### 7.1 Tools + Prompts dual fan-out + +```ts +// packages/core/src/tools/mcp-client.ts — split discover into pure +async discoverAndReturn(cliConfig: Config): Promise<{ + tools: DiscoveredMCPTool[]; + prompts: Prompt[]; +}> { + if (this.status !== MCPServerStatus.CONNECTED) throw new Error('Client is not connected.'); + try { + const [prompts, tools] = await Promise.all([ + discoverPrompts(this.serverName, this.client, /* no registry */), + discoverTools(this.client, this.serverConfig, this.serverName, this.debugMode, this.workspaceContext), + ]); + if (prompts.length === 0 && tools.length === 0) { + throw new Error('No prompts or tools found on the server.'); + } + return { tools, prompts }; + } catch (e) { + this.updateStatus(MCPServerStatus.DISCONNECTED); + throw e; + } +} + +// Legacy discover() retained, delegates to discoverAndReturn + registers (for standalone qwen) +async discover(cliConfig: Config): Promise { + const { tools, prompts } = await this.discoverAndReturn(cliConfig); + for (const t of tools) this.toolRegistry.registerTool(t); + for (const p of prompts) this.promptRegistry.registerPrompt(p); +} +``` + +```ts +class SessionMcpView { + applyTools(snapshot: DiscoveredMCPTool[]) { + this.sessionToolRegistry.removeToolsByServer(this.serverName); + for (const tool of snapshot) { + if (!this.passesFilter(tool)) continue; + // C7: per-session copy of trust (don't mutate shared snapshot) + const localTool = tool.withTrust(this.cfg.trust); + this.sessionToolRegistry.registerTool(localTool); + } + } + applyPrompts(snapshot: Prompt[]) { + this.sessionPromptRegistry.removePromptsByServer(this.serverName); + for (const p of snapshot) this.sessionPromptRegistry.registerPrompt(p); + } +} +``` + +### 7.2 Snapshot replay on attach (earlyEvents-style) + +```ts +class PoolEntry { + attach(sid: string): PooledConnection { + this.refs.add(sid); + this.cancelDrainTimer(); + const view = new SessionMcpView(...); + this.subscribers.set(sid, view); + // Immediately replay current snapshot so subscriber doesn't miss + // updates that landed between in-flight discover completion and + // attach. + if (this.state === 'active') { + view.applyTools(this.toolsSnapshot); + view.applyPrompts(this.promptsSnapshot); + } + return this.makeHandle(sid, view); + } +} +``` + +Mirrors PR 14b fix #1's `BridgeClient.earlyEvents` pattern — solves analogous race for pool attachment. + +### 7.3 Stale-handler guard (generation counter) + +```ts +class PoolEntry { + private generation = 0; + + private async reconnect(): Promise { + this.generation += 1; + const myGen = this.generation; + await this.client.disconnect(); + await this.client.connect(); + if (myGen !== this.generation) return; // superseded by another reconnect + const snap = await this.client.discoverAndReturn(this.cfg); + if (myGen !== this.generation) return; + this.toolsSnapshot = snap.tools; + this.promptsSnapshot = snap.prompts; + this.fanOut('toolsChanged'); + this.fanOut('promptsChanged'); + } + + private onServerToolsListChanged = () => { + const myGen = this.generation; + this.client + .discoverAndReturn(this.cfg) + .then((snap) => { + if (myGen !== this.generation) return; + this.toolsSnapshot = snap.tools; + this.fanOut('toolsChanged'); + }) + .catch(/* swallow + log */); + }; +} +``` + +Without this, a stale handler from a pre-reconnect Client instance could overwrite the post-reconnect snapshot with stale data. + +**Monotonicity invariant** (V21 clarification): `generation` only increments, never resets. Any in-flight operation captures `myGen` at entry, then post-`await` checks `myGen === this.generation`. Equivalent to "no superseding event has happened since I started". Bounded by Number.MAX_SAFE_INTEGER (~285k years at 1Hz reconnect), no overflow concern. + +### 7.4 Path unification (F2-1 scope expansion) + +`packages/core/src/tools/mcp-client.ts` has TWO connect-to-server paths: + +1. `McpClient` class (mcp-client.ts:100) — used by `McpClientManager` +2. `connectToMcpServer` factory function (mcp-client.ts:875) — used by `discoverMcpTools` (line 560) and `connectAndDiscover` (line 607) + +F2-1 must converge both behind `McpClient.discoverAndReturn` (with `connectToMcpServer` becoming a private helper of `McpClient` or both calling a shared `establishConnection()` primitive). Otherwise pool only covers the class path; the factory path remains per-session and undermines the whole effort. + +--- + +## 8. Global State Coexistence + +### 8.1 `serverStatuses` (mcp-client.ts:292) — collision-tolerant write + +Module-level `Map`. Pool's `ConnectionId` is `name::hash`, but `updateMCPServerStatus(name, status)` writes by name. **Multiple pool entries for same name (different fingerprints, e.g. token-divergence) would clobber each other's status.** + +**Resolution**: pool intercepts status writes: + +```ts +class PoolEntry { + updateStatus(s: MCPServerStatus) { + this.localStatus = s; + const aggregated = this.pool.aggregateStatusByName(this.serverName); + updateMCPServerStatus(this.serverName, aggregated); + } +} + +class McpTransportPool { + aggregateStatusByName(name: string): MCPServerStatus { + // Any CONNECTED ⇒ CONNECTED + // Else any CONNECTING ⇒ CONNECTING + // Else DISCONNECTED + const entries = [...this.entries.values()].filter( + (e) => e.serverName === name, + ); + if (entries.some((e) => e.localStatus === CONNECTED)) return CONNECTED; + if (entries.some((e) => e.localStatus === CONNECTING)) return CONNECTING; + return DISCONNECTED; + } +} +``` + +Status route surfaces `entryCount: number` so operators see when name → multiple entries. + +### 8.2 OAuth token storage + +`MCPOAuthTokenStorage` writes to `~/.qwen/mcp-oauth/.json` — already daemon-host-shared. Pool benefits incidentally (first session's OAuth completes → token on disk → pool entry's reconnect picks up token → all other sessions piggy-back). + +**Caveat — multi-fingerprint case**: 2 entries for same name (different headers/env) but same OAuth provider → both read the same token file. If tokens are server-scoped (OAuth typical), this works. If tokens are env-scoped (rare), explicit storage key extension needed. **Punt to F3** with a documented known-limitation. + +### 8.3 `entryCount` in snapshot + +`GET /workspace/mcp` per-server cell adds: + +```ts +{ + kind: 'mcp_server', + name: 'github', + status: 'ok', + mcpStatus: 'connected', + entryCount: 2, // NEW — N pool entries for this name + entrySummary?: [ // NEW — opaque per-entry breakdown + { entryIndex: 0, refs: 2, status: 'connected' }, + { entryIndex: 1, refs: 1, status: 'connecting' }, + ], + ... +} +``` + +**V21-7**: `entrySummary[].entryIndex` is a **stable opaque integer** assigned at entry creation (insertion order within name group), NOT the raw fingerprint. Reasoning: fingerprint changes when OAuth tokens or env vars rotate, which would leak that information through snapshot diffs (operator could infer "token rotated at T+5min" from `'a3b1' → 'f972'` transition). `entryIndex` is monotonic within name group but stays stable across rotations because old entry drains and new entry gets next index. + +Old SDK clients ignore unknown fields per PR 14 contract; new clients use `entryCount` for badges. Internal restart-by-fingerprint path uses an opaque token returned only via privileged extMethod, not exposed in HTTP snapshot. + +--- + +## 9. WorkspaceContext / ListRoots + +### 9.1 Single registration + +Pool's `McpClient` instances share **one** `WorkspaceContext` — the daemon's bound workspace context (PR #4113 invariant). `connectToMcpServer`'s `ListRootsRequestSchema` handler closes over this single context. + +`onDirectoriesChanged` listener registered **once per entry**, not once per `acquire`. Detached on entry shutdown. + +### 9.2 `roots/list_changed` fan-up + +Server notifies client of new roots → pool fans out: + +- Pool re-discovers (server may report different tool set under new roots) → `toolsChanged` event → all subscriber views re-apply + +### 9.3 Per-session `updateWorkspaceDirectories` + +**Contract**: in Mode B, per-session directory additions are a soft hint, not authoritative. Pool's `WorkspaceContext` is daemon-level. + +Two implementations choices: + +- **v1 simple**: ignore per-session adds, log warning when detected +- **v2 union**: pool maintains `extraRoots: Map>`, ListRoots handler returns union of bound workspace + all extras. Per-session removal triggers `roots/list_changed`. Adds 50-80 LOC complexity. + +**Pick v1 simple for F2**; v2 union as follow-up if user pain materializes. + +--- + +## 10. Per-session Injection + +### 10.1 `mcpServers` from `newSession({mcpServers})` + +`newSessionConfig(cwd, mcpServers, ...)` merges injected list with `settings.merged.mcpServers` (acpAgent.ts:1778-1831). Pool consumes the **per-session merged view**: + +```ts +async newSessionConfig(...) { + const config = await loadCliConfig(...); + if (this.mcpPool) config.setMcpTransportPool(this.mcpPool); + // ...existing setMcpBudgetEventCallback REMOVED — pool handles broadcast directly +} +``` + +When two sessions inject same-name server with different env/headers → different fingerprints → two pool entries. Pool sharing kicks in only when sessions agree exactly. + +### 10.2 Auth divergence + +Static `~/.qwen/settings.json` mcpServers are identical across sessions → all share → 80% case. Per-session injected mcpServers with per-user tokens → unique fingerprints → no sharing. Both safe. + +### 10.3 HTTP transport opt-in (recap from §5.2) + +Default `pooledTransports = {stdio, websocket}`. HTTP/SSE servers go through `createUnpooledConnection` path (one McpClient per session) unless operator opts in. + +### 10.4 `/mcp disable X` mid-session (V21-6) + +When operator runs `/mcp disable github` against a live session: + +1. `Config.disableMcpServer('github')` adds to per-Config `disabledMcpServers` set +2. **F2 hook**: `Config.onDisabledMcpServersChanged` fires; `SessionMcpView` for that name calls `teardown()` (removes its tool/prompt registrations from session registries) +3. Pool entry **may stay alive** if other sessions still reference it (refcount > 0) — only the disabling session's view detaches +4. If all sessions disable → refcount → 0 → drain timer starts + +Without step 2, mid-session disable would leave already-registered tools in the session's `ToolRegistry` until next session restart. Test 21.4 covers this. + +`/mcp enable github` is the inverse: triggers fresh `pool.acquire` for the session, attaches new view, re-applies snapshot. + +--- + +## 11. Budget Guardrails Graduation + +### 11.1 State machine moves to pool + +`tryReserveSlot` / `releaseSlotName` / 75% hysteresis / refused_batch coalescing / `bulkPassDepth` / `pendingRefusalNames` — all migrate from `McpClientManager` to `McpTransportPool`. `McpClientManager` retains the state only when running standalone (no pool injected). + +### 11.2 Snapshot cell scope + +```ts +{ + kind: 'mcp_budget', + scope: 'workspace', // NEW value (PR 14 v1 returned 'session') + liveCount: 5, + clientBudget: 10, + budgetMode: 'enforce', + status: 'ok', +} +``` + +Per PR 14 contract: "Consumers MUST tolerate additional entries with unrecognized scope values (drop, don't fail)." Old SDK clients see `scope: 'workspace'`, render as unknown (or fallback to top-level numbers). New SDK adds `isWorkspaceScopedBudget(cell)` helper. + +### 11.3 Event fan-out + +```ts +class QwenAgent { + constructor() { + this.mcpPool = new McpTransportPool({ + onBudgetEvent: (event) => this.broadcastBudgetEvent(event), + }); + } + + private broadcastBudgetEvent(event: McpBudgetEvent) { + for (const [sid, session] of this.sessions) { + const enriched = { + ...event, + scope: 'workspace' as const, + sessionId: sid, + }; + session.connection + .extNotification('qwen/notify/session/mcp-budget-event', enriched) + .catch((err) => + debugLogger.debug('budget event delivery failed', { sid, err }), + ); + } + } +} +``` + +### 11.4 SDK type contract changes + +PR 14b exported these (must extend additively): + +- `DaemonMcpBudgetWarningData` — add `scope?: 'workspace' | 'session'` (optional for backward compat; absent = 'session') +- `DaemonMcpChildRefusedBatchData` — same `scope?` extension +- `DaemonMcpGuardrailEvent` — discriminator unchanged + +New SDK helpers: + +```ts +export function isWorkspaceScopedBudgetEvent( + e: DaemonMcpGuardrailEvent, +): boolean; +``` + +Reducer state on `DaemonSessionViewState`: + +- **No new fields** — `mcpBudgetWarningCount` / `mcpChildRefusedBatchCount` increment regardless of scope (scope is a property of each event, not a separate stream) +- Document that under F2 these counts reflect workspace-level events fanned to every session — they will increment **simultaneously across all attached sessions** when budget pressure occurs + +**V21-12 (Q1 resolved, locked in v2.1)**: keep existing field names (`mcpBudgetWarningCount`, `mcpChildRefusedBatchCount`, `lastMcpBudgetWarning`, `lastMcpChildRefusedBatch`) with extended scope semantics documented in JSDoc: + +```ts +/** + * Count of `mcp_budget_warning` events the session has observed. + * Under F2 (`scope: 'workspace'`), this increments simultaneously + * across all attached sessions because budget events fan out at + * workspace level. Use `isWorkspaceScopedBudgetEvent(lastMcpBudgetWarning)` + * to inspect scope of the most recent event. + */ +mcpBudgetWarningCount: number; +``` + +Rationale: PR 14b already shipped these names as public SDK surface; renaming is a breaking change worse than the slightly imprecise semantics. + +--- + +## 12. OAuth — Explicit F3 Deferral + +OAuth 401 fallback in `connectToMcpServer` (mcp-client.ts:950-1010) needs interactive resolution (browser open or device-flow). Mode B daemon **must not spawn a browser** (per PR 21 design — static-source grep test fails build on `open`/`xdg-open`/`shell.openExternal`). + +**F2 behavior on OAuth-requiring server**: + +1. First acquire triggers `connectToMcpServer` → 401 detected +2. Pool catches OAuth-required exception, marks entry as `failed_auth_required` +3. Status route surfaces `errorKind: 'auth_env_error'` (existing PR 13 errorKind) +4. Pool **does not retry automatically** +5. Operator runs `/mcp auth ` (existing CLI) OR uses PR 21's device-flow route to get a token on disk → next session acquire re-attempts and succeeds + +**F3 will replace step 4-5** with `PermissionMediator` routing OAuth completion request to attached sessions for first-responder. + +This avoids F2 mixing into auth state-machine work. + +--- + +## 13. Restart Route Semantics + +### 13.1 `POST /workspace/mcp/:server/restart` under pool + +Today (PR 17): restart in bootstrap session's manager = restart the single entry for that name. + +Under pool: name → possibly multiple entries (different fingerprints for same name = different sessions with different configs). + +**Spec'd behavior**: + +| Request | Behavior | +| -------------------------------------------------- | ------------------------------------------------------------------------------------ | +| `POST /workspace/mcp/:server/restart` | Restart **all** entries matching `serverName` (parallel via `Promise.allSettled`) | +| `POST /workspace/mcp/:server/restart?entryIndex=0` | V21-3: restart only entry #0 (the opaque index from snapshot §8.3); 404 if not found | +| `POST /workspace/mcp/:server/restart?entryIndex=*` | Explicit "all" (same as no param) | + +Response shape: + +```ts +type RestartResult = { + entryIndex: number; // V21-7: opaque index, not raw fingerprint + restarted: boolean; + durationMs?: number; + reason?: string; // 'budget_would_exceed' | 'not_connected' | 'in_flight' +}; +POST /workspace/mcp/:server/restart → { entries: RestartResult[] } +``` + +Old shape `{restarted: true, durationMs}` retained when `entries.length === 1` AND no `entryIndex` query param for backward compat; clients can detect new shape by checking `'entries' in response`. + +### 13.2 In-flight restart dedupe + +```ts +class PoolEntry { + private restartInFlight?: Promise; + async restart(): Promise { + if (this.restartInFlight) return this.restartInFlight; + this.restartInFlight = this.doRestart().finally(() => { + this.restartInFlight = undefined; + }); + return this.restartInFlight; + } +} +``` + +### 13.3 Budget check (preserves PR 17 behavior) + +Pre-restart, pool checks budget: if disconnect+reconnect would still fit, OK. The current PR 17 `{restarted:false, skipped:true, reason:'budget_would_exceed'}` semantic preserved (just now applied per-entry). + +### 13.4 In-flight tool call during reconnect (V21-5, new) + +Session A invokes `pool.callTool('git.commit', args)` → request hits stdin of underlying child → child process crashes mid-write → entry transitions to reconnect: + +```ts +class MCPCallInterruptedError extends Error { + readonly serverName: string; + readonly entryIndex: number; + readonly clientGeneration: number; // pre-reconnect generation + readonly args: unknown; // original args, for caller to retry if safe + constructor(serverName, entryIndex, clientGeneration, args) { ... } +} +``` + +**Spec**: + +- The in-flight call promise rejects with `MCPCallInterruptedError` as soon as transport drop detected (don't wait for reconnect) +- Pool **does NOT auto-retry** the call; semantics unsafe for writes (commit, file edit, etc.) and pool can't distinguish read from write +- Caller (typically tool execution layer in agent loop) catches this error and decides: retry / surface to user / abort +- After reconnect: session A can re-call (same `PooledConnection.callTool`); pool routes to the new transport instance transparently +- `MCPCallInterruptedError.clientGeneration` lets caller correlate with subsequent `reconnected` event if needed + +Test 21.6 must cover: spawn a long-running stdio MCP, send tool call, kill the child mid-call, assert `MCPCallInterruptedError` rejection with non-zero `clientGeneration`. + +--- + +## 14. Status Route Refactor + +### 14.1 New query path + +```ts +// httpAcpBridge.ts:733 buildWorkspaceMcpStatus — replace data source +let accounting: McpClientAccounting | undefined; +try { + // NEW: query pool directly via bridge extMethod, not bootstrap session + accounting = await this.bridge.client.getMcpPoolAccounting(); +} catch (err) { + // Fallback to legacy bootstrap session path for non-pool daemon + const manager = config.getToolRegistry()?.getMcpClientManager(); + if (manager) accounting = manager.getMcpClientAccounting(); +} +``` + +`QwenAgent` exposes `getMcpPoolAccounting()`: + +```ts +class QwenAgent { + getMcpPoolAccounting(): McpClientAccounting | undefined { + return this.mcpPool?.getAccounting(); + } +} +``` + +ACP child bridges through `extMethod` for the daemon to call. + +### 14.2 entryCount + entrySummary + +Per §8.3. + +### 14.3 No-bootstrap-session case + +Today (PR 12), when daemon is idle (no sessions yet), `GET /workspace/mcp` returns `initialized: false` because there's no bootstrap session to query. + +Under pool: pool exists from `QwenAgent` ctor → status route can return live accounting **even with zero sessions**. Cell `initialized: true` even pre-first-session. **Documented behavior change** in PR description; not a regression. + +--- + +## 15. loadSession / resume Interaction (PR 6 #4222) + +### 15.1 Drain cancellation on resume + +``` +session-A active, holds entry-X ref +session-A disconnect (no explicit close) → eventually killSession → pool.releaseSession(A) → entry-X.refs.size === 0 → drain timer starts (30s) +session-A resume within 30s → new newSessionConfig → pool.acquire returns entry-X → attach cancels drain +session-A resume after 30s → entry-X already closed → pool spawns new entry (cold start) +``` + +### 15.2 `restoreState` cache window (5min, from PR 6) + +`acpAgent.restoreState` is held 5 min after disconnect. Pool drain (30s default) < restore window (5min) → resume between 30s and 5min pays MCP cold start. Acceptable trade-off (resume itself is rare path). + +Alternative: pool reads daemon's restore-window config and extends drain to match. Adds coupling between pool and session state machine; **defer to follow-up unless user reports cold-start pain**. + +### 15.3 `pendingRestoreIds` interaction + +`acpAgent.killSession()` must call `pool.releaseSession(sid)` AFTER cleaning `pendingRestoreIds`. Order: + +1. Session marked as restorable (`pendingRestoreIds.add(sid)`) +2. Session.close() — but pool ref still held +3. After `RESTORE_WINDOW_MS` elapses without resume: `killSession` permanently cleans → `pool.releaseSession(sid)` triggers drain + +Avoids drain firing during a restore window. + +--- + +## 16. Hot Config Reload + +### 16.1 Implicit reload via fingerprint change + +User edits `~/.qwen/settings.json` mid-flight, changes a server's env: + +1. Old sessions keep old `Config`/`McpServers` snapshot → keep acquiring old fingerprint → entry-OLD ref persists +2. New session reads fresh settings → new fingerprint → entry-NEW created → coexists with entry-OLD +3. Old sessions naturally close → entry-OLD drains → eventually closed +4. Steady state: only entry-NEW remains + +**No live-mutation of running connections** — clean separation between sessions on different config versions. + +### 16.2 Forced reload route (optional) + +``` +POST /workspace/mcp/reload-all + → for each session: re-load settings, swap Config.mcpServers + → for each entry no longer referenced: schedule eviction +``` + +Useful for "I changed env vars and want immediate effect across all sessions." Defer to F2 follow-up (not blocking). + +### 16.3 Extension uninstall orphan entries (V21-15) + +Scenario: extension `foo-ext` registers MCP server `foo-server`. Operator runs `/extension uninstall foo-ext`. Extension lifecycle removes `foo-server` from `extensionMcpServers` so future `loadCliConfig` calls don't include it. But: + +- Live sessions hold `Config` snapshots that still include `foo-server` → those sessions keep using the entry +- New sessions after uninstall don't acquire (server no longer in their merged mcpServers) → no refcount increase + +**Resolution**: rely on natural drain. As old sessions close, refcount drops; eventually entry hits `MAX_IDLE_MS = 5min` and is force-closed. **No explicit `pool.invalidateByExtension(name)` API** — keeps the model uniform with hot config reload (§16.1). + +Trade-off: extension's server may run up to 5min after uninstall if a long session keeps it alive. Acceptable; operators can `/mcp restart foo-server` then kill the session if urgency requires. + +--- + +## 17. Shutdown Ordering + +`QwenAgent.close()` sequence (must be enforced): + +``` +1. Set acceptingNewSessions = false; reject new POST /session +2. For each in-flight prompt: signal cancel, await completion (existing PR 11 lifecycle) +3. For each session: trigger close → pool.releaseSession(sid) +4. await pool.drainAll({ force: true, timeoutMs: 10_000 }) ← bypasses 30s grace + ├── For each entry: cancel drain + health timers, mark draining + ├── For each entry in parallel: listDescendantPids → SIGTERM children + ├── For each entry in parallel: client.disconnect() + └── Promise.race against timeoutMs; abandoned entries get SIGKILL +5. Bridge channel close +6. Process exit +``` + +**V21-11**: `drainAll` signature: + +```ts +async drainAll(opts?: { + force?: boolean; // default false; true bypasses 30s grace timer + timeoutMs?: number; // default 10_000; wall-clock budget; SIGKILL stragglers after +}): Promise; + +type DrainResult = { + drained: number; // entries that disconnected cleanly + forced: number; // entries SIGKILLed after timeout + errors: Array<{ entryIndex: number; serverName: string; error: string }>; +}; +``` + +Caller uses `DrainResult` for shutdown logging; on `forced > 0` log a warning so operator knows a server didn't shut down cleanly. + +--- + +## 18. File Layout + +**New files:** + +``` +packages/core/src/tools/ + mcp-transport-pool.ts # McpTransportPool main (~700 LOC) + mcp-pool-key.ts # fingerprint + canonicalize helpers (~150 LOC) + mcp-pool-entry.ts # PoolEntry: refcount + drain + health + generation (~500 LOC) + session-mcp-view.ts # SessionMcpView: filter + register tools/prompts (~200 LOC) + mcp-pool-events.ts # PoolEvent discriminated union (~80 LOC) + pid-descendants.ts # listDescendantPids cross-platform (~150 LOC, incl. tests) + +packages/core/src/tools/ + mcp-transport-pool.test.ts # ~900 LOC + mcp-pool-entry.test.ts # ~400 LOC + session-mcp-view.test.ts # ~250 LOC + mcp-pool-key.test.ts # ~150 LOC + pid-descendants.test.ts # ~200 LOC (Unix + Windows skip-gated) +``` + +**Changed files:** + +``` +packages/core/src/tools/mcp-client.ts # discoverAndReturn() split; connectToMcpServer unified +packages/core/src/tools/mcp-client-manager.ts # optional pool param; budget state conditional +packages/core/src/tools/tool-registry.ts # threads pool from config into McpClientManager +packages/core/src/config/config.ts # setMcpTransportPool / getMcpTransportPool +packages/cli/src/acp-integration/acpAgent.ts # QwenAgent.mcpPool construction; broadcastBudgetEvent; + # newSessionConfig wires pool into Config; + # killSession calls pool.releaseSession +packages/cli/src/serve/runQwenServe.ts # pass --mcp-pool-transports + budget env to ACP child +packages/cli/src/serve/httpAcpBridge.ts # buildWorkspaceMcpStatus reads pool; + # restartMcpServer extMethod returns RestartResult[] +packages/cli/src/serve/capabilities.ts # advertise mcp_workspace_pool +packages/sdk/src/daemon/mcpEvents.ts # scope?: optional field; isWorkspaceScopedBudgetEvent helper +``` + +--- + +## 19. Single-PR Delivery — Commit Breakdown (V21-1) + +Per maintainer's feature-cohesive batch guidance (#4175 branching strategy 2026-05-19), F2 ships as **one PR with 6 atomic commits**. Reviewer can step through with `git log -p HEAD~6..HEAD` and review commit-by-commit. + +| Commit # | Title | Scope | Touches | +| -------- | --------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | +| 1 | `refactor(core): split McpClient.discover into pure tool/prompt list and unify connect paths` | Add `discoverAndReturn()`; extract shared `establishConnection()` used by both `McpClient.connect()` and `connectToMcpServer()` factory; legacy `discover()` becomes thin wrapper that registers (preserves standalone qwen behavior). Zero observable behavior change. | `mcp-client.ts`, `mcp-client.test.ts` | +| 2 | `feat(core): McpTransportPool + SessionMcpView` | Pool core: `fingerprint`, refcount, `spawnInFlight` dedupe, `sessionToEntries` reverse index, drain state machine, snapshot replay on attach, generation guard, tool+prompt dual fan-out, per-session trust copy. Mock McpClient for unit tests. No production wiring. | new `mcp-transport-pool.ts`, `mcp-pool-key.ts`, `mcp-pool-entry.ts`, `session-mcp-view.ts`, `mcp-pool-events.ts` + tests | +| 3 | `feat(core): cross-platform descendant pid sweep + pool health monitor` | `listDescendantPids` (Unix `pgrep -P` recursive, Windows PowerShell CIM); unified health monitor inside `PoolEntry` (interval check + failure count + reconnect backoff per §6.6); subprocess-spawn integration tests gated on `QWEN_INTEGRATION === '1'`. | new `pid-descendants.ts` + tests; `mcp-pool-entry.ts` | +| 4 | `feat(serve): wire McpTransportPool into QwenAgent daemon mode` | `Config.setMcpTransportPool` + `getMcpTransportPool`; `ToolRegistry` threads pool into `McpClientManager`; `McpClientManager` optional `pool?` ctor param; `acpAgent.QwenAgent` constructs pool at init; `newSessionConfig` injection; `killSession` calls `pool.releaseSession`; SDK MCP + HTTP/SSE bypass via `createUnpooledConnection`; CLI flags `--mcp-pool-transports`, `--mcp-pool-drain-ms`, `--no-mcp-pool`. | `config.ts`, `tool-registry.ts`, `mcp-client-manager.ts`, `acpAgent.ts`, `runQwenServe.ts` | +| 5 | `feat(serve): pool-aware status + restart routes` | `QwenAgent.getMcpPoolAccounting` extMethod; `httpAcpBridge.buildWorkspaceMcpStatus` pool-first + bootstrap-session fallback; `restartMcpServer` accepts `?entryIndex=` and returns `RestartResult[]`; `entryCount` + `entrySummary[].entryIndex` on cell; capability tags `mcp_workspace_pool` + `mcp_pool_restart`. | `httpAcpBridge.ts`, `capabilities.ts`, SDK types | +| 6 | `feat(serve): graduate MCP budget guardrails to workspace scope` | Move `tryReserveSlot`/`releaseSlotName`/hysteresis state machine from `McpClientManager` to pool; remove per-session `setMcpBudgetEventCallback` wiring in `acpAgent.newSessionConfig`; `QwenAgent.broadcastBudgetEvent` fan-out; snapshot cell `scope: 'workspace'`; SDK `scope?` additive field; `isWorkspaceScopedBudgetEvent` helper; inline doc updates. | `mcp-transport-pool.ts`, `mcp-client-manager.ts`, `acpAgent.ts`, `httpAcpBridge.ts`, SDK | + +**Total LOC estimate**: ~4100 production + ~1900 tests = ~6000 LOC (v2 estimate ~3850; growth absorbs V21 corrections). + +**Merge target**: single PR into `daemon_mode_b_main`. Periodic batch merge to `main` per #4175 strategy. + +**Self-review process before opening PR**: + +1. After each commit, run `code-reviewer` agent on the commit diff; fold adopted findings into the same commit +2. For commit 2/4/6 (highest design risk), additionally run `silent-failure-hunter` + `type-design-analyzer` +3. After all 6 commits land: 3 full review passes by different agent combinations on the full PR diff +4. Run full test suite + typecheck + lint across all touched packages + +Mirror PR 21's specialist pre-review pattern. + +--- + +## 20. Capability Tags + SDK Contract Changes + +### 20.1 New capability tags (advertised atomically in v0.16, V21-1) + +Because F2 ships as one PR, all three tags advertise together. Pool consumers may assume **`mcp_workspace_pool` advertise ⇒ `entryCount`/`entrySummary`/`scope?` fields all present**; no per-field capability check needed. + +| Tag | When advertised | Meaning | +| -------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------ | +| `mcp_workspace_pool` | When `QwenAgent.mcpPool !== undefined` (always true in daemon mode unless `--no-mcp-pool` kill switch) | `GET /workspace/mcp` reflects pool-level state; `entryCount` + `entrySummary` fields present | +| `mcp_pool_restart` | Always when `mcp_workspace_pool` is on | `POST /workspace/mcp/:server/restart` accepts `?entryIndex=` and may return `entries: RestartResult[]` | +| (extends `mcp_guardrails`) | unchanged | Same tag, payload extended with `scope` (`'workspace'` under F2) | + +### 20.2 SDK additive surface + +```ts +// @qwen-code/sdk — additive only +export interface DaemonMcpBudgetWarningData { + // existing fields... + scope?: 'workspace' | 'session'; // NEW — absent on old daemons (means 'session') +} + +export interface DaemonMcpChildRefusedBatchData { + // existing fields... + scope?: 'workspace' | 'session'; +} + +export interface ServeWorkspaceMcpServerStatus { + // existing fields... + entryCount?: number; + entrySummary?: Array<{ + fingerprint: string; + refs: number; + status: MCPServerStatus; + }>; +} + +export function isWorkspaceScopedBudgetEvent( + e: DaemonMcpGuardrailEvent, +): boolean; +``` + +`EVENT_SCHEMA_VERSION` stays at `1` (additive). + +--- + +## 21. Test Matrix + +### 21.1 Pool key (F2-2) + +- Same cfg → same key (env-key permutation stable, header-key permutation stable) +- env value diff 1 byte → different key +- header `Authorization` value diff → different key +- `includeTools`/`excludeTools`/`trust` mutated → SAME key (per-session filter) +- Two `new MCPServerConfig(...)` with identical content → same key (canonical hash, not identity) + +### 21.2 Lifecycle (F2-2) + +- 3 sessions acquire same key → 1 spawn (verify via spy on `client.connect`) +- Release sequence n,n-1,...,1 → drain timer starts only on 1→0 +- 30s drain: acquire at 25s cancels timer; acquire at 35s spawns new entry +- `MAX_IDLE_MS` (5min) hard close even if drain flapping +- Spawn fails during in-flight: all awaiters get error; slot released; no entry stored + +### 21.3 Concurrent acquire (F2-2) + +- 5 simultaneous `acquire(sameKey)` while no entry exists → exactly 1 `spawnEntry` call, all 5 get same entry +- Spawn rejects → all 5 awaiters reject with same error; subsequent acquire re-spawns + +### 21.4 Per-session isolation (F2-2) + +- Session A `excludeTools: ['foo']`, Session B no exclusion → A's ToolRegistry omits foo, B has it; both from same `toolsSnapshot` +- Session A `trust: true`, Session B `trust: false` → Session A's `DiscoveredMCPTool.trust === true`, B's `false`; verify NOT shared reference (mutating one doesn't affect other) +- Session A acquires prompt-only server → A's PromptRegistry populated, ToolRegistry empty for that server + +### 21.5 Tool/Prompt list change (F2-2) + +- Server emits `notifications/tools/list_changed` → all subscribers' `applyTools` called with new snapshot +- Stale handler from pre-reconnect generation does NOT overwrite snapshot +- `notifications/prompts/list_changed` analog + +### 21.6 Crash + reconnect (F2-2) + +- Kill subprocess via `process.kill` → subscribers receive `disconnected` event +- 3 reconnect attempts (using existing `MCPHealthMonitorConfig`) → success → `reconnected` + fresh snapshot +- Exhausted retries → all subscribers receive `failed`; entry transitions to `failed` state; new acquires retry once then throw + +### 21.7 Descendant pid sweep (F2-2b) + +- Linux/macOS: spawn `bash -c "sleep 60 & sleep 60"` as stdio command → kill root → verify both descendants reaped (`/proc//status` poll, or `kill(0, pid) === false`) +- Windows: spawn `cmd /c "ping -t localhost"` wrapper → kill → verify ping subprocess gone +- `pgrep` unavailable (PATH missing) → graceful degradation: log warning, just SIGTERM root, don't crash + +### 21.8 Budget at workspace scope (F2-4) + +- 4 sessions × `--mcp-client-budget=2` with 3 static MCP servers → workspace total = 3 (not 12); snapshot cell `scope: 'workspace'`, `liveCount: 3` +- Budget warning fires once per 75% upward crossing across whole workspace; broadcasts to all 4 sessions simultaneously +- Hysteresis re-arm: drop to 37.5% → next crossing fires again + +### 21.9 Backward compat (F2-3) + +- Standalone `qwen` (no daemon) → `mcpPool === undefined` → all existing `mcp-client-manager.test.ts` tests pass unchanged +- `--no-mcp-pool` daemon flag → falls back to per-session, all existing daemon e2e tests pass + +### 21.10 Credential isolation (F2-3) + +- Session A injects `{name: 'github', headers: {Authorization: 'Bearer tokenA'}}`, Session B `tokenB` → 2 separate processes; verify by snapshot `entryCount: 2`; verify A's tool calls go through A's transport (by header inspection in stdin/log) + +### 21.11 LoadSession / resume (F2-3) + +- Session close → drain starts → resume within 30s → pool entry reused (no cold start, asserted via `client.connect` spy count) +- Resume after 30s but before restore-window expiry → pool cold start; restoreState content still preserved + +### 21.12 Restart route (F2-3b) + +- 1 entry for name → `POST /workspace/mcp/foo/restart` returns legacy `{restarted: true, durationMs}` shape +- 2 entries for name (different fingerprints) → returns `{entries: [{fingerprint, restarted, ...}, ...]}` +- Restart while another restart in-flight → second call returns same promise (deduped) +- Restart when budget would exceed → `{restarted: false, skipped: true, reason: 'budget_would_exceed'}` per entry + +### 21.13 Status route (F2-3b) + +- Idle daemon (no sessions) but pool has cached entries from previous session → `GET /workspace/mcp` returns `initialized: true` with live accounting +- Bootstrap session DNE → fallback to pool-direct path; no error +- Pool query throws → falls back to bootstrap-session path; never crashes snapshot + +### 21.14 SDK reducer (F2-4) + +- `mcpBudgetWarningCount` increments simultaneously across all subscriber sessions when workspace event broadcasts +- `isWorkspaceScopedBudgetEvent(e)` correctly identifies scope from payload +- Old daemon (no `scope` field) → defaults to 'session' interpretation + +### 21.15 Hot config reload (F2-3) + +- Mid-flight settings.json change → old session keeps old entry, new session creates new entry, both coexist; old drains naturally when last old session closes +- 0 sessions after old session closes → drain timer fires → old entry GC'd → only new entry remains + +### 21.16 Shutdown ordering (F2-3) + +- `QwenAgent.close()` triggers in order: stop accepting → drain prompts → close sessions → `pool.drainAll` → no zombie pids in `pgrep -P ` after exit + +--- + +## 22. Open Questions + +V21 locked Q1/Q3/Q4/Q6 in design defaults (single-PR delivery). Q2/Q5/Q7/Q8/Q9 remain. + +| # | Question | F2 design default | Decision needed before | +| ----- | ----------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ---------------------- | +| Q1 ✅ | SDK reducer field names — rename or keep? | **LOCKED v2.1**: keep `mcpBudgetWarningCount` etc. with extended scope semantics in JSDoc | resolved | +| Q2 | `mcp_workspace_pool` capability — bump `protocolVersions` ('v1' → 'v1.1'), or stay 'v1' additive? | **Stay 'v1' additive** (consistent with PR 14b precedent) | commit 5 | +| Q3 ✅ | `--no-mcp-pool` flag — default on or opt-in? | **LOCKED v2.1**: default on; `--no-mcp-pool` is kill switch | resolved | +| Q4 ✅ | HTTP/SSE default — pool off or on? | **LOCKED v2.1**: pool off; opt-in via `--mcp-pool-transports` | resolved | +| Q5 | `POST /workspace/mcp/reload-all` — include in F2 or follow-up? | **Follow-up** | n/a (deferred) | +| Q6 ✅ | Lazy pool construction — worth the conditional? | **LOCKED v2.1**: eager (always construct in `QwenAgent` ctor) | resolved | +| Q7 | `restoreState` window vs pool drain — keep separate, align, or read from settings? | **Keep separate 30s default** + config knob `--mcp-pool-drain-ms` | commit 4 | +| Q8 | OAuth handling — confirm F3 deferral, document workaround? | **Deferred to F3**, document `/mcp auth ` workaround | commit 4 | +| Q9 | `entrySummary` exposure — always include, or behind verbose flag? | **Always include** (small payload, useful for ops) | commit 5 | +| Q10 | Update `codeagents/qwen-code-daemon-design/02-architectural-decisions.md` decision #3 — coordinate with @wenshao? | F2 PR description links codeagents PR; two PRs reviewed independently | PR open | + +--- + +## 23. Risks + +### High + +- **R1 (A2 global state)**: `serverStatuses` collision on multi-entry same-name. Mitigated by aggregate-status function; remaining risk is SDK consumers reading the raw global Map (unlikely — only used via `getMCPServerStatus(name)` accessor). +- **R2 (PromptRegistry symmetry)**: forgetting prompt fan-out in any code path silently drops prompts. Mitigated by F2-2 test 21.4 third bullet + integration test asserting prompt parity vs pre-F2. +- **R3 (HTTP transport state-bleed)**: opting in HTTP pool for a server that maintains per-transport state corrupts session contexts. Mitigated by default-off + documentation; cannot detect automatically. + +### Medium + +- **R4 (path unification F2-1)**: `connectToMcpServer` factory and `McpClient` class have subtle behavioral diffs (e.g. capabilities advertised at construct time vs connect time). Mitigated by F2-1 being a pure refactor PR with full regression coverage before pool work begins. +- **R5 (Windows descendant pid)**: PowerShell `Get-CimInstance` may be slow (spawn cost) or blocked by AppLocker. Mitigated by 2s timeout + graceful degradation. +- **R6 (Pool event broadcast amplification)**: budget warning fanning out to 100 sessions causes 100 extNotification calls in tight loop. Mitigated by `Promise.all` parallelization + per-session catch (existing PR 14b pattern). + +### Low + +- **R7 (Fingerprint stability across MCPServerConfig versions)**: future fields added to `MCPServerConfig` not included in fingerprint would silently allow incorrect sharing. Mitigated by explicit canonicalization function + test that enumerates all `MCPServerConfig` fields and asserts coverage. +- **R8 (Generation counter races)**: rapid restart cycles could exhaust JS number precision (≈ 2^53 = ~285k years at 1/sec). Not a practical concern. + +### Single-PR-specific (V21-14) + +- **R9 (Review fatigue on ~6000 LOC single PR)**: Reviewer bandwidth becomes critical path. F3 blocked on F2 merge → blocking other contributors. Mitigation: (a) pre-review with 3 specialist agents and fold P0/P1 before opening, mirroring PR 21's pattern; (b) structure as 6 atomic commits so reviewer can step through; (c) coordinate review window with @wenshao in advance via #4175 comment. +- **R10 (`daemon_mode_b_main` merge conflict accumulation)**: F2 touches `acpAgent.ts`, `httpAcpBridge.ts`, `capabilities.ts`, `mcp-client*.ts` — all hot paths. F3 / F4 contributors landing concurrently risk conflicts during F2's 1–2 week review window. Mitigation: daily `git rebase origin/daemon_mode_b_main`; coordinate via #4175 update that F2 is in-flight + asks F3/F4 to defer hot-file changes until F2 merges. +- **R11 (CI execution time)**: ~1900 LOC of new tests including subprocess spawn + cross-platform pid sweep could push CI from 30min → 50min. Mitigation: (a) gate subprocess tests behind `process.env.QWEN_INTEGRATION === '1'`, run subset in PR CI + full set in nightly; (b) Vitest parallelism ≥ 4; (c) Windows pid sweep tests skip-gated on GHA Windows runner only. + +--- + +## 24. Documentation Updates + +| Doc | Update | When | +| ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- | +| `codeagents/qwen-code-daemon-design/02-architectural-decisions.md` | Decision #3 "MCP server lifetime": currently "per-session"; update to "workspace-pooled with config-hash key under daemon mode; per-session standalone" | F2-3 merges (coordinate with @wenshao codeagents PR) | +| `codeagents/qwen-code-daemon-design/06-roadmap.md` | Wave 5 PR 23 → mark as F2 series; link to PRs | F2-3 merges | +| `packages/cli/src/serve/README.md` (if exists) or new `docs/serve/mcp-pool.md` | New section: pool semantics, fingerprint key, transport opt-in, restart semantics, status snapshot interpretation | F2-3b | +| `packages/sdk/README.md` | `scope?` field on guardrail events, `entryCount` on server status, helper `isWorkspaceScopedBudgetEvent` | F2-4 | +| Issue #4175 body | Update F2 entry with sub-PR table, link to design v2 (this doc) | Before F2-1 opens | +| Issue #3803 body | Decision #3 row: update "Currently per-session" → "Workspace-pooled under daemon (F2)" | After F2-3 merges | +| `acpAgent.ts:869-936` inline comment | Remove "Wave 5 PR 23" forward reference; update to "graduated by F2 to `scope: 'workspace'`" | F2-4 PR | +| CHANGELOG / release notes (Wave 6 / F5) | "MCP processes now shared across sessions in a workspace" headline | F5 release | + +--- + +## 25. PR Description Template (single-PR delivery) + +```markdown +## feat(serve): shared MCP transport pool (workspace-scoped) [F2] + +Single feature-cohesive PR per #4175 branching strategy (2026-05-19). +Replaces what was originally planned as Wave 5 PR 23 + sub-PRs F2-1..F2-4. + +### Scope + +~4100 LOC production + ~1900 LOC tests across 6 atomic commits. +Step through with `git log -p HEAD~6..HEAD` for commit-by-commit review. + +### Design doc + +See `docs/design/f2-mcp-transport-pool.md` (v2.1). + +### Pre-review specialist agents (per PR 21 pattern) + +Folded into first commit before opening: + +- code-reviewer: N findings, all adopted +- silent-failure-hunter: N findings, all adopted +- type-design-analyzer: N findings, all adopted + +### Closes + +(none — F2 entry in #4175 stays open until PR merges into main batch) + +### Related + +- #3803 decision #3 update (codeagents PR ) +- PR 14b (#4271 merged) — budget guardrail base; F2 graduates scope to workspace +- F1 (#4319 merged) — acp-bridge package; F2 depends on injection seams + +### Backward compatibility + +- Standalone `qwen` (non-daemon): pool not constructed; existing behavior preserved +- Daemon `qwen serve --no-mcp-pool`: kill switch falls back to per-session +- SDK: all new fields additive (`entryCount`, `scope?`); EVENT_SCHEMA_VERSION stays at 1 +- Old SDK clients: unknown `scope: 'workspace'` ignored per PR 14 contract +- Old daemons: SDK consumers can detect absence of `mcp_workspace_pool` capability and fall back + +### Test plan + +- [ ] Pool key: env permutation stability, header divergence, per-session filter exclusion +- [ ] Lifecycle: 3-session sharing, drain grace, concurrent acquire dedupe, spawn failure slot release +- [ ] Tools + Prompts dual fan-out, per-session trust copy, snapshot replay on attach +- [ ] Generation guard: pre-reconnect handler doesn't overwrite post-reconnect snapshot +- [ ] Crash + reconnect with stdio backoff (5s × 3) and HTTP backoff (1/2/4/8/16s × 5) +- [ ] Descendant pid sweep: Linux/macOS pgrep recursion, Windows PowerShell CIM +- [ ] Budget at workspace scope: 4 sessions × budget=2 → 3 max (not 12); fan-out to all attached +- [ ] LoadSession resume within drain window: pool entry reused, no cold start +- [ ] Hot config reload: old/new entries coexist; old drains naturally +- [ ] Restart route: `?entryIndex=` selectivity; legacy single-entry response shape preserved +- [ ] In-flight tool call during reconnect: `MCPCallInterruptedError` rejection +- [ ] Standalone qwen: all existing mcp-client-manager tests pass unchanged +``` + +## Summary + +F2 v2.1 = single PR with 6 atomic commits (~6000 LOC), targeting `daemon_mode_b_main`. Key design pillars: + +1. **`McpTransportPool`** in `packages/core` (ACP child side), workspace-scoped, refcount + 30s drain +2. **Fingerprint key** SHA-256 over canonical config including env/headers (claude-code pattern), excluding per-session filters (includeTools/trust) +3. **`SessionMcpView`** per-session tool+prompt registry projection with trust copy +4. **Snapshot replay + generation guard** for attach race and stale notifications +5. **Cross-platform descendant pid sweep** (opencode pattern + Windows port) +6. **HTTP/SSE opt-in**, SDK MCP bypass, OAuth deferred to F3 +7. **Budget state machine** graduates to workspace scope; snapshot cell + push events extend additively (`scope?`) +8. **Status + restart routes** refactor: pool-first with bootstrap-session fallback; `entryCount` + `RestartResult[]` + +**Open questions Q1–Q10** in §22 need maintainer decisions before respective sub-PRs open. Recommend resolving Q1–Q4 before F2-3 starts (those gate the broad direction); Q5–Q10 can resolve incrementally. diff --git a/docs/design/telemetry-llm-request-timing-design.md b/docs/design/telemetry-llm-request-timing-design.md deleted file mode 100644 index 4a41b082d16..00000000000 --- a/docs/design/telemetry-llm-request-timing-design.md +++ /dev/null @@ -1,538 +0,0 @@ -# LLM Request Timing Decomposition Design (P3 Phase 4) - -> Issue #3731 — Phase 4 of hierarchical session tracing. Adds time-to-first-token, request-setup duration, sampling duration, and per-attempt retry telemetry to the `qwen-code.llm_request` span so operators can answer "why was this LLM call slow?" without guessing. -> -> Builds on Phase 1 (#4126), Phase 1.5 (#4302), Phase 2 (#4321). Independent of Phase 3 (#4410, in review) — recommended to land Phase 3 first so Phase 4's per-attempt fields aggregate cleanly under subagent subtrees. - -## Problem - -`qwen-code.llm_request` spans today carry only `model`, `prompt_id`, `input_tokens`, `output_tokens`, `success`, `error`, `duration_ms`. Operators reading a single trace cannot tell: - -1. **How much of `duration_ms` was the model thinking vs the network setup.** A 12-second `duration_ms` could be 11s of retries followed by 1s of fast generation, or 100ms of setup followed by 12s of slow streaming — the trace doesn't say. -2. **When the user saw the first token.** TTFT (time-to-first-token) is the standard latency SLO for chat UIs. We can't compute it; we don't capture it. -3. **What happened during retries.** `retryWithBackoff` (`utils/retry.ts:285`) only calls `debugLogger.warn` — no OTel event, no span attribute. The 4 LLM call sites that go through it (`client.ts:1540`, `baseLlmClient.ts:193,282`, `geminiChat.ts:1039`) have zero retry visibility in traces or metrics. `ContentRetryEvent` exists for content-recovery retries inside `geminiChat.ts:806,830` but not for the more common rate-limit / 5xx retries. -4. **That `api.request.breakdown` is dead code.** The metric is defined at `metrics.ts:242-251` with 4 `ApiRequestPhase` values, exported from `index.ts:117`, tested in `metrics.test.ts:646-675` — but `recordApiRequestBreakdown()` has zero callers in production code. The metric infrastructure is paid for; the data flow was never connected. - -These gaps make `qwen-code.llm_request` the least informative span in the trace tree. Tool spans (#4126/#4321) and subagent spans (#4410) both surface lifecycle phases; LLM spans collapse the entire request into one opaque duration. - -## Existing surface (no change) - -| Component | Location | Why we don't touch it | -| ------------------------------------------------------------ | ---------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| LLM request span lifecycle | `session-tracing.ts` `startLLMRequestSpan` / `endLLMRequestSpan` | Phase 1 (#4126) established the helpers. We extend the metadata interface, don't restructure | -| Active span propagation into provider generators | `loggingContentGenerator.ts:213,287` | Phase 1 (#4126) replaced `withSpan('api.*')` with native helpers; the active context already reaches the stream wrapper | -| `ContentRetryEvent` schema + consumers | `types.ts:626`, `qwen-logger.ts:947`, `loggers.ts:717` | Existing event keeps its shape and downstreams; we add a sibling event class for the `retryWithBackoff` path | -| `LogToSpanProcessor` log-bridge spans | `log-to-span-processor.ts` | ContentRetryEvent's existing bridge continues to nest under the active LLM span. Phase 4 does not change this | -| `ApiRequestPhase` enum | `metrics.ts:330-334` | Public surface (4 values). We populate 3 of the 4 from production code; leave the enum unchanged for backward compatibility | -| Per-provider chunk normalization → `GenerateContentResponse` | `loggingContentGenerator.ts:286-393` | Each provider already normalizes to Google's `GenerateContentResponse` shape before LoggingContentGenerator sees the stream. TTFT detection runs centrally over this normalized shape; no per-provider code | -| `retryWithBackoff` general-purpose retry | `utils/retry.ts:140` | Used by both LLM callers and non-LLM (`channels/weixin/src/api.ts`). We extend with an opt-in `onRetry` callback rather than hard-coupling to LLM telemetry | -| Non-streaming `generateContent` | `loggingContentGenerator.ts:212` | TTFT is not meaningful for non-streaming; the new fields stay `undefined`. Span lifecycle and existing attrs unchanged | - -## Out-of-scope (deferred) - -- **SDK-level retries** (openai SDK `maxRetries=3`, google-genai SDK internal retries). These happen entirely inside the third-party SDK; observing them requires disabling SDK retries and reimplementing in `retryWithBackoff`. Separate decision, not Phase 4. -- **Per-token streaming metrics** (inter-token latency, per-chunk size). Useful for inference-engine perf debugging, not for the user-perceived latency questions Phase 4 targets. -- **Separate TTFT for reasoning/thinking blocks.** "First token" includes thinking content (see D1). A future enhancement could split `ttft_to_reasoning_ms` vs `ttft_to_answer_ms`, but only after we know there's demand. -- **Sampling phase as a dedicated child span.** Computable from `duration_ms - ttft_ms - request_setup_ms`; child span adds nothing for OTel-only backends (claude-code uses one for Perfetto only). Stored as a span attribute instead — see D6. -- **Persistent retry mode (`QWEN_CODE_UNATTENDED_RETRY`) event-level rate limiting.** A single LLM request can produce 50+ `ContentRetryEvent` / `ApiRetryEvent` records under persistent retry. Capping emission is a follow-up — Phase 4 emits all events; if production volumes prove unbearable, add a per-span emission cap with a "+N more attempts (truncated)" summary event in a follow-up PR. -- **`TOKEN_PROCESSING` breakdown phase.** Enum value exists but qwen-code has no real post-stream local processing worth measuring (<10ms typical). Skipped in production callers; enum value retained for future use or for callers we don't control. -- **Migrating `ContentRetryEvent` onto LLM span as span events.** Same reasoning as Phase 3's `subagent_execution` LogRecord: existing consumers (qwen-logger RUM, future metrics) are tightly coupled to the LogRecord. Bridge-span coverage is good enough. - -## References (decision evidence) - -| Source | Key takeaway | -| --------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| claude-code (Anthropic) `claude.ts:1762, 1789, 1982, 2882` | TTFT captured as `Date.now() - start` on `message_start` SSE event; `start` reset per retry attempt. `requestSetupMs = start - startIncludingRetries`. `attemptStartTimes` array preserved per attempt. Confirms feasibility of the approach; their TTFT semantic is "first stream event" (we diverge to "first content" — see D1) | -| claude-code `perfettoTracing.ts:549-671` | Renders Request Setup → Attempt N (retry) → First Token → Sampling as nested B/E pairs. Demonstrates the visual decomposition; qwen-code does the same decomposition with OTel attributes since we have no Perfetto | -| claude-code `sessionTracing.ts:447` | Only `ttft_ms` makes it onto the OTel span (not `requestSetupMs`, not `samplingMs`, not per-attempt timing). We deliberately put more on the span — claude-code has Perfetto for visualization; we don't | -| opencode (sst/opencode) `session/llm.ts`, `route/client.ts` | No TTFT measurement. Single `LLM.run` Effect span covers everything. Validates that the gap exists across competing tools; not a reference for what to do | -| [OTel GenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) (status: Development / Experimental) | `gen_ai.usage.input_tokens` (Stable), `gen_ai.usage.output_tokens` (Stable), `gen_ai.usage.cached_tokens` (Experimental), `gen_ai.request.model` (Stable), `gen_ai.server.time_to_first_token` (Experimental, seconds as double). Dual-emit pattern follows #4410 precedent | -| [OTel Trace Spec — Span Events](https://opentelemetry.io/docs/specs/otel/trace/api/#add-events) | "Events SHOULD NOT be used to record information that's better captured as Span Attributes." Confirms per-attempt info belongs on the LLM span attributes + log-bridge spans, not as Span Events on the parent | -| Phase 3 design doc (`telemetry-subagent-spans-design.md`) | Established the dual-emit pattern (`qwen-code.subagent.id` + `gen_ai.agent.id`) and the "private name is authoritative" rule. Phase 4 follows the same convention for TTFT and token fields | - -## Design — seven decisions, each justified - -### D1 — TTFT semantic: "first chunk containing user-visible content" - -TTFT measures wall-clock from the **successful attempt's** request dispatch to the **first stream chunk that contains user-visible output**. A chunk is "user-visible" if any normalized `Part` in `candidates[0].content.parts` is one of: - -- `text` with non-empty string -- `functionCall` (tool use) -- `inlineData` (image, binary) -- `executableCode` -- `thought` / reasoning content (whatever the provider surfaces — Gemini's `thought`, Anthropic's `` block, OpenAI o1 reasoning chunk) - -Chunks containing only `role` metadata or only `usageMetadata` (final usage-summary chunk) do not trigger TTFT. - -**Why not "first stream event of any kind" (claude-code's choice)**: claude-code measures TTFT at `message_start`, an Anthropic-specific metadata event that fires 50–300ms before any actual content. Their internal `headlessProfiler.ts` already separates `time_to_first_response_ms` for the "user saw something" semantic, acknowledging the distinction. qwen-code spans multiple providers (Anthropic, OpenAI, Gemini, Qwen) — picking the metadata-event semantic means TTFT for Anthropic is fundamentally different from TTFT for OpenAI (which has no analogous metadata-only first event). The user-visible-content semantic is uniform across all 4 providers and matches "time-to-first-token" literally. - -**Why include `thought` / reasoning**: from the operator's perspective, reasoning chunks are still "the model produced output." Excluding them would understate TTFT for reasoning-heavy models (o1, Qwen thinking variants). Future split into `ttft_to_reasoning_ms` vs `ttft_to_answer_ms` is possible; not Phase 4. - -**Why include tool-call-only chunks**: agent tool-decision LLM calls (one `tool_use`, no text) are common in qwen-code's workflow. Excluding them means TTFT is undefined for these requests. The `functionCall` Part is meaningful output. - -**Cross-product comparison note**: design doc explicitly states `qwen-code.ttft_ms ≈ claude-code.time_to_first_response_ms ≠ claude-code.ttft_ms`. Operators comparing across products should align on the user-visible-content semantic. - -### D2 — TTFT measurement site: method-local variables in `LoggingContentGenerator.generateContentStream` - -The first-chunk detection runs inside the existing stream wrapper at `loggingContentGenerator.ts:393` (`async function* processStreamGenerator`). Per-call variables (`start`, `ttftMs`) live in the method's closure; **never as instance fields**. - -**Why never instance fields**: `LoggingContentGenerator` is instantiated **once per `ContentGenerator`** (`contentGenerator.ts:377`) and shared across all concurrent `generateContentStream` calls — subagent fan-out, warmup queries, side-queries from `geminiChat`. An instance field would be overwritten across concurrent calls, producing nonsense TTFT for one of every two interleaved requests. - -**Why not AsyncLocalStorage**: ALS would work but adds a context-management layer for a piece of state that doesn't need to escape the method. Method-local is simpler, zero overhead, zero risk of leakage. - -```ts -// loggingContentGenerator.ts — inside generateContentStream -const attemptStart = Date.now(); // per-call local -const requestEntryTime = Date.now(); // also per-call local — see D3 -let ttftMs: number | undefined; -const attemptStartTimes: number[] = [attemptStart]; -let retryTotalDelayMs = 0; -let finalAttempt = 1; -// stream wrapper inspects each chunk; first one matching hasUserVisibleContent: -// ttftMs = Date.now() - attemptStart; -``` - -`hasUserVisibleContent(chunk)` is a small standalone helper colocated with the wrapper, exported for tests: - -```ts -function hasUserVisibleContent(chunk: GenerateContentResponse): boolean { - const parts = chunk.candidates?.[0]?.content?.parts; - if (!parts?.length) return false; - return parts.some( - (p) => - (typeof p.text === 'string' && p.text.length > 0) || - p.functionCall !== undefined || - p.inlineData !== undefined || - p.executableCode !== undefined || - // @ts-expect-error — `thought` is not on all SDK versions but providers emit it - p.thought !== undefined, - ); -} -``` - -### D3 — `request_setup_ms` computation: entry-time vs successful-attempt-start - -`request_setup_ms` measures wall-clock from `generateContentStream`/`generateContent` entry to the **start of the successful attempt** — including all failed retries, backoff sleeps, and any pre-retry preparation work. - -```ts -request_setup_ms = attemptStart_of_successful_attempt - requestEntryTime; -``` - -When `attempt === 1` and no retries happened, `request_setup_ms` is small (just SDK setup). When retries occurred, it captures the entire retry-budget overhead. - -**Putting it on the OTel span (diverges from claude-code, which puts it only on Perfetto)**: rationale at three levels: - -1. **No Perfetto** — qwen-code has no out-of-band visualization layer. OTel attributes are the only channel. -2. **Single-trace debug** — operator sees `duration_ms=12000, request_setup_ms=11500, ttft_ms=200, sampling_ms=300` → instantly diagnoses "retries ate 11.5s, model itself was fast." Computing `request_setup_ms` from other fields requires also exposing `sampling_ms`, which we do anyway (D6). -3. **Negligible cost** — 1 INT64 attribute. Same order of magnitude as the existing `input_tokens`, `output_tokens` attributes. Backend ingest cost is not material. - -### D4 — Retry telemetry: `onRetry` callback option on `retryWithBackoff` + new `ApiRetryEvent` - -`retryWithBackoff` currently calls `logRetryAttempt` (`retry.ts:343`) which only writes to `debugLogger.warn`. We extend the `RetryOptions` interface with an opt-in callback: - -```ts -// utils/retry.ts -interface RetryOptions { - // ... existing fields ... - /** - * Optional. Called once per failed attempt, before the backoff sleep. - * Receives the attempt number (1-based), the error, and the delay before - * the next attempt. Use this to emit telemetry events for LLM call sites; - * leave undefined for non-LLM callers (e.g., channels/weixin) so they - * stay silent in LLM-specific telemetry channels. - */ - onRetry?: (info: RetryAttemptInfo) => void; -} - -interface RetryAttemptInfo { - attempt: number; // 1-based, matches debugLogger output - error: unknown; - errorStatus?: number; - delayMs: number; // backoff delay before next attempt -} -``` - -The 4 LLM call sites (`client.ts:1540`, `baseLlmClient.ts:193,282`, `geminiChat.ts:1039`) register a callback that emits a new `ApiRetryEvent`: - -```ts -// types.ts — new event class, sibling to ContentRetryEvent -export class ApiRetryEvent implements BaseTelemetryEvent { - 'event.name': typeof EVENT_API_RETRY; - 'event.timestamp': string; - model: string; - prompt_id?: string; - attempt_number: number; // 1-based - error_type: string; - error_message: string; // truncated to 256 chars - status_code?: number; - retry_delay_ms: number; - // ... duration_ms set to retry_delay_ms so LogToSpanProcessor renders - // a bridge span of meaningful width - duration_ms: number; -} -``` - -**Why a new event class, not extending `ContentRetryEvent`**: - -- `ContentRetryEvent` has 2 downstream consumers (qwen-logger, log-record export). Changing its payload risks breaking them. -- The naming "content retry" semantically refers to content-recovery retries (invalid stream, schema repair) — extending it to cover rate-limit retries would muddy the schema. -- New event is additive; no consumer surprise. - -**Why not embed callback IN `retry.ts`**: `retry.ts` is called by `channels/weixin/src/api.ts` too (microsoft messaging API retries). Hard-coupling LLM telemetry inside retry.ts would emit `ApiRetryEvent` for non-LLM retries. The `onRetry` callback is opt-in per caller — LLM callers opt in, weixin caller doesn't. - -**ContentRetryEvent coexistence**: ContentRetryEvent stays as-is for content-recovery retries inside `geminiChat.ts:806,830`. ApiRetryEvent covers the rate-limit / 5xx retries from `retryWithBackoff`. The two events fire from different layers and never duplicate. Existing log-bridge behavior for both events is preserved via `LogToSpanProcessor` — both events nest under the active LLM span automatically (Phase 1 wiring ensures the LLM span is active during retries). - -**Persistent retry mode (`QWEN_CODE_UNATTENDED_RETRY`)**: a single 429-loop request may emit 50+ events. Out of scope to rate-limit emission in Phase 4 — if production volumes prove unbearable, add a per-span cap with summary event in a follow-up PR. The aggregated `attempt` and `retry_total_delay_ms` on the parent LLM span (D5) remain accurate regardless of event cap. - -### D5 — Parent LLM span aggregation: scalar attributes only (no map-typed attrs) - -OTel span attributes are scalars (`string | number | boolean | array of these`). Map-typed attributes (like `retry_count_by_status: {429:2, 503:1}`) require JSON serialization and are awkward to query. Skip them. - -| Attribute | Type | Semantic | -| -------------------------- | ------ | ----------------------------------------------------------------------------------- | -| `attempt` | int | 1-based final attempt count (`attemptStartTimes.length`) | -| `retry_total_delay_ms` | int | Sum of all `delayMs` reported by `onRetry`; 0 if no retries | -| `ttft_ms` | int | TTFT per D1; undefined for non-streaming or aborted-before-first-chunk requests | -| `request_setup_ms` | int | Per D3 | -| `sampling_ms` | int | Per D6 | -| `output_tokens_per_second` | double | Derived; `output_tokens / (sampling_ms / 1000)`; undefined when `sampling_ms === 0` | - -Per-attempt status-code distribution (e.g., "2 of the 3 attempts were 429s") is queryable from log-bridge spans of `ApiRetryEvent` records. No need to duplicate it as a flattened attribute on the parent. - -**Why `sampling_ms` and `output_tokens_per_second` on the span**: derivable but cumbersome to compute in backend queries when summing across many spans. Same cost-benefit as `request_setup_ms` (D3). - -### D6 — Activate `recordApiRequestBreakdown()` for 3 of 4 phases - -In `endLLMRequestSpan` (or the wrapper that calls it), after computing TTFT/setup/sampling, emit: - -```ts -recordApiRequestBreakdown(config, model, [ - { phase: ApiRequestPhase.REQUEST_PREPARATION, durationMs: requestSetupMs }, - { phase: ApiRequestPhase.NETWORK_LATENCY, durationMs: ttftMs }, // ttftMs = network + first-token-generation - { phase: ApiRequestPhase.RESPONSE_PROCESSING, durationMs: samplingMs }, -]); -``` - -**Why skip `TOKEN_PROCESSING`**: qwen-code does stream chunk processing inline (consolidation happens in the wrapper at `loggingContentGenerator.ts:644`); the post-stream wrap-up phase is <10ms and not architecturally distinct. Filling it with a meaningless value pollutes the histogram. Leaving the enum value unused is safe — `apiRequestBreakdownHistogram.record(value, {model, phase})` is just a histogram with `phase` as a label; missing labels are simply absent in queries. - -**Why not redefine `NETWORK_LATENCY`**: the spec name is slightly misleading (it's network + first-token-generation, not pure network latency), but: - -- The enum is part of `metrics.ts:330-334` which is exported from `index.ts:117` and tested. -- Backend dashboards may already reference these phase names. -- Renaming or adding a new phase would be a breaking change for trivially marginal accuracy improvement. - -Document the semantic in the design doc; leave the enum unchanged. - -**Why on the span path, not parallel**: keeps `recordApiRequestBreakdown` colocated with span attribute writes — single gated emission point (see D7 idempotency), single ordering invariant. - -### D7 — `endLLMRequestSpan` idempotency: metric recording gated on existing double-end guard - -Phase 1.5 (#4302) established that `endLLMRequestSpan` may be called twice (abort path + error path collision). The existing guard at `session-tracing.ts:~470` (`if (!activeSpans.has(...)) return;`) prevents double `span.end()`. Phase 4 metric recording (D6) **must sit inside the same guarded block**, before `span.end()`: - -```ts -// session-tracing.ts — endLLMRequestSpan -const llmCtx = activeSpans.get(spanRef); -if (!llmCtx) return; // already ended — double-end guard -activeSpans.delete(spanRef); // claim the end - -// ... compute duration, set attributes ... -if (metadata) { - recordApiRequestBreakdown(config, llmCtx.attributes.model, [...]); // NEW — gated - recordTokenUsageMetrics(...); // existing -} - -span.end(); -``` - -This guarantees metric is recorded **exactly once** per LLM request, matching the span lifecycle. - -**Why not record in `loggingContentGenerator`**: it doesn't see the abort path. Recording at the span lifecycle layer ensures every LLM request that opens a span produces exactly one breakdown sample, regardless of success/failure/abort. - -### D8 — GenAI semantic conventions dual-emit (private name authoritative) - -Each Phase 4 attribute that corresponds to an OTel GenAI semconv attribute is written twice on the span: - -| qwen-code private (authoritative) | GenAI semconv (compat layer) | Unit conversion | Spec status | -| ------------------------------------------ | ----------------------------------------------- | --------------- | ------------ | -| `ttft_ms` (ms, int) | `gen_ai.server.time_to_first_token` (s, double) | `ttftMs / 1000` | Experimental | -| `input_tokens` (int) | `gen_ai.usage.input_tokens` (int) | identical | Stable | -| `output_tokens` (int) | `gen_ai.usage.output_tokens` (int) | identical | Stable | -| `cached_input_tokens` (int) (when present) | `gen_ai.usage.cached_tokens` (int) | identical | Experimental | -| `qwen-code.model` (string) | `gen_ai.request.model` (string) | identical | Stable | - -**Existing token attribute names** on the LLM span (set in `endLLMRequestSpan` before Phase 4): qwen-code uses bare `input_tokens` and `output_tokens` already. Phase 4 adds the `gen_ai.usage.*` siblings to match #4410's pattern. The bare names stay; **don't rename**. - -Fields with no GenAI semconv equivalent — `request_setup_ms`, `sampling_ms`, `retry_total_delay_ms`, `attempt`, `output_tokens_per_second` — are emitted only under the qwen-code namespace. - -**Why "private authoritative, semconv as compat"**: - -- Internal dashboards, SLOs, debugLogger output, qwen-logger RUM, ARMS queries — all reference `ttft_ms` etc. Treating those as canonical avoids a flag-day migration. -- The Experimental GenAI semconv may rename `gen_ai.server.time_to_first_token` before reaching Stable. If/when it does, we update the semconv emission; the qwen-code names don't move. -- Future spec-aware backends (Datadog AI views, Honeycomb AI, ARMS GenAI dashboards) auto-pick up the `gen_ai.*` attributes without our involvement. - -**Why dual-emit unit conversion** (ms ↔ seconds): GenAI semconv chose seconds-as-double for latency; qwen-code chose ms-as-int (matches `duration_ms` already on the span). Both representations have value; the conversion is cheap. - -## Helper API (additive to `session-tracing.ts`) - -```ts -// session-tracing.ts — LLMRequestMetadata interface extended (additive) -export interface LLMRequestMetadata { - // ... existing fields: inputTokens, outputTokens, cachedInputTokens, success, error, ... - - /** Time from successful attempt start to first user-visible content chunk (ms). Undefined for non-streaming or aborted-before-first-chunk requests. */ - ttftMs?: number; - - /** Time from generateContent entry to start of successful attempt (ms). Includes all failed retries + backoff. */ - requestSetupMs?: number; - - /** Final attempt number (1-based). 1 = no retries. */ - attempt?: number; - - /** Sum of all backoff delays before the successful attempt (ms). */ - retryTotalDelayMs?: number; -} - -// No new exported helpers — Phase 4 reuses startLLMRequestSpan / endLLMRequestSpan with extended metadata. -``` - -```ts -// types.ts — new event class -export class ApiRetryEvent implements BaseTelemetryEvent { - 'event.name': typeof EVENT_API_RETRY = EVENT_API_RETRY; - 'event.timestamp': string; - model: string; - prompt_id?: string; - attempt_number: number; - error_type: string; - error_message: string; - status_code?: number; - retry_delay_ms: number; - duration_ms: number; // = retry_delay_ms, drives LogToSpanProcessor bridge span width - - constructor(opts: { model: string; promptId?: string; attemptNumber: number; error: unknown; statusCode?: number; retryDelayMs: number }) { ... } -} - -// constants.ts -export const EVENT_API_RETRY = 'qwen-code.api_retry'; - -// loggers.ts -export function logApiRetry(config: Config, event: ApiRetryEvent): void { ... } -``` - -```ts -// utils/retry.ts — RetryOptions extension -interface RetryOptions { - // ... existing ... - onRetry?: (info: RetryAttemptInfo) => void; -} - -interface RetryAttemptInfo { - attempt: number; - error: unknown; - errorStatus?: number; - delayMs: number; -} - -// Inside retryWithBackoff, where logRetryAttempt is called today: -options.onRetry?.({ attempt, error, errorStatus, delayMs: actualDelay }); -logRetryAttempt(attempt, error, errorStatus); // existing debugLogger call unchanged -``` - -## Lifecycle wiring - -### Streaming path (the common case) - -```ts -// loggingContentGenerator.ts:283 — generateContentStream -async generateContentStream(req, userPromptId): Promise> { - const requestEntryTime = Date.now(); - let attemptStart = requestEntryTime; - const attemptStartTimes: number[] = [attemptStart]; - let retryTotalDelayMs = 0; - let finalAttempt = 1; - - // Use existing startLLMRequestSpan (Phase 1) - // Pass onRetry callback to whatever retry layer is in use: - const onRetry: RetryAttemptInfo & { invoke: ... } = (info) => { - finalAttempt = info.attempt + 1; // we're about to start attempt N+1 - retryTotalDelayMs += info.delayMs; - attemptStart = Date.now() + info.delayMs; // approximate; actual reset is at top of next attempt - attemptStartTimes.push(attemptStart); - // emit ApiRetryEvent - logApiRetry(this.config, new ApiRetryEvent({ - model: req.model, - promptId: userPromptId, - attemptNumber: info.attempt, - error: info.error, - statusCode: info.errorStatus, - retryDelayMs: info.delayMs, - })); - }; - - // stream wrapper detects first user-visible chunk: - return this.processStreamGenerator(stream, ..., { - onFirstUserVisibleChunk: (now) => { - ttftMs = now - attemptStart; - }, - }); -} -``` - -At span end (already in Phase 1's `endLLMRequestSpan` flow), include the new fields in `LLMRequestMetadata`: - -```ts -endLLMRequestSpan(llmSpan, { - success: true, - inputTokens, - outputTokens, - cachedInputTokens, - ttftMs, - requestSetupMs: attemptStart - requestEntryTime, - attempt: finalAttempt, - retryTotalDelayMs, -}); -``` - -### Non-streaming path - -`generateContent` (`loggingContentGenerator.ts:212`) does not produce streaming chunks. TTFT is `undefined`; `request_setup_ms` is still meaningful (captures retry overhead). The breakdown metric records 2 phases (REQUEST_PREPARATION + RESPONSE_PROCESSING where `RESPONSE_PROCESSING = duration_ms - request_setup_ms`), not 3. - -### Retry layer integration (4 sites) - -Each of the 4 LLM `retryWithBackoff` call sites adds `onRetry`: - -```ts -// client.ts:1540 (similar at baseLlmClient.ts:193, 282, geminiChat.ts:1039) -const result = await retryWithBackoff(apiCall, { - ...existingOptions, - onRetry: (info) => { - logApiRetry( - this.config, - new ApiRetryEvent({ - model, - promptId: userPromptId, - attemptNumber: info.attempt, - error: info.error, - statusCode: info.errorStatus, - retryDelayMs: info.delayMs, - }), - ); - // also feed back into LoggingContentGenerator's local retry accumulator - // (when in scope — for callers that don't go through LoggingContentGenerator, - // the LLM span still gets `attempt` and `retry_total_delay_ms` via the - // metadata path because endLLMRequestSpan is called at the LLM layer) - }, -}); -``` - -The non-LLM caller (`channels/weixin/src/api.ts`) **does not register `onRetry`** — no `ApiRetryEvent` is emitted for its retries, matching today's behavior. - -## Concurrent safety — the headline guarantee - -`LoggingContentGenerator` instance is shared (one per `ContentGenerator`, `contentGenerator.ts:377`). Three concurrent `generateContentStream` calls (e.g., 3 subagents fan out via `coreToolScheduler.runConcurrently`) execute three independent closures of `generateContentStream`: - -``` -call_A: attemptStart_A, ttftMs_A, ... (closure) -call_B: attemptStart_B, ttftMs_B, ... (closure) -call_C: attemptStart_C, ttftMs_C, ... (closure) -``` - -Per-call locals never overlap. Stream chunks are detected against the local `attemptStart` of each call. Span attributes are set at each call's own `endLLMRequestSpan`. - -`AsyncLocalStorageContextManager` (registered by NodeSDK at `sdk.ts:273`) already ensures the active OTel context — and thus the parent span passed to `startLLMRequestSpan` — is correct per fiber. - -## Files to change - -| File | Change | LOC est | -| -------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | -| `packages/core/src/telemetry/constants.ts` | Add `EVENT_API_RETRY` constant | +2 | -| `packages/core/src/telemetry/types.ts` | Add `ApiRetryEvent` class + union member | +40 | -| `packages/core/src/telemetry/loggers.ts` | Add `logApiRetry()` function | +20 | -| `packages/core/src/telemetry/qwen-logger/qwen-logger.ts` | Add `logApiRetryEvent()` for RUM downstream consistency | +20 | -| `packages/core/src/telemetry/session-tracing.ts` | Extend `LLMRequestMetadata` (ttftMs, requestSetupMs, attempt, retryTotalDelayMs); extend `endLLMRequestSpan` to set new attrs + breakdown metric + dual-emit gen_ai.\* | +60 | -| `packages/core/src/telemetry/metrics.ts` | Wire `recordApiRequestBreakdown` callsite inside `endLLMRequestSpan` (no change to the existing recorder) | 0 | -| `packages/core/src/utils/retry.ts` | Add `onRetry?: (info: RetryAttemptInfo) => void` to RetryOptions; export `RetryAttemptInfo`; invoke callback in the existing logRetryAttempt site | +25 | -| `packages/core/src/core/loggingContentGenerator/loggingContentGenerator.ts` | TTFT capture: method-local accumulators + `hasUserVisibleContent` helper + first-chunk detection in stream wrapper; pass new metadata to `endLLMRequestSpan` | +80 | -| `packages/core/src/core/client.ts` | Wire `onRetry` callback at `retryWithBackoff` call site (`client.ts:1540`) | +15 | -| `packages/core/src/core/baseLlmClient.ts` | Wire `onRetry` callback at 2 `retryWithBackoff` call sites | +25 | -| `packages/core/src/core/geminiChat.ts` | Wire `onRetry` callback at `retryWithBackoff` call site (`geminiChat.ts:1039`) | +15 | -| `packages/core/src/telemetry/session-tracing.test.ts` | `endLLMRequestSpan` sets ttft_ms / request_setup_ms / attempt / retry_total_delay_ms / sampling_ms / output_tokens_per_second + gen_ai dual-emit + breakdown metric (each phase) + idempotent end | +120 | -| `packages/core/src/core/loggingContentGenerator/loggingContentGenerator.test.ts` | `hasUserVisibleContent` (text / functionCall / inlineData / executableCode / thought / role-only / usage-only); concurrent calls don't cross-contaminate; TTFT undefined when aborted before first chunk; TTFT undefined on non-streaming | +100 | -| `packages/core/src/utils/retry.test.ts` | `onRetry` invoked per failed attempt with correct `attempt`, `delayMs`, `error`, `errorStatus`; absence of `onRetry` is silent (no telemetry emitted) | +50 | -| `packages/core/src/telemetry/loggers.test.ts` | `logApiRetry` emits LogRecord with expected payload; bridges through LogToSpanProcessor to nested span under active LLM span | +40 | - -Total: 14 files, ~610 LOC. Larger than Phase 2 (#4321) but comparable to Phase 3 (#4410) and justified by the breadth of integration (4 retry sites + telemetry plumbing + streaming wrapper). - -If review pushes back on size: split into **Phase 4a + 4b + 4c**: - -- **4a** (~200 LOC): TTFT capture + extended `LLMRequestMetadata` + dual-emit. Self-contained value (TTFT visibility from day one). -- **4b** (~250 LOC): `onRetry` callback + `ApiRetryEvent` + 4 caller wiring. **Independently a bug fix** for the `retryWithBackoff` telemetry gap. -- **4c** (~160 LOC): `recordApiRequestBreakdown` activation + parent span aggregation attrs (`attempt`, `retry_total_delay_ms`, `sampling_ms`, `output_tokens_per_second`). Depends on 4a + 4b. - -## Testing strategy - -| Test | What it proves | -| -------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- | -| `hasUserVisibleContent` returns true for text/functionCall/inlineData/executableCode/thought | D1 semantics across part types | -| `hasUserVisibleContent` returns false for role-only and usage-only chunks | D1 negative cases | -| streaming: TTFT measured from attempt start to first user-visible chunk | End-to-end TTFT detection | -| streaming: TTFT undefined if stream aborts before any user-visible chunk | Edge case | -| streaming: TTFT computed from final attempt's start (not first attempt) | D3 — TTFT reset on retry | -| non-streaming: TTFT remains undefined | S3 decision | -| concurrent `generateContentStream` calls don't cross-contaminate TTFT | D2 — method-local guarantee | -| `endLLMRequestSpan` sets all Phase 4 attrs (ttft_ms, request_setup_ms, sampling_ms, attempt, retry_total_delay_ms, output_tokens_per_second) | Attribute presence | -| `endLLMRequestSpan` dual-emits gen_ai.server.time_to_first_token + gen_ai.usage.\* + gen_ai.request.model | D8 dual-emit | -| `endLLMRequestSpan` records breakdown metric with 3 phases for streaming, 2 for non-streaming | D6 | -| `endLLMRequestSpan` called twice: metric recorded exactly once, attrs not re-set | D7 idempotency | -| `retryWithBackoff` with `onRetry`: callback invoked per failed attempt with correct args | D4 callback contract | -| `retryWithBackoff` without `onRetry`: no telemetry emitted (silent for non-LLM callers) | P2 — channels/weixin scope protection | -| `client.ts` / `baseLlmClient.ts` / `geminiChat.ts` retry callsites emit `ApiRetryEvent` on retry | Integration of D4 at 4 sites | -| `ApiRetryEvent` LogRecord bridges via LogToSpanProcessor to a child span under active LLM span | Trace tree correctness | -| LLM span `attempt` field correctly reflects final attempt number under retries | D5 aggregation | -| LLM span `retry_total_delay_ms` correctly sums onRetry delays | D5 aggregation | -| `output_tokens_per_second` undefined when `sampling_ms === 0` (no streaming) | Avoid divide-by-zero | - -## Edge cases - -| Case | Handling | -| ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| Stream aborts before any chunk arrives | `ttftMs = undefined`, `sampling_ms = undefined`, `output_tokens_per_second = undefined`. `attempt`, `request_setup_ms` still set. `success = false` | -| Stream aborts after first chunk | `ttftMs` set; `sampling_ms` = `duration_ms - ttftMs - request_setup_ms`; reflects partial response time. `success = false` | -| Retry succeeds on attempt 1 (no retries) | `attempt = 1`, `retry_total_delay_ms = 0`, no `ApiRetryEvent` emitted, breakdown metric records `request_setup_ms` close to 0 | -| Persistent retry mode 50+ attempts | 50+ `ApiRetryEvent` records emitted (out-of-scope cap deferred); LLM span `attempt = 51`, `retry_total_delay_ms = sum of all delays`. Operator sees aggregated view on span; full per-attempt detail in log-bridge spans | -| Non-LLM `retryWithBackoff` caller (channels/weixin) | No `onRetry` registered; only existing `debugLogger.warn` fires. No `ApiRetryEvent`; no breakdown metric (caller isn't an LLM site) | -| `endLLMRequestSpan` called twice (abort + error race) | Phase 1.5 guard at `activeSpans.delete()` returns early on second call; `recordApiRequestBreakdown` is inside the guard, recorded exactly once | -| Anthropic `message_start` chunk arrives before content | `hasUserVisibleContent` returns false for it (no parts with text/functionCall/etc.); TTFT not triggered until subsequent `content_block_delta` chunk | -| OpenAI first chunk with empty `delta.content` but `role` only | `hasUserVisibleContent` returns false; TTFT not triggered until first chunk with non-empty delta | -| Tool-call-only response (no text) | First chunk with `functionCall` Part triggers TTFT; `output_tokens_per_second` computed against tool-call token count | -| Concurrent subagents (3 calls in flight) | Each call's closure has its own `attemptStart`, `ttftMs`, `attemptStartTimes`. Per-call span receives its own metadata at `endLLMRequestSpan`. No interleaving (D2) | -| SDK-level retries inside openai-sdk (`maxRetries=3`) | Invisible to qwen-code telemetry — happens entirely inside SDK before retryWithBackoff sees the request. `attempt` reflects retryWithBackoff attempts only. Out of scope (see Out-of-scope) | -| `gen_ai.server.time_to_first_token` spec renames before reaching Stable | Single-file update: `session-tracing.ts:endLLMRequestSpan`. The qwen-code-native `ttft_ms` stays authoritative — no downstream impact | -| Subagent's LLM request | Parent is the subagent span (Phase 3). Phase 4 fields nest correctly. Aggregations grouped by `qwen-code.subagent.id` give per-subagent LLM perf — design-doc-future, easy follow-up | -| Reasoning model with long thought blocks | First `thought` Part triggers TTFT; `sampling_ms` includes both thinking + answer phases. Split into separate metrics deferred | - -## Rollback - -The change is additive at the OTel and metric level — every new attribute is optional, every new event is a new class. Existing dashboards that don't filter on the new fields keep working unchanged. - -Behavior-affecting changes: - -- New `ApiRetryEvent` LogRecord starts flowing → log volume increases proportional to retry rate (typically <1% of requests retry). Mitigate by sampling LogRecord at the SDK layer if needed. -- New breakdown metric `qwen-code.api.request.breakdown` starts producing time series → mild Prometheus cardinality bump (`{model, phase}` — bounded). -- `output_tokens_per_second` derived attribute may appear unusual on dashboards filtering "all attributes" — document. - -Rollback path: revert the single PR (or each of 4a/4b/4c independently). All new fields use defensive defaults (undefined / 0) and don't change span structure. - -## Sequencing - -- **After Phase 3 (#4410, in review)**: not a hard dependency. Phase 4 attributes attach to `qwen-code.llm_request` spans regardless of whether they're under a `qwen-code.subagent` (Phase 3) or `qwen-code.interaction` (Phase 1) parent. Recommend Phase 3 land first so per-attempt aggregation under subagent subtrees works naturally. -- **Independent of #4384** (`traceparent` + `X-Qwen-Code-Session-Id` outbound propagation). They touch the HTTP layer; Phase 4 touches the stream/retry/metric layer. -- **Independent of `clearDetailedSpanState` chat-compression follow-up** (#4097 follow-up). Different surface. - -## Open questions - -1. **`onRetry` callback firing semantics**: invoked **before** backoff sleep (current proposal) or **after** (when the next attempt is about to start)? Before is simpler — callback has all the info immediately; after would require capturing the just-completed delay separately. Pre-sleep is the recommendation; document in callback contract. -2. **Per-attempt timing on the LLM span**: should we add `attempt_durations_ms: number[]` array? OTel supports array-of-primitive attributes. Useful for "which attempt of N was slow" diagnostics. Defer until production data shows demand — log-bridge spans already carry the equivalent. -3. **Persistent retry mode emission cap**: at what `attempt > N` threshold should we start sampling? `N = 5` then 1-in-10? `N = 10` then summary-only? Defer until we have production volume data. -4. **`TOKEN_PROCESSING` phase**: keep enum value dormant or wire it to something (e.g., consolidation time)? Defer — wait for a real use case. -5. **Subagent-level LLM rollups**: trivial follow-up once Phase 4 lands — sum `ttft_ms`/`output_tokens`/`input_tokens` per subagent subtree. Not Phase 4 scope but the data flow enables it. diff --git a/docs/design/telemetry-outbound-propagation-design.md b/docs/design/telemetry-outbound-propagation-design.md deleted file mode 100644 index 91fea0d835a..00000000000 --- a/docs/design/telemetry-outbound-propagation-design.md +++ /dev/null @@ -1,878 +0,0 @@ -# Telemetry: Outbound Trace Context & Session ID Header Propagation - -> 配套 issue: [#4384](https://github.com/QwenLM/qwen-code/issues/4384) -> 父 issue: [#3731](https://github.com/QwenLM/qwen-code/issues/3731) (P3 deeper observability) -> 前置 PR: #4367 (resource attributes — merged 2026-05-21, commit `64401e1`) -> 基于 2026-05-21 对 qwen-code main 分支 + 直接验证的 claude-code 源码 - -## 修订历史 - -| 修订 | 日期 | 触发 | 摘要 | -| ---- | ---------- | --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| R1 | 2026-05-21 | 初稿 | 全广播：所有出站 LLM 请求都带 `X-Qwen-Code-Session-Id` + `traceparent` | -| R2 | 2026-05-22 | wenshao R2/R3 review | 边界安全：URL normalize、port matching、quote 对齐、staticCorrelationHeaders try/catch、host:port fallback strip | -| R3 | 2026-05-23 | LaZzyMan REQUEST_CHANGES | **重大语义改动**：`X-Qwen-Code-Session-Id` 默认作用域收窄到 first-party（Alibaba/DashScope）host 白名单。详见 §11 | -| R4 | 2026-05-25 | LaZzyMan round-8 follow-up (scope conflation) | **PR scope 大幅收窄**：本 PR 仅保留 client HTTP span + OTLP loop guard；`traceparent` 默认 off（NoopTextMapPropagator）；新增 `outboundCorrelation.*` 顶级 namespace 放安全相关 toggle；R3 落地的整套 `X-Qwen-Code-Session-Id` 机器**移除本 PR**，搬到独立 follow-up PR。详见 §12 | - -**特别提示**：阅读 §3.1（目标）/ §3.2（非目标）/ §4.3（Part B 设计）/ §4.4（配置 schema 影响）/ §5（文件改动清单）/ §9（与 claude-code 对比）/ §10（未来工作）/ §11（R3 host-allowlist scoping）时，请同时参考 §12 —— **R4 修订让 R1-R3 关于"本 PR 同时落地 traceparent + session id header"的论断不再成立**：本 PR 现仅为 telemetry observability + 独立的 outbound trace-context toggle，所有 outbound correlation header 工作（包括 R3 的 host allowlist）整体搬到独立 follow-up PR。R3 工作代码本身没浪费，挪到 follow-up PR 即可复用。 - -## 1. 背景 - -#4367 解决了**emitted telemetry 上的 attribute 与 cardinality**（操作员能给 span/log/metric 打 `user.id`/`tenant.id` 这类标签）。但有一类东西它没碰：**outbound LLM 请求的 HTTP header**。今天 qwen-code 发往 DashScope / OpenAI / Gemini / Anthropic 的请求**完全不带任何 cross-process correlation header**——既没有 W3C `traceparent`，也没有 session id。 - -后果： - -1. trace context 在 qwen-code 进程边界断开。若模型服务（如 ARMS Tracing 接入的 DashScope）本身有 OTel instrumentation，它产生的 span 与 qwen-code 的 trace 彼此独立，端到端 trace tree 不存在。 -2. 没有 session id 在 wire 上。后端要把 qwen-code 的 metric/log 与服务端日志关联，需要离线匹配 trace id 或时间戳，远不如直接读 header 简单。 -3. 本地 trace 缺一层 client-side HTTP span。今天只能看 `api.generateContent` 的总耗时，看不到网络 TTFB / 响应体大小 / 重试次数。 - -## 2. 现状 - -### 2.1 仅启用了 `HttpInstrumentation` - -`packages/core/src/telemetry/sdk.ts:330`： - -```ts -instrumentations: [new HttpInstrumentation()], -``` - -`HttpInstrumentation` 只 hook Node 内建的 `http`/`https` 模块，**不**覆盖 `globalThis.fetch` / undici 路径。 - -### 2.2 两套 LLM SDK 都走 fetch / undici - -| SDK | HTTP 实现 | `HttpInstrumentation` 是否覆盖 | -| ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ | -| `openai@5.11.0` | `globalThis.fetch`（Node 18+ 即 undici）。证据：`node_modules/openai/internal/shims.mjs` 报错 `'fetch' is not defined as a global` | ❌ | -| `@google/genai@1.30.0` | `globalThis.fetch` + `new Headers()`。证据：`dist/node/index.mjs` 内的 `new Headers()` 调用 | ❌ | -| `@anthropic-ai/sdk`（anthropicContentGenerator） | 同样基于 fetch | ❌ | - -### 2.3 代码库零 manual propagation - -``` -grep -rn "propagation\.\|setGlobalPropagator\|W3CTraceContext\|traceparent" packages/core/src --include="*.ts" | grep -v "\.test\." -``` - -→ 空。没有任何 `propagation.inject()` 调用，没有手动 traceparent 注入。 - -### 2.4 各 provider 的 `defaultHeaders` 现状 - -OpenAI 家族（用 `openai` SDK）： - -所有 OpenAI 子 provider 都 `extends DefaultOpenAICompatibleProvider`。**buildHeaders override 行为分两类**（已 grep audit 验证）： - -| Provider | 文件 | `buildHeaders()` 行为 | 影响 | -| ---------- | ---------------------- | --------------------------------------------------------------------------------------- | ---------------------------------------------- | -| 基类 | `default.ts:63-74` | 提供 `{ 'User-Agent' }` + customHeaders | 改这里 | -| DashScope | `dashscope.ts:110-124` | **`override` 但不 call `super`**——返回 `User-Agent` + `X-DashScope-*` 全新对象 | **必须单独改这里**，否则 correlation header 丢 | -| OpenRouter | `openrouter.ts:20-30` | `override` 但**先 `const baseHeaders = super.buildHeaders()`** | 改基类自动继承 ✅ | -| DeepSeek | `deepseek.ts` | 不 override `buildHeaders`（只 override `buildRequest` / `getDefaultGenerationConfig`） | 改基类自动继承 ✅ | -| Minimax | `minimax.ts` | 同 deepseek | 自动继承 ✅ | -| Mistral | `mistral.ts` | 同 deepseek | 自动继承 ✅ | -| ModelScope | `modelscope.ts` | 同 deepseek | 自动继承 ✅ | - -→ **OpenAI 家族需要触动 2 个文件**：`default.ts` 和 `dashscope.ts`。其余 5 个自动继承。 - -Google Gemini： - -| Provider | 文件 | 头注入路径 | -| -------- | ------------------------------ | -------------------------------------------------------------- | -| Gemini | `geminiContentGenerator.ts:59` | `new GoogleGenAI({ httpOptions: { headers } })` — SDK 原生支持 | - -Anthropic： - -| Provider | 文件 | 头注入路径 | -| --------- | ------------------------------------------------------------------------------------------------------ | ---------------- | -| Anthropic | `anthropicContentGenerator.ts:177` (`buildHeaders`) + `:212` (`defaultHeaders` arg to `new Anthropic`) | `defaultHeaders` | - -**总计 4 个 SDK 构造点**需要注入 session id header。所有 SDK 都已支持 `defaultHeaders` / `httpOptions.headers`，无需 fetch wrapper。 - -### 2.5 已有的 proxy 与 fetch 配置 - -`provider/default.ts:87-89`： - -```ts -const runtimeOptions = buildRuntimeFetchOptions( - 'openai', - this.cliConfig.getProxy(), -); -``` - -`buildRuntimeFetchOptions` 在用户配 proxy 时返回 `{ fetch: customFetch }` 或类似，触发 `setGlobalDispatcher(new ProxyAgent(...))`（见 `config.ts:1126-1128`）。**undici 全局 dispatcher 模式与 `UndiciInstrumentation` 兼容**——它通过 monkey-patch `globalThis.fetch` 与 undici 的 channel diagnostics 协作，不依赖具体 dispatcher。 - -## 3. 目标 / 非目标 - -### 3.1 目标 - -- 所有 outbound LLM 请求自动带 W3C `traceparent` header（OTel SDK 默认的 `W3CTraceContextPropagator`） -- ~~所有~~ 出站 LLM 请求带 `X-Qwen-Code-Session-Id` header（claude-code 同款产品命名空间） — **R3 修订**：默认仅向 first-party (Alibaba/DashScope) host 注入，第三方 provider 默认不发；详见 §11 -- 自动避免对 OTLP exporter endpoint 自身的 trace（feedback loop） -- 给 LLM 请求加一层精确的 client span（网络耗时 vs 模型耗时分离） -- 覆盖 4 个 provider 构造点：OpenAI 基类、DashScope override、Gemini、Anthropic -- streaming 请求 / proxy 模式 / 重试场景全部不退化 -- 与 #4367 的设计哲学一致：通过 `defaultHeaders` 这种 SDK-native 选项 — **R1 修订**：因 staleness 问题转用 fetch wrapper；**R3 修订**：fetch wrapper 内再叠加 host gate - -### 3.2 非目标 - -- **`baggage` header**：标准 SDK 已支持，但 qwen-code 没调 `propagation.setBaggage()`，默认不会发送。本设计不主动开启。 -- **subprocess `TRACEPARENT` env var 继承**：claude-code 给 Bash/PowerShell 子进程注入 `TRACEPARENT`。qwen-code 的 `BashTool` 没做。是独立 follow-up sub-issue。 -- **inbound `TRACEPARENT` / `TRACESTATE` 读取**：claude-code 的 `-p` 模式和 Agent SDK 从 env 读 traceparent 接续父进程 trace。qwen-code 没做。独立 follow-up。 -- **`X-Qwen-Code-Request-Id`**：claude-code 有 `x-client-request-id`，对超时容错 correlation 有用。本期不做，可作为下一个 sub-issue。 -- **自定义 propagator（B3 / Jaeger / X-Ray）**：默认 W3C 已覆盖 99% 场景。可作为 future config option。 -- ~~**per-endpoint 选择性注入**：claude-code 对第三方 endpoint (Bedrock / Vertex) 不发 traceparent；qwen-code 没有第三方区分需要，统一发即可。~~ — **R3 修订**：此论断已被推翻。LaZzyMan review 指出 qwen-code 是开源 CLI 连接多个第三方 provider（OpenAI / Anthropic / OpenRouter / 等），claude-code 的 first-party→first-party 类比不适用；session id header 必须按 host 区分。详见 §11。`traceparent` 仍按 R1 设计全注入（OTel 标准 header，且 trace id 是 `sha256(sessionId)` 哈希值），可作为独立 follow-up 加 per-destination toggle（`telemetry.propagateTraceContext`）。 - -## 4. 设计 - -### 4.1 总体分层 - -``` -┌─ qwen-code process ────────────────────────────────────────────┐ -│ │ -│ ┌─ session-tracing.ts ─┐ │ -│ │ active span ctx │ │ -│ └──────┬───────────────┘ │ -│ │ │ -│ ▼ │ -│ ┌─ propagation.inject() (called by undici instrumentation) ─┐│ -│ │ writes `traceparent: 00---01` to headers ││ -│ └─────────────────────────────────────────────────────────────┘│ -│ │ │ -│ ┌──────▼──────────────────────────────────────────────────┐ │ -│ │ fetch() — undici, instrumented │ │ -│ │ creates HTTP client span │ │ -│ │ injects traceparent into request headers │ │ -│ │ (skipped via ignoreRequestHook if endpoint is OTLP) │ │ -│ └─────────────────────────────────────────────────────────┘ │ -│ │ │ -│ │ ┌─ defaultHeaders (per SDK constructor) ───────┐ │ -│ │ │ { 'X-Qwen-Code-Session-Id': sessionId, ... } │ │ -│ └───┴────────────────────────────────────────────────┘ │ -│ │ │ -└─────────────┼──────────────────────────────────────────────────┘ - │ - ▼ outbound HTTP - POST /v1/chat/completions - traceparent: 00-... - X-Qwen-Code-Session-Id: ... - ... (existing User-Agent, X-DashScope-*, etc.) -``` - -两条注入路径独立、互不依赖： - -| Layer | 何时注入 | 由谁注入 | -| ------------------------ | ------------------------------------- | ------------------------------------------------------------- | -| `traceparent` | 每次 fetch 调用时 | `UndiciInstrumentation` 自动（来自 OTel SDK 默认 propagator） | -| `X-Qwen-Code-Session-Id` | SDK 构造时一次性写入 `defaultHeaders` | 应用代码 | - -### 4.2 Part A — `traceparent` via undici instrumentation - -**改动点**：`packages/core/src/telemetry/sdk.ts` - -```ts -import { UndiciInstrumentation } from '@opentelemetry/instrumentation-undici'; - -// ... -const otlpUrls = [ - config.getTelemetryOtlpEndpoint(), - config.getTelemetryOtlpTracesEndpoint(), - config.getTelemetryOtlpLogsEndpoint(), - config.getTelemetryOtlpMetricsEndpoint(), -] - .filter((u): u is string => !!u) - .map((u) => u.replace(/\/$/, '')); - -instrumentations: [ - new HttpInstrumentation(), - new UndiciInstrumentation({ - ignoreRequestHook: (request) => { - // request.origin = "https://collector:4318", request.path = "/v1/traces" - const url = `${request.origin}${request.path}`; - return otlpUrls.some((e) => url.startsWith(e)); - }, - }), -], -``` - -#### 为什么 `ignoreRequestHook` 必须 - -OTel SDK 自己用 fetch 把数据 POST 到 OTLP collector。如果不跳，UndiciInstrumentation 会给"上报数据"的请求也建一个 span → 这个新 span 会被再次上报 → 无限循环 / 巨量噪声。每个 OTel 项目都踩过这个坑，OTel 文档明确推荐这种 hook。 - -#### 默认 propagator - -OTel SDK `NodeSDK` 不传 `textMapPropagator` 时默认是 `CompositePropagator([W3CTraceContextPropagator, W3CBaggagePropagator])`。无需显式设置。 - -#### `traceparent` 格式 - -``` -traceparent: 00-<32hex traceId>-<16hex spanId>-<01 sampled | 00 not sampled> - ─┬─ ─┬─ - version (固定 00) flags -``` - -固定 55 bytes，无 padding。 - -#### `tracestate` 与 `baggage` - -- `tracestate`: 上游传过来才续传；自己 inject 不会主动加（OTel SDK 行为）。 -- `baggage`: 仅当 `propagation.setBaggage(ctx, ...)` 被调用过才有。qwen-code 不调，所以不会发送。 - -### 4.3 Part B — `X-Qwen-Code-Session-Id` via fetch wrapper（OpenAI / Anthropic）+ static headers（Gemini） - -> **R3 修订**：以下设计描述的是 fetch wrapper 的 staleness 解决和 4 个 provider 集成点 — 这些都保留。但 wrapper 内部增加了一道 host allowlist gate，`staticCorrelationHeaders` 也加了 `destinationUrl` 参数。带 host gate 的最新实现代码与 default allowlist 见 §11。 - -#### Critical：staleness 问题与方案选择 - -天真做法（`defaultHeaders` 直接 bake-in `getSessionId()`）有**真 bug**： - -1. `pipeline.ts:60` 在 contentGenerator 构造时一次性 `this.client = this.config.provider.buildClient()`，SDK client 的 `defaultHeaders` 在那一刻 capture 当时的 session id -2. `config.ts:1850` 的 session reset（用户 `/clear` 时触发）更新 `this.sessionId` 并 `refreshSessionContext()`，但**不重建 contentGenerator** -3. 后续 LLM 调用仍走旧 client → wire header 仍是旧 session id → 后端 correlation 错位 - -→ 必须读取 session id **per-request**，不能 bake at构造时。 - -#### 方案 - -``` - ┌─ fetch 支持 ─┐ 方案 -OpenAI SDK │ ✅ │ fetch wrapper (per-request 读 sessionId) ✅ -Anthropic SDK │ ✅ │ fetch wrapper ✅ -@google/genai SDK │ ❌ │ static httpOptions.headers + 接受 staleness - └──────────────┘ -``` - -`@google/genai`'s `HttpOptions` interface 不支持 `fetch`（已 grep `node_modules/@google/genai/dist/genai.d.ts` 验证：只有 `baseUrl`/`apiVersion`/`headers`/`timeout`/`extraParams`）。所以 Gemini 走 static headers，与 OpenAI/Anthropic 不一致——这是 **known limitation**，见 §8.6。 - -#### 集中辅助函数（per-request fetch wrapper） - -新文件 `packages/core/src/telemetry/llm-correlation-fetch.ts`： - -```ts -import type { Config } from '../config/config.js'; - -/** - * Wrap a fetch implementation so every outbound request gets correlation - * headers (`X-Qwen-Code-Session-Id`) populated from the **current** session - * id, not the value captured when the SDK client was constructed. - * - * Matches claude-code's pattern (src/services/api/client.ts:370-390 — - * `buildFetch()`). Per-request injection is necessary because `/clear` - * resets the session id mid-process; SDK clients (and their static - * `defaultHeaders`) are NOT recreated on reset. - * - * Caller responsible for choosing the base fetch — usually - * `runtimeOptions?.fetch ?? globalThis.fetch` so proxy-aware fetch is - * preserved when ProxyAgent is in use. - * - * If telemetry is disabled, returns baseFetch unchanged (no correlation - * header is added, matching the privacy stance of §3.1). - */ -export function wrapFetchWithCorrelation( - baseFetch: typeof fetch, - config: Config, -): typeof fetch { - return async function correlationFetch(input, init) { - if (!config.getTelemetryEnabled()) { - return baseFetch(input, init); - } - const sid = config.getSessionId(); - if (!sid) { - // Defensive: empty header value is rejected by some HTTP middleware. - // Skip injection rather than send `X-Qwen-Code-Session-Id: `. - return baseFetch(input, init); - } - const headers = new Headers(init?.headers); - headers.set('X-Qwen-Code-Session-Id', sid); - return baseFetch(input, { ...init, headers }); - }; -} -``` - -Companion helper for the SDKs that can only take static headers (Gemini): - -```ts -/** - * Static correlation headers. Captures the session id at call time — - * **subject to staleness** if the host SDK keeps these headers in a - * captured-at-construction slot (e.g. `@google/genai`'s `httpOptions.headers`). - * Prefer `wrapFetchWithCorrelation` whenever the SDK exposes a `fetch` hook. - */ -export function staticCorrelationHeaders( - config: Config, -): Record { - if (!config.getTelemetryEnabled()) return {}; - return { 'X-Qwen-Code-Session-Id': config.getSessionId() }; -} -``` - -#### 集成点 1: `provider/default.ts` (OpenAI 基类) - -`buildClient()` 改动——compose 现有 `runtimeOptions.fetch`（proxy）与我们的 wrapper： - -```ts -buildClient(): OpenAI { - // ... existing ... - const runtimeOptions = buildRuntimeFetchOptions('openai', this.cliConfig.getProxy()); - const baseFetch = - (runtimeOptions as { fetch?: typeof fetch } | undefined)?.fetch - ?? globalThis.fetch; - return new OpenAI({ - apiKey, - baseURL: baseUrl, - timeout, - maxRetries, - defaultHeaders, - ...(runtimeOptions || {}), - // After spread, override `fetch` so our correlation wrapper wraps the - // proxy-aware fetch (or globalThis.fetch when no proxy). - fetch: wrapFetchWithCorrelation(baseFetch, this.cliConfig), - }); -} -``` - -`buildHeaders()` itself unchanged. - -#### 集成点 2: `provider/dashscope.ts` (override) - -`buildClient()` 同样的 compose 模式（它本来就 override buildClient）。`buildHeaders()` 不动。 - -#### 集成点 3: `geminiContentGenerator/index.ts` (factory, NOT 构造器) - -**修正先前设计的过度声明**：`geminiContentGenerator.ts` 构造器**不需要**改签名。`index.ts:48` 的 factory 函数已经接收 `gcConfig: Config`（line 33 已经在用 `gcConfig?.getUsageStatisticsEnabled()`），只需要在 factory 里把 correlation 静态 headers merge 进 `httpOptions.headers`： - -```ts -// geminiContentGenerator/index.ts -let headers: Record = { ...baseHeaders }; -if (gcConfig?.getUsageStatisticsEnabled()) { - // ... existing x-gemini-api-privileged-user-id ... -} -headers = { ...headers, ...staticCorrelationHeaders(gcConfig) }; // ← 新增 -const httpOptions = config.baseUrl - ? { headers, baseUrl: config.baseUrl } - : { headers }; -// new GeminiContentGenerator(...) unchanged -``` - -零 signature 改动。 - -#### 集成点 4: `anthropicContentGenerator.ts` - -Anthropic SDK 同样接受 custom `fetch`（已经在用 `buildRuntimeFetchOptions`）。把 `buildClient` 路径里那个 fetch wrap 一下，方式同 OpenAI default.ts。`buildHeaders` 不变。 - -#### 优先级链 - -不变：用户的 `customHeaders` 在 `defaultHeaders` merge 中仍然赢（见 §8.2 spoofing 讨论）。fetch wrapper 注入的 `X-Qwen-Code-Session-Id` 在 SDK 的 headers list 之**后**追加到最终 `Headers` 对象上——以 Node `Headers.set()` 的语义，等于覆盖任何之前同名的（包括 user 的 customHeaders 里写的同名 header）。 - -**对 OpenAI/Anthropic（fetch wrapper 路径）**：correlation > customHeaders > SDK defaults。 -**对 Gemini（static headers 路径）**：customHeaders > correlation > SDK defaults（沿用既有 spread 顺序）。 - -差异是 fetch wrapper 路径下 spoofing 不再可能（fetch wrapper 在 SDK headers 之后跑）。这是 **bug 修复的副产品**，并非有意收紧——但更安全。要在 §8.2 明示。 - -### 4.4 配置 schema 影响 - -~~**几乎为零**。本设计不引入新 setting~~ — **R3 修订**：引入了一项新 setting `telemetry.sessionIdHeaderHosts: string[]`，用于覆盖默认的 first-party host 白名单。schema 项已加入 `packages/cli/src/config/settingsSchema.ts`，描述与 override 语法（`["*"]` 恢复广播 / `[]` 全关 / 自定义数组）见 §11。原文以下描述仅适用于 R3 之前： - -- `traceparent` 注入由 telemetry enabled 触发（已有 toggle） -- `X-Qwen-Code-Session-Id` 注入也由 telemetry enabled 触发 -- `ignoreRequestHook` 的 OTLP url 已经从现有 config 读 - -未来可以加的 setting（**out of scope**）： - -- `telemetry.outboundCorrelationHeader`: 自定义 header name（默认 `X-Qwen-Code-Session-Id`） -- `telemetry.outboundPropagationDisabled`: 全局关闭（如果 LLM 服务对未知 header 严格） -- ~~per-destination header scope toggle~~ — **R3 已落地**，见 §11 - -## 5. 文件改动清单 - -| 文件 | 改动类型 | 说明 | -| ------------------------------------------------------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `packages/core/package.json` | 加依赖 | `@opentelemetry/instrumentation-undici` | -| `packages/core/src/telemetry/sdk.ts` | 修改 | +`UndiciInstrumentation` + `ignoreRequestHook` | -| `packages/core/src/telemetry/llm-correlation-fetch.ts` | 新文件 | `wrapFetchWithCorrelation()` (OpenAI/Anthropic) + `staticCorrelationHeaders()` (Gemini fallback) | -| `packages/core/src/core/openaiContentGenerator/provider/default.ts` | 修改 | `buildClient()` 在 `new OpenAI({...})` 里加 `fetch: wrapFetchWithCorrelation(baseFetch, cliConfig)` | -| `packages/core/src/core/openaiContentGenerator/provider/dashscope.ts` | 修改 | 同上（override `buildClient`） | -| `packages/core/src/core/geminiContentGenerator/index.ts` | 修改 | factory 函数里 merge `staticCorrelationHeaders(gcConfig)` 进 `httpOptions.headers`（**caller 已有 Config，零 signature 改动** — 修正之前的 over-specification） | -| `packages/core/src/core/anthropicContentGenerator/anthropicContentGenerator.ts` | 修改 | `buildClient` 路径下用 `wrapFetchWithCorrelation` 包 SDK 的 `fetch` option | - -**显式 audited 但无需改动**（避免 reviewer 怀疑漏路径）： - -- `packages/core/src/qwen/qwenContentGenerator.ts` — `extends OpenAIContentGenerator`，用 `DashScopeOpenAICompatibleProvider`，**自动继承 dashscope.ts 的 buildClient 改动**。所有 Qwen OAuth 流程同样受益。 -- `packages/core/src/core/loggingContentGenerator/loggingContentGenerator.ts` — wrapper 模式，不构造 SDK client（它包装其他 contentGenerator 做 telemetry logging），无需改动。 -- `packages/core/src/core/contentGenerator.ts` — factory 入口，不持有 client。 - | `packages/core/src/telemetry/sdk.test.ts` | 修改 | 加 undici instrumentation 注册 + ignoreRequestHook 测试 | - | `packages/core/src/telemetry/llm-correlation-fetch.test.ts` | 新文件 | telemetry-on/off 行为单测 + per-request 读 sessionId 验证（critical：session reset 后 wrapped fetch 读到新 id） | - | 各 provider 的 `*.test.ts` | 修改 | 断言 SDK 构造时 `fetch` option 是 wrapped 版本（OpenAI/Anthropic）；断言 Gemini 构造时 `httpOptions.headers` 含 `X-Qwen-Code-Session-Id` | - | `docs/developers/development/telemetry.md` | 修改 | 新增 "Trace context & session correlation propagation" 段 | - | `docs/design/telemetry-outbound-propagation-design.md` | 本文件 | 设计文档 | - -## 6. 分 PR 拆分 - -按 review 友好度分两个 PR（也可以合一，规模允许）： - -### PR 1 — `traceparent` 自动注入（structural） - -- 加 `@opentelemetry/instrumentation-undici` 依赖 -- `sdk.ts` 加 `UndiciInstrumentation` + `ignoreRequestHook` -- 测试：SDK 注册、OTLP endpoint 不被 trace -- 文档片段 - -**风险**：低。Additive。已有 client span 是 net 增益，不会改变现有 span 结构。 - -### PR 2 — `X-Qwen-Code-Session-Id` header（结合 helper 函数） - -- 新文件 `llm-correlation-headers.ts` -- 4 个 provider 集成 -- 测试：每个 provider 断言 header 存在；telemetry-off 时不发 -- 文档片段 - -**风险**：低-中。要小心 `geminiContentGenerator` 构造器签名扩展可能波及调用方。 - -### PR 3（可选） — Docs + E2E verify - -- 完善 `telemetry.md` 段落 -- 加 E2E verify script（复用 `/tmp/verify-telemetry-pr-4367.mjs` 模式）：实际跑 fetch + 抓 header - -也可以合并到 PR 2 里。 - -### 顺序偏好 - -PR 1 和 PR 2 技术上**互相独立**——不共享代码。但**推荐 PR 1 先合**： - -- `traceparent` 是 OTel **标准** header，任何 OTel-aware collector / 后端立刻识别 → 用户立即获益 -- `X-Qwen-Code-Session-Id` 是**产品自定义** header，需要后端配置识别才有价值 → 价值滞后 -- 万一 PR 2 review 周期长，PR 1 已经把 cross-process trace 跑通了 -- PR 1 是 additive structural（低风险），适合先建立信心 - -## 7. 测试计划 - -### 7.1 `sdk.ts` 单测 - -- ✅ `UndiciInstrumentation` 在 `NodeSDK` 的 `instrumentations` 中存在 -- ✅ `ignoreRequestHook` 对 `https://collector:4318/v1/traces` 返回 true -- ✅ `ignoreRequestHook` 对 `https://dashscope.aliyuncs.com/...` 返回 false -- ✅ trailing slash 与无 trailing slash 都正确匹配 - -### 7.2 `llm-correlation-fetch.ts` 单测 - -**`wrapFetchWithCorrelation`**： - -| 场景 | 期望 | -| ------------------------------------------------------- | ---------------------------------------------------------------------- | -| `getTelemetryEnabled() === false` | wrapped fetch = baseFetch（不加任何 header） | -| `getTelemetryEnabled() === true`, sessionId = "abc-123" | wrapped fetch 发出的 init.headers 含 `X-Qwen-Code-Session-Id: abc-123` | -| `init.headers` 已有 `X-Qwen-Code-Session-Id: spoof` | wrapper 后覆盖为真 sessionId（fetch wrapper 路径不允许 spoof，§8.1） | -| **session reset 后 wrapped fetch 被再次调用** | **读取新 sessionId**（regression guard for staleness fix） | -| baseFetch reject | wrapper 透传 reject 不吞 | - -**`staticCorrelationHeaders`**（Gemini path）： - -| 场景 | 期望返回 | -| ------------------------------------------------------- | ---------------------------------------------------------------- | -| `getTelemetryEnabled() === false` | `{}` | -| `getTelemetryEnabled() === true`, sessionId = "abc-123" | `{ 'X-Qwen-Code-Session-Id': 'abc-123' }` | -| sessionId 中含 unicode（`會話-1`） | 原样返回——HTTP header value 由 SDK 负责编码 | -| sessionId 为空字符串 | `{ 'X-Qwen-Code-Session-Id': '' }`——业务 invariant，不在此层校验 | - -### 7.3 Per-provider 集成测试 - -每个 provider 的 `buildHeaders()` / 构造测试加： - -```ts -it('includes X-Qwen-Code-Session-Id when telemetry enabled', () => { - const config = makeFakeConfig({ - sessionId: 'sess-xyz', - telemetry: { enabled: true }, - }); - const provider = new DefaultProvider(genConfig, config); - expect(provider.buildHeaders()['X-Qwen-Code-Session-Id']).toBe('sess-xyz'); -}); - -it('omits X-Qwen-Code-Session-Id when telemetry disabled', () => { - const config = makeFakeConfig({ telemetry: { enabled: false } }); - const provider = new DefaultProvider(genConfig, config); - expect(provider.buildHeaders()).not.toHaveProperty('X-Qwen-Code-Session-Id'); -}); -``` - -### 7.4 E2E verification（tmux + local HTTP server） - -⚠️ **不要** mock `globalThis.fetch` 来抓 header：`UndiciInstrumentation` 通过 undici 的 diagnostics channel hook，monkey-patching globalThis.fetch 可能完全 bypass instrumentation（取决于 patch 顺序），让 `traceparent` 注入测不到。**正确做法是起 local HTTP server**，让 SDK 真发请求，server 端记录收到的 headers。 - -写一个仿 `/tmp/verify-telemetry-pr-4367.mjs` 的脚本： - -1. `http.createServer((req, res) => { capturedHeaders.push(req.headers); res.end('{}') })` 起本地 server -2. 启 telemetry + outfile + 把 OpenAI SDK 的 `baseURL` 指向 `http://127.0.0.1:`（或者用 mock provider 让 SDK 真发 fetch） -3. 触发一次 `client.chat.completions.create(...)`（要带最小可解析的 mock 响应，否则 SDK 解析报错——本地 server 返回合法但空的 OpenAI 响应即可） -4. 断言 `capturedHeaders[0]` 含 `traceparent: 00-...` 和 `X-Qwen-Code-Session-Id: ` -5. 另起一个 OTLP collector mock 在 different port，验证给它发的 OTLP 上报**不**触发 `traceparent` 注入（验证 `ignoreRequestHook`） -6. **额外：staleness 验证** — emit request 1 → call `config.resetSession(...)` → emit request 2 → 断言 request 2 的 `X-Qwen-Code-Session-Id` 是新 session id（**这是 #1 fix 的关键回归测试**） - -### 7.5 回归保护 - -- streaming chat completion 的 fetch（带 `stream: true`）仍正常关闭——`UndiciInstrumentation` 历史上对 streaming response 的 span lifecycle 有过 bug，**实施时需要实际跑一次 streaming completion 端到端验证 client span 正常 end + 无 leaked span + 流不被截断**；不假设具体版本号已修 -- proxy mode (`ProxyAgent`) 与 instrumentation 同时启用——`ignoreRequestHook` 仍按 endpoint 字符串匹配，proxy 不影响 -- 重试（`maxRetries`）下每次重试都得到独立 client span，但都共享同一个 `traceparent` parent（理想是 retry 作为同一个父 span 下多个 child span — 这部分由 SDK 行为决定，本设计不强制） - -## 8. 边界 / 边角 - -### 8.1 customHeaders override 与 spoofing 的不一致行为 - -不同 provider 路径的 spoofing 表面**不同**（设计后果，非原意收紧）： - -| Provider 路径 | spoofing 可能? | 原因 | -| --------------------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------- | -| OpenAI / Anthropic (fetch wrapper 路径) | ❌ 不能 spoof | fetch wrapper 在 SDK headers list 之后 `headers.set('X-Qwen-Code-Session-Id', ...)`，覆盖 user customHeaders 的同名 | -| Gemini (static headers 路径) | ✅ 可 spoof | merge 顺序 `{ ...baseHeaders, ...correlationHeaders, ...customHeaders }`——customHeaders 最后赢 | - -claude-code 同样使用 fetch wrapper 路径，行为与 OpenAI/Anthropic 一致（spoofing 不能）。这是修 staleness bug 的副产品，不是原本要做的事。 - -**不打算"对齐"两条路径**——Gemini 路径的行为是 SDK 限制（没有 `fetch` hook）导致的，反向把 OpenAI 也降级到 static 不合理。 - -Session id spoofing 不是真威胁（用户控制本地，可以直接改 source code）。文档里要明示这个差异，避免 reviewer 看到 fetch wrapper 路径无法 spoof 时质疑 customHeaders 优先级。 - -### 8.2 OTLP collector URL 匹配的两类 edge case - -#### (a) Auth token in URL - -如果用户 OTLP endpoint 形如 `https://collector/path?token=secret`，`ignoreRequestHook` 的 `url.startsWith(e)` 比对应包含 query string。但 undici 给的 `request.path` 只到 path（不含 query），所以比较时 `e` 也只用到 path 部分。为安全起见，剥掉 query： - -```ts -const otlpUrls = [...] - .map((u) => u.replace(/\?.*$/, '').replace(/\/$/, '')); -``` - -#### (b) startsWith 跨 hostname 边界的理论 false positive - -若 `e = "http://collector"`（无 port），来路 url = `http://collector-fake/v1/traces` 会被 startsWith 错误匹配。 - -**实际触发概率极低**： - -- OTLP endpoint 几乎总带 port（4317 gRPC / 4318 HTTP），`http://collector:4318` 形态后 `-fake` 这种延伸不可能（port 后跟的是 `/`） -- 用户配 endpoint 不带 port 是配置错误，本来 SDK 就要默认 fallback - -**如果想 harden**：解析 URL origin + path 分别比较，不用裸 startsWith： - -```ts -const parsed = otlpUrls.map((u) => new URL(u)); -return parsed.some( - (e) => - `${request.origin}` === e.origin && request.path.startsWith(e.pathname), -); -``` - -本期不做——开销没必要，false positive 实际触发不到。 - -### 8.3 Vertex AI 模式的 Gemini - -`@google/genai` 支持 `vertexai: true` 模式（用 GCP 凭据走 Vertex 端点而非 generative ai endpoint）。两种模式都走 fetch，所以 instrumentation 都覆盖。`httpOptions.headers` 在两种模式下都有效。 - -### 8.4 Anthropic SDK 已有 `defaultHeaders` 逻辑 - -`anthropicContentGenerator.ts:177` 已经在调 `buildHeaders()` 然后传给 `new Anthropic({ defaultHeaders })`。但 staleness 同样适用——本设计改用 `fetch` wrapper 路径（与 OpenAI 一致）。 - -### 8.5 SDK 与 fetch 之间的 trailer header - -`openai` SDK 在 streaming 时可能用 `Transfer-Encoding: chunked` 和 trailer headers。这些都不影响 request-time 的 `traceparent` / `X-Qwen-Code-Session-Id` 注入——它们都是请求头，发出时一次性写入。 - -### 8.6 ⚠️ Known limitation: Gemini 的 session id 在 `/clear` 后 stale - -由于 `@google/genai` SDK 不支持 `fetch` hook（`HttpOptions` 接口只有 `baseUrl`/`apiVersion`/`headers`/`timeout`/`extraParams`），Gemini provider 走 static `httpOptions.headers` 路径——session id 在 SDK 构造时 capture，**`/clear` 触发 session reset 后不刷新**。 - -**实际影响范围**： - -- 用户启动 qwen-code → `/clear` → 用 Gemini 模型 → wire 上的 `X-Qwen-Code-Session-Id` 是旧 session id -- 后端 correlation 错位（trace id 和 log 已正确切换到新 session，但 wire header 滞后） - -**为什么不修**（本期）： - -- OpenAI / Anthropic 路径**没有这个 bug**（fetch wrapper 路径 per-request 读 session id） -- Gemini fix path 有几个选项，全部超出本期 scope（见下） - -**Future fix path 选项**（按推荐顺序）： - -| 选项 | 描述 | 代价 | -| --------------------------------------------- | ------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------- | -| **A. Lazy invalidate** ★ 推荐 | session reset 时只 mark contentGenerator dirty，下次 LLM 调用时 lazy recreate | 小：~10 行加在 `resetSession` + LLM 调用入口；同步 API，无侵入 | -| B. Eager recreate | session reset 时立即 `await createContentGenerator(...)`，需 async 化 `resetSession` | 中：API 改动级联多处 | -| C. Proxy headers object | 给 `httpOptions.headers` 包 Proxy 拦截 getter | 风险高：`@google/genai` 内部是否 per-request 重读 headers 不可知，行为可能 silently break | -| D. 推动 `@google/genai` 上游加 `fetch` option | 提 PR 给 google-deepmind/generative-ai-js | 长期；不可控 | - -**文档要在用户面前说明**：使用 Gemini provider 时如果 `/clear` 后立刻有 LLM 调用，wire 上的 session id 在那一刻是旧的。可以靠 trace correlation 间接修正（spans/logs 上 session.id 已经是新的）。 - -应单开 follow-up sub-issue 跟踪选项 A。 - -## 9. 与 claude-code 对比 - -| 维度 | claude-code | qwen-code 本设计 | 决策依据 | -| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | -| Session id header 命名 | `X-Claude-Code-Session-Id`（产品前缀） | `X-Qwen-Code-Session-Id`（产品前缀） | ✅ 同样命名空间策略 | -| Session id 注入机制 | SDK `defaultHeaders`（`client.ts:108`）+ 自定义 `buildFetch()` wrapper（`client.ts:370-390`，per-request `randomUUID()` 注入 `x-client-request-id`） | OpenAI/Anthropic 走 fetch wrapper（per-request 读 session id，避免 `/clear` staleness）；Gemini 走 static `httpOptions.headers`（SDK 限制） | 与 claude-code 的 fetch wrapper 模式对齐。claude-code 也用 fetch wrapper 才能 per-request 加 `x-client-request-id` | -| Session id 持久性 | claude-code 没有 `/clear`-式 session reset；session = process | 有 `/clear` reset → fetch wrapper 路径自动跟随；static headers 路径会 stale（§8.6） | qwen-code 独有的复杂度 | -| Session id 编码 | HTTP header（不是 baggage） | HTTP header | ✅ 同——backend 友好 | -| `traceparent` 注入 | 闭源；公开 docs 描述存在；开源 repo 无 `propagation.inject` / `UndiciInstrumentation` 引用 | `@opentelemetry/instrumentation-undici` 自动 | claude-code 怎么实现的不可见。我们选 OTel 官方推荐路径，更轻 | -| `traceparent` 发送范围 | 仅第一方 Anthropic API；不发 Bedrock/Vertex/Foundry | 发给所有出站 fetch (W3C 标准；trace id 是 `sha256(sessionId)` 哈希)。**R3 修订**：session id header 仅向 first-party (Alibaba/DashScope) 白名单注入，第三方默认不发。详见 §11 | R3 后 qwen-code 的 session header 与 claude-code 同样的 first-party-only 语义；`traceparent` 仍待 per-destination toggle follow-up | -| `x-client-request-id` (随机) | 有，自动 | 暂不做（独立 follow-up sub-issue 价值更高） | 范围控制 | -| 子进程 `TRACEPARENT` env | 文档承认存在（实现闭源） | 不做（独立 follow-up） | 范围控制 | -| 入站 `TRACEPARENT` 读取 | 文档承认存在（`-p` / Agent SDK 模式） | 不做（独立 follow-up） | 范围控制 | - -**verified vs documented 注解**： - -| claim | 验证状态 | -| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | -| `X-Claude-Code-Session-Id` via `defaultHeaders` | ✅ Open source `src/services/api/client.ts:108` 已读 | -| `x-client-request-id` via fetch wrapper | ✅ Open source `src/services/api/client.ts:370-390` 已读 | -| `traceparent` 注入 | ⚠️ 仅 docs.claude.com/docs/en/monitoring-usage.md 提到；开源 repo `grep -rn "propagation\.inject\|UndiciInstrumentation\|traceparent" src` 返回空 | - -## 10. 未来工作 - -挂在 #3731 P3 下，本设计**不**包含但与之相关： - -- **`X-Qwen-Code-Request-Id`** 随机 UUID per request（claude-code 等价：`x-client-request-id`）。对超时/timeout error correlation 有用——超时时服务端可能还没 assign request id，客户端先发的 id 是唯一关联手段。R3 修订后这个建议变得更有意义：per-request UUID 没有"跨请求行为画像"风险，可以作为"对所有 LLM provider 发送的支持/调试 header"。 -- **`traceparent` 的 per-destination scope toggle** — R3 修订仅处理了 session id header 的作用域；`traceparent` 仍向所有出站 fetch 注入。可以加 `telemetry.propagateTraceContext: 'trusted-hosts' | 'all' | 'none'`，使用与 §11 同一份 allowlist 决定行为。 -- **Gemini 的 session id staleness lazy-invalidate fix**（§8.6 选项 A）：`/clear` 时 mark contentGenerator dirty，下次 LLM 调用 lazy recreate。让 Gemini 路径也享受 fetch wrapper 的实时性。 -- **子进程 `TRACEPARENT` env**：给 `BashTool` 执行子进程时注入 env，让外部工具能续传 trace。需要单独看 tool execution lifecycle。 -- **入站 `TRACEPARENT`**：`--prompt` 模式启动时读 env，让 CI / 外部 orchestrator 能把 qwen-code 接到更大的 trace。 -- **可配置 `correlationHeader` name**：让企业 ops 自定义 header（默认 `X-Qwen-Code-Session-Id`）。 -- **`baggage` propagation 策略**：是否主动 set baggage 让 `user.id` / `tenant.id` 等也走 baggage 传到下游。本期不做，等需求明确。 - -## 11. R3 修订 — Host-Allowlist Scoping for `X-Qwen-Code-Session-Id` - -> 触发：[LaZzyMan 在 PR #4390 的 REQUEST_CHANGES review](https://github.com/QwenLM/qwen-code/pull/4390) -> 落地 commit：`1c8528a56` (核心实现) + `cb162e716` (Vertex baseUrl fail-closed + `["*"]` trim 容错) - -### 11.1 触发与论证 - -R1 设计把 `X-Qwen-Code-Session-Id` 向**所有**出站 LLM 请求注入，仅由 `telemetry.enabled` 控制。LaZzyMan review 指出了三个递进的问题： - -1. **标签错位**：`feat(telemetry):` + `telemetry/` 路径 + `getTelemetryEnabled()` gate 让用户合理理解为"自家可观测性数据流向自家 collector"。但 `X-Qwen-Code-Session-Id` 不会到达 OTLP 后端，它走在 LLM API 请求里发给 DashScope / OpenAI / Anthropic / Gemini / OpenRouter / MiniMax / ModelScope / Mistral。两种不同的数据出口决策绑在一个开关上。 - -2. **claude-code 类比不成立**：R1 在 §9 把命名空间策略和 fetch wrapper 模式都"对齐"了 claude-code。但 claude-code 是 Anthropic 一方 → Anthropic 一方（single vendor, single direction），qwen-code 是开源 CLI → 多个第三方 provider。"一个稳定 cross-request UUID 广播到所有第三方"是 R1 没正面回答的问题。 - -3. **traceparent 是同一指纹的另一通道**：trace id = `sha256(sessionId).slice(0, 32)`，对接收方来说仍是稳定 per-session 标识符（哈希后不可逆，但同一 session 仍稳定）。 - -LaZzyMan 标定 severity：session id `high` / traceparent `medium`。 - -### 11.2 解法概要 - -**收窄默认作用域到 first-party hosts**。新增一项 setting： - -```jsonc -"telemetry": { - "sessionIdHeaderHosts": ["*"] // 恢复 R1 广播行为 - "sessionIdHeaderHosts": [] // 全关 header - "sessionIdHeaderHosts": ["api.mycompany.com", - "*.gateway.mycompany.internal"] -} -``` - -默认值（来自 `packages/core/src/telemetry/trusted-llm-hosts.ts:DEFAULT_SESSION_ID_HEADER_HOSTS`）： - -``` -dashscope.aliyuncs.com -dashscope-intl.aliyuncs.com -*.dashscope.aliyuncs.com -*.dashscope-intl.aliyuncs.com -*.alibaba-inc.com -*.aliyun-inc.com -``` - -这个集合的语义是"LLM provider、ARMS Tracing 后端、qwen-code distribution 同一法律实体"——也就是 claude-code 那个 single-vendor / single-direction 关系在 qwen-code 的对应集合。第三方 provider（OpenAI / Anthropic / OpenRouter / 等）默认**不**接收 header。 - -### 11.3 Pattern 语法（intentionally tiny） - -`matchesTrustedHost(hostname, patterns)` 只支持两种模式，与 `DashScopeOpenAICompatibleProvider.isDashScopeProvider` 对齐： - -- bare hostname → 精确匹配（case-insensitive） -- `*.suffix` → 匹配 `suffix` 自身 **AND** 任何子域；dot-anchored 拒绝 `evil-alibaba-inc.com` / `alibaba-inc.com.attacker.tld` 等 typo-suffix 攻击向量 - -不引入 regex、不引入端口/scheme 感知 globbing —— 让 settings 里的字符串就是它字面看起来的语义。 - -### 11.4 实现差异 vs R1 - -#### `wrapFetchWithCorrelation` (OpenAI / Anthropic) - -R1 的 wrapper 只有 telemetry-enabled + sessionId 两个 gate。R3 在两者之间插入第三个 gate： - -```ts -const trustedHosts = - config.getTelemetrySessionIdHeaderHosts?.() ?? - DEFAULT_SESSION_ID_HEADER_HOSTS; -const broadcastAll = trustedHosts.some((p) => p.trim() === '*'); - -return async function correlationFetch(input, init) { - if (!config.getTelemetryEnabled()) return baseFetch(input, init); - if (!broadcastAll) { - const host = extractRequestHost(input); - if (!host || !matchesTrustedHost(host, trustedHosts)) { - return baseFetch(input, init); // host gate - } - } - const sid = config.getSessionId(); - if (!sid) return baseFetch(input, init); - // ... header injection -}; -``` - -`trustedHosts` 在 wrap 时一次性 snapshot（与 session id 的"每请求实时读"不同）。中途修改 `telemetry.sessionIdHeaderHosts` 需要重建 contentGenerator 才生效。`[" * "]` 之类带空格的写法通过 `.trim()` 兜底成 broadcast，避免 settings.json 手敲笔误沉默退化。 - -#### `staticCorrelationHeaders` (Gemini) - -签名加一个 `destinationUrl?: string` 参数： - -```ts -export function staticCorrelationHeaders( - config: Config, - destinationUrl?: string, -): Record { - if (!config.getTelemetryEnabled()) return {}; - if (!destinationUrl) return {}; // fail-closed: 不知道目的地就不发 - if (!matchesTrustedHost(new URL(destinationUrl).hostname, trustedHosts)) { - return {}; - } - return { [SESSION_ID_HEADER]: config.getSessionId() }; -} -``` - -#### Gemini factory 集成 - -Gemini SDK 有两个不可见 default endpoint（`generativelanguage.googleapis.com` 与 `{region}-aiplatform.googleapis.com`，由 `vertexai` 决定），factory 层无法准确还原其中之一。R3 选择"`config.baseUrl` 没设就传 `undefined`"，让 helper fail-closed → 不发 header。运营商想要相关性必须显式设 `baseUrl`（也是 SDK 自己用来解 destination 的同一输入）。这一改动避免了猜错 Vertex destination 后被允许列表错误命中。 - -### 11.5 新文件 / 新代码 - -| 文件 | 说明 | -| -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | -| `packages/core/src/telemetry/trusted-llm-hosts.ts` (NEW) | `DEFAULT_SESSION_ID_HEADER_HOSTS` + `matchesTrustedHost` + `extractRequestHost` | -| `packages/core/src/telemetry/trusted-llm-hosts.test.ts` (NEW) | 单测，含 TLD-suffix 攻击向量、IPv6 fail-closed、port/userinfo/query 提取 | -| `packages/core/src/telemetry/llm-correlation-fetch.ts` | 加 host gate；`staticCorrelationHeaders` 加 `destinationUrl` 参数 | -| `packages/core/src/telemetry/llm-correlation-fetch.test.ts` | 加 host-gate 8 个 case；`mockConfig` 用 `'hosts' in opts` 区分 "default allowlist" vs "broadcast" | -| `packages/core/src/telemetry/config.ts` (`resolveTelemetrySettings`) | 透传 `sessionIdHeaderHosts` | -| `packages/core/src/config/config.ts` | `TelemetrySettings.sessionIdHeaderHosts` + `getTelemetrySessionIdHeaderHosts()` getter | -| `packages/core/src/core/geminiContentGenerator/index.ts` | 传 `config.baseUrl` 给 helper；fail-closed when undefined | -| `packages/core/src/core/geminiContentGenerator/index.test.ts` | 重写 telemetry-on Gemini 测试以匹配新 fail-closed 语义 | -| `packages/cli/src/config/settingsSchema.ts` | `sessionIdHeaderHosts` JSON schema 入口 | -| `packages/vscode-ide-companion/schemas/settings.schema.json` | 由 `npm run generate:settings-schema` 重新生成 | -| `docs/developers/development/telemetry.md` | "Session correlation header" 段落改写 + 默认 scope + override 语法 | - -### 11.6 对各 LazzyMan 论点的回应 - -| LazzyMan 论点 | R3 回应 | -| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| ① telemetry 标签错位 | **化解**：在 DashScope 用例下，session id header 字面就是发给 ARMS Tracing 后端（同一法律实体），`telemetry.enabled` 语义对齐 | -| ② cross-vendor stable identifier 广播 | **化解**：默认 allowlist 只含阿里系 first-party host；广播退化为 opt-in (`["*"]`) | -| ③ traceparent 是同一指纹的另一通道 | **暂保留**：traceparent 仍按 R1 全注入。理由：W3C 标准、trace id 是 sha256 哈希、in-vendor trace 续接是 W3C 的核心设计场景。per-destination traceparent toggle 列入 §10 future work | - -### 11.7 已知遗留 + 跟进 - -- **traceparent scope** — 见上文第 ③ 点，列入 §10 -- **Per-request random UUID** (`X-Qwen-Code-Request-Id`) — LazzyMan 提的替代方案，列入 §10 -- **Gemini staleness lazy-invalidate** (§8.6 选项 A) — 与 R3 解耦，独立 sub-issue -- **`matchesTrustedHost` IPv6 支持** — 当前 IPv6 destination 永不在 allowlist 上（`URL.hostname` 返回 `[::1]` 带方括号，pattern 语法无对应形式）。当前满足"命名 first-party endpoint"用例。若将来有 raw IP allowlist 需求再扩展。 - -## 12. R4 修订 — Scope Conflation Split - -> 触发：[LaZzyMan round-8 follow-up review on PR #4390](https://github.com/QwenLM/qwen-code/pull/4390) -> 落地：本 PR 收窄；R3 落地的 session-id 整套挪到独立 follow-up PR - -### 12.1 触发与论证 - -R3 化解了 LaZzyMan 第一轮 review 的「广播稳定指纹给第三方 provider」担忧（severity: high）。但在 round-8 follow-up 中他升级到更深的架构原则反对： - -> "Telemetry is not a container for adjacent features. The `traceparent` cross-process propagation and the `X-Qwen-Code-Session-Id` header injection are **not telemetry**. They are outbound-identity / outbound-correlation work that uses some OTel APIs internally as an implementation detail." - -他的核心元论点： - -- **"telemetry" namespace 暗示 recipient = 用户自己的 OTLP collector** -- 但 `traceparent` 和 `X-Qwen-Code-Session-Id` 的 recipient = **第三方 LLM provider** -- 两类不同 recipient 应该有两类不同的同意决策树 -- 即使默认行为安全（R3 已实现），把 wire-level 行为放在 `telemetry.*` 下**设了坏先例**：未来 telemetry PR 可以继续偷渡 wire 行为给第三方 -- "If we accept that principle, the split is mechanical. If we don't, this PR is the wrong place to debate it because the technical fixes are already in." - -### 12.2 解法概要（"方案 C" hybrid split） - -经过几轮内部讨论（含 yiliang 提出的 customHeader 模板替代方案，最终判定 customHeader 不能携带 runtime-dynamic 值），决定走 **方案 C**： - -**本 PR 留下**： - -- `UndiciInstrumentation` 注册（产 client HTTP span → 用户自家 OTLP collector） -- OTLP feedback-loop guard（前者的必要副作用） -- **`NoopTextMapPropagator` 默认安装** → `propagation.inject()` 是 no-op → outbound `fetch` 上**不再有 `traceparent`** -- **新增 `outboundCorrelation.propagateTraceContext: bool` (默认 false)** 作为独立 namespace 顶级设置；设 true 时安装默认 W3C composite propagator -- 整套 `R3 session-id` 代码（`llm-correlation-fetch.ts` / `trusted-llm-hosts.ts` / `telemetry.sessionIdHeaderHosts` setting / 4 个 provider 集成点 / 所有相关测试）**全部移除** - -**搬到 follow-up PR**： - -- `X-Qwen-Code-Session-Id` header 整套机器（R3 实现复用） -- 进入新 `outboundCorrelation.*` namespace（具体 setting key TBD，但**不会**叫 `telemetry.*`） -- Follow-up PR 自带：threat model section、独立 review、security-relevant 标注的 docs -- `X-Qwen-Code-Request-Id` per-request UUID（LazzyMan 在 R3 round 提出的替代设计）也归入此 follow-up 的考虑范围 - -### 12.3 与 R3 R1 论点的映射 - -| R1/R3 论点 | R4 后状态 | -| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | -| §3.1 "所有出站 LLM 请求带 traceparent" | ❌ **R4 默认 off**；需 `outboundCorrelation.propagateTraceContext: true` 才开 | -| §3.1 "所有出站 LLM 请求带 `X-Qwen-Code-Session-Id`" | ❌ **R4 整套移出本 PR**，搬到 follow-up PR | -| §4.3 fetch wrapper 注入 session id | ❌ 整段代码不在本 PR；复用到 follow-up PR | -| §11 host allowlist (R3 设计) | ❌ 同上；整体迁移 follow-up PR | -| §4.4 不引入新 setting | ❌ **本 PR 新增 `outboundCorrelation.propagateTraceContext`** 一个 boolean；session id 相关 setting 在 follow-up PR | -| §10 future work "`X-Qwen-Code-Request-Id`" | ✅ 仍是 future work；与 session-id follow-up 一起设计 | - -### 12.4 新 namespace 设计意图 - -`outboundCorrelation.*` 顶级 namespace 在本 PR 只有一个 boolean (`propagateTraceContext`)，看起来过度结构化。但这是**精心选择的**： - -- **建立命名空间作为承诺**：让后续 session-id / request-id / etc. 自然进入这个 namespace -- **标注为 security-relevant**：`settingsSchema.ts` description 显式写 "SECURITY-RELEVANT"，文档化为"安全设置"而非"observability 设置" -- **defaults 全部 off**：符合 LazzyMan 提出的"open-source 客户端不应未经显式同意向第三方发稳定 id"原则 -- **与 telemetry.\* 解耦**：用户读 settings.json 看到 `outboundCorrelation.*` 立刻能识别这是出站 wire 行为，不是 observability - -#### 隐性依赖：`telemetry.enabled` - -虽然 namespace 与 `telemetry.*` 解耦，**运行时生效仍依赖 `telemetry.enabled: true`** —— OTel SDK 只在 telemetry 启用时初始化，没有 SDK 就没有 propagator 安装、没有 `propagation.inject()` 调用，flag 等于沉默 no-op。容易踩的 footgun：运营商加 `propagateTraceContext: true` 却忘开 telemetry，trap server 上看不到任何 `traceparent`，无 error / 无 warning。 - -两个面向用户的面板都显式标注此依赖： - -- `telemetry.md` 的 `propagateTraceContext` 段附完整双 flag JSON 示例 -- `settingsSchema.ts` 的 description string **首句**即 "Requires `telemetry.enabled: true`"（前置以避免 VS Code 设置 UI 长描述折叠后看不到） - -未来若添加 session-id header 或其他 `outboundCorrelation.*` setting，**同一依赖关系适用** —— 都得在 telemetry 启用前提下才有意义（因为它们都通过 OTel instrumentation/SDK 注入）。Follow-up PR 应继承此 footgun 提示模式。 - -### 12.5 实施 - -| 文件 | 改动 | -| ------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `packages/core/src/telemetry/llm-correlation-fetch.ts` | **删除** | -| `packages/core/src/telemetry/llm-correlation-fetch.test.ts` | **删除** | -| `packages/core/src/telemetry/trusted-llm-hosts.ts` | **删除** | -| `packages/core/src/telemetry/trusted-llm-hosts.test.ts` | **删除** | -| `packages/core/src/telemetry/sdk.ts` | + `NoopTextMapPropagator`；按 `getOutboundCorrelationPropagateTraceContext()` 决定 SDK textMapPropagator | -| `packages/core/src/core/openaiContentGenerator/provider/default.ts` | 移除 `wrapFetchWithCorrelation` 引用 | -| `packages/core/src/core/openaiContentGenerator/provider/dashscope.ts` | 同上 | -| `packages/core/src/core/anthropicContentGenerator/anthropicContentGenerator.ts` | 同上 | -| `packages/core/src/core/geminiContentGenerator/index.ts` | 移除 `staticCorrelationHeaders` 引用 | -| 上述 4 个 provider 的 `*.test.ts` | 删 session-id 相关测试 case | -| `packages/core/src/config/config.ts` | 删 `TelemetrySettings.sessionIdHeaderHosts`、`getTelemetrySessionIdHeaderHosts`；**新增 `OutboundCorrelationSettings` 接口 + `outboundCorrelationSettings` 字段 + `getOutboundCorrelationPropagateTraceContext()` getter** | -| `packages/core/src/telemetry/config.ts` | 删 `resolveTelemetrySettings` 中 sessionIdHeaderHosts 透传 | -| `packages/cli/src/config/settingsSchema.ts` | 删 `sessionIdHeaderHosts` schema；**新增 `outboundCorrelation` 顶级 schema 项** | -| `packages/cli/src/config/config.ts` | 透传 `outboundCorrelation: settings.outboundCorrelation` 进 `ConfigParameters` | -| `packages/vscode-ide-companion/schemas/settings.schema.json` | `npm run generate:settings-schema` 重新生成（description 后续更新时同步刷新） | -| `docs/developers/development/telemetry.md` | 重写 "Trace context propagation" → "Client-side HTTP span on outbound fetch"；删 "Session correlation header" 整节；新增 "Outbound correlation (SECURITY-RELEVANT)" 顶级 section；附 `telemetry.enabled` 依赖说明 + JSON 配置示例 | -| `docs/design/telemetry-outbound-propagation-design.md` | 本节 + R4 表头 + 修订指针 | -| `packages/core/src/config/config.test.ts` | **新增 `OutboundCorrelation Configuration` describe block**，`it.each` 4 个 case 锁定 `getOutboundCorrelationPropagateTraceContext` 的 default-false 安全不变性（omitted / `{}` / explicit true / explicit false） | - -### 12.6 对 LazzyMan 元论点的回应 - -| 论点 | R4 后状态 | -| ----------------------------------------------- | ----------------------------------------------------------------------------------------------------- | -| "Telemetry namespace 暗示自家 collector 接收方" | ✅ wire 行为已搬出 `telemetry.*`；新 `outboundCorrelation.*` namespace 显式标识"出站第三方"语义 | -| "默认行为不应未经显式同意向第三方发标识符" | ✅ `propagateTraceContext` 默认 false；session-id 整套 follow-up PR 也将默认 off | -| "telemetry PR 不应偷渡 wire-level 行为" | ✅ 本 PR 不再添加任何"telemetry 控制 wire 行为"的代码路径；wire 行为统一由 `outboundCorrelation.*` 管 | -| "split is mechanical, work isn't wasted" | ✅ R3 落地代码物理删除自本 branch，留在 git history 里给 follow-up PR 复用（或 cherry-pick） | - -### 12.7 follow-up PR 大纲（信息性，不在本 PR 范围） - -未来 follow-up PR 应包含： - -- `outboundCorrelation.sessionIdHeader: { enabled, trustedHosts }` 或类似 setting -- 复用 R3 已实现的 `wrapFetchWithCorrelation` / `matchesTrustedHost` / `DEFAULT_SESSION_ID_HEADER_HOSTS` 代码骨架 -- threat model 一节，明确：recipient 集合、稳定 id 的去匿名化窗口、可选 per-request UUID 配套 -- **默认 off**（无 default allowlist —— 比 R3 更严，符合 LazzyMan 的开源 CLI 原则） -- security-relevant 标注 + docs/users/configuration/settings.md 收录 diff --git a/docs/design/telemetry-resource-attributes-design.md b/docs/design/telemetry-resource-attributes-design.md deleted file mode 100644 index 01ec84efa5b..00000000000 --- a/docs/design/telemetry-resource-attributes-design.md +++ /dev/null @@ -1,762 +0,0 @@ -# Telemetry: Custom Resource Attributes + Metric Cardinality Controls - -> 配套 issue: [#4365](https://github.com/QwenLM/qwen-code/issues/4365) -> 父 issue: [#3731](https://github.com/QwenLM/qwen-code/issues/3731) -> 基于 2026-05-21 对 qwen-code main 分支的代码复核 - -## 1. 背景 - -qwen-code 已经接入 OpenTelemetry SDK，但 Resource 构造方式让它在两个常见生产场景下不可用： - -1. **无法附加自定义维度**：运维侧想给所有 telemetry 数据打 `team` / `env` / `cost_center` / `user_id` 标签，今天没有任何机制可以做到。即使设置标准的 `OTEL_RESOURCE_ATTRIBUTES` 环境变量也**完全不生效**。 -2. **指标基数（cardinality）失控**：`session.id` 被注入到了 Resource 层，会自动附着到每条 metric 数据点。每个 CLI session 产生一个新值，指标后端（Prometheus / 阿里云 ARMS Metric / VictoriaMetrics）会被无界 time-series 撑爆。 - -这两个问题耦合在一起：解决前者会让用户**更容易**给数据加高基数的字段，所以必须配套提供后者。 - -## 2. 现状 - -### 2.1 Resource 构造 - -`packages/core/src/telemetry/sdk.ts:156-161`： - -```ts -const resource = resourceFromAttributes({ - [SemanticResourceAttributes.SERVICE_NAME]: SERVICE_NAME, - [SemanticResourceAttributes.SERVICE_VERSION]: - config.getCliVersion() || 'unknown', - 'session.id': config.getSessionId(), -}); -``` - -`sdk.ts:274-278`： - -```ts -sdk = new NodeSDK({ - resource, - // Disable async host/process/env resource detectors: they leave attributes - // pending and trigger an OTel diag.error on any resource attribute read - // before the detectors settle (e.g. during HttpInstrumentation span creation). - autoDetectResources: false, - ... -}); -``` - -`autoDetectResources: false` 关闭了标准 OTel 的 `envDetector`——也就是平时会读取 `OTEL_RESOURCE_ATTRIBUTES` 和 `OTEL_SERVICE_NAME` 的那一层。这是有原因的（detector 异步，会在 settle 前触发 `diag.error`），但副作用是这两个标准环境变量在 qwen-code 里**完全无效**。 - -### 2.2 `session.id` 实际是三重注入 - -| 位置 | 行号 | 影响 | -| --------------------------- | ------------------------ | ------------------------------------- | -| Resource | `sdk.ts:160` | 所有 signal（spans / logs / metrics） | -| Per-span | `session-tracing.ts:169` | spans | -| Per-log | `loggers.ts:128` | logs | -| **`getCommonAttributes()`** | `metrics.ts:57` | **每条 metric record 显式叠加** | - -也就是说**单独把 `session.id` 从 Resource 拿掉是不够的**——`metrics.ts:57` 的 `baseMetricDefinition.getCommonAttributes()` 会被 30+ 个 metric 调用点 `...spread` 进去，再次塞回 `session.id`。 - -```ts -// metrics.ts:55-59 -const baseMetricDefinition = { - getCommonAttributes: (config: Config): Attributes => ({ - 'session.id': config.getSessionId(), - }), -}; -``` - -好消息：所有 metric 调用点（30+ 个）都走这一个函数，是天然的 chokepoint。 - -### 2.3 config resolver 模式 - -`packages/core/src/telemetry/config.ts:resolveTelemetrySettings()` 用统一的优先级链： - -``` -argv (highest) > QWEN_* env > OTEL_* env > settings.json (lowest) -``` - -新加项照搬这个 pattern。 - -### 2.4 settings schema 现状 - -`packages/cli/src/config/settingsSchema.ts:998-1018` 定义 `telemetry` 的 JSON schema： - -```ts -telemetry: { - type: 'object', - // ... - jsonSchemaOverride: { - type: 'object', - properties: { - includeSensitiveSpanAttributes: { ... }, - }, - additionalProperties: true, // ← 今天对其他 telemetry.* key 不校验 - }, -} -``` - -`additionalProperties: true` 意味着今天 schema 对 `otlpEndpoint` / `otlpProtocol` / `resourceAttributes` 等其他字段全部放行不校验。新加 `resourceAttributes` / `metrics` 字段时，应同步在这里补 schema，方便 IDE 自动补全和 settings UI 渲染。 - -### 2.5 不在本设计范围的代码路径 - -`packages/core/src/telemetry/qwen-logger/qwen-logger.ts` 是 qwen-code 的**第一方使用上报通道**（基于阿里 RUM 内部协议 `RumResourceEvent`），与 OTel SDK 完全独立。它有自己的 endpoint、proxy 和数据模型，**不受本设计影响**。详见第 3 节。 - -### 2.6 已支持 / 未支持的 `OTEL_*` 环境变量 - -| 环境变量 | 现状 | -| --------------------------------------------------- | --------------------------------- | -| `OTEL_EXPORTER_OTLP_ENDPOINT` | ✅ 支持（`config.ts:79`） | -| `OTEL_EXPORTER_OTLP_{TRACES,LOGS,METRICS}_ENDPOINT` | ✅ 支持 | -| `OTEL_EXPORTER_OTLP_HEADERS` | ✅ 底层 exporter 直接读取 | -| `OTEL_TRACES_SAMPLER` | ✅ 支持（`tracer.ts:247`） | -| **`OTEL_RESOURCE_ATTRIBUTES`** | ❌ 完全不支持 | -| **`OTEL_SERVICE_NAME`** | ❌ 完全不支持 | -| **`OTEL_METRICS_INCLUDE_*`** | ❌ 完全不支持（claude-code 风格） | - -## 3. 目标 / 非目标 - -### 3.1 目标 - -- 让运维通过标准 `OTEL_RESOURCE_ATTRIBUTES` 和自家 `settings.json` 给所有 OTLP 导出的 span / log / metric 附加自定义 resource attributes -- 让 `OTEL_SERVICE_NAME` 按 OTel 规范工作（包括与 `OTEL_RESOURCE_ATTRIBUTES` 里的 `service.name` 的优先级） -- 默认情况下，metric 上**不**携带 `session.id`（保护后端基数） -- 提供显式开关让需要 metric-level session correlation 的用户重新打开 -- 保留 spans 和 logs 上的 `session.id`（trace correlation 必须） -- 保留 `autoDetectResources: false`，不退化 `diag.error` 那个已修的 bug -- 配套更新 `settingsSchema.ts` 让新字段对 settings UI 和 IDE 可见 - -### 3.2 非目标 - -- **`qwen-logger` 第一方上报**：完全独立的 RUM 通道，不在本设计范围。其上报字段（device id、user agent 等）由 RUM 协议决定，不应被用户 resource attribute 干扰。若未来要给 `qwen-logger` 增加自定义维度，是另一条独立的设计。 -- **Per-span 动态 attribute hook**：让用户写代码 / hook 给每个 span 计算 attribute。claude-code 也没解决这块，复杂度高、收益低。 -- **`service.version` cardinality 控制**：版本变化频率有限（月级），time series 增长可控。需要时走 v2，引入 OTel View API。 -- **Agent SDK 形态的 per-query resource attrs**：qwen-code 目前没有 SDK 调用场景。 -- **OTLP 请求头（auth headers）配置**：是另一条 issue 线（#3731 P1），与本设计独立。 -- **CLI flag 形式的 resource attribute**：env var + settings.json 已覆盖临时与基线两种场景，CLI flag 会让命令行变得啰嗦，无明显增益。 - -## 4. 设计 - -### 4.1 总体分层 - -``` -┌─ Resource（sdk.ts:156）────────────────────────────────────────┐ -│ service.name ← OTEL_SERVICE_NAME │ -│ > OTEL_RESOURCE_ATTRIBUTES.service.name│ -│ > 'qwen-code' │ -│ service.version ← config.getCliVersion() [reserved] │ -│ ...user attrs ← OTEL_RESOURCE_ATTRIBUTES │ -│ + settings.resourceAttributes │ -│ ✗ session.id 移走 │ -└────────────────────────────────────────────────────────────────┘ - │ - ├──→ Spans ＋ session.id（session-tracing.ts:169，保留） - ├──→ Logs ＋ session.id（loggers.ts:128，保留） - └──→ Metrics ＋ getCommonAttributes() — 默认 {} - toggle ON: { session.id } -``` - -### 4.2 优先级 / merge 顺序 - -#### 一般 attribute - -低 → 高： - -1. `OTEL_RESOURCE_ATTRIBUTES`（标准 OTel env var） -2. `settings.telemetry.resourceAttributes` -3. 内建保留键（覆盖以上任何同名） - -**理由**：环境变量是 ops-time 临时覆盖（CI / 单机 debug），settings.json 是 fleet-baked 基线，内建是产品契约——基线优先级应高于临时变量，内建优先级应高于一切。 - -#### `service.name` 特殊处理 - -`service.name` 必须遵守 [OTel 规范](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/)： - -> **`OTEL_SERVICE_NAME` takes precedence over `service.name` defined with the `OTEL_RESOURCE_ATTRIBUTES` variable.** - -因此对 `service.name` 单独应用这条优先级链（高 → 低）： - -1. `OTEL_SERVICE_NAME`（最高，标准 OTel 规范规定） -2. `settings.resourceAttributes.service.name`（settings 优先于 env，沿用本设计一般规则） -3. `OTEL_RESOURCE_ATTRIBUTES.service.name` -4. 内建默认 `'qwen-code'` - -`service.name` 允许通过 settings 覆盖——它是 service 身份，企业 fleet 用统一 settings.json 配置 service.name 是常见且合理的做法，禁止反而会阻断 GitOps 分发场景。`OTEL_SERVICE_NAME` 作为标准 OTel 规范规定的"最高优先级"通道，仍然可以在 CI / 单机调试时临时覆盖 settings。 - -具体规则： - -| 来源 | 写入 `service.name` 是否生效 | -| ------------------------------------------------------- | -------------------------------------- | -| `OTEL_SERVICE_NAME=foo` | ✅ 最高优先级（覆盖任何其他来源） | -| `settings.resourceAttributes={ "service.name": "foo" }` | ✅ 仅在没有 `OTEL_SERVICE_NAME` 时生效 | -| `OTEL_RESOURCE_ATTRIBUTES=service.name=foo` | ✅ 仅在以上两者都没有时生效 | - -### 4.3 保留键策略 - -| 键 | 用户能否覆盖 | 理由 | -| ----------------- | ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | -| `service.name` | ✅ env var + settings 都可（见 §4.2 优先级链） | service 身份，应允许 ops 控制 | -| `service.version` | ❌ 任何来源都丢弃 + warn | 遥测可信度——不允许用户谎报版本 | -| `session.id` | ❌ 任何来源都丢弃 + warn（在 metric 上额外有 toggle 控制 runtime 注入） | runtime-only；用户写到 Resource 会绕过 metric cardinality toggle（Resource attr 自动附到所有 signal） | -| `qwen.*` 前缀 | ⚠️ 不强制保留，但 docs 建议留给产品自用 | 避免未来内建 attr 与用户 attr 冲突 | - -**保留键以常量集中维护**： - -```ts -// telemetry/resource-attributes.ts (new file) -/** Keys that cannot be overridden from any source (env or settings). */ -export const RESERVED_RESOURCE_ATTRIBUTE_KEYS = new Set([ - 'service.version', - 'session.id', -]); -``` - -`service.name` **不**在 RESERVED 列表里——它走自己的优先级链（§4.2），不属于"全局禁止覆盖"语义。RESERVED 是"任何来源写了都警告并丢弃"，统一适用于 env 和 settings 两个入口。 - -### 4.4 `OTEL_RESOURCE_ATTRIBUTES` 解析 - -同步实现，绕开 OTel 自带的异步 envDetector： - -```ts -function parseOtelResourceAttributes( - raw: string | undefined, -): Record { - if (!raw) return {}; - const out: Record = {}; - for (const pair of raw.split(',')) { - const trimmed = pair.trim(); - if (!trimmed) continue; - const idx = trimmed.indexOf('='); - if (idx <= 0) { - diag.warn( - `Skipping malformed OTEL_RESOURCE_ATTRIBUTES entry: ${trimmed}`, - ); - continue; - } - const key = trimmed.slice(0, idx).trim(); - const valueRaw = trimmed.slice(idx + 1).trim(); - if (!key) continue; - let value: string; - try { - value = decodeURIComponent(valueRaw); - } catch { - diag.warn( - `Invalid percent-encoding in OTEL_RESOURCE_ATTRIBUTES for key "${key}", using raw value`, - ); - value = valueRaw; - } - out[key] = value; // duplicate keys: last wins (matches OTel reference impls) - } - return out; -} -``` - -格式严格按 OTel 规范：`key1=val1,key2=val2`，值 percent-encoded。 - -### 4.5 Metric attribute filter - -唯一改动点 `metrics.ts:55-59`： - -```ts -const baseMetricDefinition = { - getCommonAttributes: (config: Config): Attributes => { - const out: Attributes = {}; - if (config.getTelemetryMetricsIncludeSessionId()) { - out['session.id'] = config.getSessionId(); - } - return out; - }, -}; -``` - -调用点（30+ 个）零改动——`...spread` 一个空对象等价于不展开任何字段。 - -### 4.6 边界情况与校验 - -| 输入 | 行为 | -| ---------------------------------------------------------------- | ----------------------------------------------------------------------- | -| `OTEL_RESOURCE_ATTRIBUTES=""` (空字符串) | 返回 `{}`，正常启动 | -| `OTEL_RESOURCE_ATTRIBUTES="a"` (无 `=`) | 跳过该项 + `diag.warn`，继续解析其余 | -| `OTEL_RESOURCE_ATTRIBUTES="=val"` (空 key) | 跳过该项，继续解析其余 | -| `OTEL_RESOURCE_ATTRIBUTES="a=,b=2"` (空 value) | `a=''`, `b='2'`（OTel 规范允许空 value） | -| `OTEL_RESOURCE_ATTRIBUTES="a=val%ZZbad"` (无效 percent-encoding) | 保留原始 `val%ZZbad` + `diag.warn` | -| `OTEL_RESOURCE_ATTRIBUTES="a=1,a=2"` (duplicate key) | 后写胜出 `a=2`（与 OTel SDK 参考实现一致） | -| `OTEL_RESOURCE_ATTRIBUTES="a=1, b=2 "` (含空格) | 自动 trim | -| `OTEL_RESOURCE_ATTRIBUTES=service.version=x` | 静默丢弃 `service.version` + `diag.warn`，保留其他键 | -| `settings.resourceAttributes={ "service.name": "x" }` | 接受（settings 可设 service.name，见 §4.2） | -| `settings.resourceAttributes={ "service.version": "x" }` | 静默丢弃 + `diag.warn` | -| `settings.resourceAttributes={ "team": 123 }` (非 string) | TypeScript 类型阻挡；runtime 传入则 settings JSON schema validator 拒绝 | -| Resource 总大小 > OTel 限制 (4KB?) | 由底层 OTel SDK 处理，不在本层校验 | - -**为什么不在本层做 attribute key 命名校验**（如 OTel 推荐的 `[a-z][a-z0-9_.]*` 模式）：OTel SDK 自己会在 export 时校验，本层重复校验既慢又容易和 SDK 行为偏移。我们只做格式解析，不做语义校验。 - -**RESERVED 键的强制保护对两个入口都生效**： - -```ts -// 应用于 env-parsed attrs -for (const k of RESERVED_RESOURCE_ATTRIBUTE_KEYS) { - if (k in envAttrs) { - diag.warn(`OTEL_RESOURCE_ATTRIBUTES cannot override "${k}"; ignoring`); - delete envAttrs[k]; - } -} - -// 应用于 settings attrs -for (const k of RESERVED_RESOURCE_ATTRIBUTE_KEYS) { - if (k in settingsAttrs) { - diag.warn( - `settings.telemetry.resourceAttributes cannot override "${k}"; ignoring`, - ); - delete settingsAttrs[k]; - } -} -``` - -### 4.7 生命周期与多进程 - -- **SDK init 时机**：Resource 在 `initializeTelemetry()` 时一次性构造，**进程内不可变**。这与 OTel SDK 设计一致。 -- **Subagent fork**：qwen-code 的 subagent 是同进程内的 (`subagent-runtime.ts`)，共享 Resource。若未来引入跨进程 subagent，子进程会**重新 init SDK**，重新读 env var 和 settings——只要 env 透传过去，行为一致。 -- **Hot reload**：settings 修改后**不会重新构造 Resource**。需要操作员重启 CLI 才能生效。文档应明确说明。 -- **`refreshSessionContext()`** (`sdk.ts:306`)：仅刷新 session ALS context，**不重建 Resource**——因为 Resource 上已经没有 `session.id` 了（本设计的核心改动之一）。 - -## 5. Config schema 改动 - -### 5.1 `TelemetrySettings` 接口（`packages/core/src/config/config.ts:293`） - -```ts -export interface TelemetrySettings { - // ... existing fields - /** Static resource attributes attached to every span/log/metric. */ - resourceAttributes?: Record; - /** Per-signal cardinality controls. */ - metrics?: { - /** Include session.id on metric data points (default: false). */ - includeSessionId?: boolean; - }; -} -``` - -### 5.2 `Config` getter（同文件） - -```ts -class Config { - getTelemetryResourceAttributes(): Record { - return this.telemetrySettings.resourceAttributes ?? {}; - } - getTelemetryMetricsIncludeSessionId(): boolean { - return this.telemetrySettings.metrics?.includeSessionId ?? false; - } -} -``` - -### 5.3 `resolveTelemetrySettings()` 新增 - -```ts -const envResourceAttrs = parseOtelResourceAttributes( - env['OTEL_RESOURCE_ATTRIBUTES'], -); -const settingsResourceAttrs = { ...(settings.resourceAttributes ?? {}) }; - -// Strip RESERVED keys from both sources (warn if user tried to set them). -for (const k of RESERVED_RESOURCE_ATTRIBUTE_KEYS) { - if (k in envResourceAttrs) { - diag.warn(`OTEL_RESOURCE_ATTRIBUTES cannot override "${k}"; ignoring`); - delete envResourceAttrs[k]; - } - if (k in settingsResourceAttrs) { - diag.warn( - `settings.telemetry.resourceAttributes cannot override "${k}"; ignoring`, - ); - delete settingsResourceAttrs[k]; - } -} - -// Merge: env < settings (settings wins on conflict). -const merged: Record = { - ...envResourceAttrs, - ...settingsResourceAttrs, -}; - -// service.name precedence: OTEL_SERVICE_NAME (env-only escape) wins over -// everything else. settings already overwrote env in the spread above. -if (env['OTEL_SERVICE_NAME']) { - merged['service.name'] = env['OTEL_SERVICE_NAME']; -} - -const resourceAttributes = merged; - -const metricsIncludeSessionId = - parseBooleanEnvFlag(env['QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID']) ?? - settings.metrics?.includeSessionId ?? - false; - -return { - // ... existing fields - resourceAttributes, - metrics: { includeSessionId: metricsIncludeSessionId }, -}; -``` - -### 5.4 `sdk.ts` Resource 构造改动 - -```ts -const userAttrs = config.getTelemetryResourceAttributes(); -// service.version is always built-in; service.name flows through userAttrs -// (it was already resolved with OTEL_SERVICE_NAME precedence in resolver). -const builtinServiceName = userAttrs['service.name'] ?? SERVICE_NAME; -const { 'service.name': _, 'service.version': __, ...nonReserved } = userAttrs; - -const resource = resourceFromAttributes({ - ...nonReserved, - [SemanticResourceAttributes.SERVICE_NAME]: builtinServiceName, - [SemanticResourceAttributes.SERVICE_VERSION]: - config.getCliVersion() || 'unknown', - // session.id deliberately NOT placed on Resource — see design doc §4.1 -}); -``` - -### 5.5 `settingsSchema.ts` 改动 - -`packages/cli/src/config/settingsSchema.ts:998-1018` 的 `telemetry.jsonSchemaOverride.properties` 加： - -```ts -{ - // ... existing includeSensitiveSpanAttributes - resourceAttributes: { - type: 'object', - additionalProperties: { type: 'string' }, - description: - 'Static resource attributes attached to all telemetry data. ' + - 'Keys must be strings; values must be strings. ' + - 'Reserved keys (service.name, service.version) are silently dropped.', - default: {}, - }, - metrics: { - type: 'object', - additionalProperties: false, - properties: { - includeSessionId: { - type: 'boolean', - default: false, - description: - 'Include session.id on every metric data point. ' + - 'WARNING: each CLI session creates a new value, causing unbounded ' + - 'metric time-series fan-out. Only enable for short-term debugging.', - }, - }, - }, -} -``` - -也要把 `additionalProperties: true` 重新评估——目前是 permissive，可以保留也可以转 strict。建议保留 permissive，避免对其他未在 schema 中声明的 `telemetry.*` 字段产生破坏性变更，但 docs 里明确"未声明字段会被忽略"。 - -## 6. 文件改动清单 - -| 文件 | 改动 | -| -------------------------------------------------------------- | -------------------------------------------------------------------------- | -| `packages/core/src/telemetry/sdk.ts` | 改 Resource 构造（合并 user attrs，删 `session.id`） | -| `packages/core/src/telemetry/resource-attributes.ts` (新文件) | `parseOtelResourceAttributes()` + `RESERVED_RESOURCE_ATTRIBUTE_KEYS` 常量 | -| `packages/core/src/telemetry/config.ts` | resolver 加 `resourceAttributes` + `metrics.includeSessionId` 解析与 merge | -| `packages/core/src/telemetry/metrics.ts` | `getCommonAttributes()` 加 toggle gate | -| `packages/core/src/config/config.ts` | `TelemetrySettings` schema + 两个 getter | -| `packages/cli/src/config/settingsSchema.ts` | `jsonSchemaOverride` 加 `resourceAttributes` + `metrics` | -| `docs/developers/development/telemetry.md` | 加 "Resource attributes" + "Cardinality controls" 两节 + 迁移说明 + 示例 | -| `packages/core/src/telemetry/resource-attributes.test.ts` (新) | 解析器单元测试（覆盖 §4.6 全部用例） | -| `packages/core/src/telemetry/sdk.test.ts` | merge 优先级 / 保留键 / `OTEL_SERVICE_NAME` | -| `packages/core/src/telemetry/metrics.test.ts` | toggle off/on 时 `session.id` 出现与否 | -| `packages/core/src/telemetry/config.test.ts` | env / settings 合并 | -| `CHANGELOG.md` 或 release notes | PR 2 的 breaking change 说明 | - -## 7. 分 PR 拆分 - -按 review 友好性与 blast radius 分三个 PR： - -### PR 1 — Custom resource attributes（additive，零破坏） - -- 新文件 `resource-attributes.ts`：`parseOtelResourceAttributes()` + `RESERVED_RESOURCE_ATTRIBUTE_KEYS` -- `TelemetrySettings.resourceAttributes` 字段 + resolver merge 逻辑 -- `OTEL_SERVICE_NAME` / `OTEL_RESOURCE_ATTRIBUTES` 接入，按 §4.2 优先级 -- 合并进 Resource（`sdk.ts`） -- `settingsSchema.ts` 加 `resourceAttributes` JSON schema -- **不动** `session.id` 在 Resource 上的位置 -- Docs 加 "Resource attributes" 一节 - -**风险**：低。完全 additive，不改任何现有行为。除非用户主动设置环境变量或 settings，否则导出的数据无变化。 - -### PR 2 — Cardinality controls（semantic break） - -- 从 Resource 删 `session.id` (`sdk.ts:160` 那一行) -- 加 `metrics.includeSessionId` toggle（settings + env）+ `getCommonAttributes()` gate -- `settingsSchema.ts` 加 `metrics` JSON schema -- CHANGELOG / 迁移说明 -- 快照测试锁定 metric attribute 集合（防回归） -- Docs 加 "Cardinality controls" 一节 + 迁移指南 - -**风险**：中等。任何依赖 metric 上 `session.id` 的 Prometheus query / Grafana dashboard / 告警规则会失效。需要显式 release note 与 1-2 个版本的迁移窗口。 - -**Opt-in 过渡方案**（候选，本期建议**不采用**）： - -> PR 2 可先以"opt-out"形式落地——默认仍把 `session.id` 注入 metric，但加 warn log "this default will flip in v0.X"。一个 release 后再翻转默认。 - -不建议采用的原因：（1）当前 qwen-code 用户群不大，破坏面有限；（2）这是 cardinality bug，越早默认安全越好；（3）双段式发布会增加文档负担。如果父 issue owner 想要保守一些，可以采纳。 - -### PR 3 — Docs polish + samples（cleanup） - -- `docs/developers/development/telemetry.md` 补示例（见 §10） -- 阿里云 ARMS / Prometheus / Grafana 接入示例 -- 把所有典型 use case 的 settings.json 片段加进去 - -## 8. 测试计划 - -### 8.1 `parseOtelResourceAttributes()` 单元测试 - -参数化覆盖 §4.6 表格全部行（建议用 vitest `it.each`）： - -```ts -it.each([ - ['', {}], - ['a=1', { a: '1' }], - ['a=1,b=2', { a: '1', b: '2' }], - ['a=hello%20world', { a: 'hello world' }], - ['a=val%ZZbad', { a: 'val%ZZbad' }], // invalid percent - ['malformed', {}], - ['=val', {}], - ['a=', { a: '' }], - ['a=1,a=2', { a: '2' }], - [' a = 1 , b = 2 ', { a: '1', b: '2' }], -])('parses %j → %j', (input, expected) => { - expect(parseOtelResourceAttributes(input)).toEqual(expected); -}); -``` - -### 8.2 Resolver merge 测试 - -| 场景 | 期望 `service.name` | 期望 user attr | -| ----------------------------------------------------------------------- | ----------------------------------------------------- | ------------------------------------ | -| 全空 | `'qwen-code'` | 不存在 | -| 仅 env `OTEL_SERVICE_NAME=A` | `'A'` | — | -| 仅 env `OTEL_RESOURCE_ATTRIBUTES=service.name=B` | `'B'` | — | -| `OTEL_SERVICE_NAME=A` + `OTEL_RESOURCE_ATTRIBUTES=service.name=B` | `'A'`（OTEL_SERVICE_NAME 优先） | — | -| `OTEL_SERVICE_NAME=A` + `settings={service.name:C}` | `'A'`（OTEL_SERVICE_NAME 优先） | — | -| `OTEL_RESOURCE_ATTRIBUTES=service.name=B` + `settings={service.name:C}` | `'C'`（settings 优先于 env，无 OTEL_SERVICE_NAME 时） | — | -| `OTEL_RESOURCE_ATTRIBUTES=team=x` + `settings={team:y}` | `'qwen-code'` | `team='y'`（settings 优先） | -| `OTEL_RESOURCE_ATTRIBUTES=service.version=fake` | `'qwen-code'` + warn | service.version 仍为真实 cli version | -| `settings={service.version:fake}` | `'qwen-code'` + warn | service.version 仍为真实 cli version | - -### 8.3 Resource 内容快照测试 - -用 `InMemorySpanExporter` 拿一个 span，断言： - -```ts -expect(span.resource.attributes['service.name']).toBe('qwen-code'); -expect(span.resource.attributes['service.version']).toBe(EXPECTED_VERSION); -expect(span.resource.attributes['session.id']).toBeUndefined(); // 关键 -expect(span.resource.attributes['team']).toBe('platform'); // 用户加的 -``` - -### 8.4 Metric attribute toggle 测试 - -```ts -it('does not emit session.id on metrics by default', async () => { - // emit one tool call counter - recordToolCallMetrics(...); - const data = await metricReader.collect(); - const dp = data.resourceMetrics.scopeMetrics[0].metrics[0].dataPoints[0]; - expect(dp.attributes['session.id']).toBeUndefined(); -}); - -it('emits session.id when toggle is true', async () => { - config.telemetrySettings.metrics = { includeSessionId: true }; - recordToolCallMetrics(...); - const data = await metricReader.collect(); - const dp = data.resourceMetrics.scopeMetrics[0].metrics[0].dataPoints[0]; - expect(dp.attributes['session.id']).toBe(KNOWN_SESSION_ID); -}); -``` - -### 8.5 Spans / Logs 行为保持测试 - -- spans 仍有 `session.id`（不受 metric toggle 影响） -- logs 仍有 `session.id`（不受 metric toggle 影响） - -### 8.6 回归保护 - -- `autoDetectResources: false` 保持不变（assertion on config） -- 启动期间不出现新增 `diag.error`（捕获 OTel diag 日志做 assertion） -- 现有所有 telemetry 测试通过（CI） - -### 8.7 Diag warn 测试 - -校验下列输入都触发 `diag.warn` 一次： - -- `settings.resourceAttributes = { 'service.version': 'x' }`（reserved） -- `OTEL_RESOURCE_ATTRIBUTES=service.version=x`（reserved，env 也要 warn） -- `OTEL_RESOURCE_ATTRIBUTES=malformed`（无 `=`） -- `OTEL_RESOURCE_ATTRIBUTES=a=val%ZZ`（无效 percent-encoding） - -校验下列输入**不**触发 warn（合法路径）： - -- `settings.resourceAttributes = { 'service.name': 'x' }`（settings 允许设 service.name） -- `OTEL_SERVICE_NAME=foo` + `settings.resourceAttributes = { 'service.name': 'bar' }`（OTEL_SERVICE_NAME 优先即可，不需要 warn） - -## 9. 迁移 / 破坏性变更 - -### 9.1 破坏性变更（PR 2） - -**指标上的 `session.id` 默认消失**。这会影响： - -- Prometheus query 中 `by (session_id)` / `group_left(session_id)` 的聚合 -- Grafana dashboard 中按 session 切片的图 -- 任何按 session.id 做告警分组的规则 - -注：spans 和 logs 上的 `session.id` **不受影响**。 - -### 9.2 迁移路径 - -文档里给两个选项： - -**选项 A**：恢复旧行为（短期 debug 推荐） - -```bash -export QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID=true -``` - -或 `settings.json`： - -```json -{ - "telemetry": { - "metrics": { "includeSessionId": true } - } -} -``` - -⚠️ **警告**：长期开启会让 metric time-series 数量 = 历史 session 数量，撑爆后端。仅短期 debug 用。 - -**选项 B**：改用 spans / logs 做 session 切片（推荐） - -- spans / logs 上仍有 `session.id`，可在 trace backend（如 Jaeger / Aliyun ARMS Tracing）/ log backend（如 Loki / SLS）按 session 切片 -- 这两类数据本来就是 per-event 存储，cardinality 不会爆炸 -- 适合做 session-level drill-down 分析 - -### 9.3 Release note 模板 - -``` -**Breaking change (metric attribute):** - -The `session.id` attribute is no longer attached to metric data -points by default. This protects metric backends from unbounded -time-series fan-out. - -- Spans and logs are unaffected — `session.id` is still present. -- To restore the previous behavior (short-term debugging only), set - `QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID=true` or in settings.json: - `telemetry.metrics.includeSessionId: true`. -- For long-term session correlation, query against trace / log - backends instead of metric backends. - -See docs/developers/development/telemetry.md "Migration" for details. -``` - -## 10. 示例配置（用于文档） - -### 10.1 按 team / env 切片所有 telemetry - -```bash -export OTEL_RESOURCE_ATTRIBUTES="team=platform,env=prod,cost_center=eng-123" -``` - -效果：所有 span / log / metric 都带 `team=platform` `env=prod` `cost_center=eng-123`。 - -### 10.2 用 `OTEL_SERVICE_NAME` 在共享 collector 中路由 - -```bash -export OTEL_SERVICE_NAME=qwen-code-ci -``` - -效果：`service.name=qwen-code-ci`，多租户 OTel collector 可按 service.name 路由到不同后端。 - -### 10.3 Fleet baseline + 单机 override - -公司 fleet 的 `~/.qwen/settings.json`（GitOps 分发）： - -```json -{ - "telemetry": { - "resourceAttributes": { - "deployment.environment": "production", - "service.namespace": "engineering-tooling" - } - } -} -``` - -单机 ops 临时覆盖（不修改 settings）： - -```bash -export OTEL_RESOURCE_ATTRIBUTES="debug_run=true" -# settings 里的 deployment.environment / service.namespace 仍然生效 -# 同时这次运行额外带 debug_run=true -``` - -### 10.4 短期 debug 打开 metric session.id - -```bash -# 一次性 debug run -QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID=true qwen "投资分析" -``` - -完事即关闭，不要持久化到 settings。 - -### 10.5 阿里云 ARMS Metric 接入（推荐配置） - -```json -{ - "telemetry": { - "enabled": true, - "otlpEndpoint": "http:///api/v1/...", - "otlpProtocol": "http", - "resourceAttributes": { - "team": "platform", - "deployment.environment": "production" - }, - "metrics": { - "includeSessionId": false - } - } -} -``` - -## 11. 与 claude-code 实现的对比 - -| 维度 | claude-code | qwen-code 本设计 | 决策依据 | -| -------------------------- | ------------------------------------------------ | ------------------------------------------------ | -------------------------------------------------- | -| 标准 OTel env var | `OTEL_RESOURCE_ATTRIBUTES` / `OTEL_SERVICE_NAME` | ✅ 一致 | 标准契约 | -| `OTEL_SERVICE_NAME` 优先级 | 遵守 OTel 规范 | ✅ 遵守 | spec 明确规定 | -| Cardinality 开关命名 | `OTEL_METRICS_INCLUDE_*` | `QWEN_TELEMETRY_METRICS_INCLUDE_*` | 不污染标准 OTel 命名空间 | -| 开关作用域 | 仅 metric | ✅ 仅 metric | spans / logs 是 per-event，无 cardinality 爆炸问题 | -| 默认值 | 高基数 attribute 默认 false | ✅ 默认 false | 安全优先 | -| Per-attribute granularity | 每 attribute 一个 toggle | ✅ 一致 | 灵活，符合实际诊断需求 | -| settings.json 等价物 | ❌ 无 | ✅ 有 `telemetry.resourceAttributes` + `metrics` | 企业 fleet 部署 base config | -| Per-span 动态 hook | ❌ 无 | ❌ 无 | 复杂度高，claude-code 也没解，本期不做 | -| 多租户 `account_uuid` | 有 | ❌ 无 | qwen-code metric 里没有此 attr | -| Agent SDK `options.env` | 有 | ❌ 无 | qwen-code 没有等价模式 | -| 保留键策略 | 不允许覆盖 built-in id | ✅ 一致 | 遥测可信度 | -| 第一方上报通道 | claude-code 也有独立第一方通道（与 OTel 隔离） | ✅ qwen-logger 同样隔离 | 第一方与第三方通道职责分离 | - -**最值得借的两点**： - -1. **命名约定**：`*_INCLUDE_*` 一眼能看出语义，比反义命名（`*_EXCLUDE_*` / `*_DROP_*`）清晰 -2. **范围克制**：只 gate metric，不 gate span/log——claude-code 显然踩过这个边界，我们直接受益 - -**qwen-code 做得更好的点**： - -- settings.json 支持：claude-code 完全靠 env var，对企业 fleet 场景不友好 -- 明确的保留键策略（`service.version` 不可覆盖）：减少遥测被污染的可能 -- 第一方上报隔离：qwen-logger 走独立通道，与用户 OTLP 设置完全解耦 - -## 12. 未来工作（v2 + 候选） - -- **`service.version` cardinality 控制**：用 OTel View API 在 metric 层 drop attribute -- **更多 cardinality toggle**：未来若 metric 上引入 `user.account_uuid` / `model` 等，按需补 toggle -- **Per-span 动态 attribute hook**：可借鉴 qwen-code 自家 hooks 系统，加 `OnSpanStart(span, context) => attrs` 回调。需要独立设计。 -- **Resource attribute schema 校验**：限制 key 命名空间（如禁止覆盖 `service.*` 前缀以外的内建 attr），目前靠保留键列表硬编码够用。 -- **Hot reload Resource**：当 settings.json 在进程内被修改（设想 qwen-serve daemon 场景），目前不会重建 Resource。若 daemon 场景成熟，可以增加一条 reload 路径。 -- **跨进程 subagent context 传播**：subagent 跨进程时，把 parent 的 trace context（包括 resource）通过 OTel context propagation 标准 header 传过去。需要独立设计。 diff --git a/docs/design/worktree.md b/docs/design/worktree.md index 8c9d4f0d56d..4de9968b7aa 100644 --- a/docs/design/worktree.md +++ b/docs/design/worktree.md @@ -8,23 +8,23 @@ qwen-code 目前仅有面向 Arena 多模型对比场景的内部 worktree 实 ## 现状对比 -| 功能 | qwen-code | claude-code | 阶段 | -| --------------------------------- | --------------- | ----------- | ------- | -| `EnterWorktree` 工具 | ✅（Phase A） | ✅ | — | -| `ExitWorktree` 工具 | ✅（Phase A） | ✅ | — | -| AgentTool `isolation: 'worktree'` | ✅（Phase B） | ✅ | — | -| 过期 worktree 自动清理 | ✅（Phase B） | ✅ | — | -| worktree 会话状态持久化与恢复 | ❌ | ✅ | Phase C | -| Post-creation setup（hooks 配置） | ❌ | ✅ | Phase C | -| StatusLine worktree 状态展示 | ❌ | ✅ | Phase C | -| WorktreeExitDialog（退出提示） | ❌ | ✅ | Phase C | -| `--worktree` CLI 启动标志 | ❌ | ✅ | Phase D | -| 符号链接目录（node_modules 等） | ❌ | ✅ | Phase D | -| sparse checkout | ❌ | ✅ | Future | -| tmux 集成 | ❌ | ✅ | Future | -| Arena 多模型 worktree 隔离 | ✅（qwen 独有） | ❌ | — | -| 脏状态覆盖（stash + copy） | ✅ | ✅ | — | -| Baseline commit 追踪 | ✅（qwen 独有） | ❌ | — | +| 功能 | qwen-code | claude-code | +| --------------------------------- | --------------- | ----------- | +| `EnterWorktree` 工具 | ❌ | ✅ | +| `ExitWorktree` 工具 | ❌ | ✅ | +| AgentTool `isolation: 'worktree'` | ❌ | ✅ | +| worktree 会话状态持久化与恢复 | ❌ | ✅ | +| 过期 worktree 自动清理 | ❌ | ✅ | +| Post-creation setup（hooks 配置） | ❌ | ✅ | +| StatusLine worktree 状态展示 | ❌ | ✅ | +| WorktreeExitDialog（退出提示） | ❌ | ✅ | +| 符号链接目录（node_modules 等） | ❌ | ✅ | +| sparse checkout | ❌ | ✅ | +| `--worktree` CLI 启动标志 | ❌ | ✅ | +| tmux 集成 | ❌ | ✅ | +| Arena 多模型 worktree 隔离 | ✅（qwen 独有） | ❌ | +| 脏状态覆盖（stash + copy） | ✅ | ✅ | +| Baseline commit 追踪 | ✅（qwen 独有） | ❌ | ## 设计原则 @@ -54,14 +54,14 @@ AgentTool 的 `isolation: 'worktree'` 只走通用路径，Arena 内部不经过 Arena 的 worktree 路径由 `agents.arena.worktreeBaseDir` 控制，默认 `~/.qwen/arena`（`ArenaManager.ts:125`），与通用路径完全独立，不做任何改动。 -### 扩展配置 +### 扩展配置（暂缓至 Phase C/D） | 配置项 | 类型 | 用途 | 阶段 | | ----------------------------- | ---------- | -------------------------------------------------------------- | ------- | -| `worktree.symlinkDirectories` | `string[]` | 符号链接指定目录（如 `node_modules`）到 worktree，避免磁盘浪费 | Phase D | -| `worktree.sparsePaths` | `string[]` | git sparse-checkout cone 模式，大型 monorepo 只写入指定路径 | Future | +| `worktree.symlinkDirectories` | `string[]` | 符号链接指定目录（如 `node_modules`）到 worktree，避免磁盘浪费 | Phase C | +| `worktree.sparsePaths` | `string[]` | git sparse-checkout cone 模式，大型 monorepo 只写入指定路径 | Phase D | -Phase A / B / C 不新增任何配置项。 +Phase A / B 不新增任何配置项。 ## 工具设计 @@ -187,88 +187,27 @@ _无需改动：_ --- -### Phase C：会话完整性（SessionService 持久化 + UI 安全网） +### Phase C：体验优化（Post-creation setup + UI） -**目标：** worktree 状态在会话中断后可恢复，用户在界面上始终知道自己在哪个 worktree 里，退出会话时有安全提示。 +**目标：** worktree 创建后自动初始化环境，状态在界面上可见。 **要实现的功能：** -_SessionService worktree 状态持久化 + `--resume` 恢复：_ - -- `SessionService` 扩展 `WorktreeSession` 字段，记录 `{ slug, worktreePath, worktreeBranch, originalCwd, originalBranch }` -- `EnterWorktreeTool` 调用 `sessionService.setWorktreeSession()` 写入状态 -- `ExitWorktreeTool` 调用 `sessionService.clearWorktreeSession()` 清除状态 -- `--resume` 启动路径读取该字段，恢复 `targetDir` 并向模型注入上下文提示 - -_Post-creation setup：_ - -- 创建 worktree 后自动执行 `git config core.hooksPath /.git/hooks`，确保 worktree 内的提交与主仓库 hooks 行为一致 - -_StatusLine worktree 展示：_ - -- `UIStateContext` 新增 `activeWorktree` 字段（从 session 状态读取），在会话进入 / 退出 worktree 时更新 -- `StatusLineCommandInput` payload 新增 `worktree?: { slug: string; branch: string }` 字段，供用户 statusline 脚本使用 -- `Footer` 在 `activeWorktree` 非空时内置展示一行 `⎇ ()`，无需用户配置 statusline 脚本即可获得基本可见性 - -_WorktreeExitDialog：_ - -- 新增 `WorktreeExitDialog.tsx` 组件，参考现有 Dialog 写法 -- 修改退出键（Ctrl+C / Ctrl+D）处理逻辑：检测到 `activeWorktree` 非空时，拦截第二次确认，展示 Dialog 提示用户选择 keep 或 remove -- keep / remove 操作复用 `ExitWorktreeTool` 的现有路径 - -**影响文件：** - -| 文件 | 变更类型 | -| ------------------------------------------------------------- | ----------------------------------------------------------------------------- | -| `packages/core/src/services/sessionService.ts` | 新增 `WorktreeSession` 字段及读写方法 | -| `packages/core/src/tools/enter-worktree.ts` | 调用 `sessionService.setWorktreeSession()` | -| `packages/core/src/tools/exit-worktree.ts` | 调用 `sessionService.clearWorktreeSession()` | -| `packages/core/src/services/gitWorktreeService.ts` | `createUserWorktree()` / `createAgentWorktree()` 后追加 `core.hooksPath` 配置 | -| `packages/cli/src/ui/contexts/UIStateContext.tsx` | 新增 `activeWorktree` 字段及 set/clear action | -| `packages/cli/src/ui/hooks/useStatusLine.ts` | `StatusLineCommandInput` 新增 `worktree` 字段 | -| `packages/cli/src/ui/components/Footer.tsx` | 内置 worktree 行展示 | -| `packages/cli/src/ui/components/WorktreeExitDialog.tsx` | 新建 | -| `packages/cli/src/ui/components/DialogManager.tsx` | 注册 `WorktreeExitDialog` | -| `packages/cli/src/ui/components/ExitWarning.tsx` 或退出键处理 | 检测 `activeWorktree` 并拦截退出 | +- Post-creation setup：配置 `core.hooksPath` 指向主仓库（qwen-code 无 `settings.local.json` 概念，不需要复制） +- StatusLine 展示当前 worktree 名称 / 分支 +- WorktreeExitDialog：会话退出时（检测到 worktree 仍活跃）提示用户选择 keep 或 remove +- 新增 `worktree.symlinkDirectories` 配置项，实现目录符号链接 --- -### Phase D：启动时配置（`--worktree` CLI 标志 + 目录符号链接） +### Phase D：高级功能 -**目标：** 支持在启动时直接进入 worktree，并通过目录符号链接减少大型项目的磁盘开销。 +**目标：** 对齐 claude-code 的完整特性集。 **要实现的功能：** -_`--worktree [name]` CLI 启动标志：_ - -- `packages/cli/src/args.ts` 新增 `--worktree [name]` 参数 -- 启动流程在进入主循环前调用 `createUserWorktree()`，将 `targetDir` 设为 worktree 路径，并写入 SessionService 状态 -- 整个会话从启动即在 worktree 环境中运行，退出时触发 WorktreeExitDialog - -_`worktree.symlinkDirectories` 配置项：_ - -- settings schema 新增 `worktree.symlinkDirectories: string[]` -- `createUserWorktree()` 后遍历配置，调用 `fs.symlink()` 将主仓库目录链接进 worktree -- 跳过目标不存在的项；目标已存在时跳过（不覆盖） - -**影响文件：** - -| 文件 | 变更类型 | -| -------------------------------------------------- | ------------------------------------------- | -| `packages/cli/src/args.ts` | 新增 `--worktree [name]` 参数 | -| `packages/cli/src/main.ts`（或启动入口） | 解析 `--worktree` 并在主循环前创建 worktree | -| `packages/core/src/services/gitWorktreeService.ts` | `createUserWorktree()` 后追加 symlink 逻辑 | -| `packages/core/src/config/`（settings schema） | 新增 `worktree.symlinkDirectories` 字段 | - ---- - -### Future：高级功能（按需实现） - -以下功能面向更特定的使用场景，当前阶段不纳入排期，待用户需求明确后再评估实现。 - -| 功能 | 说明 | -| ----------------------- | ------------------------------------------------------------------------------------------- | -| sparse checkout | `worktree.sparsePaths` 配置项，大型 monorepo 只 checkout 指定路径，缩短创建时间和磁盘占用 | -| `.worktreeinclude` 文件 | 将 gitignore 的文件（`.env`、`secrets.json` 等）自动复制进 worktree | -| tmux 集成 | `--worktree --tmux` 在新 tmux 窗口启动 worktree 会话 | -| PR 引用解析 | `--worktree=#123` 自动 fetch PR 分支并基于它创建 worktree（依赖 Phase D `--worktree` 标志） | +- `--worktree [name]` CLI 启动标志：启动时直接创建 worktree，整个会话在隔离环境中运行 +- sparse checkout 支持：新增 `worktree.sparsePaths` 配置项 +- `.worktreeinclude` 文件：支持将 gitignore 的文件复制到 worktree +- tmux 集成：`--worktree --tmux` 在 tmux 会话中启动 +- PR 引用解析：`--worktree=#123` 自动 fetch 并基于 PR 创建 worktree diff --git a/docs/developers/daemon-client-adapters/tui.md b/docs/developers/daemon-client-adapters/tui.md deleted file mode 100644 index c9d223b9262..00000000000 --- a/docs/developers/daemon-client-adapters/tui.md +++ /dev/null @@ -1,96 +0,0 @@ -# TUI Daemon Adapter Draft - -## Goal - -Add a flag-gated TUI transport that talks to `qwen serve` through -`DaemonSessionClient` instead of creating an in-process `Config` + agent -runtime. - -This is a dogfood path for Mode B client migration. It must not replace the -default TUI path until output sinks, typed daemon events, session-scoped -permission, and lifecycle diagnostics are stable. - -## Proposed Entry Point - -```bash -QWEN_DAEMON_URL=http://127.0.0.1:4170 qwen --experimental-daemon-tui -``` - -Optional: - -```bash -QWEN_DAEMON_TOKEN=... QWEN_DAEMON_WORKSPACE=/repo qwen --experimental-daemon-tui -``` - -The CLI should refuse this mode unless both are true: - -- `QWEN_DAEMON_URL` or `--daemon-url` is set. -- `GET /capabilities` advertises `session_create`, `session_prompt`, and - `session_events`. - -## Minimal Flow - -1. Create `DaemonClient` with daemon URL and token. -2. Fetch `/capabilities`. -3. Create or attach with `DaemonSessionClient.createOrAttach()`. -4. Subscribe to `session.events()`. -5. Submit user prompts through `session.prompt()`. -6. Route cancel through `session.cancel()`. -7. Route model switch through `session.setModel()`. -8. Route permission votes through `session.respondToPermission()`. - -## Rendering Contract - -The first implementation adds `DaemonTuiAdapter`, a locally verifiable reducer -and transport spike. It maps only these daemon events: - -| Daemon event | TUI handling | -| ---------------------------------------- | -------------------------------------------- | -| `session_update` / `agent_message_chunk` | Append assistant text | -| `session_update` / `agent_thought_chunk` | Append thinking text | -| `session_update` / `tool_call` | Show tool call lifecycle | -| `permission_request` | Show existing confirmation UI where possible | -| `permission_resolved` | Close or update confirmation UI | -| `model_switched` | Update footer/model display | -| `session_died` | Show disconnected state and stop streaming | - -Unknown events must be ignored, not fatal. Typed event reducers will land in a -later protocol PR. - -The adapter is not wired into the default Ink app yet. Existing interactive TUI, -JSONL, stream-json, and dual-output behavior remains unchanged. - -## Explicit Non-Goals - -- Do not remove the current TUI in-process runtime. -- Do not change JSONL, stream-json, or dual-output behavior in this PR. -- Do not expose file CRUD, MCP management, memory CRUD, or provider/auth - mutation through TUI yet. -- Do not make browser/web direct-to-daemon assumptions; this is terminal only. - -## Merge Safety - -- Default off. -- Additive code path. -- No existing CLI flags change behavior. -- If the daemon is unavailable, the experimental path fails before starting the - TUI and tells the user to run `qwen serve`. - -## Validation Plan - -- Unit-test event-to-TUI-state mapping with synthetic daemon events. -- Unit-test prompt, cancel, model switch, and permission vote forwarding. -- Unit-test flag/env parsing when the feature flag is wired. -- Smoke-test against a local `qwen serve`: - - prompt text streams into the TUI - - cancel resolves the active prompt - - permission request can be accepted or rejected - - reconnect sends the tracked `Last-Event-ID` - -## Blockers Before Default Migration - -- Typed daemon event schema. -- Session-scoped permission route. -- Output sink refactor for JSONL / stream-json / dual-output parity. -- Session lifecycle close/delete semantics. -- Runtime diagnostics for MCP, skills, providers, and workspace env. diff --git a/docs/developers/daemon-client-adapters/web-ui.md b/docs/developers/daemon-client-adapters/web-ui.md new file mode 100644 index 00000000000..2aa022fa371 --- /dev/null +++ b/docs/developers/daemon-client-adapters/web-ui.md @@ -0,0 +1,118 @@ +# Daemon Web UI Adapter + +## Goal + +Web chat and web terminal clients should consume `qwen serve` through the +daemon HTTP/SSE APIs and render a client-side transcript. Native local TUI, +channel, and IDE integrations keep their existing default paths for now. + +## Shared UI Contract + +Use the TypeScript SDK daemon UI exports as the common boundary: + +```ts +import { + DaemonClient, + DaemonSessionClient, + createDaemonTranscriptStore, + normalizeDaemonEvent, +} from '@qwen-code/sdk/daemon'; +``` + +The split is: + +- `DaemonClient` handles daemon HTTP routes. +- `DaemonSessionClient` owns session creation/attachment and SSE replay. +- `normalizeDaemonEvent()` converts daemon wire events into UI events. +- `createDaemonTranscriptStore()` reduces UI events into transcript blocks. + +React clients can use the optional `@qwen-code/webui` binding: + +```tsx +import { + DaemonSessionProvider, + useDaemonActions, + useDaemonConnection, + useDaemonPendingPermissions, + useDaemonTranscriptBlocks, +} from '@qwen-code/webui'; +``` + +Minimal React shape: + +```tsx +function App() { + return ( + + + + + ); +} + +function Transcript() { + const blocks = useDaemonTranscriptBlocks(); + return blocks.map((block) => ); +} +``` + +The provider creates or attaches a daemon session, subscribes to SSE, keeps the +last event id on `DaemonSessionClient`, and reconnects the stream by default. +Callers can disable that with `autoReconnect={false}` for tests or custom +connection management. + +## Browser Deployment Shapes + +### Same-Origin Local POC + +A daemon-served page can call the daemon directly because the page and API share +one origin. This is the preferred early POC shape for local web chat and web +terminal validation. + +### Remote Web Chat / Web Terminal + +A production remote web app should normally talk to a backend-for-frontend. The +BFF owns daemon URL, token, workspace routing, and session metadata, then +forwards browser-safe app events to the browser. This keeps bearer tokens out of +browser storage and lets the deployment decide which daemon/workspace a user is +allowed to reach. + +### Local Browser Against Local Daemon + +A separate local dev server is cross-origin from `qwen serve`; it must either +proxy daemon routes through the same origin or be served by the daemon. The +daemon intentionally rejects arbitrary browser `Origin` requests. + +## Rendering Responsibilities + +The shared transcript model is semantic, not visual. UI clients decide how to +render: + +- user and assistant message blocks +- collapsed thought blocks +- tool status cards +- shell output blocks +- permission request controls +- status/error/debug blocks + +The web terminal is a browser-native semantic renderer. It should look and feel +terminal-like with monospace layout, scrollback, prompt input, shortcuts, and +streaming blocks, but it is not a raw PTY proxy and does not require server-side +Ink rendering. + +## Merge Safety + +- The native `qwen` TUI remains direct and unchanged. +- `--acp`, channel, and IDE paths remain unchanged by default. +- The SDK UI core is additive. +- The WebUI React binding is optional and only runs in clients that import it. +- Removed daemon TUI spike code should not be treated as a product migration. + +## Follow-Ups + +- Add a daemon-served local `/web` POC or equivalent same-origin web app. +- Build first-class chat and terminal renderers on top of transcript blocks. +- Add richer typed events only where existing daemon events are too low-level + for stable browser UI behavior. +- Consider a dedicated `@qwen-code/daemon-ui-core` package if non-SDK consumers + need the UI core as an independent dependency. diff --git a/docs/developers/development/telemetry.md b/docs/developers/development/telemetry.md index 266ce05cc83..1cd31a96916 100644 --- a/docs/developers/development/telemetry.md +++ b/docs/developers/development/telemetry.md @@ -66,8 +66,6 @@ These settings can be overridden by environment variables or CLI flags. | `outfile` | `QWEN_TELEMETRY_OUTFILE` | `--telemetry-outfile ` | Save telemetry to file (overrides OTLP export) | file path | - | | `logPrompts` | `QWEN_TELEMETRY_LOG_PROMPTS` | `--telemetry-log-prompts` / `--no-telemetry-log-prompts` | Include prompts in telemetry logs | `true`/`false` | `true` | | `includeSensitiveSpanAttributes` | `QWEN_TELEMETRY_INCLUDE_SENSITIVE_SPAN_ATTRIBUTES` | - | Include user prompts, system prompts, tool I/O, and model output as native span attributes (in addition to log-to-span bridge spans) | `true`/`false` | `false` | -| `resourceAttributes` | `OTEL_RESOURCE_ATTRIBUTES` (+ `OTEL_SERVICE_NAME`) | - | Static resource attributes attached to every exported span / log / metric. See [Resource attributes](#resource-attributes) below. | `key=value,…` | `{}` | -| `metrics.includeSessionId` | `QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID` | - | Include `session.id` on metric data points. **Disabled by default** to protect metric backends from time-series fan-out. | `true`/`false` | `false` | **Note on boolean environment variables:** For the boolean settings (`enabled`, `logPrompts`, `includeSensitiveSpanAttributes`), setting the @@ -127,234 +125,6 @@ The `QWEN_TELEMETRY_OTLP_*` variants take precedence over the `OTEL_*` variants. For detailed information about all configuration options, see the [Configuration Guide](./cli/configuration.md). -### Resource attributes - -Resource attributes are static key-value pairs attached to every span, log, -and metric exported via OTLP. Use them to slice telemetry by team, environment, -deployment region, or any other dimension your backend cares about. - -Two sources, merged in priority order (lowest → highest): - -1. The standard `OTEL_RESOURCE_ATTRIBUTES` env var -2. `telemetry.resourceAttributes` in `.qwen/settings.json` (overrides env on - key conflict) - -`OTEL_SERVICE_NAME` is a separate escape hatch — when set, it overrides -`service.name` from any other source (per the OpenTelemetry spec). - -#### Examples - -**Slice all telemetry by team / environment:** - -```bash -export OTEL_RESOURCE_ATTRIBUTES="team=platform,env=prod,cost_center=eng-123" -``` - -**Route to a per-tenant collector via `service.name`:** - -```bash -export OTEL_SERVICE_NAME=qwen-code-ci -``` - -**Fleet baseline (`~/.qwen/settings.json`) + per-host override:** - -```json -{ - "telemetry": { - "resourceAttributes": { - "deployment.environment": "production", - "service.namespace": "engineering-tooling" - } - } -} -``` - -```bash -# Add a one-off tag without touching settings: -export OTEL_RESOURCE_ATTRIBUTES="debug_run=true" -``` - -#### Reserved keys - -Some keys are runtime-controlled and cannot be overridden: - -- `service.version` — always set to the running CLI version. Setting it from - any source is silently dropped with a warning. -- `session.id` — runtime-injected per session. User-provided values from - either env or settings are dropped with a warning. The reason is that - Resource attributes auto-attach to every metric data point; allowing user - override would bypass [Cardinality controls](#cardinality-controls) below. - Spans and logs always carry `session.id`. - -`service.name` is **not** reserved; it follows the precedence chain above. - -#### Format - -`OTEL_RESOURCE_ATTRIBUTES` follows the OpenTelemetry spec: -`key1=value1,key2=value2` with values percent-encoded. Spaces in values must -be encoded as `%20`, **commas as `%2C`** (unencoded commas split the value at -the wrong boundary and the second half is dropped as malformed). Malformed -pairs are skipped with a warning rather than failing telemetry startup. - -#### Troubleshooting: when a user-provided attribute appears not to take effect - -Reserved keys (`service.version`, `session.id`), malformed pairs, non-string -settings values, and invalid percent-encoding are all silently dropped with a -warning logged via the OpenTelemetry diagnostics channel. That channel routes -to the debug log file (`~/.qwen/log/otel-*.log`), **not** the console, so the -behavior can look like silent failure. - -If a custom resource attribute isn't appearing on exported telemetry: - -1. Check `~/.qwen/log/otel-*.log` for lines matching `cannot override` (reserved - key dropped), `Skipping malformed` (bad env var pair), or `must be a string` - (non-string settings value). -2. Verify the env var is set in the qwen-code process's environment (not just - your shell) and that values are percent-encoded. -3. Confirm `telemetry.enabled` is `true` — telemetry init only runs if enabled. - -### Cardinality controls - -Metrics are aggregated by attribute set at the backend — every distinct -combination of attribute values produces a new time series. Attaching a -high-cardinality field like `session.id` to a metric causes time-series fan-out -proportional to the number of sessions, which quickly exhausts metric backend -storage. - -To prevent this, Qwen Code keeps high-cardinality attributes off metric data -points by default. Spans and logs are per-event and unaffected, so they -continue to carry `session.id` for trace and log correlation. - -#### `telemetry.metrics.includeSessionId` (default: `false`) - -Setting this to `true` (via settings or -`QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID=true`) re-attaches `session.id` to -every metric data point. - -⚠️ **Warning:** each CLI session creates a new value. Leaving this on for a -fleet will blow up metric storage. Recommended only for short-term debugging. -For long-term session correlation, query trace or log backends instead. - -#### Migration from earlier versions - -Prior to this release, `session.id` was attached to metrics by default. If -your Prometheus queries / Grafana dashboards / alert rules reference -`session_id` on a metric, you have two options: - -**Option A** — restore the previous behavior for short-term debugging: - -```bash -export QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID=true -``` - -or: - -```json -{ - "telemetry": { - "metrics": { "includeSessionId": true } - } -} -``` - -**Option B (recommended)** — move session-level analysis off metrics. Spans -and logs still carry `session.id`, and trace / log backends (Jaeger, Tempo, -Loki, Aliyun SLS / ARMS Tracing) handle per-session slicing natively without -cardinality pressure. - -### Client-side HTTP span on outbound fetch - -When telemetry is enabled, Qwen Code registers `UndiciInstrumentation` -which creates a client-side HTTP span for every outbound `fetch()` -request originated by the process — including the LLM SDKs (`openai`, -`@google/genai`, `@anthropic-ai/sdk`), the MCP StreamableHTTP client, the -`WebFetch` tool, and any IDE-extension out-of-process calls. The span -lets you see network latency (TTFB / response body transfer) separately -from upstream model processing time, which the existing -`api.generateContent` span alone can't distinguish. - -These spans go to your **own** OTLP collector (or file outfile) just like -the rest of the telemetry — they do not affect what is written onto the -outbound HTTP request itself. Whether the W3C `traceparent` header is -also written into the outgoing request stream is controlled by a -**separate, security-relevant setting** documented in -[outbound correlation](#outbound-correlation-security-relevant) below. - -**Feedback-loop avoidance.** OTel SDK uses `fetch` internally to upload OTLP -data. Without protection, instrumenting `fetch` would trace those uploads, -which would themselves be uploaded, causing an infinite loop. Qwen Code's -undici instrumentation is configured with an `ignoreRequestHook` that skips -URLs matching the configured `telemetry.otlpEndpoint` / -`telemetry.otlpTracesEndpoint` / `telemetry.otlpLogsEndpoint` / -`telemetry.otlpMetricsEndpoint` prefixes. In file-outfile mode there are no -outbound HTTP uploads, so the hook is a no-op. - -## Outbound correlation (SECURITY-RELEVANT) - -These settings live in a **separate top-level namespace** from `telemetry.*` -on purpose: telemetry controls data flow into the operator's own -observability backend, while `outboundCorrelation.*` controls what -client-side correlation data qwen-code writes **into outbound LLM API -request streams** that reach third-party LLM provider endpoints -(DashScope, OpenAI, Anthropic, etc.). Different recipients, different -consent decision. **All values default to off.** See PR #4390 review -discussion for the framing rationale. - -### `outboundCorrelation.propagateTraceContext` - -```jsonc -"outboundCorrelation": { - "propagateTraceContext": false // default -} -``` - -When `false` (default), Qwen Code installs a no-op `TextMapPropagator` on -the OTel SDK. UndiciInstrumentation still creates client HTTP spans for -your OTLP collector, but `propagation.inject()` is a no-op so **no -`traceparent` is written onto outbound requests**. Trace IDs stay -internal to the operator's collector. - -When `true`, the SDK's default W3C composite propagator -(`tracecontext` + `baggage`) is installed and the standard `traceparent` -header is written on every outbound `fetch`: - -``` -traceparent: 00-<32-hex traceId>-<16-hex parentSpanId>-<01-sampled | 00-not-sampled> -``` - -Opt in only when the LLM provider also reports into your OTel collector -for cross-process trace stitching — e.g. ARMS Tracing serving DashScope. -For most operators the value is `false`; cross-vendor trace continuation -is niche. - -**Depends on `telemetry.enabled: true`.** The OTel SDK only initializes -when telemetry is enabled, so `propagateTraceContext` only takes effect -in that state. Setting it to `true` while telemetry is disabled is a -silent no-op — no SDK, no propagator, no `traceparent` on the wire. -Verify both flags when wiring an ARMS+DashScope correlation setup: - -```jsonc -{ - "telemetry": { - "enabled": true, - "otlpTracesEndpoint": "http://tracing-analysis-...", - }, - "outboundCorrelation": { - "propagateTraceContext": true, - }, -} -``` - -### Other outbound correlation headers - -`X-Qwen-Code-Session-Id` and `X-Qwen-Code-Request-Id` are **not part of -this PR**. They will be designed and proposed in their own follow-up -PR(s) under the same `outboundCorrelation.*` namespace, each with its -own threat model and operator-consent flow. PR #4390 review (LaZzyMan) -established the principle: "telemetry's scope of work doesn't include -sending identifiers to LLM providers"; correlation-header work moves to -its own design discussion rather than landing under telemetry. - ## Aliyun Telemetry ### Manual OTLP Export diff --git a/docs/developers/qwen-serve-protocol.md b/docs/developers/qwen-serve-protocol.md index f6d6315827e..84eaddc56c7 100644 --- a/docs/developers/qwen-serve-protocol.md +++ b/docs/developers/qwen-serve-protocol.md @@ -104,9 +104,17 @@ registry. Clients **must** gate UI off `features`, not off `mode` (per design 'mcp_guardrail_events', 'workspace_file_read', 'workspace_file_bytes', 'workspace_file_write', 'session_approval_mode_control', 'workspace_tool_toggle', - 'workspace_init', 'workspace_mcp_restart'] + 'workspace_init', 'workspace_mcp_restart', + 'auth_device_flow', 'permission_mediation'] ``` +> The conditional `require_auth` tag (PR 15) appears only when the daemon +> is started with `--require-auth`. F3's `permission_mediation` tag is +> always-on and carries `modes: ['first-responder', 'designated', +'consensus', 'local-only']` so SDK clients can introspect the +> build-supported set; the runtime-active strategy is at +> `body.policy.permission`. + `session_scope_override` is the negotiation handle for the per-request `sessionScope` field on `POST /session` (see below). Older daemons silently ignore the field, so SDK clients should pre-flight `caps.features` for this tag before sending it. `session_load` and `unstable_session_resume` advertise the explicit-restore routes (`POST /session/:id/load` and `POST /session/:id/resume`). Older daemons return `404` for these paths, so SDK clients should pre-flight `caps.features` before calling. The `unstable_` prefix on `unstable_session_resume` mirrors the underlying ACP method (`connection.unstable_resumeSession`) — the daemon's wire shape is committed for v1, but the ACP method name itself may change before ACP marks resume stable. @@ -1121,7 +1129,7 @@ Request: { "mode": "auto-edit", "persist": false } ``` -`mode` must be one of `'plan' | 'default' | 'auto-edit' | 'auto' | 'yolo'` (mirror of core's `ApprovalMode` enum; the SDK exports `DAEMON_APPROVAL_MODES` for runtime validation). `persist` defaults to `false`. +`mode` must be one of `'plan' | 'default' | 'auto-edit' | 'yolo'` (mirror of core's `ApprovalMode` enum; the SDK exports `DAEMON_APPROVAL_MODES` for runtime validation). `persist` defaults to `false`. Response (200): @@ -1278,17 +1286,19 @@ data: {"v":1,"type":"client_evicted","data":{"reason":"queue_overflow","droppedA The SSE-level `id:` / `event:` lines duplicate `envelope.id` / `envelope.type` for EventSource compatibility. Raw-`fetch` consumers (the SDK's `parseSseStream`) read everything off the JSON envelope and ignore the SSE preamble lines. -| Event type | Trigger | -| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `session_update` | Any ACP `sessionUpdate` notification (LLM chunks, tool calls, usage) | -| `permission_request` | Agent asked for tool approval | -| `permission_resolved` | Some client voted on a permission via `POST /permission/:requestId` | -| `model_switched` | `POST /session/:id/model` succeeded | -| `model_switch_failed` | `POST /session/:id/model` rejected | -| `session_died` | Agent child crashed unexpectedly. **Terminal: SSE stream closes after this frame; the session is gone from `byId`.** Subscribers should reconnect via `POST /session` to spawn a fresh one. | -| `slow_client_warning` | Subscriber-local: queue ≥ 75% full. **Non-terminal** — the stream continues; the warning is a heads-up before eviction. Carries `{queueSize, maxQueued, lastEventId}`. Fires ONCE per overflow episode; re-arms after the queue drains below 37.5%. No `id` (synthetic). Pre-flight `caps.features.slow_client_warning`. | -| `client_evicted` | Subscriber-local: queue overflow. **Terminal: SSE stream closes after this frame** (no `id` — synthetic). Other subscribers on the same session continue. | -| `stream_error` | Daemon-side error during fan-out. **Terminal: SSE stream closes after this frame** (no `id` — synthetic). | +| Event type | Trigger | +| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `session_update` | Any ACP `sessionUpdate` notification (LLM chunks, tool calls, usage) | +| `permission_request` | Agent asked for tool approval | +| `permission_resolved` | Some client voted on a permission via `POST /permission/:requestId` | +| `permission_partial_vote` | (consensus only) A vote was recorded but quorum not yet reached. Carries `{requestId, sessionId, votesReceived, votesNeeded, quorum, optionTallies}`. Pre-flight `caps.features.permission_mediation`. | +| `permission_forbidden` | A vote was rejected by the active policy (`designated` mismatch, `local-only` non-loopback, or `consensus` voter not in snapshot). Carries `{requestId, sessionId, clientId?, reason}`. Pre-flight `caps.features.permission_mediation`. | +| `model_switched` | `POST /session/:id/model` succeeded | +| `model_switch_failed` | `POST /session/:id/model` rejected | +| `session_died` | Agent child crashed unexpectedly. **Terminal: SSE stream closes after this frame; the session is gone from `byId`.** Subscribers should reconnect via `POST /session` to spawn a fresh one. | +| `slow_client_warning` | Subscriber-local: queue ≥ 75% full. **Non-terminal** — the stream continues; the warning is a heads-up before eviction. Carries `{queueSize, maxQueued, lastEventId}`. Fires ONCE per overflow episode; re-arms after the queue drains below 37.5%. No `id` (synthetic). Pre-flight `caps.features.slow_client_warning`. | +| `client_evicted` | Subscriber-local: queue overflow. **Terminal: SSE stream closes after this frame** (no `id` — synthetic). Other subscribers on the same session continue. | +| `stream_error` | Daemon-side error during fan-out. **Terminal: SSE stream closes after this frame** (no `id` — synthetic). | Reconnect semantics: @@ -1305,20 +1315,32 @@ Backpressure: ### `POST /permission/:requestId` -Cast a vote on a pending `permission_request`. **First responder wins** — once one client answers, every other client trying to answer the same id gets `404`. +Cast a vote on a pending `permission_request`. The active **mediation policy** decides who wins: + +| Policy | Behavior | +| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `first-responder` (default) | Any validated voter wins; later voters get `404`. Pre-F3 baseline. | +| `designated` | Only the prompt originator (`originatorClientId`) decides; non-originators get `403 permission_forbidden / designated_mismatch`. Falls back to first-responder for anonymous prompts. | +| `consensus` | N-of-M voters must agree (default `N = floor(M/2) + 1`, override via `policy.consensusQuorum`). First option to reach `N` wins. Non-resolving votes get `200` + `permission_partial_vote` SSE frames. | +| `local-only` | Only loopback voters decide; remote callers get `403 permission_forbidden / remote_not_allowed`. | + +The active policy is configured in `settings.json` under `policy.permissionStrategy` and surfaced on `/capabilities` at `body.policy.permission`. Pre-flight `caps.features.permission_mediation` (with `modes: [...]`) for the build-supported set. + +> **F3 (#4175): multi-client permission coordination.** F3 added the four policies above. Pre-F3 daemons hardcoded first-responder; the wire shape stays bit-for-bit unchanged when the configured policy is `first-responder`. New events (`permission_partial_vote`, `permission_forbidden`) are additive — old SDKs see them as `unrecognized_known_event` and gracefully ignore. -> **Stage 1 limitation — no permission timeout.** A `permission_request` +> **Permission timeout (default 5 minutes).** A `permission_request` > stays pending until: (a) some client votes here, (b) `POST /session/:id/cancel` -> fires, (c) the HTTP client driving the prompt -> disconnects (mid-prompt cancel resolves outstanding permissions as -> `cancelled`), (d) the session is killed, or (e) the daemon shuts -> down. **In a fully-headless deployment with no SSE subscriber, -> `requestPermission` blocks the agent indefinitely** — there's nothing -> to time out the wait. Stage 2 will add a configurable -> `permissionTimeoutMs`. Until then, headless callers should keep an -> SSE subscription open or wrap their prompt loop in their own timeout -> -> - `POST /session/:id/cancel` on expiry. +> fires, (c) the HTTP client driving the prompt disconnects +> (mid-prompt cancel resolves outstanding permissions as `cancelled`), +> (d) the session is killed, (e) the daemon shuts down, **or +> (f) the per-session permission timeout fires** (`DEFAULT_PERMISSION_TIMEOUT_MS`, +> 5 minutes). On timeout fire the agent's `requestPermission` resolves +> as `{outcome: 'cancelled'}`, the audit ring records a +> `permission.timeout` entry, daemon stderr emits a one-line +> breadcrumb, and the SSE bus fans out the standard +> `permission_resolved` cancelled frame so subscribers clean up. The +> timeout is configurable via `BridgeOptions.permissionResponseTimeoutMs`; +> headless callers running long-form prompts may want to extend it. Request: @@ -1338,10 +1360,13 @@ Outcomes: Response: -- `200 {}` — your vote was accepted +- `200 {}` — your vote was accepted (resolved OR recorded under consensus quorum) +- `403 { "code": "permission_forbidden", "reason": "designated_mismatch" | "remote_not_allowed", "requestId", "sessionId" }` — F3: the active policy rejected your vote - `404 { "error": "..." }` — the requestId is unknown (already resolved, never existed, or session torn down) +- `500 { "code": "cancel_sentinel_collision", ... }` — F3: the agent's `allowedOptionIds` contains the reserved sentinel `'__cancelled__'`; agent / daemon contract violation +- `501 { "code": "permission_policy_not_implemented", "policy": "" }` — F3 forward-compat: a policy literal landed in the schema but its mediator branch isn't built yet (currently unreachable; reserved for future policies) -After a successful vote, every connected client sees `permission_resolved` with the same `requestId` and the chosen `outcome`. +After a successful vote, every connected client sees `permission_resolved` with the same `requestId` and the chosen `outcome`. Under `consensus`, intermediate votes additionally fan out `permission_partial_vote` until quorum. ### Auth device-flow routes (issue #4175 PR 21) diff --git a/docs/developers/sdk-java.md b/docs/developers/sdk-java.md index 8e2a8dbfec2..d3eab983c57 100644 --- a/docs/developers/sdk-java.md +++ b/docs/developers/sdk-java.md @@ -144,7 +144,7 @@ The SDK supports different permission modes for controlling tool execution: - **`default`**: Write tools are denied unless approved via `canUseTool` callback or in `allowedTools`. Read-only tools execute without confirmation. - **`plan`**: Blocks all write tools, instructing AI to present a plan first. -- **`auto-edit`**: Auto-approve edit tools (`edit`, `write_file`, `notebook_edit`) while other tools require confirmation. +- **`auto-edit`**: Auto-approve edit tools (edit, write_file) while other tools require confirmation. - **`yolo`**: All tools execute automatically without confirmation. ### Session Event Consumers and Assistant Content Consumers diff --git a/docs/developers/sdk-typescript.md b/docs/developers/sdk-typescript.md index a3de0f2e57e..8ba590997d6 100644 --- a/docs/developers/sdk-typescript.md +++ b/docs/developers/sdk-typescript.md @@ -158,7 +158,7 @@ The SDK supports different permission modes for controlling tool execution: - **`default`**: Write tools are denied unless approved via `canUseTool` callback or in `allowedTools`. Read-only tools execute without confirmation. - **`plan`**: Blocks all write tools, instructing AI to present a plan first. -- **`auto-edit`**: Auto-approve edit tools (`edit`, `write_file`, `notebook_edit`) while other tools require confirmation. +- **`auto-edit`**: Auto-approve edit tools (edit, write_file) while other tools require confirmation. - **`yolo`**: All tools execute automatically without confirmation. ### Permission Priority Chain diff --git a/docs/developers/tools/_meta.ts b/docs/developers/tools/_meta.ts index 9f68d150ce9..2662563769f 100644 --- a/docs/developers/tools/_meta.ts +++ b/docs/developers/tools/_meta.ts @@ -3,7 +3,6 @@ export default { 'file-system': 'File System', 'multi-file': 'Multi-File Read', shell: 'Shell', - monitor: 'Monitor', 'todo-write': 'Todo Write', task: 'Task', 'exit-plan-mode': 'Exit Plan Mode', diff --git a/docs/developers/tools/file-system.md b/docs/developers/tools/file-system.md index d07fd805c6b..118f5e0b6d8 100644 --- a/docs/developers/tools/file-system.md +++ b/docs/developers/tools/file-system.md @@ -44,72 +44,7 @@ Qwen Code provides a comprehensive suite of tools for interacting with the local - For other binary files: A message like `Cannot display content of binary file: /path/to/data.bin`. - **Confirmation:** No. -### Jupyter notebook reads - -For Jupyter notebooks (`.ipynb`), `read_file` parses the notebook JSON and returns a structured, model-readable notebook view instead of raw JSON. The rendered output includes the notebook language, ordered cells, cell IDs, source, and summarized outputs. - -Notebook cells can then be edited with `notebook_edit`. The model should use the cell IDs shown by `read_file` when targeting a cell. - -`offset` and `limit` are not supported for `.ipynb` files. Notebook reads are treated as structured full-file reads; if the rendered notebook output is internally truncated because it is too large, `notebook_edit` will reject cell-level edits and ask you to reduce outputs or split the notebook before editing. - -## 3. `notebook_edit` (NotebookEdit) - -`notebook_edit` edits Jupyter notebook (`.ipynb`) files safely at the cell level. Use it instead of `edit` or `write_file` when changing notebook cells. - -- **Tool name:** `notebook_edit` -- **Display name:** NotebookEdit -- **File:** `notebook-edit.ts` -- **Parameters:** - - `notebook_path` (string, required): The absolute path to the `.ipynb` file. - - `cell_id` (string, optional): The target cell ID shown by `read_file`. Required for `replace` and `delete`. For `insert`, the new cell is inserted after this cell; if omitted, the new cell is inserted at the beginning. - - `new_source` (string, optional): The new cell source for `replace` and `insert`. Not required for `delete`. - - `cell_type` (`code` or `markdown`, optional): The cell type for inserted cells, or the target type when replacing a cell. - - `edit_mode` (`replace`, `insert`, or `delete`, optional): The edit operation. Defaults to `replace`. -- **Behavior:** - - Requires the notebook to have been read first with `read_file` in the current session. - - Targets cells using the IDs rendered by `read_file`, including real notebook cell IDs and displayed `cell-N` fallback IDs. - - Rejects ambiguous rendered cell IDs instead of guessing. - - For code cells, clears stale outputs and resets `execution_count` when source changes. - - Preserves notebook JSON formatting, line endings, encoding, and BOM where possible. - - Invalidates the prior-read state after structural edits when displayed fallback IDs can shift, so the next notebook edit requires a fresh `read_file`. -- **Output (`llmContent`):** A success message describing the edited notebook cell and, for non-delete operations, the updated source. -- **Confirmation:** Yes. Shows a notebook JSON diff and asks for user approval before writing, unless the current permission mode or rules auto-approve edit tools. - -### `notebook_edit` examples - -Replace a code cell: - -``` -notebook_edit( - notebook_path="/path/to/analysis.ipynb", - cell_id="load-data", - new_source="result = 41 + 1\nprint(result)" -) -``` - -Insert a markdown cell after an existing cell: - -``` -notebook_edit( - notebook_path="/path/to/analysis.ipynb", - edit_mode="insert", - cell_id="summary", - cell_type="markdown", - new_source="## Findings\n\nThe cleaned data is ready for modeling." -) -``` - -Delete a cell: - -``` -notebook_edit( - notebook_path="/path/to/analysis.ipynb", - edit_mode="delete", - cell_id="old-experiment" -) -``` - -## 4. `write_file` (WriteFile) +## 3. `write_file` (WriteFile) `write_file` writes content to a specified file. If the file exists, it will be overwritten. If the file doesn't exist, it (and any necessary parent directories) will be created. @@ -121,12 +56,11 @@ notebook_edit( - `content` (string, required): The content to write into the file. - **Behavior:** - Writes the provided `content` to the `file_path`. - - Does not write raw Jupyter notebook JSON. Use `notebook_edit` for `.ipynb` cell edits. - Creates parent directories if they don't exist. - **Output (`llmContent`):** A success message, e.g., `Successfully overwrote file: /path/to/your/file.txt` or `Successfully created and wrote to new file: /path/to/new/file.txt`. - **Confirmation:** Yes. Shows a diff of changes and asks for user approval before writing. -## 5. `glob` (Glob) +## 4. `glob` (Glob) `glob` finds files matching specific glob patterns (e.g., `src/**/*.ts`, `*.md`), returning absolute paths sorted by modification time (newest first). @@ -144,7 +78,7 @@ notebook_edit( - **Output (`llmContent`):** A message like: `Found 5 file(s) matching "*.ts" within /path/to/search/dir, sorted by modification time (newest first):\n---\n/path/to/file1.ts\n/path/to/subdir/file2.ts\n---\n[95 files truncated] ...` - **Confirmation:** No. -## 6. `grep_search` (Grep) +## 5. `grep_search` (Grep) `grep_search` searches for a regular expression pattern within the content of files in a specified directory. Can filter files by a glob pattern. Returns the lines containing matches, along with their file paths and line numbers. @@ -197,7 +131,7 @@ Search for a pattern with file filtering and custom result limiting: grep_search(pattern="function", glob="*.js", limit=10) ``` -## 7. `edit` (Edit) +## 6. `edit` (Edit) `edit` replaces text within a file. By default it requires `old_string` to match a single unique location; set `replace_all` to `true` when you intentionally want to change every occurrence. This tool is designed for precise, targeted changes and requires significant context around the `old_string` to ensure it modifies the correct location. @@ -214,7 +148,6 @@ grep_search(pattern="function", glob="*.js", limit=10) - `replace_all` (boolean, optional): Replace all occurrences of `old_string`. Defaults to `false`. - **Behavior:** - - Does not edit raw Jupyter notebook JSON. Use `notebook_edit` for `.ipynb` cell edits. - If `old_string` is empty and `file_path` does not exist, creates a new file with `new_string` as content. - If `old_string` is provided, it reads the `file_path` and attempts to find exactly one occurrence unless `replace_all` is true. - If the match is unique (or `replace_all` is true), it replaces the text with `new_string`. diff --git a/docs/developers/tools/introduction.md b/docs/developers/tools/introduction.md index 2a6b3e2faeb..1dafb14c885 100644 --- a/docs/developers/tools/introduction.md +++ b/docs/developers/tools/introduction.md @@ -45,7 +45,6 @@ Qwen Code's built-in tools can be broadly categorized as follows: - **[File System Tools](./file-system.md):** For interacting with files and directories (reading, writing, listing, searching, etc.). - **[Shell Tool](./shell.md) (`run_shell_command`):** For executing shell commands. -- **[Monitor Tool](./monitor.md) (`monitor`):** For running long-lived shell commands that stream output back as background task notifications. - **[Web Fetch Tool](./web-fetch.md) (`web_fetch`):** For retrieving content from URLs. - **[Multi-File Read Tool](./multi-file.md) (`read_many_files`):** A specialized tool for reading content from multiple files or directories, often used by the `@` command. - **[Memory Tool](./memory.md) (`save_memory`):** For saving and recalling information across sessions. diff --git a/docs/developers/tools/monitor.md b/docs/developers/tools/monitor.md deleted file mode 100644 index c004552aa14..00000000000 --- a/docs/developers/tools/monitor.md +++ /dev/null @@ -1,154 +0,0 @@ -# Monitor Tool (`monitor`) - -This document describes the `monitor` tool for Qwen Code. - -## Description - -Use `monitor` to start a long-running shell command that streams stdout and -stderr lines back to the agent as background task notifications. It is intended -for watch-style commands where new output matters over time, such as tailing -logs, watching build output, polling a health endpoint, or observing file -changes. - -The monitor runs in the background, so the agent can continue working while -events arrive. Each non-empty output line becomes a notification event, subject -to throttling. - -### Arguments - -`monitor` takes the following arguments: - -- `command` (string, required): The shell command to run and monitor. -- `description` (string, optional): A brief description of what the monitor is - watching. The display text is truncated to 80 characters. -- `max_events` (number, optional): Stop after this many notification events. - Must be a positive integer. Defaults to `1000`; maximum `10000` (values - outside this range are rejected, not silently clamped). -- `idle_timeout_ms` (number, optional): Stop if the command produces no output - for this many milliseconds. Must be a positive integer. Defaults to `300000` - (5 minutes); maximum `600000` (10 minutes), and values outside this range are - rejected. -- `directory` (string, optional): An absolute path to run the command in. Must - resolve (after symlink canonicalization) inside one of the registered - workspace directories, and must not be inside the user-skills directory. If - omitted, Qwen Code uses the project root. - -## How to use `monitor` with Qwen Code - -The model chooses the `monitor` tool when it needs to observe a process over -time instead of collecting a single command result. A successful invocation -returns a monitor ID, the command, the event limit, and the idle timeout. - -Usage: - -``` -monitor(command="tail -f logs/app.log", description="app log stream") -``` - -Monitor output is visible in the conversation as task notifications. You can -also inspect running and completed monitors with `/tasks` or the interactive -Background tasks dialog. - -To stop a running monitor, use the `task_stop` tool with the monitor ID: - -``` -task_stop(task_id="mon_abc123def4567890") -``` - -## `monitor` examples - -Watch an application log: - -``` -monitor( - command="tail -f logs/app.log", - description="application log stream", - max_events=200 -) -``` - -Monitor a dev server or build watcher: - -``` -monitor( - command="npm run build -- --watch", - description="watch build output", - idle_timeout_ms=600000 -) -``` - -Poll a local health endpoint: - -``` -monitor( - command="while true; do curl -s http://localhost:8080/health; sleep 5; done", - description="local health check", - max_events=120 -) -``` - -Run from a specific workspace directory: - -``` -monitor( - command="npm run dev", - description="frontend dev server", - directory="/absolute/path/to/workspace/packages/web" -) -``` - -## Monitor vs. background shell commands - -Use `monitor` when the agent needs to react to streaming output while the -command keeps running. Use `run_shell_command` instead when you need a one-shot -result or the complete command output. - -| Need | Use | -| :----------------------------------------------------- | :--------------------------------------- | -| Watch logs, build output, or periodic status updates | `monitor` | -| Run a one-time command and read the full output | `run_shell_command(is_background=false)` | -| Start a daemon that does not produce meaningful output | `run_shell_command(is_background=true)` | - -Do not add `&` to monitor commands. A trailing `&`, such as -`tail -f log &`, is stripped because the monitor manages backgrounding itself. -A non-final `&`, such as `cmd1 & cmd2`, is rejected outright; restructure such -commands without backgrounding instead. - -## Important notes - -- **Auto-stop behavior:** Monitors stop automatically when they reach - `max_events`, when `idle_timeout_ms` elapses without output, or when the - underlying command exits on its own. A monitor's status reflects the - command's outcome, not a tool error: a clean exit (`code 0`) becomes - `completed`, a non-zero exit code becomes `failed` with message - `Exit code N`, and termination by signal becomes `failed` with message - `Killed by signal SIG`. Commands cannot be interactive because stdin is - closed. When a monitor stops, Qwen Code sends `SIGTERM` to the command's - process group and escalates to `SIGKILL` after about 200 ms. On Windows, it - uses `taskkill /f /t`. If the Qwen Code process itself is hard-killed, - crashes, or runs out of memory, the detached process group is not cleaned up - automatically; recover by stopping the monitor with `task_stop` before exit - or by terminating the process group manually. -- **Concurrency limit:** Qwen Code allows up to 16 running monitors per CLI - session as a single shared pool. Monitors started by subagents count against - the same cap as monitors started by the main agent. Stop an existing monitor - before starting another if the limit is reached. -- **Output handling:** Stdout and stderr are merged into a single notification - stream with no stream prefix. Empty lines are ignored, ANSI color and control - characters are stripped, and individual lines longer than 2000 characters are - truncated. High-volume output is rate-limited with a burst of 5 events and - about 1 event per second after that; lines beyond the rate limit are dropped, - not buffered. Monitor output flows into the agent context as - `` content. Structural notification tags are defanged, but - the model still reads each line's text, so avoid monitoring streams that - external parties can write to unless you trust the model to ignore embedded - instructions. -- **Permissions:** `monitor` has its own permission boundary and permission - rules, such as `Monitor(git status)`. Read-only commands are automatically - allowed; commands that modify state require user approval; commands containing - command substitution (`$(...)`, backticks, `<(...)`, or `>(...)`) are rejected - outright. The `tools.core` and `tools.exclude` settings for - `run_shell_command` do not apply to `monitor`. -- **Workspace restriction:** The optional `directory` must be an absolute path - that resolves inside a registered workspace directory and outside the - user-skills directory. Symlinks that point outside the workspace are rejected. diff --git a/docs/e2e-tests/2026-05-18-qwen-memory-benchmark-report.md b/docs/e2e-tests/2026-05-18-qwen-memory-benchmark-report.md deleted file mode 100644 index 1a7aaf32533..00000000000 --- a/docs/e2e-tests/2026-05-18-qwen-memory-benchmark-report.md +++ /dev/null @@ -1,286 +0,0 @@ -# Qwen Code Runtime Memory Benchmark Report - -Date: 2026-05-18 - -## Summary - -This report records local memory benchmarks for Qwen Code runtime behavior. It -compares Qwen Code across models and compares Qwen Code with Claude Code on the -same task shapes where equivalent model endpoints were available. - -The headline result is consistent across the latest matrix (single run per cell, -not statistically repeated): - -- Qwen Code process-tree RSS peak: about `852-1062 MiB` (`0.83-1.04 GiB`). -- Claude Code process-tree RSS peak: about `279-366 MiB` (`0.27-0.36 GiB`). -- Qwen Code was about `2.3x-3.6x` higher in the tested - non-interactive CLI task benchmarks. - -Note: process-tree RSS includes MCP child processes (~350 MiB overhead on the -Qwen side). This inflates the absolute numbers but the relative comparison -remains informative since both CLIs were measured the same way. - -The difference reproduced in small PR review, code navigation, and synthetic -diff workloads. It is therefore unlikely to be explained only by one large PR -or by one model provider. - -This report is intended to make the current performance investigation visible: -what has been measured, what conclusion is already supported, what remains -unknown, and what diagnostics should be added next. - -## Test Environment - -| Item | Value | -| --------------------------------------------- | ------------------------------------------ | -| Date | 2026-05-18 | -| Platform | macOS local development machine | -| Qwen Code version | `0.15.11` | -| Qwen Code binary | PATH-resolved `qwen` binary | -| Claude Code version used in the latest matrix | `2.1.129` | -| Claude Code binary used in the latest matrix | PATH-resolved `claude` binary | -| Node.js version | v22.x (default system install) | -| Sampling method | External `ps` RSS sampling once per second | -| Headline metric | Process-tree RSS peak | - -Process-tree RSS is used as the headline metric because Qwen Code launches a -root wrapper and a child Node/Qwen worker. Looking only at the root process can -understate the memory footprint seen by users. - -Temporary CLI config directories were used for matrix runs so the benchmarks -did not depend on global CLI state. - -## Benchmark Artifacts - -Five local reports were produced before this consolidated report: - -1. Qwen Code PR review memory run. -2. Qwen Code model comparison run. -3. Strict Qwen Code vs Claude Code comparison with `pai/glm-5`. -4. Qwen Code vs Claude Code, two CLIs by two models. -5. Qwen Code vs Claude Code, five-case matrix. - -This consolidated report covers the conclusions and headline metrics from all -five reports. It does not embed every raw sample row, terminal transcript, or -temporary runner artifact. Those raw artifacts stayed in local `tmp/` -directories because they are experiment outputs rather than stable repository -fixtures. - -The latest matrix is the strongest evidence because it covers multiple task -shapes rather than only one PR review workload. - -## Preliminary Conclusion - -The current data is strong enough to say that Qwen Code has a higher runtime -memory footprint than Claude Code in these local non-interactive CLI task -benchmarks. It is not strong enough to name one final root cause yet. - -The leading explanation is a Qwen Code runtime/path difference rather than a -model provider difference: - -- the gap reproduces with both `pai/glm-5` and `qwen3.6-plus`; -- the gap reproduces in small PR and code-navigation tasks, not only in large - diff tasks; -- Qwen Code repeatedly sends or accounts for more tokens than Claude Code for - similar work; -- Qwen Code's largest observed component is the child Node/Qwen worker process, - which points toward task-time process footprint, module loading, context - assembly, live history, tool-result retention, or subagent/saved-output - paths. - -The most useful next measurement is therefore not another external RSS-only -run. The next measurement should split RSS into V8 heap, native memory, -session/history size, retained tool-result size, and subagent/process-tree -activity. - -## Initial Cause Analysis - -The benchmark does not yet prove one root cause, but it does narrow the likely -problem area. - -| Signal | What it suggests | What it does not prove | -| -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------- | -| Qwen remains near `1 GiB` in small PR and code-navigation cases | A high non-interactive task-time runtime cost is likely involved | It does not identify whether the footprint is V8 heap, native memory, module loading, or retained state | -| Diff size from 100 KiB to 5 MiB does not scale linearly with RSS | Raw diff bytes alone are probably not the primary driver | Large outputs can still amplify memory in real PR review flows | -| Qwen uses more tokens than Claude in every matrix cell | Qwen likely constructs or retains larger prompt/context/tool-result state for similar work | Token count is not the same as process memory and may be an effect rather than the cause | -| Tool call counts are similar, and Claude sometimes uses more turns/tool calls with lower RSS | A longer tool-call chain is unlikely to be the main explanation by itself | Tool output size and retention still need to be measured | -| Earlier large PR runs showed saved-output recovery and subagent amplification | Tool-output truncation and saved-output paths are likely heavy-workload amplifiers | They do not explain the entire small-task execution footprint | - -The current best explanation is therefore: - -1. **Task-time runtime cost first**: Qwen Code likely initializes or retains - more runtime state during non-interactive CLI task execution than Claude - Code. This may include agent runtime, tool registry, provider adapters, - session services, or UI/history structures that are not strictly needed for - a short non-interactive task. -2. **Context/tool-result volume second**: Qwen Code appears to carry larger - model-facing or session-facing context for similar work. The token gap makes - context assembly, tool result normalization, and history retention important - suspects. -3. **Large-output amplification third**: Large PR review can trigger additional - saved-output and subagent paths. These are probably not the only cause, but - they can make memory and token pressure worse in realistic review tasks. - -The next diagnostic run should answer where the `~1 GiB` sits: - -- high immediately after startup: module/runtime startup cost; -- jumps after tool execution: tool-output retention or result normalization; -- jumps during request assembly: context construction or duplicated histories; -- grows after streaming/compression: response retention or compression state; -- mostly RSS outside V8 heap: native buffers, loaded modules, or external - memory. - -## Latest Matrix - -The latest benchmark ran: - -- 2 CLIs: Qwen Code and Claude Code. -- 2 model labels: `pai/glm-5` and `qwen3.6-plus`. -- 5 cases: - - small PR review: PR `#4268`, one-line change - - code navigation: `rg` plus `sed` on compression-related files - - synthetic local diff, about 100 KiB - - synthetic local diff, about 1 MiB - - synthetic local diff, about 5 MiB - -All 20 runs exited `0` with no timeout. - -## Matrix Results - -| Case | Model | Qwen tree peak | Claude tree peak | Qwen / Claude | -| ---------------- | -------------- | -------------: | ---------------: | ------------: | -| small PR `#4268` | `pai/glm-5` | 1032.7 MiB | 357.8 MiB | 2.89x | -| small PR `#4268` | `qwen3.6-plus` | 852.2 MiB | 365.5 MiB | 2.33x | -| code navigation | `pai/glm-5` | 993.1 MiB | 359.6 MiB | 2.76x | -| code navigation | `qwen3.6-plus` | 996.9 MiB | 349.0 MiB | 2.86x | -| diff 100 KiB | `pai/glm-5` | 1012.1 MiB | 350.8 MiB | 2.89x | -| diff 100 KiB | `qwen3.6-plus` | 1001.1 MiB | 336.2 MiB | 2.98x | -| diff 1 MiB | `pai/glm-5` | 1008.3 MiB | 278.8 MiB | 3.62x | -| diff 1 MiB | `qwen3.6-plus` | 1003.3 MiB | 340.5 MiB | 2.95x | -| diff 5 MiB | `pai/glm-5` | 858.8 MiB | 323.2 MiB | 2.66x | -| diff 5 MiB | `qwen3.6-plus` | 1062.0 MiB | 331.2 MiB | 3.21x | - -Average process-tree RSS peak by case: - -| Case | Avg Qwen tree peak | Avg Claude tree peak | -| ---------------- | -----------------: | -------------------: | -| small PR `#4268` | 942.5 MiB | 361.6 MiB | -| code navigation | 995.0 MiB | 354.3 MiB | -| diff 100 KiB | 1006.6 MiB | 343.5 MiB | -| diff 1 MiB | 1005.8 MiB | 309.6 MiB | -| diff 5 MiB | 960.4 MiB | 327.2 MiB | - -## Runtime And Token Signals - -The same matrix also showed Qwen Code using more model-side tokens in every -tested case. - -Selected examples: - -| Case | Model | CLI | Duration | Turns | Total tokens | Tool calls | -| --------------- | -------------- | ------ | -------: | ----: | -----------: | ---------: | -| small PR | `pai/glm-5` | Qwen | 25.2s | 2 | 32,567 | 3 | -| small PR | `pai/glm-5` | Claude | 21.1s | 4 | 7,899 | 3 | -| code navigation | `qwen3.6-plus` | Qwen | 25.2s | 2 | 38,151 | 3 | -| code navigation | `qwen3.6-plus` | Claude | 46.9s | 6 | 25,861 | 5 | -| diff 100 KiB | `qwen3.6-plus` | Qwen | 16.5s | 3 | 57,185 | 2 | -| diff 100 KiB | `qwen3.6-plus` | Claude | 17.2s | 3 | 6,377 | 2 | -| diff 5 MiB | `pai/glm-5` | Qwen | 23.2s | 2 | 38,574 | 2 | -| diff 5 MiB | `pai/glm-5` | Claude | 9.8s | 3 | 5,285 | 2 | - -This token gap does not prove that token volume is the memory root cause, but it -does suggest that context assembly, tool result retention, or response -normalization should be measured alongside RSS and V8 heap statistics. - -## Token Usage Analysis - -The token gap is one of the strongest clues, but it needs internal request -metrics before it can be treated as a root cause. - -What the data supports today: - -- Qwen Code used more total tokens than Claude Code in every matrix cell. -- The gap appears even when tool-call counts are similar. -- Claude sometimes used more turns or tool calls while still using less memory. - -What this suggests: - -- The token delta is unlikely to come only from a longer tool-call chain. -- Qwen may be carrying larger static prompt/context state, larger tool schemas, - larger serialized tool results, or more retained conversation/session content. -- Large-output flows may add another layer through truncation, saved-output - recovery, or subagent paths. - -What is still missing: - -- per-request input token breakdown; -- system prompt and tool schema token sizes; -- retained message and tool-result sizes before each model request; -- whether large outputs are retained in multiple places, such as model history, - UI history, session recording, or saved-output storage. - -Those missing metrics are why the next step should add internal diagnostics -rather than only repeat the external RSS benchmark. - -## Earlier Large PR Review Signal - -An earlier strict PR review benchmark used PR `#4186` and showed the same broad -shape: - -| Model | CLI | Process-tree RSS peak | -| -------------- | ----------- | --------------------: | -| `pai/glm-5` | Qwen Code | 1000.7 MiB | -| `pai/glm-5` | Claude Code | 349.0 MiB | -| `qwen3.6-plus` | Qwen Code | 1095.8 MiB | -| `qwen3.6-plus` | Claude Code | 341.1 MiB | - -That earlier run was not enough by itself because a large PR can trigger unusual -tool-output and saved-output paths. The latest five-case matrix makes the -finding stronger because small PR and code-navigation tasks also reproduce the -gap. - -## Working Hypothesis - -The current evidence supports these hypotheses, in priority order: - -1. Qwen Code has a higher non-interactive task-time process footprint than - Claude Code. The Qwen child Node worker was typically the largest process in - local sampling, often around `0.7-0.8 GiB`. -2. Model choice is not the main explanation. Both `pai/glm-5` and - `qwen3.6-plus` showed the same broad Qwen-vs-Claude gap. -3. Large diff size alone is not the main explanation. The synthetic diff size - did not scale linearly from 100 KiB to 5 MiB, likely because tool-output - truncation caps how much output reaches the model. -4. Context/tool-result handling is still a likely contributor. Qwen Code used - more tokens than Claude Code in every matrix cell, and earlier large-PR runs - showed saved tool-output recovery and subagent amplification paths. -5. The next diagnostic layer should separate V8 heap, native RSS, loaded - module/runtime startup cost, session history, UI history, tool-result - retention, and subagent activity. External RSS alone cannot distinguish - those causes. - -## Caveats - -- These are single runs per matrix cell, not repeated statistical samples. -- RSS is external process RSS. It cannot distinguish V8 heap, native buffers, - module loading, retained tool output, UI state, or session history. -- Claude Code and Qwen Code use different runtime implementations and protocol - adapters, even when the model labels are the same. -- The benchmark was run locally on macOS. Linux servers should be tested before - drawing deployment-specific conclusions. - -## Recommended Follow-Up Measurements - -The next local investigation branch should add or use diagnostics for: - -- `process.memoryUsage()` before and after startup, tool execution, streaming, - compression, and session finalization. -- V8 heap statistics and heap spaces. -- Active handles and requests. -- Session message count and approximate retained character/token volume. -- Tool result count, total retained tool-result size, largest tool-result size, - and whether large outputs are retained by UI history or model history. -- Subagent count and child process/process-tree RSS. -- Tool-output truncation and saved-output recovery events. - -These measurements should be collected with the same benchmark matrix so the -current RSS comparison can be connected to internal Qwen Code state. diff --git a/docs/e2e-tests/2026-05-19-oom-reproduction-report.md b/docs/e2e-tests/2026-05-19-oom-reproduction-report.md deleted file mode 100644 index 8716e208f56..00000000000 --- a/docs/e2e-tests/2026-05-19-oom-reproduction-report.md +++ /dev/null @@ -1,437 +0,0 @@ -# OOM 压力测试与长任务 Replay 报告 - -**日期**: 2026-05-19 -**分支**: `codex/memory-diagnostics-local-run` -**测试人**: yiliang114 -**结论**: 成功复现并定位根因。v0.15.7 (#3735) 引入的 auto-compaction 使 `structuredClone` -调用频率倍增，在高 heap 压力时形成正反馈死循环导致 OOM。真实 debug 日志完整佐证了该机制。 - ---- - -## 一、背景 - -多个 issue（#4309, #4276, #4185, #4315, #4322, #2868）报告 qwen-code 在长会话中出现 V8 heap OOM crash： - -``` -FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory -``` - -用户报告的崩溃特征： -| Issue | 崩溃时 Heap | 运行时长 | 平台 | -|-------|------------|---------|------| -| #4276 | 4014 MB | ~110 分钟 | Linux x64 | -| #4315 | 2027 MB | ~19.6 小时 | macOS (默认 2GB limit) | -| #4322 | 4023 MB | ~7 小时 | Windows | -| #2868 | 2035 MB | ~1.7 分钟 | Linux | -| #4309 | 7020 MB | 未知 | Windows (设了 8GB limit 仍崩) | - ---- - -## 二、方法论修正 - -本报告区分两类测试： - -1. **低 heap 压力测试**：通过降低 `--max-old-space-size` 放大问题，用于快速定位 - “history 很大时整段复制导致瞬时峰值”的代码路径。它是诊断工具，不等价于用户真实 - 4G/8G OOM 复现。 -2. **默认 heap 长任务 replay**：不设置 `NODE_OPTIONS`，使用真实 JSONL 历史恢复并 - 继续执行 review 任务，同时从进程外采样 process-tree RSS。这类结果才用于判断 - 用户侧实际内存量级。 - -因此，低 heap 结果不能单独作为“真实 OOM 已修复”的证明。它只能说明某条路径在 -history 足够大时会产生峰值放大，需要再用默认 heap 长任务验证。 - -## 三、低 heap 压力测试条件 - -| 参数 | 值 | -| ------------------------ | ------------------------------------------------------------ | -| CLI 版本 | 0.15.11 (从 `codex/memory-diagnostics-local-run` 分支 build) | -| Model | `qwen3.6-plus` (128K context window) | -| Heap limit | `--max-old-space-size=512` | -| Heap-pressure safety net | **禁用** (HEAP_PRESSURE_COMPRESSION_RATIO 设为 99.0) | -| 操作模式 | YOLO + 自动化多轮 Read 文件任务 | -| 工作目录 | qwen-code monorepo (3538 .ts files, 1.26M lines) | - -### 关键配置修改 - -`packages/core/src/core/geminiChat.ts` 中将 heap-pressure compaction 阈值从 0.7 改为 99.0（使其永远不触发），模拟 #4186 修复前的状态。 - ---- - -## 四、低 heap 压力测试结果 - -### 崩溃时间线 - -``` -[21:26:59] #1 RSS:193.6MB Ctx:0% → Read geminiChat.ts (1500 行) -[21:27:46] #2 RSS:270.4MB Ctx:4.2% → Read agent.ts -[21:28:32] #3 RSS:397.5MB Ctx:4.3% → grep + Read 3 个文件 -[21:29:18] #4 RSS:452.7MB Ctx:5.7% → Read slashCommandProcessor.ts -[21:30:04] #5 RSS:515.0MB Ctx:5.9% → Read chatCompressionService.ts -[21:30:50] #6 RSS:649.1MB Ctx:4.0% ← TOKEN COMPACTION 触发 (5.9%→4.0%) - RSS 反增 134MB (structuredClone 峰值) -[21:31:36] #7 RSS:666.7MB Ctx:3.2% ← 再次 compaction, RSS 继续涨 -[21:32:22] CRASH — FATAL ERROR: Ineffective mark-compacts near heap limit -``` - -**总耗时**: ~5.5 分钟，7 轮任务后崩溃。 - -这证明在受限 heap 下，长 history + compaction/history clone 可以触发 V8 heap OOM。 -但该结果不代表默认 heap 下的真实用户 OOM 已经被完整复现。 - -### 更大 heap 的 synthetic 复现 - -为避免只依赖 512 MiB 低 heap 结论，补充了更大 heap 的 synthetic runtime -pressure 测试。该测试不调用模型，而是构造类似长 review/subagent 任务的历史： - -- root review turns: 10 -- subagent calls: 30 -- subagent transcript records: 780 -- retained tool result bytes: 193,986,560 -- serialized history bytes: 195,620,061 -- pressure mode: retained `structuredClone(history)` copies - -| Heap limit | Clone pressure | 结果 | 关键 GC / stack | -| ---------- | -----------------: | ---------------------------------------- | ------------------------------------------------------------ | -| 2 GiB | 8 retained clones | 未崩溃，RSS 2.42 GiB，heap used 1.87 GiB | 接近 heap limit | -| 2 GiB | 10 retained clones | OOM | `Reached heap limit`, `ValueDeserializer`, `StructuredClone` | -| 4 GiB | 20 retained clones | OOM | `Reached heap limit`, `ValueDeserializer`, `StructuredClone` | - -2 GiB 复现的 GC 摘要： - -``` -Mark-Compact 2042.9 (2081.9) -> 2042.9 (2081.1) MB -Mark-Compact 2048.9 (2087.2) -> 2048.9 (2087.2) MB -FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory -... -node::worker::(anonymous namespace)::StructuredClone -``` - -4 GiB 复现的 GC 摘要： - -``` -Mark-Compact 4082.5 (4126.8) -> 4082.5 (4126.3) MB -Mark-Compact 4095.1 (4139.0) -> 4095.1 (4139.0) MB -FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory -... -node::worker::(anonymous namespace)::StructuredClone -``` - -这组结果比 512 MiB 压力测试更接近用户报告的 2 GiB / 4 GiB heap OOM： -只要 history 中保留足够多的大 tool result / subagent transcript，对整段 history -做 retained 或瞬时 clone 都可以在 2-4 GiB heap 下触发 V8 OOM。它仍然是 synthetic -复现，不等价于完整业务长任务 replay，但能直接证明问题不是“小 heap 人为制造”的。 - -### 崩溃时 GC 状态 - -``` -[41381:0x130008000] 342468 ms: Mark-Compact 508.6 (526.7) -> 507.0 (526.9) MB, - pooled: 1 MB, 86.42 / 0.00 ms (average mu = 0.175, current mu = 0.150) - task; scavenge might not succeed - -[41381:0x130008000] 342568 ms: Mark-Compact 509.1 (526.9) -> 507.1 (528.2) MB, - pooled: 0 MB, 93.79 / 0.12 ms (average mu = 0.121, current mu = 0.068) - allocation failure; scavenge might not succeed - -FATAL ERROR: Ineffective mark-compacts near heap limit -Allocation failed - JavaScript heap out of memory -``` - -Mark-Compact 只能回收 1-2 MB（几乎所有对象都是 reachable），证明内存确实被合法持有的对象占满。 - ---- - -## 五、默认 heap 长任务 replay - -为了避免低 heap 结论过度外推，补充了默认 heap 的真实 JSONL replay： - -- 不设置 `NODE_OPTIONS` -- 不启用内部 runtime profiler，避免采样器自身影响 heap -- 每个 CLI 从同一份 rewound JSONL 复制出 fresh session -- 使用临时 `QWEN_HOME`，禁用 MCP 和 hooks，避免本地全局配置污染 -- 只用进程外采样统计 process-tree RSS - -| CLI | 结果 | 时长 | Tree RSS 峰值 | Root RSS 峰值 | Worker RSS 峰值 | 备注 | -| -------------------- | ---- | -----: | ------------: | ------------: | --------------: | ----------------------------------------------------------- | -| installed `qwen` | 成功 | 167.3s | 838.0 MiB | 230.2 MiB | 566.3 MiB | 第一次 fresh run 遇到模型服务端错误，未纳入结论；retry 成功 | -| local rebuilt bundle | 成功 | 106.3s | 527.5 MiB | 182.1 MiB | 345.4 MiB | 包含本地 clone 热路径修复 | - -默认 heap replay 的结论： - -1. 当前这份 review JSONL 可以稳定跑出数百 MiB 到约 0.8 GiB 的 process-tree RSS， - 但没有复现 4G/8G OOM。 -2. 本地 rebuilt bundle 在同起点 replay 上的峰值低于 installed CLI，说明减少 - history clone 热路径有实际收益。 -3. 这还不能证明所有用户 OOM 都已解决。真实 4G/8G OOM 仍需要更长任务、更大 - tool-result 累积，或保留 MCP/tool schema 压力的 replay 继续验证。 - -## 六、根因分析 - -### OOM 的三层机制 - -``` -┌─────────────────────────────────────────────────────────┐ -│ Layer 3: V8 Heap Limit (512MB/2GB/4GB) │ ← 用户最终撞到这里 -├─────────────────────────────────────────────────────────┤ -│ Layer 2: structuredClone() 峰值放大 (瞬时 ~2x) │ ← 直接诱因 -├─────────────────────────────────────────────────────────┤ -│ Layer 1: History 中 tool result 累积 (线性增长) │ ← 基础增长 -├─────────────────────────────────────────────────────────┤ -│ Layer 0: Token compaction 触发时机 │ ← 控制点 -└─────────────────────────────────────────────────────────┘ -``` - -### 精确崩溃路径 - -``` -sendMessage() - → tryCompress() - → heapPressureRatio < threshold (safety net disabled) - → ChatCompressionService.compress() - → chat.getHistory(true) - → structuredClone(this._history) ← 峰值分配！ - → V8 需要额外 ~N MB 来容纳 clone - → 如果 existing heap + N > limit → OOM -``` - -### 关键证据 - -| 观察 | 含义 | -| --------------------------------------- | ---------------------------------------------- | -| Task #5→#6: Context 5.9%→4.0% (降了) | Token compaction **成功执行**了 | -| Task #5→#6: RSS 515→649 MB (涨了 134MB) | Compaction 过程的 `structuredClone` 制造了峰值 | -| GC 只能回收 1-2 MB | 所有对象都是 live（history + clone 都在） | -| #4309 设 8GB limit 仍崩 | history 足够大时，clone 峰值可超任何 limit | - -需要注意：以上证据来自低 heap 压力测试和 issue 现象的组合推断。默认 heap replay -目前支持”clone 热路径会显著影响峰值 RSS”，但尚未单独复现 4G/8G OOM。 - -### 为什么 128K context window 更容易触发 - -- 128K × 70% = ~90K tokens 触发 compaction -- 大 context window (1M) 的 70% = 700K tokens，几乎不会触发 -- **compaction 越频繁 → structuredClone 越频繁 → OOM 风险越高** -- DeepSeek 等未配置 contextWindowSize 的模型默认 128K，更易触发 - ---- - -## 六.5、真实运行日志佐证 - -以下日志提取自本地 crash session 的 debug 输出。为避免泄露本地路径和 session id， -报告只保留时间线和关键日志内容。 - -该 session 启动于 `2026-05-19T13:26:35Z` (本地 21:26:35)，crash 于 -`2026-05-19T13:32:10Z` (本地 21:32:10)。 - -### Heap Pressure 与 Auto-Compaction 事件时间线 - -``` -13:29:43 [WARN] Heap pressure at 74.9%; attempting auto-compaction before token threshold. -13:30:06 [DEBUG] [FILE_READ_CACHE] clear after auto tryCompress ← compaction #1 执行成功 -13:30:13 [WARN] Heap pressure at 70.7%; attempting auto-compaction before token threshold. - ← 刚压完 heap 从 74.9% 仅降到 70.7%，仍超阈值，立即再次尝试 -13:30:52 [DEBUG] Heap pressure at 86.0%; skipping heap-pressure auto-compaction during cooldown. - ← 30s cooldown 期间拒绝执行 -13:30:56 [WARN] Heap pressure at 85.3%; attempting auto-compaction before token threshold. - ← cooldown 过期，heap 已升至 85.3% -13:31:21 [DEBUG] [FILE_READ_CACHE] clear after auto tryCompress ← compaction #2 执行成功 -13:31:37 [WARN] Heap pressure at 88.8%; attempting auto-compaction before token threshold. - ← 压完后 heap 反弹至 88.8% -13:32:09 [DEBUG] Heap pressure at 90.2%; skipping heap-pressure auto-compaction during cooldown. - ← heap 已达 90.2%，cooldown 中无法执行 -13:32:10 ← 日志终止（进程 OOM crash） -``` - -### 日志证据解读 - -| 日志观察 | 含义 | -| ------------------------------------------------------------------------------------- | --------------------------------------------------------- | -| 2.5 分钟内触发 **4 次** heap-pressure auto-compaction 尝试（另有 2 次 cooldown 拒绝） | #3735 引入的 `tryCompress` 在高压时频繁触发 | -| 每次 compaction 执行后 heap 占比仍 >70% | `structuredClone()` 制造的临时峰值抵消了压缩收益 | -| 74.9% → 70.7% → 86% → 85.3% → 88.8% → 90.2% → crash | 正反馈循环：压缩→clone 峰值→heap 更高→再压缩→更高 | -| 日志在 90.2% 后 1 秒内断裂 | 下一次 `getHistory(true)` 的 `structuredClone()` 瞬间超限 | -| `[FILE_READ_CACHE] clear after auto tryCompress` 出现 2 次 | 证实 compaction 确实走了完整的 compress → setHistory 路径 | - -### 正反馈死循环机制 - -``` -heap 占比高 (>70%) - → 触发 heap-pressure auto-compaction - → tryCompress() 内部调用 getHistory(true) - → structuredClone(this._history) ← 瞬时 heap 峰值 +30~40% - → compaction 成功，释放旧 history - → 但 clone 峰值已经把 heap 推高到更危险的水位 - → 下一轮 send 继续累积 - → heap 占比更高 → 更频繁触发 → crash -``` - ---- - -## 六.6、版本归因：为什么 0.15.7 ~ 0.15.11 期间 OOM 报告增多 - -### 关键 commit 时间线 - -| 版本 | PR | 改动 | 对 `structuredClone` 调用频率的影响 | -| ------------ | ---------------------------------------------------- | ----------------------------------------------------------------------------------- | ----------------------------------- | -| **v0.15.6** | — | `getHistory(true)` 仅在 `sendMessage` 入口调用 1 次 | 基线：每次 send 1 次 clone | -| **v0.15.7** | **#3735** `auto-compact subagent context` | 将 `tryCompress()` 下沉到 `GeminiChat`，**每次 send 前**先执行一次 compaction 检查 | **+1 次**：send 前 compress 检查 | -| **v0.15.10** | **#3879** `reactive compression on context overflow` | 当 provider 返回 context overflow 时，再次触发 `tryCompress()` + `getHistory(true)` | **+1~2 次**：overflow retry 路径 | -| **v0.15.10** | **#3985** `harden reactive compression` | 强化 reactive compression 重试逻辑 | 同上 | - -### v0.15.6 vs v0.15.11 的 `getHistory(true)` 调用点对比 - -**v0.15.6** (2 处)： - -``` -L367: const requestContents = this.getHistory(true); ← send 构造 request -L618: const recoveryContents = self.getHistory(true); ← MAX_TOKENS escalation (极少触发) -``` - -**v0.15.11** (5 处)： - -``` -L467: ChatCompressionService.compress() 内部调用 ← #3735: 每次 send 前的 auto-compact -L574: requestContents = this.getHistory(true); ← send 构造 request -L724: reactive tryCompress() 内部调用 ← #3879: context overflow 后 retry -L739: requestContents = self.getHistory(true); ← #3879: retry 构造新 request -L943: const recoveryContents = self.getHistory(true); ← MAX_TOKENS escalation -``` - -### 最坏路径：一次 send 可触发 4 次 `structuredClone` - -``` -sendMessage() - → tryCompress() ← #3735: getHistory(true) [clone #1] - → getHistory(true) ← 构造 request [clone #2] - → API 返回 context overflow - → reactive tryCompress() ← #3879: getHistory(true) [clone #3] - → getHistory(true) ← retry request [clone #4] -``` - -### 结论 - -**#3735 (v0.15.7)** 是 OOM 频率显著上升的最可能触发因素（非唯一根因）——它使每次 -`sendMessage` 都会先跑一次 `tryCompress()`，而 `tryCompress` 内部通过 -`ChatCompressionService.compress()` → `chat.getHistory(true)` 做全量 `structuredClone`。 -在 history 较大时，这个 “先 clone 再判断是否需要压缩” 的设计让内存峰值从 ~1.3x 升至 ~2x+。 -注：issue history 显示 OOM 报告在 #3735 之前就已存在，但 #3735 大幅增加了 structuredClone -的调用频率，从而显著提高了 OOM 的触发概率。 - -**#3879 (v0.15.10)** 进一步恶化了问题——在已经处于 heap 边界时 (provider 返回 context overflow) -再触发一次全量 clone，使原本就危险的 session 更容易 crash。 - ---- - -## 七、#4186 修复效果验证（对比测试） - -启用 heap-pressure safety net (HEAP_PRESSURE_COMPRESSION_RATIO = 0.7) 后的对比测试： - -| 指标 | 禁用 safety net | 启用 safety net | -| --------------- | ------------------ | ------------------------- | -| OOM 发生 | 是（7 轮后 crash） | 否（持续运行 >10 分钟） | -| RSS 峰值 | 666 MB → crash | 555 MB → GC 回收到 280 MB | -| Compaction 触发 | 仅 token threshold | heap 70% 时提前触发 | -| Context 行为 | 5.9%→4.0%→crash | 22.7%→17.0%（安全回落） | - -**结论**: #4186 的 heap-pressure safety net 有效防止了 OOM，但它是一个**缓解**而非根治： - -- 如果 history 本身已经占了 heap 的 60%+，即使提前 compact，clone 的峰值仍然可能超限 -- 这解释了为什么 #4309 用户设了 8GB limit 后仍然 crash - ---- - -## 八、内存占用分布 - -基于测试中的 RSS 增长模式估算： - -| 内存位置 | 占比 | 增长特征 | -| -------------------------------- | ------ | --------------------------- | -| `this._history[]` (tool results) | 40-50% | 线性累积，每轮 +30-100MB | -| `structuredClone()` 临时拷贝 | 30-40% | 瞬时峰值，compaction 时出现 | -| V8 runtime (GC metadata, code) | ~15% | 基本恒定 | -| UI/logging/stream buffers | ~5% | 缓慢增长 | - ---- - -## 九、复现脚本与环境 - -### 自动化驱动脚本 - -```bash -#!/bin/bash -# /tmp/oom-simple-driver.sh -SESSION="$1" - -TASKS=( - "用 Read 工具完整读取 packages/core/src/core/geminiChat.ts" - "用 Read 工具完整读取 packages/core/src/tools/agent/agent.ts" - "用 grep -rn structuredClone packages/core/src 然后 Read 前 3 个文件" - "用 Read 完整读取 packages/cli/src/ui/hooks/slashCommandProcessor.ts" - "用 Read 完整读取 packages/core/src/services/chatCompressionService.ts" - "用 find packages/cli/src/ui/commands -name '*.ts' 然后逐一 Read" - "用 Read 完整读取 packages/core/src/core/turn.ts" - # ... 更多任务 -) - -i=0 -while true; do - TASK="${TASKS[$((i % ${#TASKS[@]}))]}" - i=$((i + 1)) - - QWEN_PID=$(ps aux | grep "dist/index.js" | grep -v grep | awk '{print $2}' | sort -rn | head -1) - RSS=$(ps -o rss= -p $QWEN_PID 2>/dev/null) - [ -z "$RSS" ] && { echo "CRASH after $((i-1)) tasks!"; exit 0; } - - RSS_MB=$(echo "scale=1; $RSS/1024" | bc) - CTX=$(tmux capture-pane -t "$SESSION:1" -p 2>/dev/null | grep -oE "[0-9]+\.[0-9]+% 已用" | tail -1) - echo "[$(date +%H:%M:%S)] #$i RSS:${RSS_MB}MB Ctx:$CTX | ${TASK:0:55}" - - tmux send-keys -t "$SESSION:1" C-u - sleep 0.2 - tmux send-keys -t "$SESSION:1" "$TASK" Enter - sleep 0.5 - tmux send-keys -t "$SESSION:1" Enter - sleep 45 -done -``` - -### 启动命令 - -```bash -# 1. 禁用 heap-pressure safety net -# geminiChat.ts: HEAP_PRESSURE_COMPRESSION_RATIO = 99.0 - -# 2. Build -npm run build --workspace=packages/core && npm run build --workspace=packages/cli - -# 3. 启动 qwen (128K context model, 512MB heap) -SESSION="oom-test" -tmux new-session -d -s "$SESSION" -c "$REPO_DIR" -tmux send-keys -t "$SESSION" \ - "NODE_OPTIONS='--max-old-space-size=512' node packages/cli/dist/index.js --model 'qwen3.6-plus'" Enter - -# 4. 等待启动后运行驱动 -sleep 10 -bash /tmp/oom-simple-driver.sh "$SESSION" -``` - ---- - -## 十、后续建议 - -### 短期缓解（已有） - -- [x] #4186: heap-pressure auto-compaction safety net (0.7 threshold) -- [x] #4188: fileReadCache / crawlCache 上限 - -### 中期修复（建议） - -- [ ] 减少 `structuredClone()` 调用 — `nextSpeakerChecker` 只需最后一条消息，不需 clone 全量 -- [ ] Compaction 使用 slice + 引用替代全量 deep clone -- [ ] 大 tool result (>100KB) 写入临时文件，history 中只保留摘要引用 - -### 长期方向 - -- [ ] Tool result offload 到磁盘 + lazy load (#4184) -- [ ] 基于 RSS 的分级压缩策略（不仅是 token count） -- [ ] History 分段存储，避免单次全量操作 diff --git a/docs/e2e-tests/2026-05-19-qwen-runtime-diagnostics-benchmark-report.md b/docs/e2e-tests/2026-05-19-qwen-runtime-diagnostics-benchmark-report.md deleted file mode 100644 index e482f0f94c3..00000000000 --- a/docs/e2e-tests/2026-05-19-qwen-runtime-diagnostics-benchmark-report.md +++ /dev/null @@ -1,904 +0,0 @@ -# Qwen Code Runtime Diagnostics Benchmark Report - -Date: 2026-05-19 - -## Scope - -This run repeats the previous Qwen Code benchmark shapes with the new opt-in -runtime diagnostics enabled. It only tests Qwen Code, not Claude Code. - -Initial model matrix: - -- `pai/glm-5` -- `qwen3.6-plus` - -Additional PR-size follow-up: - -- `DeepSeek/deepseek-v4-pro` through Anthropic-compatible protocol - -Cases: - -- small GitHub PR review: PR `#4268` -- code navigation: compression / compaction related code search and reads -- synthetic local diff: about 94.6 KiB -- synthetic local diff: about 968.5 KiB -- synthetic local diff: about 4.84 MiB - -The run used the local bundled CLI from the diagnostics branch, with -`QWEN_CODE_PROFILE_RUNTIME=1` and a temporary CLI home. Global MCP servers and -hooks were not loaded for this benchmark. - -Important caveat: these absolute RSS numbers are lower than the previous -PATH-resolved `qwen` runs because this run used `node dist/cli.js` from the -local branch plus a stripped temporary config. Treat this report as an internal -diagnostics distribution run, not a direct replacement for the earlier installed -CLI RSS comparison. - -## Installed CLI vs Local Bundle Sanity Check - -A follow-up sanity check used the same minimal prompt, model, and non-interactive -mode across the installed CLI and the local diagnostics bundle. The only -intentional variable was whether Qwen Code loaded a stripped temporary CLI home -or the normal user config. - -| CLI | Config mode | Total tokens | Tree RSS peak | Root RSS peak | Process count peak | Runtime diagnostics | -| ------------------- | --------------- | -----------: | ------------: | ------------: | -----------------: | ------------------- | -| PATH `qwen` | stripped config | 33,965 | 542.4 MiB | 249.9 MiB | 3 | no | -| local `dist/cli.js` | stripped config | 47,281 | 455.2 MiB | 214.2 MiB | 4 | yes | -| PATH `qwen` | normal config | 97,615 | 1,099.9 MiB | 250.1 MiB | 6 | no | -| local `dist/cli.js` | normal config | 97,954 | 1,105.4 MiB | 212.7 MiB | 8 | yes | - -This check changes the attribution: the earlier 1 GiB user-visible peak is -reproducible with the normal config even on the local diagnostics bundle. It is -therefore not primarily explained by the local branch including PR `#4186`. - -At the normal-config peak, the local process-tree sample was dominated by -multiple Node/MCP processes rather than the Qwen root process alone: - -| Role | Command shape | RSS at tree peak | -| ----- | ------------------------- | ---------------: | -| child | Node process | 252.9 MiB | -| child | Chrome DevTools MCP | 219.7 MiB | -| child | Node process | 219.2 MiB | -| root | Qwen Node process | 215.1 MiB | -| child | Chrome DevTools MCP setup | 175.2 MiB | - -PR `#4186` is present in the local diagnostics branch, but it is a V8 heap -pressure auto-compaction safety net. It triggers at about 70% V8 heap pressure; -on this environment the Node heap limit is about 4.1 GiB, while the stripped -benchmark end heap was about 99-143 MiB. Based on these numbers, the lower -stripped-config RSS is not caused by `#4186` actively compressing context during -these benchmark runs. - -### Bare Mode Config Attribution Check - -A second follow-up used `qwen3.6-plus` with the same PR-review prompt shape on -both the installed CLI and the local bundle. This is not a normal end-to-end -business benchmark. It is a controlled attribution check for startup/config -memory only. - -`--bare` changes the runtime inputs: it skips normal global settings discovery, -MCP startup, hooks, implicit context, skills, and other startup integrations. It -can therefore fail or behave differently when a model provider is configured -only in global settings. For this run, model credentials were supplied only -through the child-process environment because bare mode intentionally does not -load the normal provider settings. Nothing was written back to the user's global -config. - -This run did not produce useful token/tool-call statistics: the model completed -in one turn and did not call the requested shell command. Do not use these rows -as normal task benchmark results, and do not compare their token/tool-call -behavior with the matrix above. They are only useful for estimating how much -process-tree RSS comes from normal config and configured child processes. - -| CLI | Mode | Wall | Turns | Tool uses | Tree RSS peak | Root RSS peak | Process count peak | -| ------------------- | -------- | ---: | ----: | --------: | ------------: | ------------: | -----------------: | -| PATH `qwen` | normal | 5.5s | 1 | 0 | 1,021.3 MiB | 251.5 MiB | 5 | -| PATH `qwen` | `--bare` | 2.4s | 1 | 0 | 525.7 MiB | 246.4 MiB | 2 | -| local `dist/cli.js` | normal | 4.9s | 1 | 0 | 1,046.2 MiB | 213.3 MiB | 5 | -| local `dist/cli.js` | `--bare` | 2.3s | 1 | 0 | 454.3 MiB | 216.5 MiB | 3 | - -The result confirms the process-tree hypothesis for startup/config attribution. -On this machine, normal config adds roughly 0.50-0.59 GiB of user-visible -process-tree RSS over `--bare`, while root RSS stays in the same 0.21-0.25 GiB -band. At the normal-config peak, the extra RSS again came from additional -Node/MCP child processes, including a Chrome DevTools MCP process and its setup -wrapper. `--bare` removes those startup/config children and brings -installed/local runs back into the 0.45-0.53 GiB tree-RSS range. - -### Temporary Settings MCP / Hooks Isolation - -Because `--bare` changes too many runtime inputs to be treated as a normal -benchmark, a follow-up used temporary `QWEN_HOME` directories with generated -settings files derived from the normal settings. The run stayed on the normal -settings-loading path, but toggled only two config dimensions: - -- MCP disabled: `mcpServers` cleared and MCP allow/exclude lists emptied. -- Hooks disabled: `disableAllHooks` set to true. - -No global settings were modified. The case used `qwen3.6-plus` and a minimal -startup prompt, so it measures startup/config process-tree cost, not task -reasoning quality. - -| CLI | Temporary config | MCP servers | Tools | Tree RSS peak | Root RSS peak | Process count peak | -| ------------------- | -------------------- | ----------: | ----: | ------------: | ------------: | -----------------: | -| PATH `qwen` | full | 4 | 46 | 1,017.4 MiB | 249.8 MiB | 5 | -| PATH `qwen` | MCP disabled | 0 | 17 | 548.7 MiB | 252.4 MiB | 2 | -| PATH `qwen` | hooks disabled | 4 | 46 | 1,003.8 MiB | 246.4 MiB | 5 | -| PATH `qwen` | MCP + hooks disabled | 0 | 17 | 542.5 MiB | 248.0 MiB | 2 | -| local `dist/cli.js` | full | 4 | 48 | 865.9 MiB | 220.4 MiB | 6 | -| local `dist/cli.js` | MCP disabled | 0 | 19 | 442.9 MiB | 209.6 MiB | 2 | -| local `dist/cli.js` | hooks disabled | 4 | 48 | 848.3 MiB | 212.6 MiB | 5 | -| local `dist/cli.js` | MCP + hooks disabled | 0 | 19 | 447.2 MiB | 217.8 MiB | 2 | - -Interpretation: - -1. Disabling MCP is the dominant change. It removes 4 MCP servers, reduces the - advertised tool count by about 29 tools, and lowers process-tree RSS by about - 0.42-0.47 GiB in this startup/config case. -2. Disabling hooks alone barely changes RSS in this case. That is expected - because the prompt did not produce tool calls, so `PreToolUse` / - `PostToolUse` hooks were not executed. -3. The root process stays around 0.21-0.25 GiB across all rows. The large - difference is again process-tree composition, not root Qwen RSS. - -Two attempted code-navigation follow-ups with `qwen3.6-plus` and `pai/glm-5` -also reproduced the same MCP-vs-no-MCP memory split, but neither model produced -tool calls in those runs. Those rows are therefore not used as hooks execution -evidence. A valid hooks benchmark still needs a task/model combination that -reliably emits tool calls. - -### Per-MCP Isolation - -The previous row showed MCP as a group is the dominant startup/config memory -factor. A follow-up isolated each configured MCP server while keeping hooks -disabled for all rows. This keeps the test on the normal settings-loading path -but changes only the MCP server subset. - -Configured MCP server names: - -- `approval-bridge` -- `env-center` -- `chrome-devtools` -- `code` - -Single-pass isolation: - -| Variant | Enabled MCPs | Tools | MCP servers | Tree RSS peak | Root RSS peak | Interpretation | -| ------------------------- | -------------------------------------------------- | ----: | ----------: | ------------: | ------------: | ------------------------------------ | -| none | none | 19 | 0 | 444.4 MiB | 211.7 MiB | baseline without MCP | -| full | all 4 | 48 | 4 | 857.3 MiB | 215.9 MiB | full MCP startup shape | -| only `approval-bridge` | `approval-bridge` | 19 | 1 | 455.5 MiB | 214.0 MiB | near baseline | -| only `env-center` | `env-center` | 19 | 1 | 452.3 MiB | 214.4 MiB | near baseline | -| only `chrome-devtools` | `chrome-devtools` | 48 | 1 | 824.4 MiB | 209.5 MiB | large RSS increase and tool increase | -| only `code` | `code` | 19 | 1 | 452.1 MiB | 216.6 MiB | near baseline | -| without `approval-bridge` | `env-center`, `chrome-devtools`, `code` | 48 | 3 | 997.1 MiB | 215.4 MiB | still high; run showed variance | -| without `env-center` | `approval-bridge`, `chrome-devtools`, `code` | 48 | 3 | 863.8 MiB | 220.9 MiB | still high | -| without `chrome-devtools` | `approval-bridge`, `env-center`, `code` | 19 | 3 | 463.4 MiB | 221.6 MiB | returns near baseline | -| without `code` | `approval-bridge`, `env-center`, `chrome-devtools` | 48 | 3 | 858.1 MiB | 219.5 MiB | still high | - -Because startup RSS has some variance, the key variants were repeated twice: - -| Variant | Samples | Tree RSS range | Avg tree RSS | Result | -| ------------------------- | ------: | ------------------- | -----------: | ------------------------------ | -| none | 2 | 443.3-451.9 MiB | 447.6 MiB | stable no-MCP baseline | -| full | 2 | 856.1-922.8 MiB | 889.5 MiB | stable high-MCP range | -| only `chrome-devtools` | 2 | 1,007.1-1,021.2 MiB | 1,014.2 MiB | enough alone to reproduce high | -| without `chrome-devtools` | 2 | 461.1-461.6 MiB | 461.4 MiB | removes the high RSS | -| only `approval-bridge` | 2 | 449.1-449.9 MiB | 449.5 MiB | near baseline | -| only `env-center` | 2 | 438.7-449.5 MiB | 444.1 MiB | near baseline | -| only `code` | 2 | 450.6-451.3 MiB | 451.0 MiB | near baseline | - -Interpretation: - -1. `chrome-devtools` is the dominant MCP contributor in this environment. It is - sufficient by itself to reproduce the high process-tree RSS. -2. Removing `chrome-devtools` from the full MCP set returns RSS to the no-MCP - band. Removing other MCPs while keeping `chrome-devtools` does not. -3. The advertised tool count follows the same pattern: baseline is 19 tools, - while `chrome-devtools` raises the tool count to 48. That means this MCP is - also likely to increase request tool schema size and token pressure, not just - process-tree RSS. -4. `approval-bridge`, `env-center`, and `code` individually stay near the - no-MCP baseline in these startup/config runs. They emitted startup warnings - in this environment, so this result should be interpreted as "no persistent - startup RSS owner observed" rather than proof that they have zero cost in all - workflows. - -## Runtime Summary - -| Case | Model | Wall | Turns | Total tokens | Tree RSS peak | Root RSS peak | End heap | End RSS | -| ---------------- | -------------- | ----: | ----: | -----------: | ------------: | ------------: | --------: | --------: | -| small PR `#4268` | `pai/glm-5` | 20.1s | 7 | 173,216 | 362.1 MiB | 359.8 MiB | 103.1 MiB | 216.5 MiB | -| code navigation | `pai/glm-5` | 18.4s | 2 | 49,127 | 378.0 MiB | 376.0 MiB | 102.4 MiB | 313.4 MiB | -| diff 94.6 KiB | `pai/glm-5` | 16.6s | 6 | 135,716 | 367.9 MiB | 366.0 MiB | 99.1 MiB | 295.0 MiB | -| diff 968.5 KiB | `pai/glm-5` | 11.4s | 2 | 42,590 | 373.2 MiB | 362.5 MiB | 106.4 MiB | 345.6 MiB | -| diff 4.84 MiB | `pai/glm-5` | 12.0s | 4 | 95,119 | 414.2 MiB | 412.0 MiB | 123.6 MiB | 410.7 MiB | -| small PR `#4268` | `qwen3.6-plus` | 35.0s | 6 | 156,556 | 358.9 MiB | 356.9 MiB | 102.6 MiB | 293.1 MiB | -| code navigation | `qwen3.6-plus` | 28.9s | 4 | 99,800 | 370.3 MiB | 368.3 MiB | 105.8 MiB | 298.2 MiB | -| diff 94.6 KiB | `qwen3.6-plus` | 28.3s | 4 | 90,808 | 358.8 MiB | 356.9 MiB | 105.9 MiB | 307.0 MiB | -| diff 968.5 KiB | `qwen3.6-plus` | 30.9s | 6 | 151,782 | 366.1 MiB | 364.1 MiB | 101.0 MiB | 316.9 MiB | -| diff 4.84 MiB | `qwen3.6-plus` | 24.1s | 4 | 93,271 | 372.8 MiB | 366.0 MiB | 142.8 MiB | 366.0 MiB | - -Average by model: - -| Model | Avg tree RSS peak | Avg root RSS peak | Avg turns | Avg total tokens | Avg max wire body | Avg total tool result | -| -------------- | ----------------: | ----------------: | --------: | ---------------: | ----------------: | --------------------: | -| `pai/glm-5` | 379.1 MiB | 375.3 MiB | 4.2 | 99,154 | 111.8 KiB | 335.1 KiB | -| `qwen3.6-plus` | 365.4 MiB | 362.4 MiB | 4.8 | 118,443 | 119.3 KiB | 344.3 KiB | - -Overlapping small PR `#4268` model snapshot: - -| Model | Protocol | Wall | Turns | Total tokens | Tree RSS peak | Root RSS peak | Max wire body | -| -------------------------- | --------- | ----: | ----: | -----------: | ------------: | ------------: | ------------: | -| `pai/glm-5` | OpenAI | 20.1s | 7 | 173,216 | 362.1 MiB | 359.8 MiB | 113.8 KiB | -| `qwen3.6-plus` | OpenAI | 35.0s | 6 | 156,556 | 358.9 MiB | 356.9 MiB | 134.1 KiB | -| `DeepSeek/deepseek-v4-pro` | Anthropic | 39.7s | 2 | 43,362 | 346.9 MiB | 344.8 MiB | 103.0 KiB | - -## Request And Tool Diagnostics - -| Case | Model | Requests | Max wire body | Max system prompt | Max tool schema | Tool calls | Total tool result | Max tool result | Max function response in request | -| ---------------- | -------------- | -------: | ------------: | ----------------: | --------------: | ---------: | ----------------: | --------------: | -------------------------------: | -| small PR `#4268` | `pai/glm-5` | 7 | 113.8 KiB | 51.4 KiB | 40.2 KiB | 9 | 4.7 KiB | 3.9 KiB | 15.3 KiB | -| code navigation | `pai/glm-5` | 2 | 114.6 KiB | 51.5 KiB | 40.2 KiB | 3 | 17.5 KiB | 6.2 KiB | 18.4 KiB | -| diff 94.6 KiB | `pai/glm-5` | 6 | 111.2 KiB | 39.1 KiB | 37.2 KiB | 9 | 94.9 KiB | 92.6 KiB | 29.2 KiB | -| diff 968.5 KiB | `pai/glm-5` | 2 | 104.8 KiB | 39.1 KiB | 37.2 KiB | 2 | 772.1 KiB | 771.9 KiB | 25.6 KiB | -| diff 4.84 MiB | `pai/glm-5` | 4 | 114.7 KiB | 39.1 KiB | 37.2 KiB | 4 | 786.3 KiB | 783.2 KiB | 34.7 KiB | -| small PR `#4268` | `qwen3.6-plus` | 6 | 134.1 KiB | 51.4 KiB | 40.2 KiB | 5 | 34.6 KiB | 15.6 KiB | 36.6 KiB | -| code navigation | `qwen3.6-plus` | 4 | 114.9 KiB | 51.5 KiB | 40.2 KiB | 3 | 17.5 KiB | 6.2 KiB | 18.4 KiB | -| diff 94.6 KiB | `qwen3.6-plus` | 4 | 112.8 KiB | 39.1 KiB | 37.2 KiB | 3 | 92.9 KiB | 92.6 KiB | 33.0 KiB | -| diff 968.5 KiB | `qwen3.6-plus` | 6 | 113.1 KiB | 39.1 KiB | 37.2 KiB | 5 | 778.0 KiB | 771.9 KiB | 32.1 KiB | -| diff 4.84 MiB | `qwen3.6-plus` | 4 | 121.5 KiB | 39.1 KiB | 37.2 KiB | 4 | 798.5 KiB | 783.2 KiB | 41.3 KiB | - -## Observations - -1. Process-tree RSS is almost the same as root RSS in this local bundle run. - The root/tree gap is usually below 10 MiB. That means these runs did not - show a persistent child-process memory owner. The dominant process is the - main Node process. -2. The local bundle run peaks around 0.36-0.41 GiB, not the earlier - 0.83-1.04 GiB, because the matrix used a stripped temporary config. A - follow-up normal-config sanity check reproduced about 1.1 GiB tree RSS on - both PATH `qwen` and local `dist/cli.js`, with the extra memory coming from - child MCP/Node processes in the process tree. -3. V8 heap is much smaller than RSS. End heap is about 99-143 MiB while end RSS - is about 216-411 MiB. The remaining footprint is likely loaded modules, - native allocations, external buffers, or runtime overhead outside live JS - heap. -4. Static request overhead is large and repeated. The system prompt is about - 39-51 KiB per request, and tool schema is about 37-40 KiB per request. This - explains why even small tasks can produce high accumulated token counts when - the model takes several turns. -5. Large diff output is capped before it reaches the model request. The 968 KiB - and 4.84 MiB diff cases produced around 772-799 KiB of captured tool result, - but the largest model-facing function response in a request stayed around - 25-41 KiB, and max wire body stayed around 105-122 KiB. This points to - truncation / saved-output handling working on the model-facing path. -6. Memory still increases on large-output cases even though wire body remains - bounded. For example, the 4.84 MiB GLM run reached 414.2 MiB tree RSS and - 410.7 MiB end RSS, and the 4.84 MiB qwen3.6-plus run ended with 142.8 MiB - heap. That suggests large tool output can still affect local capture, - normalization, or retained runtime state even when the final request payload - is capped. -7. Model choice changed turns and token totals more than RSS in this run. - `qwen3.6-plus` averaged more tokens and turns than `pai/glm-5`, but its - average tree RSS peak was slightly lower. This supports the earlier - conclusion that model choice is not the main explanation for process memory. - -## Updated Working Inference - -The new diagnostics make the earlier hypothesis more precise: - -- The installed-CLI user-visible 1 GiB peak is now reproducible with the normal - config on the local diagnostics bundle. The stripped run should be used for - internal Qwen runtime attribution; the normal-config run should be used for - user-visible process-tree attribution. -- The largest observed difference between stripped and normal config is - process-tree shape: normal config starts additional MCP/Node child processes. - Those children explain most of the absolute jump from about 0.35-0.55 GiB to - about 1.1 GiB in the minimal prompt sanity check. -- The `--bare` follow-up confirms the same direction on `qwen3.6-plus`: normal - config costs about 0.50-0.59 GiB more process-tree RSS than bare mode for the - same prompt shape, while root RSS changes only slightly. -- The temporary-settings isolation is a better attribution test than `--bare`: - disabling MCP alone reduces process-tree RSS by about 0.42-0.47 GiB while - keeping the normal settings-loading path. Disabling hooks alone does not show - a meaningful RSS change in no-tool-call cases. -- Per-MCP isolation points to `chrome-devtools` as the dominant MCP contributor: - it is enough by itself to reproduce the high RSS band, and removing it returns - the run near the no-MCP baseline. -- Within the local Qwen runtime, the most suspicious areas are no longer "raw - diff bytes sent to the model". The model-facing request body is bounded. -- The stronger suspects are static per-request context cost, repeated request - rounds, tool schema size, and local retention/capture of large tool outputs - before or outside model-facing truncation. -- Because RSS remains much higher than V8 heap, the next profiling layer should - include module/startup accounting, external memory, and heap snapshots around - tool execution and final response emission. - -## RSS Attribution From Current Diagnostics - -The current counters do not identify an exact retained object or source file, -but they do narrow what is and is not driving RSS in these local runs: - -| Signal | Current evidence | RSS implication | -| ---------------------------- | --------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | -| Root RSS vs process-tree RSS | Root and tree peaks are usually within about 2-10 MiB; DeepSeek large PR is the widest gap at about 23.6 MiB | No persistent child process explains the RSS in this local bundle run; the main Node process dominates | -| Normal config process tree | Minimal-prompt normal-config runs reach about 1.1 GiB tree RSS while root RSS stays about 213-250 MiB | User-visible 1 GiB peaks can be dominated by MCP/Node child processes rather than Qwen root RSS alone | -| `--bare` comparison | `qwen3.6-plus` normal runs peak around 1.02-1.05 GiB tree RSS; bare runs peak around 0.45-0.53 GiB | Loading normal config adds about 0.50-0.59 GiB process-tree RSS in this environment | -| Temporary MCP isolation | Clearing MCP servers drops startup/config tree RSS from 865-1,017 MiB to 443-549 MiB | MCP startup and MCP child processes explain about 0.42-0.47 GiB of process-tree RSS in the controlled config check | -| Per-MCP isolation | `chrome-devtools` alone reaches about 1.0 GiB in repeated samples; without it the run stays around 461 MiB | `chrome-devtools` is the dominant MCP process-tree RSS contributor in this environment | -| Temporary hooks isolation | `disableAllHooks=true` with MCP still enabled changes tree RSS by only about 13-18 MiB in no-tool-call cases | Hook config alone is not a visible startup RSS driver here; hook execution still needs a tool-call benchmark | -| V8 heap vs RSS | End heap is about 99-143 MiB while end RSS is about 216-411 MiB | Live JS heap is not the whole footprint; loaded modules, native allocations, external buffers, or runtime overhead are likely significant | -| PR/diff size vs RSS | DeepSeek small/medium/large PRs scale from 1 to 4,750 changed lines, but tree RSS stays in a narrow 340.7-360.0 MiB band | Raw PR size is not linearly driving RSS once tool output is bounded | -| Tool output size | Large diff runs capture about 772-799 KiB tool results and show some higher end RSS / heap, but RSS does not scale linearly | Tool result capture/normalization contributes pressure, especially large-output cases, but is unlikely to be the only RSS driver | -| Request body size | Max model-facing body ranges from about 103-289 KiB while RSS stays near the same band | Request serialization size affects tokens and latency more clearly than RSS peak | -| Static per-request context | System prompt is about 39-51 KiB and tool schema about 37-48 KiB per request | Repeated rounds are a token/cost amplifier; this alone does not explain RSS but is a likely optimization target for token pressure | - -Working attribution: in the stripped local bundle benchmark, the RSS floor looks -mostly like task-time runtime/module/native footprint, with large tool output -adding incremental pressure. In the normal-config run, the user-visible 1 GiB -tree peak is mostly process-tree composition: Qwen root plus MCP/Node child -processes. The next targeted measurement should split Qwen root diagnostics -from configured MCP server diagnostics, then add startup/module/external-memory -checkpoints inside the Qwen root process. - -## Progress Snapshot - -Current confirmed signals: - -1. The user-visible 1 GiB startup/config peak is reproducible with both the - installed CLI and the local diagnostics bundle when the normal config is - loaded. It is not primarily explained by the diagnostics branch or PR `#4186`. -2. In this environment, that 1 GiB peak is mostly process-tree composition: - Qwen root process plus relaunch child process plus MCP child processes. -3. `chrome-devtools` is the dominant configured MCP contributor in the current - config. It is enough by itself to reproduce the high process-tree RSS band, - even when the prompt does not explicitly use that MCP. -4. The no-MCP normal relaunch shape still sits around 0.45 GiB process-tree RSS. - A single Qwen runtime process without the relaunch parent is closer to - 0.22-0.24 GiB in the startup attribution check. This means the 0.45 GiB - baseline is not a single-process root RSS number. -5. In stripped non-interactive task runs, model choice changes turns, token - totals, latency, and request sizes more clearly than RSS. RSS stayed in a - relatively narrow range across `pai/glm-5`, `qwen3.6-plus`, and - `DeepSeek/deepseek-v4-pro`. -6. Current short-task diagnostics show model-facing tool/function responses are - bounded, but local tool-result capture and runtime state can still increase - heap/RSS on large-output cases. This keeps large-output retention on the - investigation path. - -Current gaps: - -1. The short-task benchmark matrix is still short-lived. A later interactive - long-review run did reproduce a 41.9 min failure, but it is still one sample - and needs repeat runs plus heap/object attribution. -2. The current counters are enough to attribute process-tree RSS and request - size, but not enough to name the retained JS object graph during long - sessions. -3. Startup/config RSS and long-session OOM must remain separate tracks. MCP and - relaunch explain a large idle/startup RSS band; they do not by themselves - explain V8 heap OOM after long tasks. -4. Interactive TUI memory still needs a separate run from non-interactive mode, - because UI history and Ink static output are not exercised the same way. - -## Long-Task OOM Evidence From Issues And PRs - -Issue/PR evidence points to several different OOM shapes, not one single -failure mode: - -| Source | Evidence summary | Hypothesis to test | -| ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | -| [`#4309`](https://github.com/QwenLM/qwen-code/issues/4309) | User reports 5.84 GiB memory usage / 7.02 GiB warning with YOLO mode and DeepSeek backend; increasing Node memory to 8 GiB did not remove the symptom | Long autonomous tool loops can retain enough state that simply raising old-space limit is not a root fix | -| [`#4149`](https://github.com/QwenLM/qwen-code/issues/4149) | Multiple reports show `Ineffective mark-compacts near heap limit`, including 4 GiB and much larger heap-limit cases | A large fraction of heap is reachable application state, not immediately collectible garbage | -| [`#4116`](https://github.com/QwenLM/qwen-code/issues/4116) | OOM occurred while context display was around 9.5%; analysis points to `structuredClone`, UI history, Ink static tree, and large context windows | Token usage can be low while JS heap pressure is high; token threshold alone is not a reliable memory guard | -| [`#4167`](https://github.com/QwenLM/qwen-code/issues/4167) | User says the crash happened while compressing; analysis identifies compression peak memory as a distinct shape | Compression can itself create a peak when heap is already high, especially if history is cloned/stringified around the same time | -| [`#2128`](https://github.com/QwenLM/qwen-code/issues/2128) | Report identifies unbounded UI history, retained file diffs / terminal output, string-width caches, and checkpoint serialization | Interactive TUI long sessions may retain memory outside model history and outside non-interactive benchmarks | -| [`#2562`](https://github.com/QwenLM/qwen-code/issues/2562) | Report focuses on `GeminiChat.getHistory()` deep-cloning full history in long sessions | Full-history cloning can amplify memory peaks and should be measured separately from retained steady-state size | -| [`#4185`](https://github.com/QwenLM/qwen-code/issues/4185) | Tracks V8 heap pressure exceeding limit before token-based compaction runs | Heap-pressure guard is necessary, but it only mitigates symptoms if retained data remains large | -| [`#4184`](https://github.com/QwenLM/qwen-code/issues/4184) | Proposes diagnostics and offload/preview for large retained tool results | Large tool output may be bounded for model requests while still retained in local hot memory | -| [`#4186`](https://github.com/QwenLM/qwen-code/pull/4186) | Merged heap-pressure auto-compaction safety net and O(1) last-history access for `nextSpeakerChecker` | Covers part of heap-pressure and clone amplification, but does not claim to solve all OOM classes | -| [`#4127`](https://github.com/QwenLM/qwen-code/pull/4127), [`#4168`](https://github.com/QwenLM/qwen-code/pull/4168) | Open compaction-threshold PRs; one uses fixed heap thresholds, the other redesigns token thresholds and compression behavior | Useful related work, but long-task testing must verify whether heap, token, and compression signals line up in real runs | -| [`#3000`](https://github.com/QwenLM/qwen-code/issues/3000), [`#4183`](https://github.com/QwenLM/qwen-code/issues/4183) | Diagnostic roadmap calls out `/doctor memory`, heap snapshot, and bounded memory timeline | Snapshot/timeline support is needed to move from RSS attribution to retained-object attribution | - -Initial interpretation: - -- Unused configured MCP can consume memory because normal startup connects to - configured MCP servers and advertises their tools before the task needs them. - In the measured config, `chrome-devtools` starts extra Node/npm MCP processes - and also increases the tool schema count from 19 to 48. This explains a large - startup/config RSS band and can also increase repeated request overhead. -- The long-session OOM reports are a different layer. GC logs where - Mark-Compact frees very little memory suggest the heap is full of reachable - state. The strongest candidates are retained history/tool/UI objects, - full-history clones, compression intermediates, and streaming/logging - accumulators. -- PR `#4186` is a useful mitigation because it can compact based on heap - pressure before token thresholds trigger, and it removes one unnecessary - full-history clone. It should not be treated as proof that large tool-output - retention, UI history retention, or compression peak memory is already solved. - -## Long-Task Validation Plan - -The next benchmark should keep two tracks separate: - -1. Startup/config attribution: normal config vs MCP-disabled vs - `chrome-devtools`-only vs no-relaunch attribution. This explains what users - see before meaningful work begins. -2. Long-task runtime growth: repeated tool calls, large outputs, compression, - resume, and interactive UI history. This explains OOM after real work. - -Recommended long-task cases: - -| Case | Shape | Why it matters | -| ----------------------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | -| Long PR review loop | Repeat medium/large PR review prompts for 30, 60, and 120 minutes, with fixed model and fixed config | Closest to reported agent workflows; captures turns, tool calls, token growth, and RSS/heap trend | -| Large tool-output retention | Repeatedly produce bounded 1 MiB / 5 MiB / 20 MiB command outputs, then ask follow-up questions | Tests whether raw output is retained locally after model-facing truncation | -| Compression pressure | Use a lower controlled old-space limit and large-context prompts to trigger heap-pressure compaction | Verifies PR `#4186` triggers before OOM and whether compression itself creates a new peak | -| Interactive TUI history | Run the same long loop in tmux TUI mode and compare with non-interactive mode | Isolates UI history, Ink static output, rendered diffs, and terminal-output display retention | -| Resume stress | Resume a large saved session and immediately continue work | Targets `/resume` OOM reports and session reconstruction cost | -| Streaming/logging accumulator | Force long streamed responses with telemetry/logging enabled vs disabled | Tests the suspected `collected responses` / logging-retention path from issue analysis | -| MCP idle vs MCP active | Run no-MCP, `chrome-devtools` configured-but-unused, and `chrome-devtools` actively used variants | Separates idle MCP child RSS from actual MCP tool execution and tool schema/token overhead | - -Metrics that should be recorded per turn or per sampling interval: - -- Root RSS current/peak and process-tree RSS current/peak. -- Child process count and top child command shapes. -- V8 `heapUsed`, `heapTotal`, `heap_size_limit`, `external`, and - `arrayBuffers`. -- Turn count, request count, tool-call count, and tool-call rounds. -- Input/output/cache/total tokens by request and by whole task. -- Request body bytes, system prompt bytes, tool schema bytes, and function - response bytes. -- Tool-result count, total captured tool-result bytes, max tool-result bytes, - and retained tool-result bytes if available. -- Conversation history message count and approximate history byte size. -- Interactive-only UI history item count and approximate retained display size. -- Compression attempts, compression trigger reason, tokens before/after, heap - pressure before/after, and compression failure status. -- Heap snapshot or bounded memory timeline artifacts when heap pressure crosses - a configured threshold. - -Validation criteria: - -1. Repeat at least the key long-task cases twice. Startup RSS has visible - variance, so single-run conclusions should be avoided. -2. Report root RSS and process-tree RSS separately. User-facing memory pressure - can come from child processes, while V8 OOM comes from the Qwen root heap. -3. Treat a flat RSS line as important evidence. If tokens and tool calls grow - but heap/RSS stays flat, the issue is likely elsewhere. -4. When RSS or heap grows, correlate the growth with a specific signal: - tool-result bytes, history bytes, UI history count, compression event, - streaming accumulator size, or MCP process start. -5. If a heap snapshot is taken, write a structured diagnostics JSON first, then - the snapshot. Heap snapshots may be large and can contain sensitive strings, - so they should remain opt-in and local. - -## Interactive Long-Review Reproduction - -After the short non-interactive prompts kept finishing before the target window, -an interactive TUI benchmark was run with remote input. The CLI process stayed -alive in one session while a controller submitted one real PR-review turn at a -time. The next turn was only submitted after the assistant emitted that turn's -completion marker. This avoids treating a short one-shot prompt as a long-task -reproduction. - -Setup: - -- Installed Qwen Code `0.15.11`, model `qwen-latest-series-invite-beta-v28`. -- Temporary CLI home derived from the normal settings, with MCP and hook config - removed. No global config was modified. -- Interactive TUI mode with dual JSON event output and remote JSONL input. -- Static PR review only. The prompt disallowed dependency install, build, test, - Playwright, Docker, and other long external build commands. -- External RSS samplers recorded both process-tree RSS and the Qwen Node root - RSS every 5 seconds. - -Outcome: - -| Signal | Value | -| ----------------------------- | ----------: | -| Wall time before exit | 41.9 min | -| Exit status | 1 | -| Completed PR-review turns | 6 | -| Main chat records | 1,076 | -| API response telemetry | 335 | -| Tool-call telemetry | 607 | -| MCP tool-call telemetry | 0 | -| Main/root API responses | 36 | -| Subagent API responses | 299 | -| Root total tokens | 2.08M | -| Subagent total tokens | 17.24M | -| Total API telemetry tokens | 19.32M | -| Max root input tokens | 85,655 | -| Max subagent input tokens | 215,207 | -| `/usr/bin/time -l` max RSS | 1,072.4 MiB | -| Sampled Qwen root RSS peak | 1,028.2 MiB | -| Sampled process-tree RSS peak | 1,038.1 MiB | - -The process exited with: - -```text -libc++abi: terminating due to uncaught exception of type std::__1::system_error: thread constructor failed: Resource temporarily unavailable -``` - -This is a **thread exhaustion** error, not a V8 heap OOM. The failure mechanism -is distinct: the OS refused to create a new thread, likely due to per-process -resource limits (`RLIMIT_NPROC`) or memory fragmentation preventing stack -allocation. It is still relevant because it occurred in a disabled-MCP, -no-build/test, interactive long-session review where the Qwen Node process -itself crossed about 1 GiB RSS. -The failure happened during the final summary phase, after the controller had -already completed six review turns. - -Turn timeline and sampled Qwen root RSS: - -| Window | Turn state | Qwen root RSS max | Qwen root RSS at window end | -| ------------- | -------------------- | ----------------: | --------------------------: | -| 0.0-9.0 min | turn 1 completed | 701.2 MiB | 255.3 MiB | -| 9.0-15.1 min | turn 2 completed | 503.2 MiB | 494.4 MiB | -| 15.1-24.1 min | turn 3 completed | 468.7 MiB | 457.5 MiB | -| 24.1-31.9 min | turn 4 completed | 619.3 MiB | 602.3 MiB | -| 31.9-40.3 min | turn 5 completed | 955.5 MiB | 955.5 MiB | -| 40.3-40.4 min | turn 6 completed | 988.6 MiB | 988.6 MiB | -| 40.4-41.9 min | final summary / exit | 1,028.2 MiB | 1,028.2 MiB | - -Token and tool distribution: - -| Owner | API responses | Input tokens | Output tokens | Total tokens | Max input | -| ------------ | ------------: | -----------: | ------------: | -----------: | --------: | -| Root session | 36 | 2.06M | 22.2K | 2.08M | 85,655 | -| Subagents | 299 | 17.08M | 154.6K | 17.24M | 215,207 | - -Tool-call telemetry by function: - -| Tool | Calls | Captured content length | -| ------------------- | ----: | ----------------------: | -| `read_file` | 271 | 1.46 MB | -| `run_shell_command` | 181 | 164.4 KB | -| `web_fetch` | 80 | 846.3 KB | -| `grep_search` | 25 | 15.0 KB | -| `glob` | 15 | 27.8 KB | -| `todo_write` | 16 | 16.1 KB | -| `list_directory` | 8 | 6.2 KB | -| `agent` | 10 | 0 | -| `tool_search` | 1 | 2.1 KB | - -The top visible TUI token counter for a single agent reached about 3.83M -tokens. Telemetry also shows the heaviest subagent at about 4.05M total tokens -with a 215K-token max input request. That makes subagent amplification the -dominant signal in this reproduction. - -Interpretation: - -1. This run separates long-session growth from MCP startup/config memory. MCP - was disabled and there were no MCP tool calls, yet the Qwen root process - still reached about 1 GiB RSS. -2. The late memory peak aligns with subagent-heavy review turns and final - summary/merge-back, not with external build/test child processes. -3. The RSS curve is not a simple linear leak. It falls after early turns, then - rises sharply after later subagent turns and remains high near exit. -4. The failure mode is native resource exhaustion rather than a V8 heap-limit - stack, so the next run should add heap/external/arrayBuffer/thread-count - sampling. RSS alone cannot distinguish JS heap from native allocations or - thread-resource pressure. -5. The strongest code paths to inspect remain subagent transcript retention, - agent-result merge-back, full-history cloning, checkpoint/session recording, - and final summary/history assembly. - -## Deterministic Huge-Task Clone-Pressure Reproduction - -A deterministic stress harness was added as -`scripts/memory-pressure-repro.mjs`. It does not call a model. Instead, it -constructs a Qwen-like long-session object graph with root review turns, -subagent transcripts, large tool results, checkpoint JSON, and retained -`structuredClone()` copies. This gives a repeatable reproduction for the clone -and checkpoint peak suspected from the user-provided OOM stack. - -The harness has a lightweight script test: - -```bash -npx vitest run --config ./scripts/tests/vitest.config.ts \ - scripts/tests/memory-pressure-repro.test.js -``` - -Result: passed, 1 test. - -Controlled runs used `node --max-old-space-size=256` unless otherwise noted. - -| Case | History shape | Clone/checkpoint pressure | Result | Max RSS | -| ------------------------------------------------- | ----------------------------------------------------------------------- | -------------------------------------------------- | --------------------------------- | --------: | -| Small sanity | 2 turns, 2 KiB tool result, 1 subagent | 1 clone + 1 checkpoint | passed; 2.6 MiB history JSON | 89.7 MiB | -| Huge build only | 12 turns, 256 KiB tool result, 2 subagents x 12 subagent turns | no retained clone/checkpoint | passed; 76.2 MiB history JSON | 491.5 MiB | -| Huge + 1 clone | same as above | 1 retained `structuredClone()` | passed | 569.6 MiB | -| Huge + 2 clones | same as above | 2 retained `structuredClone()` copies | OOM, exit 134 | 496.5 MiB | -| Huge + 1 checkpoint | same as above | one checkpoint with original + cloned history JSON | passed; 152.5 MiB checkpoint JSON | 926.9 MiB | -| Huge + 2 checkpoints | same as above | two checkpoint copies | OOM, exit 134 | 920.1 MiB | -| Huge + 2 clones, no retained subagent transcripts | same generated subagent output, but parent history keeps only summaries | passed; parent history JSON drops to 3.8 MiB | 136.8 MiB | - -The failing huge-clone run produced: - -```text -FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory -``` - -The native stack included: - -- `v8::internal::ValueDeserializer::ReadObjectInternal` -- `v8::internal::ValueDeserializer::ReadDenseJSArray` -- `node::worker::Message::Deserialize` -- `node::worker::StructuredClone` - -This matches the same stack family as the user-provided OOM log. The controlled -reproduction also shows why 4 GiB / 8 GiB user reports are plausible: the -failure is not caused by a single large object, but by large retained -history/tool-result/subagent state plus one or more full-history clone or -checkpoint copies. Raising `--max-old-space-size` can delay the crash while -preserving the same amplification pattern. - -Important attribution from this deterministic run: - -1. Building a 76.2 MiB parent history JSON can succeed under the reduced heap. - The OOM appears when additional full-history clone/checkpoint copies are - retained. -2. A single checkpoint copy can push RSS close to 1 GiB even before OOM. -3. Removing retained subagent transcripts from the parent hot history changes - the same generated workload from OOM to a small 136.8 MiB RSS run. That is - the clearest mitigation signal so far. -4. This reproducer is synthetic and intentionally adversarial, but it exercises - the same object-graph shape as the long interactive review: parent session, - subagents, large tool outputs, transcript merge-back, and full-history clone - pressure. - -## DeepSeek PR-Size Follow-Up - -After the initial model matrix, an additional Qwen Code-only run tested -`DeepSeek/deepseek-v4-pro` across three real PR sizes. This model is configured -through the Anthropic-compatible protocol; OpenAI-compatible execution returned -404 in a smoke check, so the successful benchmark uses `--auth-type anthropic`. - -The diagnostics branch was extended to record Anthropic wire request summaries -with the same privacy rule as the OpenAI path: aggregate counts and byte sizes -only, no prompt text, diff content, tool arguments, headers, base URL, or API -key. - -PR sizes: - -| Size | PR | State | Files | Changed lines | Title | -| ------ | ------- | ------ | ----: | ------------: | ----------------------------------------------------------------------- | -| small | `#4268` | merged | 1 | 1 | fix(serve): add mcp_guardrails to E2E capabilities expectation | -| medium | `#4186` | merged | 6 | 494 | fix(core): add heap-pressure auto-compaction safety net | -| large | `#4168` | open | 25 | 4,750 | feat(core)!: redesign auto-compaction thresholds with three-tier ladder | - -Runtime: - -| Size | PR | Wall | Turns | Total tokens | Cache-read tokens | Tree RSS peak | Root RSS peak | End heap | End RSS | -| ------ | ------- | -----: | ----: | -----------: | ----------------: | ------------: | ------------: | --------: | --------: | -| small | `#4268` | 39.7s | 2 | 43,362 | 28,672 | 346.9 MiB | 344.8 MiB | 115.2 MiB | 304.3 MiB | -| medium | `#4186` | 142.6s | 4 | 135,120 | 115,840 | 340.7 MiB | 337.3 MiB | 103.5 MiB | 285.6 MiB | -| large | `#4168` | 191.1s | 8 | 386,891 | 332,928 | 360.0 MiB | 336.3 MiB | 119.3 MiB | 237.9 MiB | - -Request and tool diagnostics: - -| Size | PR | Requests | Anthropic wire requests | Max Anthropic body | Max system | Max tool schema | Tool calls | Total tool result | Max tool result | Max function response in request | -| ------ | ------- | -------: | ----------------------: | -----------------: | ---------: | --------------: | ---------: | ----------------: | --------------: | -------------------------------: | -| small | `#4268` | 2 | 2 | 103.0 KiB | 50.8 KiB | 47.6 KiB | 3 | 0.6 KiB | 0.5 KiB | 1.1 KiB | -| medium | `#4186` | 4 | 4 | 159.8 KiB | 50.8 KiB | 47.6 KiB | 5 | 30.2 KiB | 29.3 KiB | 56.7 KiB | -| large | `#4168` | 8 | 8 | 289.5 KiB | 50.8 KiB | 47.6 KiB | 11 | 235.0 KiB | 232.1 KiB | 182.4 KiB | - -DeepSeek observations: - -1. PR size scaled turns, tokens, Anthropic wire body size, and tool result size - clearly, but did not scale RSS proportionally. The small/medium/large tree - RSS peaks stayed in a narrow `340.7-360.0 MiB` band. -2. The large PR was expensive mostly in model rounds and token volume: - 8 requests and 386,891 total tokens. Its max Anthropic body was 289.5 KiB, - much larger than the OpenAI-compatible runs, but RSS still stayed near the - same local-bundle band. -3. The static Anthropic request cost is also visible: system prompt is about - 50.8 KiB and tool schema about 47.6 KiB per request. Repeated rounds are - therefore a major token amplifier. -4. The large PR produced 235.0 KiB of captured tool results and 182.4 KiB max - function response in a request. This is higher than the earlier small PR / - code-navigation cases and shows large PRs still put pressure on local - tool-result handling and request assembly, even when RSS does not spike. -5. The DeepSeek run reinforces the model-choice conclusion: provider/model - choice strongly changes turns, latency, token volume, and wire payload shape, - but the local bundle RSS peak remains dominated by Qwen Code runtime shape - rather than scaling linearly with PR size. - -## Long-Review JSONL Replay: History Clone Pressure - -A recent long PR-review chat record was analyzed as a post-mortem shape for -the reported OOM class. The raw JSONL is not included here because it contains -prompt and tool output text. The aggregate shape is: - -| Signal | Value | -| ----------------------- | ----------------------------- | -| Duration | 87.0 min | -| Qwen Code version | 0.15.10 | -| Model | qwen-latest-series beta model | -| API responses | 380 | -| Tool-call telemetry | 507 events | -| MCP tool-call telemetry | 4 events | -| Subagent API responses | 313 | -| Root API responses | 67 | -| Root prompt growth | 38,622 -> 168,555 tokens | -| Max prompt tokens | 168,555 | -| Total response tokens | 31.28M | - -This shape does not support MCP as the primary OOM cause for this case. Only -4 of 507 tool-call telemetry events were MCP, and all four recorded -`content_length=0`. The dominant shape is long-session/subagent amplification: -15 `agent` calls produced 313 subagent API responses and 403 subagent tool-call -events. - -The replay then rebuilt the chat `Content[]` message shape from the JSONL and -ran controlled clone/stringify pressure tests. The base retained message payload -is small, so it is not itself enough to OOM: - -| Replay scale | Retained clones | History JSON | Checkpoint JSON | End heap | End RSS | -| ------------ | --------------: | -----------: | --------------: | -------: | -------: | -| 1x | 8 | 0.54 MB | 1.08 MB | 18.0 MB | 88.8 MB | -| 30x | 8 | 14.46 MB | 28.92 MB | 260.0 MB | 577.8 MB | -| 60x | 8 | 28.86 MB | 57.71 MB | 510.3 MB | 960.8 MB | - -The scaled replay is not a user-data claim; it is a controlled amplification of -the observed JSONL shape to test whether full-history clone and checkpoint -serialization can create the same failure mode as the reports. - -A low-heap reproduction with `--max-old-space-size=256` confirms the mechanism: - -| Case | History JSON | Result | -| ------------------------- | -----------: | ----------------------------------------------------- | -| Build history only | 38.4 MB | Succeeded; heap 131.6 MB, RSS 378.2 MB | -| Build + one clone | 38.4 MB | Succeeded; heap 183.3 MB, RSS 463.4 MB | -| Build + repeated clones | 38.4 MB | OOM after several retained `structuredClone()` copies | -| Checkpoint double-history | 38.4 MB | OOM while holding history plus cloned client history | - -The repeated-clone OOM stack contains `ValueDeserializer::ReadObjectInternal`, -`ValueDeserializer::ReadDenseJSArray`, -`node::worker::Message::Deserialize`, and -`node::worker::StructuredClone`, matching the same stack family seen in the -user-provided OOM log. This proves that full-history `structuredClone()` can be -the immediate OOM trigger without any MCP server involvement. - -Current working hypothesis for this JSONL class: - -1. MCP can explain normal-config startup RSS in separate benchmarks, but it is - not the likely trigger for this long-review OOM shape. -2. Long task growth comes from retained chat history, large tool outputs, - subagent histories, observable agent messages, and UI/tool-result state. -3. The immediate OOM trigger can be a full-history clone or checkpoint-style - double serialization after the heap is already high. -4. Compression can mitigate retained history, but compression itself may create - a temporary peak if it first clones or serializes large history. - -### Local Mitigation Validation: Disabled-MCP PR Review Case - -Two targeted mitigations were applied locally and validated before rerunning a -disabled-MCP PR review case: - -1. `checkNextSpeaker()` now reads only the last curated message with - `getHistoryTail(1, true)` and sends only that message to the next-speaker - side query. The next-speaker prompt only asks about the immediately previous - model response, so sending full history was unnecessary clone and token - pressure. -2. `AgentToolInvocation` no longer retains full `responseParts` arrays inside - the live `task_execution.toolCalls` display. The real response parts still - flow through transcript/history paths, but the parent UI display now keeps - only a bounded text summary for nested tool-result streaming instead of - holding another full copy of large subagent tool outputs during long runs. -3. `GeminiChat.sendMessageStream()` now builds model request contents through - an internal curated-history view instead of calling public - `getHistory(true)`. Public `getHistory()` still returns a defensive - `structuredClone()` for external callers, but the request hot path no longer - deep-clones the whole retained chat history before every model call. - -TDD checks added for these mitigations: - -| Test | Expected protection | -| -------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | -| `checkNextSpeaker > should send only the last curated model message to the side query` | Prevents full-history clone/send in next-speaker checks | -| `AgentTool > should not retain responseParts in live tool call display after TOOL_RESULT` | Prevents live subagent display from retaining large tool responses | -| `AgentTool > should keep only a bounded result summary in live tool call display` | Preserves nested result readability without retaining the full response body | -| `GeminiChat > sendMessageStream > does not deep-clone the full curated history when building request contents` | Prevents request setup from hitting the `ValueDeserializer` / `StructuredClone` OOM path | - -Additional reproduction and fix validation: - -| Step | Command shape | Result | -| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | -| Pre-fix deterministic clone pressure | `node --max-old-space-size=256 scripts/memory-pressure-repro.mjs ... --clone-count=2 --mode=clone` | OOM, exit 134; stderr contained `Reached heap limit` and `ValueDeserializer` / `StructuredClone`; max RSS 528.1 MiB in the repeat run | -| Red test | targeted `GeminiChat` test with `structuredClone` forced to throw during request setup | failed at `GeminiChat.getHistory()` before the mitigation | -| Green test | same targeted `GeminiChat` test after the mitigation | passed | -| Built-code smoke | `node --max-old-space-size=256` against the built core package, with a 96-entry / about 48 MiB history and `structuredClone` forced to throw | passed; request had 97 contents; process RSS 161.4 MiB, `/usr/bin/time -l` max RSS 161.6 MiB | - -This narrows the earlier "same stack family" statement: the deterministic -synthetic OOM still proves retained full-history clones can fail in the same V8 -stack family as the user log, while the new `GeminiChat` red/green test proves -one real production request-setup path no longer reaches that clone point. -Checkpoint/resume and compression internals still need separate long-run -validation because they can legitimately need durable copied history. - -Verification commands: - -| Command | Result | -| ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | -| `npx vitest run src/core/geminiChat.test.ts` | passed, 89 tests | -| `npx vitest run src/utils/nextSpeakerChecker.test.ts --coverage=false` | passed, 13 tests | -| `npx vitest run src/tools/agent/agent.test.ts --coverage=false` | passed, 77 tests | -| `npx vitest run --config ./scripts/tests/vitest.config.ts scripts/tests/memory-pressure-repro.test.js` | passed, 1 test | -| `npm run build --workspace=packages/core` | passed | -| `npm run build --workspace=packages/cli` | passed | -| `npm run typecheck --workspace=packages/core` | passed | -| `npm run typecheck --workspace=packages/cli` | passed | -| `npm run bundle` | passed | -| `npm run build` | failed in `packages/vscode-ide-companion` lint on existing internal-module import rules; core, CLI, bundle, and targeted tests above passed | - -The full root `npm run build` was not clean in this worktree because the -`vscode-ide-companion` package hit pre-existing `import/no-internal-modules` -lint errors. The core/CLI build and bundle needed for the local runtime test -completed successfully. - -The same PR review prompt was then run with a temporary config where MCP and -hooks were disabled. Both rows were interrupted after a bounded long-run window -instead of waiting for a full review to finish. **Caveat**: the two runs are -confounded by workload size (79K vs 390K tokens) and cannot be compared as a -controlled experiment. The comparison only shows directional evidence. - -| Variant | Runtime | MCP servers | Tools | Assistant messages | Tool use/result blocks | Parent tool ids | Total tokens | Max input tokens | Root max RSS | -| ----------------- | ------: | ----------: | ----: | -----------------: | ---------------------: | --------------: | -----------: | ---------------: | -----------: | -| before mitigation | 365.08s | 0 | 19 | 42 | 42 / 42 | 3 | 79,439 | 26,807 | 357.7 MiB | -| after mitigation | 404.52s | 0 | 19 | 58 | 52 / 42 | 2 | 390,339 | 54,000 | 310.5 MiB | - -This is not a deterministic apples-to-apples model benchmark: the patched run -did more work and consumed substantially more total tokens before the manual -cutoff. The useful signal is narrower: under a disabled-MCP review case with -more observed work, root max RSS did not increase and was about 47.2 MiB lower. -That supports the mitigation direction, but it does not prove the whole -long-task OOM class is fixed. - -Remaining high-risk clone/retention paths to inspect next: - -1. Compression still calls full `getHistory(true)` before summarization. If the - heap is already high, the compression attempt can create the peak that trips - OOM. -2. Checkpoint creation can hold original history, cloned client history, and a - serialized checkpoint payload at the same time. -3. Fork subagents still seed from parent history with `getHistory(true)`. -4. ACP/history export/summary/copy paths still call full `getHistory()` and - should be audited separately from the normal review loop. - -Version timing: - -| Issue | Created | Reported version | Signal | -| ----- | ---------- | ------------------------ | ---------------------------------------- | -| #2128 | 2026-03-05 | not specified | Long-session UI memory growth | -| #2562 | 2026-03-21 | not specified | `structuredClone` OOM in long sessions | -| #2868 | 2026-04-03 | 0.13.2 | Heap OOM | -| #2945 | 2026-04-07 | 0.14.0 | V8 heap OOM | -| #4116 | 2026-05-13 | 0.15.11 | OOM with structured-clone-style analysis | -| #4134 | 2026-05-14 | 0.15.11 | OOM | -| #4149 | 2026-05-14 | 0.15.10-nightly.20260513 | V8 heap OOM | -| #4167 | 2026-05-15 | 0.15.11 | Crash near compression | -| #4185 | 2026-05-15 | 0.15.11 | Heap pressure before token compaction | -| #4254 | 2026-05-17 | not specified | Memory keeps rising | -| #4276 | 2026-05-18 | 0.15.11 | V8 heap OOM | -| #4309 | 2026-05-19 | 0.15.11 | High memory warning around 7 GiB | - -The issue history does not prove that 0.15.10 introduced the OOM class; similar -reports existed in March and April. It does support a recent cluster beginning -around 2026-05-13, overlapping `v0.15.10`/`v0.15.11` releases. The relevant -diff between `v0.15.9` and `v0.15.10` touched subagent runtime, -non-interactive execution, `GeminiChat`, and compression code heavily, so this -range is a reasonable first bisect window. - -## Notes - -- The first code-navigation prompt allowed open-ended exploration and hit - `maxSessionTurns`; the successful rows above use a constrained command list. -- The first synthetic-diff attempt used a relative bundle path from inside the - temporary repositories; those failed immediately and are excluded from the - tables. The successful rows use the absolute local bundle path. -- Raw JSONL streams are not committed because they contain prompts, tool - commands, and tool output. The report only includes aggregate diagnostics. diff --git a/docs/e2e-tests/2026-05-21-qwen-0.15.11-default-heap-oom-stress-report.md b/docs/e2e-tests/2026-05-21-qwen-0.15.11-default-heap-oom-stress-report.md deleted file mode 100644 index e9579dee1b8..00000000000 --- a/docs/e2e-tests/2026-05-21-qwen-0.15.11-default-heap-oom-stress-report.md +++ /dev/null @@ -1,338 +0,0 @@ -# Qwen Code 0.15.11 默认 Heap OOM 压测报告 - -日期：2026-05-21 - -## 测试范围 - -本报告记录了针对 Qwen Code `0.15.11` 最新本地构建的一轮默认 heap 压测。 -这轮测试的目标是验证：在不人为降低内存上限的情况下，当前代码是否还能复现 -issue 中提到的长会话 OOM，以及在更极端的大输出场景下还有没有新的风险。 - -本轮覆盖三个模型： - -- `pai/glm-5` -- `qwen3.6-plus` -- `DeepSeek/deepseek-v4-pro` - -测试分为两部分： - -1. 真实长任务、多 agent 并发 review 循环。 -2. amplified foreground stdout 压测，即用大规模前台 shell stdout 放大 - tool-output 路径压力。 - -## 测试环境 - -| 项目 | 值 | -| --------------------------- | --------------------------------------------- | -| 分支 | `codex/memory-investigation-draft-pr` | -| Commit | `c161e0aa4` | -| CLI | 本地 `dist/cli.js` | -| CLI 版本 | `0.15.11` | -| Node 默认 heap limit | `4144 MiB` | -| `NODE_OPTIONS` | 未设置 | -| 显式 `--max-old-space-size` | 未设置 | -| runner `ulimit` | runner 未设置 | -| 配置模式 | 临时复制 `~/.qwen`，并隔离 `QWEN_RUNTIME_DIR` | -| MCP / 正常配置 | 尽量按复制后的正常配置加载 | - -注意：这里的 CLI 版本显示为 `0.15.11`，是因为 package version 尚未 bump。 -实际测试对象是 commit `c161e0aa4` 下本地编译出的 `dist/cli.js`，不是 PATH -里的全局 `qwen` 可执行文件。 - -本轮没有修改全局 Qwen 配置。原始 runtime artifacts 在： - -- `.qwen/runtime-bench/2026-05-20T13-51-58-731Z-oom-stress` -- `.qwen/runtime-bench/2026-05-20T15-20-37-790Z-oom-amplified` - -注意：本轮里 `env-center` MCP server 启动失败，但其他内置工具和部分 -MCP/child process 仍然加载。因此这些结果代表当前本地环境，不是完全 stripped -的 `--bare` 环境。 - -## 核心结论 - -最新本地构建在 issue 最关心的“长会话 V8 heap OOM”路径上表现明显更好。 -基于这轮默认 heap、多模型、多 agent、长任务压测，可以认为本 PR 对此前遇到的 -long-session heap OOM 问题已经基本解决，至少在当前复现维度下已经不能再复现 -原始 heap OOM。 - -真实长任务、多 agent 并发测试一共执行了： - -- 23 个 worker turn -- 约 `719,094,118` reported total tokens -- 77 次 agent tool call -- 856 次总 tool call - -这部分没有复现任何传统 V8 heap OOM 特征： - -- `JavaScript heap out of memory` -- `Reached heap limit` -- `Ineffective mark-compacts near heap limit` -- `Allocation failed` - -真实长任务阶段最高 process-tree RSS 为 `874.7 MiB`，最高 root-process RSS 为 -`219.1 MiB`。这说明在默认 heap 下，当前代码没有轻易复现原 issue 中那种长任务 -跑挂的 heap OOM。 - -第二阶段 amplified stdout 压测更激进。它一共执行了 18 个 payload attempt， -覆盖三个模型和 `128 MiB` 到 `2048 MiB` 的 foreground stdout payload。 - -结果是： - -- 三个模型都成功跑过 `1536 MiB` payload。 -- 最高成功 process-tree RSS 是 `5964.7 MiB`，出现在 `qwen3.6-plus` - 的 `1536 MiB` payload。 -- 到 `2048 MiB` payload 时，出现了一个新的 extreme large-output failure。 - -`2048 MiB` 的结果： - -- `pai/glm-5`：`exit=1`，stdout 为空，没有标准 OOM 文本。 -- `qwen3.6-plus`：`exit=1`，stdout 为空，没有标准 OOM 文本。 -- `DeepSeek/deepseek-v4-pro`：出现 V8 fatal： - `Check failed: i::kMaxInt >= len`，栈在 - `v8::String::NewFromOneByte` / `node::StringBytes::Encode` / - `DecodeUTF8`。 - -这个新问题不是原 issue 中的传统 long-session heap OOM。它更像是 -multi-GiB foreground stdout 被解码/构造成 JS string 时触发的 V8 字符串长度 -限制或大输出处理问题。建议作为 large-output follow-up 跟踪，而不是把它当作 -当前长会话 heap-pressure 修复失败。 - -## Phase 1：真实长任务、多 Agent 并发压测 - -### 测试形态 - -每个模型 worker 都复用同一个 session，不断 `--resume`。每一轮要求 Qwen Code： - -- 进行只读代码审查和代码搜索； -- 在同一轮中并发启动至少 4 个 `agent` tool call； -- 重点检查 chat history、compaction、subagent runtime、non-interactive - streaming、provider adapters 等 memory 相关区域； -- 保留足够详细的最终回答，让 session history 自然增长。 - -runner 每秒采样 process-tree RSS，没有设置任何额外 heap cap。 - -这部分在观察到内存比较稳定后用 `SIGTERM` 主动停止，以便切换到第二阶段的 -amplified stdout 压测。因此表里的 `SIGTERM` 不是 OOM。 - -### 汇总结果 - -| Model | Worker turns | Total tokens | Agent calls | Tool calls | Peak tree RSS | Peak root RSS | Last exit | OOM | -| -------------------------- | -----------: | --------------: | ----------: | ---------: | ------------: | ------------: | --------- | ------ | -| `pai/glm-5` | 9 | 444,614,704 | 36 | 362 | 874.7 MiB | 217.4 MiB | `SIGTERM` | no | -| `qwen3.6-plus` | 7 | 101,425,927 | 17 | 346 | 862.7 MiB | 219.1 MiB | `SIGTERM` | no | -| `DeepSeek/deepseek-v4-pro` | 7 | 173,053,487 | 24 | 148 | 864.5 MiB | 213.8 MiB | `SIGTERM` | no | -| **Total / max** | **23** | **719,094,118** | **77** | **856** | **874.7 MiB** | **219.1 MiB** | - | **no** | - -### 分轮结果 - -| Model | Turn | Exit | Timed out | OOM | Peak tree RSS | Peak root RSS | Total tokens | Agent calls | Tool calls | -| -------------------------- | ---: | --------- | --------- | --- | ------------: | ------------: | -----------: | ----------: | ---------: | -| `DeepSeek/deepseek-v4-pro` | 1 | `0` | no | no | 709.1 MiB | 167.3 MiB | 5,565,147 | 4 | 37 | -| `DeepSeek/deepseek-v4-pro` | 2 | `0` | no | no | 674.5 MiB | 118.8 MiB | 13,989,721 | 4 | 29 | -| `DeepSeek/deepseek-v4-pro` | 3 | `0` | no | no | 734.1 MiB | 148.0 MiB | 22,621,542 | 4 | 24 | -| `DeepSeek/deepseek-v4-pro` | 4 | `0` | no | no | 771.1 MiB | 107.5 MiB | 33,470,249 | 4 | 22 | -| `DeepSeek/deepseek-v4-pro` | 5 | `0` | no | no | 864.5 MiB | 212.9 MiB | 43,540,313 | 4 | 19 | -| `DeepSeek/deepseek-v4-pro` | 6 | `0` | no | no | 807.6 MiB | 167.9 MiB | 53,866,515 | 4 | 17 | -| `DeepSeek/deepseek-v4-pro` | 7 | `SIGTERM` | no | no | 785.1 MiB | 213.8 MiB | n/a | n/a | n/a | -| `pai/glm-5` | 1 | `SIGTERM` | yes | no | 742.8 MiB | 170.5 MiB | 17,071,519 | 4 | 142 | -| `pai/glm-5` | 2 | `0` | no | no | 874.7 MiB | 217.4 MiB | 27,438,727 | 4 | 60 | -| `pai/glm-5` | 3 | `0` | no | no | 699.7 MiB | 102.1 MiB | 35,627,222 | 4 | 38 | -| `pai/glm-5` | 4 | `0` | no | no | 796.0 MiB | 194.0 MiB | 44,130,101 | 4 | 23 | -| `pai/glm-5` | 5 | `0` | no | no | 743.4 MiB | 152.1 MiB | 50,465,979 | 4 | 26 | -| `pai/glm-5` | 6 | `0` | no | no | 714.9 MiB | 125.2 MiB | 56,357,372 | 4 | 18 | -| `pai/glm-5` | 7 | `0` | no | no | 694.5 MiB | 96.6 MiB | 64,047,037 | 4 | 20 | -| `pai/glm-5` | 8 | `0` | no | no | 756.0 MiB | 136.8 MiB | 71,891,505 | 4 | 15 | -| `pai/glm-5` | 9 | `SIGTERM` | no | no | 755.7 MiB | 157.3 MiB | 77,585,242 | 4 | 20 | -| `qwen3.6-plus` | 1 | `0` | no | no | 735.1 MiB | 153.1 MiB | 3,890,508 | 4 | 83 | -| `qwen3.6-plus` | 2 | `0` | no | no | 702.4 MiB | 142.5 MiB | 4,300,186 | 1 | 9 | -| `qwen3.6-plus` | 3 | `0` | no | no | 862.7 MiB | 219.1 MiB | 8,635,953 | 4 | 88 | -| `qwen3.6-plus` | 4 | `SIGTERM` | yes | no | 685.8 MiB | 106.5 MiB | n/a | n/a | n/a | -| `qwen3.6-plus` | 5 | `0` | no | no | 610.5 MiB | 93.1 MiB | 40,191,337 | 4 | 87 | -| `qwen3.6-plus` | 6 | `0` | no | no | 723.6 MiB | 121.9 MiB | 44,407,943 | 4 | 79 | -| `qwen3.6-plus` | 7 | `SIGTERM` | no | no | 810.4 MiB | 116.0 MiB | n/a | n/a | n/a | - -### Phase 1 解读 - -这是本轮里最能说明原始 long-session OOM 已明显改善的数据。 - -这组测试比 5 月 18 日的小 PR review / code navigation 更重：它包含更多 -`--resume`、更多 subagent activity、更大的 reported token 量和更多 tool call。 -但 process-tree RSS 始终低于 `0.9 GiB`，也没有出现传统 V8 heap OOM。 - -这不能证明所有用户 OOM 都不可能再发生，但至少说明当前构建在默认 heap 下， -已经无法轻易复现 issue 中那类长会话 heap-pressure OOM。 - -## Phase 2：Amplified Foreground Stdout 压测 - -### 测试形态 - -第二阶段故意放大 shell-output 路径压力。每个模型、每个 payload size 都要求 -parent session 和并发 agents 运行前台 shell 命令，输出大量 `x` 到 stdout： - -```bash -node -e "const chunk='x'.repeat(1024*1024); for (let i=0; i= len. -... -v8::String::NewFromOneByte -node::StringBytes::Encode -node::encoding_binding::BindingData::DecodeUTF8 -``` - -触发条件： - -- Model：`DeepSeek/deepseek-v4-pro` -- Payload：`2048 MiB` -- Peak tree RSS：`4660.4 MiB` -- Largest process RSS：`4527.6 MiB` -- runner 记录 exit：`SIGTERM`，因为 fatal 输出已经捕获后，剩余子进程仍在高 CPU - 空转，被手动终止。 - -`pai/glm-5` 和 `qwen3.6-plus` 在 `2048 MiB` 也失败，表现为 stdout 为空、 -exit code `1`，但 stderr 没有捕获到 V8 fatal stack。 - -### 严重程度 - -这是一个真实的 robustness 问题，但触发条件是 multi-GiB foreground stdout， -不是正常代码审查任务。它也不能证明当前 long-session heap-pressure 修复失败。 - -### 是否是本 PR 引入？ - -本轮没有证据表明 `2048 MiB` stdout failure 是当前 memory PR 引入的回归。 - -原因： - -- 失败路径是 foreground shell stdout decode / string construction。 -- 原 issue 路径是 long-session history、compaction、clone pressure。 -- 本轮没有做同 payload 的 pre-PR baseline，因此不能归因成 regression。 -- 该 failure 只在刻意极端的 `2048 MiB` payload 出现；`128 MiB` 到 - `1536 MiB` 都能完成。 - -建议把它作为 dedicated large-output follow-up：更早 stream / spool / hard-cap -foreground shell output，避免在内存里构造 multi-GiB JS string。除非当前 PR 的目标 -明确包含“任意 multi-GiB 前台 stdout 都必须可处理”，否则不建议把它作为当前 PR 的 -blocker。 - -## 结论 - -1. 最新本地 `0.15.11` 构建在 issue 报告的 long-session heap OOM 方向上明显更好。 - 基于当前默认 heap 压测结果，可以认为本 PR 已经基本解决此前遇到的 - long-session heap OOM 复现路径。 - -2. 在默认 Node heap 下，真实长任务 + 多 agent review loop 没有在 - `pai/glm-5`、`qwen3.6-plus`、`DeepSeek/deepseek-v4-pro` 三个模型上复现传统 - V8 heap OOM。 - -3. synthetic foreground stdout 压测仍能把 process-tree RSS 推得很高。当前构建在 - 三模型上都撑过了 `1536 MiB` payload，最高成功 tree RSS 是 `5964.7 MiB`。 - -4. 仍然存在一个独立的极端 large-output 问题：`2048 MiB` stdout 附近，Qwen Code - 可能在输出 JSON 结果前失败；DeepSeek case 捕获到了 V8 string-length fatal。 - -5. 这个新发现重要，但更像是后续 large-output robustness 问题，不应直接作为 - long-session heap-pressure mitigation 的 blocker。 - -## 建议发到 PR 的评论摘要 - -建议 PR 评论里只放精简摘要，完整数据放本文档： - -```markdown -I reran default-heap stress tests on the latest local build with -`pai/glm-5`, `qwen3.6-plus`, and `DeepSeek/deepseek-v4-pro`. - -No `NODE_OPTIONS`, `--max-old-space-size`, or runner `ulimit` was used. The -local Node heap limit was about 4144 MiB. - -Results: - -- Realistic long-session + multi-agent review loop: 23 worker turns, - ~719M reported total tokens, 77 agent calls, 856 total tool calls. - No traditional V8 heap OOM was reproduced. Peak process-tree RSS was - 874.7 MiB; peak root RSS was 219.1 MiB. -- Amplified stdout stress: 18 payload attempts across 128 MiB -> 2048 MiB. - All three models completed through 1536 MiB payloads without traditional - heap OOM. Highest successful process-tree RSS was 5964.7 MiB. -- At 2048 MiB foreground stdout, an extreme large-output failure remains. - DeepSeek captured a V8 fatal `Check failed: i::kMaxInt >= len` stack in - `String::NewFromOneByte` / `StringBytes::Encode` / `DecodeUTF8`. - -Conclusion: this PR appears to have effectively addressed the previously -observed long-session heap OOM reproduction path under default heap. The -2048 MiB stdout failure is a separate large-output/string-limit robustness issue -and should be tracked as a follow-up rather than treated as the same -long-session heap OOM regression. -``` diff --git a/docs/e2e-tests/worktree-phase-c.md b/docs/e2e-tests/worktree-phase-c.md deleted file mode 100644 index 8f7ae8d6c80..00000000000 --- a/docs/e2e-tests/worktree-phase-c.md +++ /dev/null @@ -1,594 +0,0 @@ -# Worktree Phase C E2E Test Plan - -## Scope - -End-to-end verification of Phase C features against the local build at -`/Users/mochi/code/qwen-code/.claude/worktrees/romantic-burnell-b6e48c/dist/cli.js`. - -Phase C delivers: - -- **Task 1, 3, 4** — `WorktreeSession` sidecar JSON file at - `~/.qwen/tmp//chats/.worktree.json` -- **Task 2** — `core.hooksPath` configured inside new worktrees -- **Task 5–6** — `useWorktreeSession` hook, `UIState.activeWorktree`, Footer - worktree indicator, `StatusLineCommandInput.worktree` field -- **Task 7** — `--resume` injects an INFO history item when active worktree - still exists; cleans up stale sidecar otherwise -- **Task 8** — `WorktreeExitDialog` with dirty-state inspection, intercepts - second Ctrl+C in active worktree - -## Binaries - -- **Local build**: `node /Users/mochi/code/qwen-code/.claude/worktrees/romantic-burnell-b6e48c/dist/cli.js` -- **Baseline (for pre-impl comparison if needed)**: globally installed `qwen` - -## Test environment template - -Each group runs in its own temp git repo and tmux session: - -```bash -TEST_DIR=$(mktemp -d -t qwen-wt-phc-XXXXXX) -TEST_DIR=$(cd "$TEST_DIR" && pwd -P) # resolve symlinks (macOS /var → /private/var) -cd "$TEST_DIR" -git init -q -b main -git config user.email t@e.com -git config user.name t -git config commit.gpgsign false -echo "hello" > README.md -git add README.md -git commit -q -m "initial" --no-verify -``` - -`QWEN=/Users/mochi/code/qwen-code/.claude/worktrees/romantic-burnell-b6e48c/dist/cli.js` - ---- - -## Group A: WorktreeSession sidecar (headless) - -**Mode:** headless, `--approval-mode yolo`, `--output-format json` - -### A1: enter_worktree writes sidecar with all fields - -**Steps:** - -```bash -SESSION=$(node $QWEN "use the enter_worktree tool with name='a1-test' to create a worktree" \ - --approval-mode yolo --output-format json 2>/dev/null \ - | jq -r '.[] | select(.type=="system") | .session_id' | head -1) - -PROJECT_ID=$(node -e "console.log(process.argv[1].replace(/[^a-zA-Z0-9]/g,'-'))" "$TEST_DIR") -SIDECAR=~/.qwen/projects/$PROJECT_ID/chats/$SESSION.worktree.json - -# Verify all fields present -cat "$SIDECAR" | jq '.slug, .worktreePath, .worktreeBranch, .originalCwd, .originalBranch, .originalHeadCommit' -``` - -**Expected:** - -- `slug` = "a1-test" -- `worktreePath` ends with `.qwen/worktrees/a1-test` -- `worktreeBranch` = "worktree-a1-test" -- `originalCwd` = `$TEST_DIR` (resolved) -- `originalBranch` = "main" -- `originalHeadCommit` matches `[0-9a-f]{40}` - -### A2: exit_worktree (keep) clears sidecar - -**Steps:** - -```bash -SESSION=$(node $QWEN "create a worktree named 'a2-test' using enter_worktree, then immediately exit it with action='keep' using exit_worktree" \ - --approval-mode yolo --output-format json 2>/dev/null \ - | jq -r '.[] | select(.type=="system") | .session_id' | head -1) - -SIDECAR=~/.qwen/projects/$PROJECT_ID/chats/$SESSION.worktree.json -test ! -f "$SIDECAR" && echo "PASS: sidecar removed" || echo "FAIL: sidecar still exists" -``` - -**Expected:** sidecar file does not exist after the exit_worktree call. - -### A3: exit_worktree (remove) clears sidecar - -**Steps:** - -```bash -SESSION=$(node $QWEN "create a worktree named 'a3-test' using enter_worktree, then immediately exit it with action='remove' and discard_changes=true using exit_worktree" \ - --approval-mode yolo --output-format json 2>/dev/null \ - | jq -r '.[] | select(.type=="system") | .session_id' | head -1) - -SIDECAR=~/.qwen/projects/$PROJECT_ID/chats/$SESSION.worktree.json -test ! -f "$SIDECAR" && echo "PASS: sidecar removed" || echo "FAIL: sidecar still exists" -# Also verify the worktree dir is gone -test ! -d "$TEST_DIR/.qwen/worktrees/a3-test" && echo "PASS: worktree dir removed" -``` - -**Expected:** both the sidecar AND the worktree directory are gone. - ---- - -## Group B: hooksPath configuration (headless) - -### B1: Without `.husky/`, hooksPath = `/.git/hooks` - -**Steps:** - -```bash -node $QWEN "use enter_worktree with name='b1-test' to create a worktree" \ - --approval-mode yolo --output-format json 2>/dev/null > /dev/null - -HOOKS_PATH=$(git -C "$TEST_DIR/.qwen/worktrees/b1-test" config --local core.hooksPath) -echo "Got hooksPath: $HOOKS_PATH" -test "$HOOKS_PATH" = "$TEST_DIR/.git/hooks" && echo "PASS" || echo "FAIL" -``` - -**Expected:** `$TEST_DIR/.git/hooks` - -### B2: With `.husky/`, hooksPath = `/.husky` - -**Steps:** - -```bash -mkdir -p "$TEST_DIR/.husky" -echo '#!/bin/sh' > "$TEST_DIR/.husky/pre-commit" -chmod +x "$TEST_DIR/.husky/pre-commit" - -node $QWEN "use enter_worktree with name='b2-test' to create a worktree" \ - --approval-mode yolo --output-format json 2>/dev/null > /dev/null - -HOOKS_PATH=$(git -C "$TEST_DIR/.qwen/worktrees/b2-test" config --local core.hooksPath) -test "$HOOKS_PATH" = "$TEST_DIR/.husky" && echo "PASS" || echo "FAIL got=$HOOKS_PATH" -``` - -**Expected:** `$TEST_DIR/.husky` - -### B3: Hooks in main repo actually fire from inside worktree - -**Steps:** - -```bash -# Set up a hook that writes a marker file -mkdir -p "$TEST_DIR/.git/hooks" -cat > "$TEST_DIR/.git/hooks/pre-commit" <<'EOF' -#!/bin/sh -echo "hook-fired" > /tmp/qwen-wt-hook-marker -EOF -chmod +x "$TEST_DIR/.git/hooks/pre-commit" - -node $QWEN "use enter_worktree with name='b3-test' to create a worktree" \ - --approval-mode yolo --output-format json 2>/dev/null > /dev/null - -# Commit something inside the worktree -WT="$TEST_DIR/.qwen/worktrees/b3-test" -echo "x" > "$WT/file.txt" -git -C "$WT" add file.txt -rm -f /tmp/qwen-wt-hook-marker -git -C "$WT" commit -m "trigger hook" 2>&1 -test -f /tmp/qwen-wt-hook-marker && echo "PASS: hook fired" || echo "FAIL: hook did not fire" -rm -f /tmp/qwen-wt-hook-marker -``` - -**Expected:** `/tmp/qwen-wt-hook-marker` exists after the commit. - ---- - -## Group C: --resume worktree restoration (headless) - -### C1: --resume injects worktree context when sidecar present and dir alive - -**Steps:** - -```bash -# Create initial session with worktree -INIT_OUT=$(node $QWEN "use enter_worktree with name='c1-test' to create a worktree" \ - --approval-mode yolo --output-format json 2>/dev/null) -SESSION=$(echo "$INIT_OUT" | jq -r '.[] | select(.type=="system") | .session_id' | head -1) - -# Resume the session and ask "what's my context?" -RESUMED=$(node $QWEN --resume "$SESSION" "say SIDECAR-CONFIRM" \ - --approval-mode yolo --output-format json 2>/dev/null) - -# Look for the injected INFO message text in the conversation -echo "$RESUMED" | grep -q "Resumed.*Active worktree.*c1-test" && echo "PASS" || echo "FAIL: no context injection" -``` - -**Expected:** the JSON stream contains an INFO message referencing `c1-test`. - -### C2: --resume cleans up stale sidecar when worktree dir is gone - -**Steps:** - -```bash -INIT_OUT=$(node $QWEN "use enter_worktree with name='c2-test' to create a worktree" \ - --approval-mode yolo --output-format json 2>/dev/null) -SESSION=$(echo "$INIT_OUT" | jq -r '.[] | select(.type=="system") | .session_id' | head -1) -SIDECAR=~/.qwen/projects/$PROJECT_ID/chats/$SESSION.worktree.json - -# Delete the worktree directory out-of-band -rm -rf "$TEST_DIR/.qwen/worktrees/c2-test" -test -f "$SIDECAR" || { echo "SKIP: sidecar was already gone"; exit 0; } - -# Resume — should clean up the stale sidecar -node $QWEN --resume "$SESSION" "hello" --approval-mode yolo --output-format json 2>/dev/null > /dev/null -test ! -f "$SIDECAR" && echo "PASS: stale sidecar cleaned" || echo "FAIL: stale sidecar still present" -``` - -**Expected:** sidecar file is removed. - ---- - -## Group D: Footer worktree indicator (interactive tmux) - -### D1: Footer shows worktree indicator after enter_worktree - -**Steps:** - -```bash -tmux new-session -d -s wt-d1 -x 200 -y 50 \ - "cd $TEST_DIR && node $QWEN --approval-mode yolo" -sleep 3 - -tmux send-keys -t wt-d1 "use enter_worktree with name='d1-test'" -sleep 0.5 -tmux send-keys -t wt-d1 Enter - -for i in $(seq 1 30); do - sleep 2 - tmux capture-pane -t wt-d1 -p | grep -q "Type your message" && break -done - -# Capture and look for the worktree indicator line in Footer area -tmux capture-pane -t wt-d1 -p -S -100 > /tmp/wt-d1.out -grep -E "⎇.*worktree-d1-test.*\(d1-test\)" /tmp/wt-d1.out && echo "PASS" || \ - { echo "FAIL — captured output:"; cat /tmp/wt-d1.out; } -tmux kill-session -t wt-d1 -``` - -**Expected:** Footer contains a line like `⎇ worktree-d1-test (d1-test)`. - -### D2: Footer indicator disappears after exit_worktree (keep) - -**Steps:** - -```bash -tmux new-session -d -s wt-d2 -x 200 -y 50 \ - "cd $TEST_DIR && node $QWEN --approval-mode yolo" -sleep 3 - -tmux send-keys -t wt-d2 "use enter_worktree with name='d2-test'" -sleep 0.5 -tmux send-keys -t wt-d2 Enter -for i in $(seq 1 30); do sleep 2; tmux capture-pane -t wt-d2 -p | grep -q "Type your message" && break; done - -# Verify indicator showed -tmux capture-pane -t wt-d2 -p -S -100 | grep -q "⎇.*d2-test" || { echo "FAIL: indicator missing before exit"; tmux kill-session -t wt-d2; exit 1; } - -# Exit the worktree (keep) -tmux send-keys -t wt-d2 "use exit_worktree with name='d2-test' action='keep'" -sleep 0.5 -tmux send-keys -t wt-d2 Enter -for i in $(seq 1 30); do sleep 2; tmux capture-pane -t wt-d2 -p | grep -q "Kept worktree" && break; done - -sleep 2 # give Footer a tick to refresh after sidecar removal -tmux capture-pane -t wt-d2 -p -S -100 > /tmp/wt-d2-after.out -# After exit, the indicator should be gone from the bottom panel area -tail -5 /tmp/wt-d2-after.out | grep -q "⎇.*d2-test" && \ - echo "FAIL: indicator still showing" || echo "PASS" -tmux kill-session -t wt-d2 -``` - -**Expected:** worktree indicator disappears from Footer within ~2s of `exit_worktree`. - ---- - -## Group E: WorktreeExitDialog (interactive tmux) - -### E1: Second Ctrl+C in worktree shows dialog instead of quitting - -**Steps:** - -```bash -tmux new-session -d -s wt-e1 -x 200 -y 50 \ - "cd $TEST_DIR && node $QWEN --approval-mode yolo" -sleep 3 - -tmux send-keys -t wt-e1 "use enter_worktree with name='e1-test'" -sleep 0.5 -tmux send-keys -t wt-e1 Enter -for i in $(seq 1 30); do sleep 2; tmux capture-pane -t wt-e1 -p | grep -q "Type your message" && break; done - -# First Ctrl+C (cleanup; should show "Press Ctrl+C again to exit") -tmux send-keys -t wt-e1 C-c -sleep 0.3 -tmux capture-pane -t wt-e1 -p | grep -q "Press Ctrl+C again" || \ - { echo "FAIL: first Ctrl+C didn't show warning"; tmux kill-session -t wt-e1; exit 1; } - -# Second Ctrl+C — should show the WorktreeExitDialog, NOT quit -tmux send-keys -t wt-e1 C-c -sleep 2 - -# Verify the dialog rendered -tmux capture-pane -t wt-e1 -p -S -50 > /tmp/wt-e1.out -grep -q "Active worktree.*e1-test" /tmp/wt-e1.out && \ - grep -q "Keep worktree" /tmp/wt-e1.out && \ - grep -q "Remove worktree" /tmp/wt-e1.out && \ - echo "PASS" || { echo "FAIL — captured:"; cat /tmp/wt-e1.out; } -tmux kill-session -t wt-e1 -``` - -**Expected:** dialog shows three options (Keep / Remove / Cancel) and process is still alive. - -### E2: Dialog shows dirty-state counts (commits + files) - -**Steps:** - -```bash -tmux new-session -d -s wt-e2 -x 200 -y 50 \ - "cd $TEST_DIR && node $QWEN --approval-mode yolo" -sleep 3 - -tmux send-keys -t wt-e2 "use enter_worktree with name='e2-test'" -sleep 0.5 -tmux send-keys -t wt-e2 Enter -for i in $(seq 1 30); do sleep 2; tmux capture-pane -t wt-e2 -p | grep -q "Type your message" && break; done - -# Make the worktree dirty: 1 new commit + 1 uncommitted file -WT="$TEST_DIR/.qwen/worktrees/e2-test" -echo "new" > "$WT/new.txt" -git -C "$WT" add new.txt -git -C "$WT" commit -q -m "test commit" --no-verify -echo "dirty" > "$WT/uncommitted.txt" - -# Trigger exit dialog via Ctrl+C double-press -tmux send-keys -t wt-e2 C-c -sleep 0.3 -tmux send-keys -t wt-e2 C-c -sleep 3 # allow time for git status / rev-list - -tmux capture-pane -t wt-e2 -p -S -50 > /tmp/wt-e2.out -grep -qE "new commit|uncommitted file" /tmp/wt-e2.out && echo "PASS" || \ - { echo "FAIL — captured:"; cat /tmp/wt-e2.out; } -tmux kill-session -t wt-e2 -``` - -**Expected:** dialog body contains both "X new commit(s)" and "Y uncommitted file(s)". - -### E3: Cancel option dismisses dialog without exiting - -**Steps:** - -```bash -tmux new-session -d -s wt-e3 -x 200 -y 50 \ - "cd $TEST_DIR && node $QWEN --approval-mode yolo" -sleep 3 - -tmux send-keys -t wt-e3 "use enter_worktree with name='e3-test'" -sleep 0.5 -tmux send-keys -t wt-e3 Enter -for i in $(seq 1 30); do sleep 2; tmux capture-pane -t wt-e3 -p | grep -q "Type your message" && break; done - -# Trigger dialog -tmux send-keys -t wt-e3 C-c -sleep 0.3 -tmux send-keys -t wt-e3 C-c -sleep 3 - -# Navigate to Cancel (DOWN DOWN) and press Enter -tmux send-keys -t wt-e3 Down -sleep 0.2 -tmux send-keys -t wt-e3 Down -sleep 0.2 -tmux send-keys -t wt-e3 Enter -sleep 2 - -# Dialog should be gone; input prompt should be back -tmux capture-pane -t wt-e3 -p | grep -q "Type your message" && echo "PASS" || \ - { echo "FAIL — captured:"; tmux capture-pane -t wt-e3 -p; } - -# Verify the worktree was NOT removed -test -d "$TEST_DIR/.qwen/worktrees/e3-test" && echo "worktree intact" || echo "FAIL: worktree gone" -tmux kill-session -t wt-e3 -``` - -**Expected:** dialog closes, input prompt returns, worktree directory still exists. - -### E4: Keep option exits session but preserves worktree - -**Steps:** - -```bash -tmux new-session -d -s wt-e4 -x 200 -y 50 \ - "cd $TEST_DIR && node $QWEN --approval-mode yolo" -sleep 3 - -tmux send-keys -t wt-e4 "use enter_worktree with name='e4-test'" -sleep 0.5 -tmux send-keys -t wt-e4 Enter -for i in $(seq 1 30); do sleep 2; tmux capture-pane -t wt-e4 -p | grep -q "Type your message" && break; done - -# Trigger dialog and pick Keep (first option, already selected) -tmux send-keys -t wt-e4 C-c -sleep 0.3 -tmux send-keys -t wt-e4 C-c -sleep 3 -tmux send-keys -t wt-e4 Enter - -# Wait for process to exit -for i in $(seq 1 20); do - sleep 1 - tmux has-session -t wt-e4 2>/dev/null || break - tmux capture-pane -t wt-e4 -p | grep -q "\$ " && break # shell prompt back -done - -# Worktree directory should still exist -test -d "$TEST_DIR/.qwen/worktrees/e4-test" && echo "PASS: worktree preserved" || \ - echo "FAIL: worktree was removed" -tmux kill-session -t wt-e4 2>/dev/null || true -``` - -**Expected:** process exits, worktree directory remains on disk. - -### E5: Remove option exits session and deletes worktree - -**Steps:** - -```bash -tmux new-session -d -s wt-e5 -x 200 -y 50 \ - "cd $TEST_DIR && node $QWEN --approval-mode yolo" -sleep 3 - -tmux send-keys -t wt-e5 "use enter_worktree with name='e5-test'" -sleep 0.5 -tmux send-keys -t wt-e5 Enter -for i in $(seq 1 30); do sleep 2; tmux capture-pane -t wt-e5 -p | grep -q "Type your message" && break; done - -# Trigger dialog and pick Remove (DOWN, Enter) -tmux send-keys -t wt-e5 C-c -sleep 0.3 -tmux send-keys -t wt-e5 C-c -sleep 3 -tmux send-keys -t wt-e5 Down -sleep 0.2 -tmux send-keys -t wt-e5 Enter - -# Wait for exit -for i in $(seq 1 20); do - sleep 1 - tmux has-session -t wt-e5 2>/dev/null || break - tmux capture-pane -t wt-e5 -p | grep -q "\$ " && break -done - -# Worktree directory should be GONE -test ! -d "$TEST_DIR/.qwen/worktrees/e5-test" && echo "PASS: worktree removed" || \ - echo "FAIL: worktree still on disk" -# Branch should also be deleted -git -C "$TEST_DIR" branch --list | grep -q "worktree-e5-test" && \ - echo "FAIL: branch still present" || echo "PASS: branch removed" -tmux kill-session -t wt-e5 2>/dev/null || true -``` - -**Expected:** process exits, worktree directory deleted, branch `worktree-e5-test` deleted. - ---- - -## Group F: Real-user workflow simulation (interactive tmux) - -### F1: Full enter → edit → commit → resume → exit (keep) flow - -**Steps:** - -```bash -tmux new-session -d -s wt-f1 -x 200 -y 50 \ - "cd $TEST_DIR && node $QWEN --approval-mode yolo" -sleep 3 - -# Step 1: enter worktree -tmux send-keys -t wt-f1 "use enter_worktree with name='f1-feature' to create a worktree" -sleep 0.5 -tmux send-keys -t wt-f1 Enter -for i in $(seq 1 30); do sleep 2; tmux capture-pane -t wt-f1 -p | grep -q "Type your message" && break; done - -# Step 2: read the absolute worktree path so the model knows where to write -WT="$TEST_DIR/.qwen/worktrees/f1-feature" -tmux send-keys -t wt-f1 "write the file $WT/hello.txt with content 'hi from worktree'" -sleep 0.5 -tmux send-keys -t wt-f1 Enter -for i in $(seq 1 60); do sleep 2; tmux capture-pane -t wt-f1 -p | grep -q "Type your message" && break; done - -# Verify the file was actually written INSIDE the worktree -test -f "$WT/hello.txt" && grep -q "hi from worktree" "$WT/hello.txt" && \ - echo "PASS: file written inside worktree" || echo "FAIL: file not in worktree" - -# Step 3: Exit with keep via the tool -tmux send-keys -t wt-f1 "use exit_worktree with name='f1-feature' action='keep'" -sleep 0.5 -tmux send-keys -t wt-f1 Enter -for i in $(seq 1 30); do sleep 2; tmux capture-pane -t wt-f1 -p | grep -q "Kept worktree" && break; done - -# Step 4: Verify worktree still on disk after exit -test -d "$WT" && echo "PASS: worktree kept" || echo "FAIL: worktree removed" -test -f "$WT/hello.txt" && echo "PASS: file persists" || echo "FAIL" - -tmux kill-session -t wt-f1 -``` - -**Expected:** - -- File written to worktree directory (not main repo) -- After exit `keep`, both the worktree directory and the file remain - -### F2: Custom statusline receives `worktree` payload - -**Steps:** - -```bash -# Create a statusline script that prints the JSON it receives via stdin -SETTINGS_DIR=~/.qwen -SETTINGS_FILE=$SETTINGS_DIR/settings.json -cp -f "$SETTINGS_FILE" /tmp/qwen-settings-backup.json 2>/dev/null || true -mkdir -p "$SETTINGS_DIR" -SL_SCRIPT=/tmp/qwen-wt-statusline.sh -cat > $SL_SCRIPT <<'EOF' -#!/bin/sh -INPUT=$(cat) -echo "$INPUT" > /tmp/qwen-wt-statusline-input.json -WT_NAME=$(echo "$INPUT" | jq -r '.worktree.name // "no-worktree"') -echo "WT=$WT_NAME" -EOF -chmod +x $SL_SCRIPT - -cat > "$SETTINGS_FILE" <