Skip to content

feat(telemetry): emit a Goal lifecycle event and metrics - #11927

Merged
qqqys merged 7 commits into
QwenLM:mainfrom
qqqys:feat/goal-telemetry
Sep 16, 2026
Merged

qqqys merged 7 commits into
QwenLM:mainfrom
qqqys:feat/goal-telemetry

Conversation

@qqqys

@qqqys qqqys commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

What this PR does

Goals now show up in telemetry. Every Goal state transition an operator would act on becomes a qwen-code.goal_state log event, and three metrics follow from it: a transition counter, and the tokens spent and turns finished at the moment a Goal completes, is blocked, or reaches a usage limit.

The reported transitions are the user's controls (create, replace, edit, pause, resume, clear) and the stops (complete, blocked, usage_limited, verifier_reject). Per-turn turn_finished and checkpoint are left out: they happen every turn and would multiply the volume without adding a transition anyone acts on. verifier_accept is left out because it is always followed by the complete or blocked it accepted.

The event carries identifiers, enums and numbers only: the Goal id and revision, the resulting status, the limit kind, turn count and turn budget, tokens used and token budget, active time and active-time budget, and the objective's length. The objective, stop reasons and checkpoint failure text are never included. telemetry.logPrompts, which gates prompt text elsewhere, defaults to on, so it could not be the switch that keeps a user's objective out. Metrics carry only cause, status and limit_kind; the Goal id stays on the log record, for the same cardinality reason session.id is opt-in on metrics.

The subscriber lives where the Config builds the Goal runtime, the one place that holds both. It needed one change in the runtime. When a session resumes, restore() republishes the recovered Goal with the cause of the record it came from, so a paused Goal comes back as a pause broadcast; a subscriber cannot tell that from a live pause and would count it again on every resume. Broadcast listeners now receive an optional third argument, and the restore broadcast is the only one that marks itself replayed. Existing listeners that take two arguments are unaffected.

Why it's needed

A Goal runs on its own for many turns, and every way it ends is recorded only in the session transcript. There was no Goal event and no Goal metric, so questions like how often Goals end blocked, how much a completed Goal spends, or whether a turn or token budget is set too low could only be answered by reading transcripts one at a time. Codex counts goal outcomes in its metrics and Claude Code emits tengu_goal_* analytics events; neither carries the objective text, and neither does this.

Design: English · 简体中文

Reviewer Test Plan

How to verify

cd packages/core
npx vitest run src/telemetry/goal-events.test.ts src/telemetry/loggers.test.ts src/telemetry/qwen-logger/qwen-logger.test.ts src/telemetry/metrics.test.ts src/goals/goal-runtime.test.ts
npx vitest run src/config/config.test.ts -t Goal
 ✓ src/telemetry/qwen-logger/qwen-logger.test.ts (45 tests)
 ✓ src/telemetry/goal-events.test.ts (20 tests)
 ✓ src/telemetry/metrics.test.ts (54 tests)
 ✓ src/telemetry/loggers.test.ts (89 tests)
 ✓ src/goals/goal-runtime.test.ts (201 tests)
 Test Files  5 passed (5)
      Tests  409 passed (409)

 ✓ src/config/config.test.ts (761 tests | 731 skipped)
      Tests  30 passed | 731 skipped (761)

The two guards that keep a resumed session from reporting its Goal twice were checked by mutation. Removing the replayed mark from the restore broadcast fails the runtime case and the Config case; removing the subscriber's skip fails the Config case.

To see it live, run a Goal headless with --telemetry --telemetry-target local --telemetry-outfile <file> and look for qwen-code.goal_state records.

Evidence (Before & After)

Before — released 0.22.3. The same Goal runs to completion and telemetry records its turns and tool calls, but nothing about the Goal itself:

$ qwen --yolo -p '/goal set Read a.txt, b.txt and c.txt with your file tools and quote the exact contents of each.' \
    --telemetry --telemetry-target local --telemetry-outfile before.jsonl
event names: api_request 3, api_response 4, auth 1, config 1, file_operation 3, tool_call 6, user_prompt 1, session.start 1, session.end 1
goal_state records: 0
goal metric data points: 0

After — this branch. The same run reports the creation and the completion, with the figures and without the objective text. The metric counter shows each transition once per export; goal_id appears on no metric:

$ node dist/cli.js --yolo -p '/goal set Read a.txt, b.txt and c.txt with your file tools and quote the exact contents of each.' \
    --telemetry --telemetry-target local --telemetry-outfile after.jsonl
goal_state records: 2
  {"cause":"create","goal_id":"bde016a2-…","revision":1,"status":"active","turn_count":0,"tokens_used":0,"token_budget":30000000,"active_time_ms":9,"objective_length":86}
  {"cause":"complete","goal_id":"bde016a2-…","revision":1,"status":"complete","turn_count":1,"tokens_used":77402,"token_budget":30000000,"active_time_ms":20977,"objective_length":86}
goal metric data points:
  qwen-code.goal.transition.count {"cause":"create","status":"active"} 1
  qwen-code.goal.transition.count {"cause":"complete","status":"complete"} 1
  qwen-code.goal.tokens_used      {"cause":"complete"} count 1, sum 77402
  qwen-code.goal.turn_count       {"cause":"complete"} count 1, sum 1
goal_state records containing the objective text: 0
metric data points carrying goal_id: 0

Resuming that session recovers a completed Goal, and reports nothing for it. The user's next control on the recovered Goal is reported once, under the same Goal id:

$ node dist/cli.js -c -p '/goal' --telemetry --telemetry-target local --telemetry-outfile resume.jsonl
Goal complete: Read a.txt, b.txt and c.txt with your file tools and quote the exact contents of each.
Usage: 1 turn · 77,402 of 30,000,000 tokens
goal_state records: 0

$ node dist/cli.js -c -p '/goal clear' --telemetry --telemetry-target local --telemetry-outfile clear.jsonl
Goal cleared.
goal_state records: 1
  {"cause":"clear","goal_id":"bde016a2-…","revision":1}
  qwen-code.goal.transition.count {"cause":"clear"} 1

The records were read with a small script that decodes the telemetry file's JSON objects; timestamps and session.id are left out above. The objective text does appear in the same file on the user_prompt and API records, which telemetry.logPrompts governs as before.

Tested on

OS Status
🍏 macOS ⚠️
🪟 Windows ⚠️
🐧 Linux

Environment (optional)

Linux, npm run bundle, headless --output-format text against a live model with --telemetry --telemetry-target local --telemetry-outfile. Baseline against the globally installed 0.22.3.

Risk & Scope

  • Main risk or tradeoff: a new telemetry surface. It is active only where telemetry or usage statistics already are, and it carries no text a user or model wrote. The runtime change is additive: the third listener argument is optional and set on one broadcast.
  • Not validated / out of scope: spans for Goal turns, events for Goal proposal approvals, and counting subagent or verifier spend in tokens_used (the figure is the runtime's own meter, unchanged here).
  • Breaking changes / migration notes: none.

Linked Issues

Part of #4228

中文说明

这个 PR 做了什么

Goal 现在会出现在遥测里。每个运维人员会关心的 Goal 状态转换都会成为一条 qwen-code.goal_state 日志事件,并由它派生出三个指标:一个转换计数器,以及 Goal 完成、被 blocked 或达到用量上限那一刻已花费的 token 数和已完成的轮数。

上报的转换是用户的控制操作(createreplaceeditpauseresumeclear)和各种停止(completeblockedusage_limitedverifier_reject)。每轮都有的 turn_finishedcheckpoint 不上报:它们每轮都发生,会成倍放大事件量,却不代表任何人需要处理的转换。verifier_accept 也不上报,因为它之后总会紧跟它所接受的 completeblocked

事件只携带标识、枚举和数字:Goal id 与 revision、转换后的状态、限制类型、轮数与轮数预算、已用 token 与 token 预算、活跃时长与活跃时长预算,以及目标的长度。目标、停止原因和 checkpoint 失败文本一律不包含。别处用来控制 prompt 文本的 telemetry.logPrompts 默认开启,所以它不能充当把用户目标挡在外面的开关。指标只带 causestatuslimit_kind;Goal id 只留在日志记录上,理由与指标上 session.id 需要显式开启相同,都是基数问题。

订阅者放在 Config 构建 Goal 运行时的地方,这是唯一同时持有两者的位置。为此运行时需要一处改动。会话续跑时,restore() 会以所恢复记录的 cause 重新发布恢复出来的 Goal,所以一个 paused 的 Goal 会以 pause 广播回来;订阅者无法把它和一次实时 pause 区分开,每次续跑都会重复计数。现在广播监听器会收到一个可选的第三个参数,只有 restore 广播会把自己标为 replayed。只接收两个参数的现有监听器不受影响。

为什么需要

Goal 会自主运行很多轮,而它的每种结束方式都只记录在会话 transcript 里。之前没有 Goal 事件,也没有 Goal 指标,所以诸如"Goal 有多大比例以 blocked 结束"、"一个完成的 Goal 花了多少"、"轮数或 token 预算是否设得太低"这类问题,只能逐个翻 transcript 来回答。Codex 在指标里统计 goal 结果,Claude Code 发出 tengu_goal_* 分析事件;两者都不携带目标文本,本 PR 也不携带。

设计文档:English · 简体中文

评审验证方式

如何验证

cd packages/core
npx vitest run src/telemetry/goal-events.test.ts src/telemetry/loggers.test.ts src/telemetry/qwen-logger/qwen-logger.test.ts src/telemetry/metrics.test.ts src/goals/goal-runtime.test.ts
npx vitest run src/config/config.test.ts -t Goal
 ✓ src/telemetry/qwen-logger/qwen-logger.test.ts (45 tests)
 ✓ src/telemetry/goal-events.test.ts (20 tests)
 ✓ src/telemetry/metrics.test.ts (54 tests)
 ✓ src/telemetry/loggers.test.ts (89 tests)
 ✓ src/goals/goal-runtime.test.ts (201 tests)
 Test Files  5 passed (5)
      Tests  409 passed (409)

 ✓ src/config/config.test.ts (761 tests | 731 skipped)
      Tests  30 passed | 731 skipped (761)

防止续跑会话重复上报 Goal 的两道保护用变异验证过:去掉 restore 广播上的 replayed 标记,runtime 与 Config 各一条用例失败;去掉订阅者的跳过,Config 那条用例失败。

想看实际效果,用 --telemetry --telemetry-target local --telemetry-outfile <file> 以 headless 方式跑一个 Goal,然后查找 qwen-code.goal_state 记录。

证据(前后对比)

改动前——已发布的 0.22.3。 同一个 Goal 运行到完成,遥测记录了它的轮次与工具调用,但没有任何关于 Goal 本身的记录:

$ qwen --yolo -p '/goal set Read a.txt, b.txt and c.txt with your file tools and quote the exact contents of each.' \
    --telemetry --telemetry-target local --telemetry-outfile before.jsonl
event names: api_request 3, api_response 4, auth 1, config 1, file_operation 3, tool_call 6, user_prompt 1, session.start 1, session.end 1
goal_state records: 0
goal metric data points: 0

改动后——本分支。 同样的运行上报了创建与完成,带数值、不带目标文本。指标计数器每次导出显示一次各个转换;任何指标上都没有 goal_id

$ node dist/cli.js --yolo -p '/goal set Read a.txt, b.txt and c.txt with your file tools and quote the exact contents of each.' \
    --telemetry --telemetry-target local --telemetry-outfile after.jsonl
goal_state records: 2
  {"cause":"create","goal_id":"bde016a2-…","revision":1,"status":"active","turn_count":0,"tokens_used":0,"token_budget":30000000,"active_time_ms":9,"objective_length":86}
  {"cause":"complete","goal_id":"bde016a2-…","revision":1,"status":"complete","turn_count":1,"tokens_used":77402,"token_budget":30000000,"active_time_ms":20977,"objective_length":86}
goal metric data points:
  qwen-code.goal.transition.count {"cause":"create","status":"active"} 1
  qwen-code.goal.transition.count {"cause":"complete","status":"complete"} 1
  qwen-code.goal.tokens_used      {"cause":"complete"} count 1, sum 77402
  qwen-code.goal.turn_count       {"cause":"complete"} count 1, sum 1
goal_state records containing the objective text: 0
metric data points carrying goal_id: 0

续跑这个会话会恢复出一个已完成的 Goal,但不会为它上报任何记录。用户对恢复出来的 Goal 的下一次控制操作上报一次,Goal id 相同:

$ node dist/cli.js -c -p '/goal' --telemetry --telemetry-target local --telemetry-outfile resume.jsonl
Goal complete: Read a.txt, b.txt and c.txt with your file tools and quote the exact contents of each.
Usage: 1 turn · 77,402 of 30,000,000 tokens
goal_state records: 0

$ node dist/cli.js -c -p '/goal clear' --telemetry --telemetry-target local --telemetry-outfile clear.jsonl
Goal cleared.
goal_state records: 1
  {"cause":"clear","goal_id":"bde016a2-…","revision":1}
  qwen-code.goal.transition.count {"cause":"clear"} 1

以上记录用一个解码遥测文件中 JSON 对象的小脚本读取,省略了时间戳和 session.id。目标文本确实出现在同一文件的 user_prompt 与 API 记录里,这部分和以前一样由 telemetry.logPrompts 控制。

测试平台

OS Status
🍏 macOS ⚠️
🪟 Windows ⚠️
🐧 Linux

环境(可选)

Linux,npm run bundle,headless --output-format text 连接真实模型,使用 --telemetry --telemetry-target local --telemetry-outfile。基线为全局安装的 0.22.3。

风险与范围

  • 主要风险或权衡:新增一个遥测面。它只在已经启用遥测或使用统计的地方生效,并且不携带任何用户或模型写的文本。运行时改动是纯增量:第三个监听参数是可选的,且只在一次广播上设置。
  • 未验证 / 不在范围内:Goal 轮的 span、Goal 提案审批的事件,以及把子代理或 verifier 的花费计入 tokens_used(该数字是运行时自身的计量,本 PR 未改动)。
  • 破坏性变更 / 迁移说明:无。

关联 Issue

Part of #4228

A Goal runs on its own, and every way it stops is recorded only in the
session transcript: telemetry had no Goal event and no Goal metric.

Each committed transition an operator acts on (the user's controls and
the stops: complete, blocked, usage_limited, verifier_reject) is now a
`qwen-code.goal_state` event, subscribed where the Config builds the
runtime. Per-turn causes are left out. The event carries identifiers,
enums and numbers only; the objective, stop reasons and checkpoint
failure text are never sent, because `telemetry.logPrompts` defaults to
on. A transition counter and two outcome histograms keep the Goal id off
metric attributes.

`restore()` republishes a resumed session's Goal with the cause of the
record it recovered, which a subscriber cannot tell from the live
transition that first produced it. Broadcast listeners now receive an
optional third argument, and the restore broadcast is the only one that
marks itself `replayed`, so a resumed session does not report its Goal
a second time.
@qwen-code-ci-bot

qwen-code-ci-bot commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Qwen Triage finishedview run. See the stage comments in this thread for the result.

Qwen Triage 已完成 —— 查看运行。结果见本线程中的各阶段评论。

@qwen-code-ci-bot

Copy link
Copy Markdown
Collaborator

Thanks for the PR — this is an unusually well-documented one, and the design doc answers most of the questions I'd normally have to ask.

Template looks good ✓ — every required heading is filled in, the before/after evidence is real output rather than a claim, and the design doc ships in both English and Chinese with matching structure.

Problem. This isn't a bug fix, so there's no reproduction to demand — but the gap it closes is grounded rather than hypothetical. #4228 is the open /goal roadmap and it lists "observability for why the loop continued or ended" as motivation and "Goal progress events for logs/telemetry/debugging" as an actual deliverable, so this is a slice of planned work, not an invented one. The before/after is concrete: 0.22.3 emits zero Goal records for a Goal that runs to completion, this branch emits two plus four metric points.

One claim I could not verify, for the record: that Claude Code emits tengu_goal_*. Analytics event names don't appear in its public CHANGELOG, so that's unconfirmable from here. It doesn't carry much weight either way — /goal itself is all over that CHANGELOG (restore-on-resume, idle check-ins, stop conditions), so the area is clearly live upstream and that's the signal that matters.

Direction — this is where I'm stopping and handing it to a human. The PR creates a new telemetry surface: a new event name, three new metric names, and a documented entry in the telemetry reference, which together make it an external contract. Telemetry is one of the areas this gate escalates instead of deciding, so I'm not approving and not requesting changes — the call on whether we want this surface belongs to a maintainer.

My read is favourable, and I'd rather say why than just hand over a flag:

  • The privacy posture is the conservative one, and it's argued rather than asserted. Identifiers, enums and numbers only; the objective contributes just its code-point length; stop reasons and checkpoint text never appear. Most importantly it declines to gate the objective behind telemetry.logPrompts, because that setting defaults to on — gating there would mean shipping the text by default. That's the right call, and it's worth a maintainer confirming it as the house position rather than letting it settle as one PR's local decision.
  • Cardinality discipline follows existing precedent: goal_id stays on the log record and off the metrics, which is the same reasoning as session.id being opt-in.
  • Volume is bounded by user actions. The ten reported causes are controls and stops; per-turn turn_finished and checkpoint are deliberately excluded, and verifier_accept is dropped because the stop it accepted always follows.

So the open questions are product questions, not code questions: do we want Goal telemetry now, is this the field set we're prepared to be stuck with, and is the analytics-sink payload — which drops goal_id but keeps revision, cause, status and the spend figures — what we want leaving the machine when usage statistics are on?

Size. Core paths are touched (packages/core/src/**, single package, not cross-package). Breakdown: 347 production lines in core, 459 test lines, 114 docs lines. It's a feat, so no size block applies, and it sits under both the 500-line maintainer-awareness threshold and the 1000-line large-PR advisory.

Worth recording that Tier 2's "name every downstream consumer" is satisfiable here, because the one core-contract change is the broadcast signature. subscribe gains an optional third meta argument, and it has exactly one production consumer — packages/core/src/core/client.ts:2967, which passes a two-argument listener and is unaffected. Everything else is the new Config subscriber. That's a genuinely bounded core change.

Approach. The scope feels right. I looked for the 80% cut and couldn't find it: the two histograms are the part that actually answers "how much does a completed Goal spend", which is the stated motivation, so dropping them would leave the counter answering nothing a transcript couldn't. It also reuses rather than duplicates — elapsedActiveTime from goal-reducer, the existing counter and histogram definition tables, createActionEvent/enqueueLogEvent, and the established makeXEvent factory shape. No drive-by refactors or formatting churn; the index.ts changes are pure export additions.

One genuine question, not a blocker: the runtime needed a new GoalBroadcastMeta so the Config subscriber can skip the restore replay and not count a resumed Goal twice. That widens a core protocol surface for a single telemetry consumer. The design doc explains why the alternatives are worse — comparing against getRecoveryCause() gets ambiguous as soon as a later live transition carries the same cause — and I agree the double-count-on-resume problem is real. But if a maintainer would rather not grow the broadcast contract, the alternative is dedupe state inside the subscriber. Worth saying out loud before merge, because the protocol change outlives the telemetry code that motivated it.

Risk. No elevated risk signals — none of the changed files match the high-risk paths (no shell, sandbox, MCP, ACP, geminiChat, or streaming-parser surface).

Escalating to @zjunothing for the direction call above. Two notes on how that landed, since neither is obvious from the PR:

  • The deterministic owner resolver could not run. It matches on labels, and this PR carries none; the latest-human-reviewer fallback was empty too, there being no reviews yet. .github/issue-owners.json maps packages/core/src/telemetry/ to the core-telemetry area under that owner, which is where the mention comes from. Adding scope/core would let the resolver route it on its own next time — I've left the labels alone rather than guess at them.
  • The fork-refactor approval guardrail does not apply: this is a feat, not a refactor. Flagging only because the PR comes from a fork and a reader might assume otherwise. The author does hold write access, and /verify is available directly rather than as a sponsored run if anyone wants the behavioural claims settled in a sandbox.

Stopping here rather than continuing into code review, since the direction question is the one that decides whether this surface should exist at all.

中文说明

感谢贡献!这个 PR 的文档非常完整,设计文档基本回答了我通常会提的问题。

模板完整 ✓ —— 所有必填小节都写了,before/after 是真实输出而不是声明,设计文档中英文双语齐备且结构一致。

问题。 这不是 bug fix,所以不需要复现——但它要补的缺口是有依据的,不是假想的。#4228 是仍处于 open 状态的 /goal roadmap,里面明确把"observability for why the loop continued or ended"列为动机,把"Goal progress events for logs/telemetry/debugging"列为实际交付项,所以这是既定工作的一部分,而不是凭空造出来的需求。前后对比很具体:0.22.3 对一个跑到完成的 Goal 发不出任何 Goal 记录,本分支发出 2 条记录加 4 个指标点。

有一条声明我无法核实,在此记录:Claude Code 发出 tengu_goal_*。分析事件名不会出现在它的公开 CHANGELOG 里,所以从这里无法确认。这一点两种方向都不影响判断——/goal 本身在那个 CHANGELOG 里出现得非常频繁(续跑恢复、空闲 check-in、停止条件),说明这个方向在上游是活跃的,这才是有意义的信号。

方向——我在这里停下,交给人类判断。 这个 PR 新建了一个遥测面:一个新事件名、三个新指标名,以及在遥测参考文档里新增一条记录,这些加起来构成了对外契约。遥测属于本 gate 需要上报而不是自行决定的领域,所以我既不 approve 也不 request changes——这个面该不该存在,应该由 maintainer 决定。

我的判断是正面的,与其只丢一个标记过来,不如说清理由:

  • 隐私姿态是保守的那一种,而且是论证出来的,不是宣称出来的。只带标识、枚举和数字;objective 只贡献它的码点长度;停止原因和 checkpoint 文本一律不出现。最关键的是它拒绝把 objective 放在 telemetry.logPrompts 后面,因为那个开关默认开启——放在它后面等于默认就上报文本。这个选择是对的,值得 maintainer 把它确认为项目的一致立场,而不是让它作为某一个 PR 的局部决定沉淀下来。
  • 基数控制沿用了既有先例:goal_id 只留在日志记录上、不进指标,理由与 session.id 需要显式开启相同。
  • 事件量由用户操作限定。上报的十个 cause 都是控制操作和停止;每轮的 turn_finishedcheckpoint 被明确排除,verifier_accept 也被排除,因为它所接受的那次停止总会紧随其后。

所以待决的是产品问题,不是代码问题:我们现在是否要 Goal 遥测;这套字段是否是我们准备好长期背下来的;以及在开启使用统计时,分析 sink 的载荷(去掉 goal_id,但保留 revisioncausestatus 和花费数值)是否是我们愿意让它离开本机的内容。

规模。 触及核心路径(packages/core/src/**,单个 package,非跨包)。明细:核心生产代码 347 行测试 459 行文档 114 行。类型是 feat,不适用规模硬阻断,也低于 500 行的维护者关注阈值和 1000 行的大 PR 提示阈值。

有一点值得记录:Tier 2 要求的"点名每一个下游消费者"在这里是可以满足的,因为唯一的核心契约改动就是广播签名。subscribe 增加了可选的第三个 meta 参数,而它在生产代码里只有一个消费者——packages/core/src/core/client.ts:2967,它传的是两参数监听器,不受影响。其余都是新增的 Config 订阅者。这是一个边界确实很小的核心改动。

方案。 范围是合理的。我找过能不能砍掉 80%,没找到:那两个直方图正是真正回答"一个完成的 Goal 花了多少"的部分,而这正是本 PR 的动机所在,砍掉它们之后计数器就回答不了任何 transcript 回答不了的问题。它也复用而不是重复造轮子——goal-reducerelapsedActiveTime、既有的 counter/histogram 定义表、createActionEvent/enqueueLogEvent,以及既有的 makeXEvent 工厂形态。没有夹带无关重构或格式抖动;index.ts 的改动纯粹是导出新增。

一个真问题,不是阻断项:运行时为此需要新增 GoalBroadcastMeta,好让 Config 订阅者跳过 restore 重播、不把续跑恢复出来的 Goal 重复计数。这是为一个遥测消费者拓宽了核心协议面。设计文档解释了为什么别的做法更差——一旦之后的实时转换带着相同 cause,与 getRecoveryCause() 比较就会产生歧义——我也同意"续跑重复计数"这个问题是真实存在的。但如果 maintainer 不愿意扩大广播契约,替代方案是把去重状态放在订阅者内部。这一点值得在合并前说出来,因为协议改动会比催生它的遥测代码活得更久。

风险。 无升级风险信号——改动文件都不匹配高风险路径(不涉及 shell、sandbox、MCP、ACP、geminiChat 或流式解析器)。

转交 @zjunothing 做上面的方向判断。关于这个转交是怎么得出的,有两点在 PR 上看不出来,一并说明:

  • 确定性的 owner 解析器没能运行。它按 label 匹配,而这个 PR 没有任何 label;"最近一位人类 reviewer"这个兜底也是空的,因为目前还没有 review。.github/issue-owners.jsonpackages/core/src/telemetry/ 映射到 core-telemetry 这个 area,其 owner 即上述维护者,@mention 来源于此。加上 scope/core 之后,下次解析器就能自行路由——我没有擅自改动 label,避免猜测。
  • fork refactor 的审批护栏适用:这是 feat,不是 refactor。之所以专门说明,是因为 PR 来自 fork,读者容易误以为适用。作者确实拥有 write 权限,因此如果需要在沙箱里核实那些行为声明,/verify 可以直接触发,不必走 sponsored run。

到此为止,不再进入代码审查——因为方向问题才是决定这个遥测面该不该存在的问题。

Qwen Code · qwen3.8-max-2026-09-02

Reviewed at 4b9d4a104bae8cc71f45a80f6cf615530fefae92 · re-run with @qwen-code /triage

@qwen-code-ci-bot

Copy link
Copy Markdown
Collaborator

Qwen Code review never started. The job ended before a runner picked it up, so no review ran for this head. This can happen when a queued run is cancelled or expires; this step cannot determine the cause. See workflow logs. Check the latest qwen-review-runner-schedule.yml run. If you still want a review, re-request with @qwen-code /review when the ecs-review pool is available.

@doudouOUC doudouOUC left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read all 20 changed files at the PR head, including the runtime broadcast sites the subscriber depends on and the six existing GoalRuntime.subscribe listeners. No defects found.

Verified, so they do not need re-arguing:

  • Privacy. The event carries only the Goal id (a randomUUID), enums and numbers; objective contributes [...goal.objective].length, makeGoalStateEvent strips undefined, and getCommonAttributes is session.id only. I also checked the new attributes against log-to-span-processor.ts's sensitive-key list — all primitives, no collision.
  • Replay dedupe. { replayed: true } is set on exactly one broadcast (goal-runtime.ts:1714, the restore replay) and the subscriber skips it; restore()/activateRestoredWork are memoized, and the subscriber is registered before either restore path starts, so the replay cannot be missed or double-counted.
  • Terminal paths all reach telemetry. complete/blocked through broadcast(attempt.proposal.status), usage_limited only through commitUsageLimitedSettle, the no-progress pause and every cancel/error/headless-end pause through dispatch({action:'pause'}), clear through broadcast(request.action). reduceGoalTurnFinished cannot change status, so no stop slips past the budget gate.
  • verifier_reject cannot double-emit. The broadcast('verifier_reject') and the turn-start broadcast(cause) in admitAfterRejection are mutually exclusive: activityBefore !== snapshot.activity is true exactly when flushContinuation already emitted it.
  • limit_kind cannot be stale on a complete/blocked event — the parser rejects limitKind unless status === 'usage_limited', and every exit from that status clears it. Metrics carry no goal_id (cardinality), and they no-op unless initializeMetrics ran — the same SDK init that gates logGoalState.

One observation, not filed as a finding. cause is the runtime's transition name rather than the reason a stop happened, and the no-progress bound shares pause with the user's /goal pause (goal-runtime.ts:2037 broadcasts 'pause', and :2246 reaches the same cause from request.action), distinguished only by a stop reason this event deliberately omits. So goal.transition.count{cause="pause"} counts runtime self-stops alongside deliberate user pauses.

I decided that is not a defect after checking the design doc's Problem section, which names the operator's questions as "how often Goals end blocked, how much a completed Goal spends, or whether a budget is set too low" — all three are answerable from the emitted series, and the cause list's own phrasing ("the user's controls and the stops a user acts on") can be read to include a bound the user must resume. If you do want an operator to separate the no-progress stops from deliberate pauses, it needs a second bounded dimension (an initiator or stop kind) rather than a new cause, since the reason strings are the thing you deliberately keep out; if not, the design doc's "the ways it stops" sentence is where a reader will look for that and it currently reads as if it were quantified.

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test Plan (not a blocker): npm run bundleno package defines this script; Tests 409 passed — this review observed 26489, 2113, 31595, 1028, 2040, 587, 8178 passed; Tests 30 passed — this review observed 26489, 2113, 31595, 1028, 2040, 587, 8178 passed.

中文说明

Test Plan(非阻断):npm run bundleno package defines this script; Tests 409 passed — this review observed 26489, 2113, 31595, 1028, 2040, 587, 8178 passed; Tests 30 passed — this review observed 26489, 2113, 31595, 1028, 2040, 587, 8178 passed

— qwen3.8-max via Qwen Code /review (v0.23.3)

Comment thread docs/users/features/goals.md Outdated
Comment thread docs/users/features/goals.md Outdated
Comment thread docs/users/features/goals.md Outdated
Comment thread docs/developers/development/telemetry.md Outdated
Comment thread packages/core/src/config/config.test.ts
Comment thread packages/core/src/telemetry/goal-events.ts
Comment thread packages/core/src/telemetry/goal-events.ts
Comment thread packages/core/src/telemetry/metrics.test.ts Outdated
Comment thread packages/core/src/telemetry/metrics.ts
Comment thread packages/core/src/telemetry/qwen-logger/qwen-logger.test.ts

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially reviewed — gaps disclosed. Suggestions are inline.

2 Suggestion-level finding(s) this review confirmed are already reported on this PR and are not repeated:

  • R1-3 no-progress stall shares the pause cause (packages/core/src/telemetry/goal-events.ts:61) — still stands in code; author scope-deferred and now disclosed in both references, already reported (comment 4015837671)
  • R1-16 replace reports only the successor Goal (packages/core/src/telemetry/goal-events.ts:51) — still stands in code; author scope-deferred and now disclosed in both references, already reported (comment 4015837664)

Not explored to full depth (tool budget reached): "agent reverse-audit (round 2)": could not execute a probe (test or live run) proving a *reported* cause settles on a superseded Goal runtime after a session rotation — finding 1 rests on readi…; "agent reverse-audit (round 3)": an executed positive control for the documented negatives — a live run showing a turn_finished / checkpoint / replayed broadcast reaching the config.ts:9851 ….

Not reviewed: reverse audit — did not converge within the reverse-audit round cap of 5.

Test Plan (not a blocker): src/telemetry/goal-events.test.tsno such file or directory; src/telemetry/loggers.test.tsno such file or directory; src/telemetry/qwen-logger/qwen-logger.test.tsno such file or directory; src/telemetry/metrics.test.tsno such file or directory; src/goals/goal-runtime.test.tsno such file or directory; and 3 more.

中文说明

仅完成部分审查,审查缺口已披露。 建议见行内评论。

本轮确认的 2 条建议级发现已在 PR 上报告过,不再重复发布(列表见上方英文部分)。

未探索到全部深度(达到工具调用预算):"agent reverse-audit (round 2)"could not execute a probe (test or live run) proving a *reported* cause settles on a superseded Goal runtime after a session rotation — finding 1 rests on readi…"agent reverse-audit (round 3)"an executed positive control for the documented negatives — a live run showing a turn_finished / checkpoint / replayed broadcast reaching the config.ts:9851 …

未审查:反向审计——在 5 轮的反审轮数上限内未收敛。

Test Plan(非阻断):src/telemetry/goal-events.test.tsno such file or directory; src/telemetry/loggers.test.tsno such file or directory; src/telemetry/qwen-logger/qwen-logger.test.tsno such file or directory; src/telemetry/metrics.test.tsno such file or directory; src/goals/goal-runtime.test.tsno such file or directory; and 3 more。

— qwen3.8-max via Qwen Code /review (v0.23.3)

Comment thread docs/developers/development/telemetry.md
Comment thread docs/users/features/goals.md Outdated
Comment thread packages/core/src/telemetry/goal-events.ts
Comment thread packages/core/src/telemetry/metrics.test.ts Outdated
Comment thread packages/core/src/telemetry/metrics.ts Outdated
Comment thread packages/core/src/telemetry/metrics.ts
Comment thread packages/core/src/telemetry/metrics.ts Outdated

@yiliang114 yiliang114 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM at 52df103 — no blocking issues. All 19 threads resolved; the one Critical (R1-15, the privacy claim) is fixed with precision at this head: the design doc now says the event carries the objective's length in code points and never the text, and a test pins that no objective/reason text appears in the serialized event. Test suite still running at approval time; approval is on the code, merge on green.

@qqqys
qqqys enabled auto-merge September 16, 2026 02:54

@qwen-code-dev-bot qwen-code-dev-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE

核对基线:head 52df103750628caadbc3979323d0a6e2f1623b44(base 49e8448c,20 个文件 +1226/-14)。required 档全部完成且成功(Test (ubuntu-latest, Node 22.x)Lint & StaticIntegration Tests (no-AK, No Sandbox) 等 13 项 pass,其余按改动面跳过;只有评审机器人自己的 review-pr 还在跑)。19 条线程 0 open,其中 1 条 [Critical] 我按当前 head 复核过。

那条 Critical 是要害:docs/users/features/goals.md 原来写「objective 文本与 stop reason 从不被记录」,而 logToolCall 不加闸门就导出 function_args,所以 propose_goal / update_goal 的参数(就是目标文本与模型写的理由)会进 OTLP,telemetry.logPrompts:false 也拦不住(shouldLogPrompts 那个开关只有一个读取点,在用户 prompt 记录器里)。当前 head 已把话说准:goals.md:98 明确「Goal 事件本身不含 objective 文本与 stop reason,只有码点长度;其它遥测可能含这些文本」,另外两份文档同样带范围限定,不再是无条件否定。

事件侧我逐字段核过,与文档一致:goal-events.ts:64 只放 objective_length: [...goal.objective].length,cause 必须命中 REPORTED_CAUSES 白名单才发(不在表里直接返回 undefined,不会把任意字符串灌进标签),outcome 直方图只收 complete/blocked/usage_limited;clear 事件只带被清掉的 Goal 与其 revision、不带用量。qwen-logger 那一侧显式省掉 Goal id,注释写明的理由就是「这个 sink 按安装聚合,逐 Goal 标识只会给每行加唯一值」——基数与隐私两个方向都收住了。另外我独立确认了那句「其它遥测可能含这些文本」不是套话:telemetry/types.ts:224function_args 只对 structured_output 一个工具做占位替换,其余工具(含 propose_goal/update_goal)原样带上 call.request.args,而 shouldLogUserPromptsloggers.ts 只有一个读取点(:276),确实管不到它 —— 所以文档选择如实描述边界,而不是继续声称不记录。没有发现新的阻塞问题。

@qqqys
qqqys added this pull request to the merge queue Sep 16, 2026
Merged via the queue into QwenLM:main with commit e18fa2c Sep 16, 2026
321 of 330 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants