feat(telemetry): emit a Goal lifecycle event and metrics - #11927
Conversation
A Goal runs on its own, and every way it stops is recorded only in the session transcript: telemetry had no Goal event and no Goal metric. Each committed transition an operator acts on (the user's controls and the stops: complete, blocked, usage_limited, verifier_reject) is now a `qwen-code.goal_state` event, subscribed where the Config builds the runtime. Per-turn causes are left out. The event carries identifiers, enums and numbers only; the objective, stop reasons and checkpoint failure text are never sent, because `telemetry.logPrompts` defaults to on. A transition counter and two outcome histograms keep the Goal id off metric attributes. `restore()` republishes a resumed session's Goal with the cause of the record it recovered, which a subscriber cannot tell from the live transition that first produced it. Broadcast listeners now receive an optional third argument, and the restore broadcast is the only one that marks itself `replayed`, so a resumed session does not report its Goal a second time.
|
Thanks for the PR — this is an unusually well-documented one, and the design doc answers most of the questions I'd normally have to ask. Template looks good ✓ — every required heading is filled in, the before/after evidence is real output rather than a claim, and the design doc ships in both English and Chinese with matching structure. Problem. This isn't a bug fix, so there's no reproduction to demand — but the gap it closes is grounded rather than hypothetical. #4228 is the open One claim I could not verify, for the record: that Claude Code emits Direction — this is where I'm stopping and handing it to a human. The PR creates a new telemetry surface: a new event name, three new metric names, and a documented entry in the telemetry reference, which together make it an external contract. Telemetry is one of the areas this gate escalates instead of deciding, so I'm not approving and not requesting changes — the call on whether we want this surface belongs to a maintainer. My read is favourable, and I'd rather say why than just hand over a flag:
So the open questions are product questions, not code questions: do we want Goal telemetry now, is this the field set we're prepared to be stuck with, and is the analytics-sink payload — which drops Size. Core paths are touched ( Worth recording that Tier 2's "name every downstream consumer" is satisfiable here, because the one core-contract change is the broadcast signature. Approach. The scope feels right. I looked for the 80% cut and couldn't find it: the two histograms are the part that actually answers "how much does a completed Goal spend", which is the stated motivation, so dropping them would leave the counter answering nothing a transcript couldn't. It also reuses rather than duplicates — One genuine question, not a blocker: the runtime needed a new Risk. No elevated risk signals — none of the changed files match the high-risk paths (no shell, sandbox, MCP, ACP, Escalating to @zjunothing for the direction call above. Two notes on how that landed, since neither is obvious from the PR:
Stopping here rather than continuing into code review, since the direction question is the one that decides whether this surface should exist at all. 中文说明感谢贡献!这个 PR 的文档非常完整,设计文档基本回答了我通常会提的问题。 模板完整 ✓ —— 所有必填小节都写了,before/after 是真实输出而不是声明,设计文档中英文双语齐备且结构一致。 问题。 这不是 bug fix,所以不需要复现——但它要补的缺口是有依据的,不是假想的。#4228 是仍处于 open 状态的 有一条声明我无法核实,在此记录:Claude Code 发出 方向——我在这里停下,交给人类判断。 这个 PR 新建了一个遥测面:一个新事件名、三个新指标名,以及在遥测参考文档里新增一条记录,这些加起来构成了对外契约。遥测属于本 gate 需要上报而不是自行决定的领域,所以我既不 approve 也不 request changes——这个面该不该存在,应该由 maintainer 决定。 我的判断是正面的,与其只丢一个标记过来,不如说清理由:
所以待决的是产品问题,不是代码问题:我们现在是否要 Goal 遥测;这套字段是否是我们准备好长期背下来的;以及在开启使用统计时,分析 sink 的载荷(去掉 规模。 触及核心路径( 有一点值得记录:Tier 2 要求的"点名每一个下游消费者"在这里是可以满足的,因为唯一的核心契约改动就是广播签名。 方案。 范围是合理的。我找过能不能砍掉 80%,没找到:那两个直方图正是真正回答"一个完成的 Goal 花了多少"的部分,而这正是本 PR 的动机所在,砍掉它们之后计数器就回答不了任何 transcript 回答不了的问题。它也复用而不是重复造轮子—— 一个真问题,不是阻断项:运行时为此需要新增 风险。 无升级风险信号——改动文件都不匹配高风险路径(不涉及 shell、sandbox、MCP、ACP、 转交 @zjunothing 做上面的方向判断。关于这个转交是怎么得出的,有两点在 PR 上看不出来,一并说明:
到此为止,不再进入代码审查——因为方向问题才是决定这个遥测面该不该存在的问题。 — Qwen Code · qwen3.8-max-2026-09-02 Reviewed at |
|
Qwen Code review never started. The job ended before a runner picked it up, so no review ran for this head. This can happen when a queued run is cancelled or expires; this step cannot determine the cause. See workflow logs. Check the latest |
doudouOUC
left a comment
There was a problem hiding this comment.
Read all 20 changed files at the PR head, including the runtime broadcast sites the subscriber depends on and the six existing GoalRuntime.subscribe listeners. No defects found.
Verified, so they do not need re-arguing:
- Privacy. The event carries only the Goal id (a
randomUUID), enums and numbers;objectivecontributes[...goal.objective].length,makeGoalStateEventstripsundefined, andgetCommonAttributesissession.idonly. I also checked the new attributes againstlog-to-span-processor.ts's sensitive-key list — all primitives, no collision. - Replay dedupe.
{ replayed: true }is set on exactly onebroadcast(goal-runtime.ts:1714, the restore replay) and the subscriber skips it;restore()/activateRestoredWorkare memoized, and the subscriber is registered before either restore path starts, so the replay cannot be missed or double-counted. - Terminal paths all reach telemetry.
complete/blockedthroughbroadcast(attempt.proposal.status),usage_limitedonly throughcommitUsageLimitedSettle, the no-progress pause and every cancel/error/headless-end pause throughdispatch({action:'pause'}),clearthroughbroadcast(request.action).reduceGoalTurnFinishedcannot change status, so no stop slips past the budget gate. verifier_rejectcannot double-emit. Thebroadcast('verifier_reject')and the turn-startbroadcast(cause)inadmitAfterRejectionare mutually exclusive:activityBefore !== snapshot.activityis true exactly whenflushContinuationalready emitted it.limit_kindcannot be stale on acomplete/blockedevent — the parser rejectslimitKindunlessstatus === 'usage_limited', and every exit from that status clears it. Metrics carry nogoal_id(cardinality), and they no-op unlessinitializeMetricsran — the same SDK init that gateslogGoalState.
One observation, not filed as a finding. cause is the runtime's transition name rather than the reason a stop happened, and the no-progress bound shares pause with the user's /goal pause (goal-runtime.ts:2037 broadcasts 'pause', and :2246 reaches the same cause from request.action), distinguished only by a stop reason this event deliberately omits. So goal.transition.count{cause="pause"} counts runtime self-stops alongside deliberate user pauses.
I decided that is not a defect after checking the design doc's Problem section, which names the operator's questions as "how often Goals end blocked, how much a completed Goal spends, or whether a budget is set too low" — all three are answerable from the emitted series, and the cause list's own phrasing ("the user's controls and the stops a user acts on") can be read to include a bound the user must resume. If you do want an operator to separate the no-progress stops from deliberate pauses, it needs a second bounded dimension (an initiator or stop kind) rather than a new cause, since the reason strings are the thing you deliberately keep out; if not, the design doc's "the ways it stops" sentence is where a reader will look for that and it currently reads as if it were quantified.
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Test Plan (not a blocker): npm run bundle — no package defines this script; Tests 409 passed — this review observed 26489, 2113, 31595, 1028, 2040, 587, 8178 passed; Tests 30 passed — this review observed 26489, 2113, 31595, 1028, 2040, 587, 8178 passed.
中文说明
Test Plan(非阻断):npm run bundle — no package defines this script; Tests 409 passed — this review observed 26489, 2113, 31595, 1028, 2040, 587, 8178 passed; Tests 30 passed — this review observed 26489, 2113, 31595, 1028, 2040, 587, 8178 passed。
— qwen3.8-max via Qwen Code /review (v0.23.3)
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Partially reviewed — gaps disclosed. Suggestions are inline.
2 Suggestion-level finding(s) this review confirmed are already reported on this PR and are not repeated:
- R1-3 no-progress stall shares the
pausecause (packages/core/src/telemetry/goal-events.ts:61) — still stands in code; author scope-deferred and now disclosed in both references, already reported (comment 4015837671) - R1-16
replacereports only the successor Goal (packages/core/src/telemetry/goal-events.ts:51) — still stands in code; author scope-deferred and now disclosed in both references, already reported (comment 4015837664)
Not explored to full depth (tool budget reached): "agent reverse-audit (round 2)": could not execute a probe (test or live run) proving a *reported* cause settles on a superseded Goal runtime after a session rotation — finding 1 rests on readi…; "agent reverse-audit (round 3)": an executed positive control for the documented negatives — a live run showing a turn_finished / checkpoint / replayed broadcast reaching the config.ts:9851 ….
Not reviewed: reverse audit — did not converge within the reverse-audit round cap of 5.
Test Plan (not a blocker): src/telemetry/goal-events.test.ts — no such file or directory; src/telemetry/loggers.test.ts — no such file or directory; src/telemetry/qwen-logger/qwen-logger.test.ts — no such file or directory; src/telemetry/metrics.test.ts — no such file or directory; src/goals/goal-runtime.test.ts — no such file or directory; and 3 more.
中文说明
仅完成部分审查,审查缺口已披露。 建议见行内评论。
本轮确认的 2 条建议级发现已在 PR 上报告过,不再重复发布(列表见上方英文部分)。
未探索到全部深度(达到工具调用预算):"agent reverse-audit (round 2)":could not execute a probe (test or live run) proving a *reported* cause settles on a superseded Goal runtime after a session rotation — finding 1 rests on readi…;"agent reverse-audit (round 3)":an executed positive control for the documented negatives — a live run showing a turn_finished / checkpoint / replayed broadcast reaching the config.ts:9851 …。
未审查:反向审计——在 5 轮的反审轮数上限内未收敛。
Test Plan(非阻断):src/telemetry/goal-events.test.ts — no such file or directory; src/telemetry/loggers.test.ts — no such file or directory; src/telemetry/qwen-logger/qwen-logger.test.ts — no such file or directory; src/telemetry/metrics.test.ts — no such file or directory; src/goals/goal-runtime.test.ts — no such file or directory; and 3 more。
— qwen3.8-max via Qwen Code /review (v0.23.3)
yiliang114
left a comment
There was a problem hiding this comment.
LGTM at 52df103 — no blocking issues. All 19 threads resolved; the one Critical (R1-15, the privacy claim) is fixed with precision at this head: the design doc now says the event carries the objective's length in code points and never the text, and a test pins that no objective/reason text appears in the serialized event. Test suite still running at approval time; approval is on the code, merge on green.
qwen-code-dev-bot
left a comment
There was a problem hiding this comment.
APPROVE
核对基线:head 52df103750628caadbc3979323d0a6e2f1623b44(base 49e8448c,20 个文件 +1226/-14)。required 档全部完成且成功(Test (ubuntu-latest, Node 22.x)、Lint & Static、Integration Tests (no-AK, No Sandbox) 等 13 项 pass,其余按改动面跳过;只有评审机器人自己的 review-pr 还在跑)。19 条线程 0 open,其中 1 条 [Critical] 我按当前 head 复核过。
那条 Critical 是要害:docs/users/features/goals.md 原来写「objective 文本与 stop reason 从不被记录」,而 logToolCall 不加闸门就导出 function_args,所以 propose_goal / update_goal 的参数(就是目标文本与模型写的理由)会进 OTLP,telemetry.logPrompts:false 也拦不住(shouldLogPrompts 那个开关只有一个读取点,在用户 prompt 记录器里)。当前 head 已把话说准:goals.md:98 明确「Goal 事件本身不含 objective 文本与 stop reason,只有码点长度;其它遥测可能含这些文本」,另外两份文档同样带范围限定,不再是无条件否定。
事件侧我逐字段核过,与文档一致:goal-events.ts:64 只放 objective_length: [...goal.objective].length,cause 必须命中 REPORTED_CAUSES 白名单才发(不在表里直接返回 undefined,不会把任意字符串灌进标签),outcome 直方图只收 complete/blocked/usage_limited;clear 事件只带被清掉的 Goal 与其 revision、不带用量。qwen-logger 那一侧显式省掉 Goal id,注释写明的理由就是「这个 sink 按安装聚合,逐 Goal 标识只会给每行加唯一值」——基数与隐私两个方向都收住了。另外我独立确认了那句「其它遥测可能含这些文本」不是套话:telemetry/types.ts:224 的 function_args 只对 structured_output 一个工具做占位替换,其余工具(含 propose_goal/update_goal)原样带上 call.request.args,而 shouldLogUserPrompts 在 loggers.ts 只有一个读取点(:276),确实管不到它 —— 所以文档选择如实描述边界,而不是继续声称不记录。没有发现新的阻塞问题。
What this PR does
Goals now show up in telemetry. Every Goal state transition an operator would act on becomes a
qwen-code.goal_statelog event, and three metrics follow from it: a transition counter, and the tokens spent and turns finished at the moment a Goal completes, is blocked, or reaches a usage limit.The reported transitions are the user's controls (
create,replace,edit,pause,resume,clear) and the stops (complete,blocked,usage_limited,verifier_reject). Per-turnturn_finishedandcheckpointare left out: they happen every turn and would multiply the volume without adding a transition anyone acts on.verifier_acceptis left out because it is always followed by thecompleteorblockedit accepted.The event carries identifiers, enums and numbers only: the Goal id and revision, the resulting status, the limit kind, turn count and turn budget, tokens used and token budget, active time and active-time budget, and the objective's length. The objective, stop reasons and checkpoint failure text are never included.
telemetry.logPrompts, which gates prompt text elsewhere, defaults to on, so it could not be the switch that keeps a user's objective out. Metrics carry onlycause,statusandlimit_kind; the Goal id stays on the log record, for the same cardinality reasonsession.idis opt-in on metrics.The subscriber lives where the Config builds the Goal runtime, the one place that holds both. It needed one change in the runtime. When a session resumes,
restore()republishes the recovered Goal with the cause of the record it came from, so a paused Goal comes back as apausebroadcast; a subscriber cannot tell that from a live pause and would count it again on every resume. Broadcast listeners now receive an optional third argument, and the restore broadcast is the only one that marks itselfreplayed. Existing listeners that take two arguments are unaffected.Why it's needed
A Goal runs on its own for many turns, and every way it ends is recorded only in the session transcript. There was no Goal event and no Goal metric, so questions like how often Goals end blocked, how much a completed Goal spends, or whether a turn or token budget is set too low could only be answered by reading transcripts one at a time. Codex counts goal outcomes in its metrics and Claude Code emits
tengu_goal_*analytics events; neither carries the objective text, and neither does this.Design: English · 简体中文
Reviewer Test Plan
How to verify
The two guards that keep a resumed session from reporting its Goal twice were checked by mutation. Removing the
replayedmark from the restore broadcast fails the runtime case and the Config case; removing the subscriber's skip fails the Config case.To see it live, run a Goal headless with
--telemetry --telemetry-target local --telemetry-outfile <file>and look forqwen-code.goal_staterecords.Evidence (Before & After)
Before — released 0.22.3. The same Goal runs to completion and telemetry records its turns and tool calls, but nothing about the Goal itself:
After — this branch. The same run reports the creation and the completion, with the figures and without the objective text. The metric counter shows each transition once per export;
goal_idappears on no metric:Resuming that session recovers a completed Goal, and reports nothing for it. The user's next control on the recovered Goal is reported once, under the same Goal id:
The records were read with a small script that decodes the telemetry file's JSON objects; timestamps and
session.idare left out above. The objective text does appear in the same file on theuser_promptand API records, whichtelemetry.logPromptsgoverns as before.Tested on
Environment (optional)
Linux,
npm run bundle, headless--output-format textagainst a live model with--telemetry --telemetry-target local --telemetry-outfile. Baseline against the globally installed 0.22.3.Risk & Scope
tokens_used(the figure is the runtime's own meter, unchanged here).Linked Issues
Part of #4228
中文说明
这个 PR 做了什么
Goal 现在会出现在遥测里。每个运维人员会关心的 Goal 状态转换都会成为一条
qwen-code.goal_state日志事件,并由它派生出三个指标:一个转换计数器,以及 Goal 完成、被 blocked 或达到用量上限那一刻已花费的 token 数和已完成的轮数。上报的转换是用户的控制操作(
create、replace、edit、pause、resume、clear)和各种停止(complete、blocked、usage_limited、verifier_reject)。每轮都有的turn_finished与checkpoint不上报:它们每轮都发生,会成倍放大事件量,却不代表任何人需要处理的转换。verifier_accept也不上报,因为它之后总会紧跟它所接受的complete或blocked。事件只携带标识、枚举和数字:Goal id 与 revision、转换后的状态、限制类型、轮数与轮数预算、已用 token 与 token 预算、活跃时长与活跃时长预算,以及目标的长度。目标、停止原因和 checkpoint 失败文本一律不包含。别处用来控制 prompt 文本的
telemetry.logPrompts默认开启,所以它不能充当把用户目标挡在外面的开关。指标只带cause、status和limit_kind;Goal id 只留在日志记录上,理由与指标上session.id需要显式开启相同,都是基数问题。订阅者放在 Config 构建 Goal 运行时的地方,这是唯一同时持有两者的位置。为此运行时需要一处改动。会话续跑时,
restore()会以所恢复记录的 cause 重新发布恢复出来的 Goal,所以一个 paused 的 Goal 会以pause广播回来;订阅者无法把它和一次实时 pause 区分开,每次续跑都会重复计数。现在广播监听器会收到一个可选的第三个参数,只有 restore 广播会把自己标为replayed。只接收两个参数的现有监听器不受影响。为什么需要
Goal 会自主运行很多轮,而它的每种结束方式都只记录在会话 transcript 里。之前没有 Goal 事件,也没有 Goal 指标,所以诸如"Goal 有多大比例以 blocked 结束"、"一个完成的 Goal 花了多少"、"轮数或 token 预算是否设得太低"这类问题,只能逐个翻 transcript 来回答。Codex 在指标里统计 goal 结果,Claude Code 发出
tengu_goal_*分析事件;两者都不携带目标文本,本 PR 也不携带。设计文档:English · 简体中文
评审验证方式
如何验证
防止续跑会话重复上报 Goal 的两道保护用变异验证过:去掉 restore 广播上的
replayed标记,runtime 与 Config 各一条用例失败;去掉订阅者的跳过,Config 那条用例失败。想看实际效果,用
--telemetry --telemetry-target local --telemetry-outfile <file>以 headless 方式跑一个 Goal,然后查找qwen-code.goal_state记录。证据(前后对比)
改动前——已发布的 0.22.3。 同一个 Goal 运行到完成,遥测记录了它的轮次与工具调用,但没有任何关于 Goal 本身的记录:
改动后——本分支。 同样的运行上报了创建与完成,带数值、不带目标文本。指标计数器每次导出显示一次各个转换;任何指标上都没有
goal_id:续跑这个会话会恢复出一个已完成的 Goal,但不会为它上报任何记录。用户对恢复出来的 Goal 的下一次控制操作上报一次,Goal id 相同:
以上记录用一个解码遥测文件中 JSON 对象的小脚本读取,省略了时间戳和
session.id。目标文本确实出现在同一文件的user_prompt与 API 记录里,这部分和以前一样由telemetry.logPrompts控制。测试平台
环境(可选)
Linux,
npm run bundle,headless--output-format text连接真实模型,使用--telemetry --telemetry-target local --telemetry-outfile。基线为全局安装的 0.22.3。风险与范围
tokens_used(该数字是运行时自身的计量,本 PR 未改动)。关联 Issue
Part of #4228