Skip to content

feat(workflows): add an orchestration policy layer to the Workflow tool description - #8694

Merged
wenshao merged 6 commits into
QwenLM:mainfrom
qqqys:feat/workflow-policy-layer
Aug 9, 2026
Merged

feat(workflows): add an orchestration policy layer to the Workflow tool description#8694
wenshao merged 6 commits into
QwenLM:mainfrom
qqqys:feat/workflow-policy-layer

Conversation

@qqqys

@qqqys qqqys commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

What this PR does

Rewrites the Workflow tool description so it carries an orchestration policy layer on top of the runtime facts it already stated. Every capability fact from the previous text is preserved — concurrency window, agent ceiling, per-call options, resume semantics, the background-tasks and /workflows surfaces, the sandbox boundary — and the environment variables that override the limits are now named inline. On top of that the description explains what a workflow is actually for, tells the model to scout the work list in the main loop before orchestrating over it, makes staggered per-item pipelining the default while spelling out the narrow cases where a synchronizing barrier is genuinely required, describes how to verify a subagent's answer instead of believing it, gives the deduplication rule that keeps unknown-size discovery loops terminating, and asks for honest reporting when a run bounds its own coverage.

This is prompt text only. No runtime, sandbox, orchestrator, or schema behavior changes.

Why it's needed

The runtime is already capable, but a model that is only handed the API reliably picks the naive shape: fan everything out through a single barrier, take the first answer at face value, and stop. That pays barrier latency on stages that never needed synchronizing, and lets plausible-but-wrong subagent findings through unchallenged. Nothing in the description previously pushed against either failure, so the quality of an orchestration depended entirely on whatever the model happened to improvise.

A test pins the load-bearing claims so a future edit cannot quietly reduce the description back to a capability list — there is no other signal that would catch it.

Reviewer Test Plan

How to verify

Read the new description as if you were the model deciding how to structure a fan-out: it should tell you when not to orchestrate at all, why per-item pipelining beats a barrier by default, and what to do with a finding before reporting it. Then confirm nothing regressed on the capability side — the concurrency default, the 1000-agent ceiling, the per-call options, resumeFromRunId, run_in_background, and the sandbox statement should all still be present and unchanged in meaning.

Unit tests for the tool: cd packages/core && npx vitest run src/tools/workflow/workflow.test.ts — 40 passed locally, including the new description test.

Evidence (Before & After)

N/A — not user-visible; the description is read by the model, not rendered in the TUI.

Tested on

OS Status
🍏 macOS N/A
🪟 Windows N/A
🐧 Linux

Environment (optional)

Unit tests only.

Risk & Scope

  • Main risk or tradeoff: a longer tool description costs context on every turn that has workflows enabled. The tradeoff is deliberate — the policy layer is what makes the runtime usable well — but the text was kept tighter than upstream's equivalent and omits guidance for capabilities this build does not have.
  • Not validated / out of scope: no measurement of how model-authored scripts change in practice; that needs live runs across a range of prompts.
  • Breaking changes / migration notes: none.

Linked Issues

Closes #8690

中文说明

这个 PR 做了什么

重写 Workflow 工具描述,在原有的运行时事实之上补齐编排策略层。此前文本中的每一条能力事实都被保留——并发窗口、agent 上限、按调用的选项、恢复语义、后台任务视图与 /workflows 界面、沙箱边界——并且把覆盖这些上限的环境变量就地写明。在此之上,描述还解释了 workflow 究竟是为什么而存在,要求模型先在主循环里侦察出工作清单再对其编排,把逐项错峰的 pipeline 定为默认并写清哪些情形才真正需要同步 barrier,说明如何验证子 agent 的答案而不是直接采信,给出让未知规模的发现循环得以终止的去重规则,并要求在一趟扫描自我设限时如实汇报。

纯提示词改动。运行时、沙箱、orchestrator、schema 的行为均无变化。

为什么需要

运行时本身已经够用,但只拿到 API 的模型必然选最朴素的形状:一个 barrier 一把梭,直接采信第一个答案,然后收工。这既在根本不需要同步的阶段上付出 barrier 延迟,也让"看着合理其实是错的"子 agent 结论毫无阻拦地通过。此前的描述对这两种失败模式都没有任何约束,一次编排的质量完全取决于模型临场发挥。

新增的测试锚定了几条承重论断,避免日后的编辑把描述悄悄退回成一份能力清单——除此之外没有任何信号能发现这种退化。

评审验证计划

如何验证

以"模型正在决定如何组织扇出"的视角读一遍新描述:它应当告诉你什么时候根本不该编排、为什么默认逐项 pipeline 优于 barrier、以及一条发现在上报之前该做什么。然后确认能力侧没有退化——并发默认值、1000 agent 上限、按调用的选项、resumeFromRunIdrun_in_background 以及沙箱声明都应仍然存在且语义不变。

工具单测:cd packages/core && npx vitest run src/tools/workflow/workflow.test.ts——本地 40 项通过,含新增的描述测试。

证据(前后对比)

N/A——非用户可见;该描述由模型读取,不在 TUI 中渲染。

测试环境

OS Status
🍏 macOS N/A
🪟 Windows N/A
🐧 Linux

运行环境(可选)

仅单元测试。

风险与范围

  • 主要风险或取舍:更长的工具描述会在每一个启用 workflow 的轮次上占用上下文。这个取舍是刻意的——策略层正是让运行时被用好的关键——但文本比上游同类描述更紧凑,且不含本 build 不具备的能力对应的指引。
  • 未验证/不在范围:没有测量模型实际产出的脚本因此发生了怎样的变化;那需要跨多种 prompt 的实机运行。
  • 破坏性变更/迁移说明:无。

关联 Issue

Closes #8690

…ol description

The description was an accurate technical specification that said nothing
about when to orchestrate, which shape to use, or how to trust what comes
back. With only the API in view, the naive shape wins every time: fan
everything out through one barrier and take the first answer at face value.

Adds the decision layer on top of the existing runtime facts — purpose
framing, pipeline-by-default with an explicit test for when a barrier is
genuinely required, scout-then-orchestrate, reusable shapes, adversarial
and perspective-diverse verification, the deduplicate-against-seen rule
that keeps discovery loops terminating, and honest reporting of bounded
coverage.

Prompt text only: no runtime, sandbox, or schema change.

Closes QwenLM#8690

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions github-actions Bot added the review/self-reported The linked issue was opened by the PR author (self-reported) label Aug 7, 2026
@qwen-code-ci-bot

qwen-code-ci-bot commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

🚫 Qwen Triage was cancelledview run. The run was cancelled before finishing. Check for a newer run before re-running.

🚫 Qwen Triage 已取消 —— 查看运行。运行未完成即被取消。重跑前请先确认是否有更新的运行。

@qwen-code-ci-bot

qwen-code-ci-bot commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the PR!

Template looks good ✓

Problem: real, with a linked issue (#8690, already triaged as priority/P2 and on the roadmap/subagents-tools roadmap). The gap is observable rather than theoretical — the current description in workflow.ts is a pure API specification with nothing on when to orchestrate, which shape to pick, or how to verify what comes back. Self-reported by the same author, which is fine here because the gap is verifiable by reading the current text.

Direction: aligned. The workflow runtime is already shipped and capable; the missing decision layer is what makes it usable well. The area is active upstream too — no direct reference to a policy-layer description, but clearly relevant.

Size: core path touched (packages/core/src/tools/workflow/): 84 production lines (+64/−20 in workflow.ts) plus 88 test lines. Well under every threshold; no maintainer-awareness flag needed.

Approach: minimal — the description moves into a WORKFLOW_TOOL_DESCRIPTION constant that carries the policy layer, the env knobs and agent cap are interpolated from the orchestrator's exported constants so there is no prose to hand-sync, and two tests pin the load-bearing claims (tool description + script parameter). Every item in the issue's six-point scope is covered; no unrelated edits. Nothing to cut.

Risk: no elevated risk signals — none of the changed files match the high-risk paths from the revert analysis.

Moving on to code review. 🔍

中文说明

感谢贡献!

模板完整 ✓

问题:真实存在,有关联 issue(#8690,已被 triage 标为 priority/P2 并列入 roadmap/subagents-tools 路线图)。缺口是可观察的而非理论性的——当前 workflow.ts 里的描述是一份纯 API 说明书,没有关于何时该编排、选哪种形状、如何验证结果的指引。issue 由同一作者自报,但缺口读一遍现有文本即可验证,没有问题。

方向:对齐。workflow 运行时已落地且能力完整,缺的正是让它被用好的决策层。上游在该领域也持续活跃——虽无直接对应的描述,但该领域明显相关。

规模:触及核心路径(packages/core/src/tools/workflow/):84 生产行(workflow.ts +64/−20),另有 88 行测试。远低于所有阈值,无需维护者关注标记。

方案:最小化——描述移入携带策略层的 WORKFLOW_TOOL_DESCRIPTION 常量,env 开关与 agent 上限直接从 orchestrator 导出的常量插值,无需手工同步文本;两个测试锚定承重论断(工具描述 + script 参数)。issue 六点范围全覆盖,无无关改动,无可裁剪项。

风险:无升级风险信号——改动文件均未命中 revert 分析中的高风险路径。

进入代码审查。🔍

Qwen Code · qwen3.8-max

Reviewed at 5a466444e7b1c18e667e1e17ce76f6d39089632f · re-run with @qwen-code /triage

@qwen-code-ci-bot

qwen-code-ci-bot commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Code review

Clean pass — no blocking findings, and this round re-verified the diff against the runtime end to end.

The shape is exactly what the issue asked for: the description moves out of the constructor into WORKFLOW_TOOL_DESCRIPTION, gains the policy layer, and the new tests pin the load-bearing claims in both model-visible copies (the tool description and the script parameter). What stands out is how the caps are kept honest: the agent ceiling and both env knobs are interpolated from the orchestrator's exported constants (DEFAULT_MAX_AGENTS_PER_RUN, MAX_WORKFLOW_AGENTS_ENV, MAX_WORKFLOW_CONCURRENCY_ENV) in both halves, so raising a cap moves every model-visible copy at once. The one literal left — "30-minute wall-clock cap" — is unavoidable (DEFAULT_MAX_WALL_CLOCK_MS is private to workflow-sandbox.ts) and is documented as hand-synced on both the code and the test side.

Every capability fact in the new text was checked against the implementation:

  • Concurrency default max(1, min(16, cpus-2)) and the 1000-agent ceiling match resolveConcurrencyLimit / DEFAULT_MAX_AGENTS_PER_RUN; the env knobs are named in the right places.
  • The one-level workflow() nesting limit is real — the nested sandbox is created without a workflow impl, so a second-level call throws.
  • Saved-workflow resolution (<projectRoot>/.qwen/workflows, ~/.qwen/workflows, project shadows user) matches workflow-saved.ts, and the slash-command surface is SavedWorkflowLoader.
  • budget.total === null = uncapped, budget refusal semantics (WorkflowBudgetExceededError; a refused slot becomes null inside parallel()/pipeline()), resumeFromRunId prefix-hash journal caching, run_in_background TUI gating, the /workflows dialog with cooperative pause/resume, and the node:vm sandbox statement all match the code.
  • All six policy sections from the issue are anchored by the tests, each with at least one assertion that goes red if the section is dropped.

Nothing outside workflow.ts depends on the old description string.

CI test evidence

The previous blocker is resolved: the branch now carries main (including the check:voice-guard-sync step that failed the earlier head), and the suite ran green on this commit.

Check Conclusion
Test (ubuntu-latest, Node 22.x) ✅ success
Desktop Shell (ubuntu-22.04) ✅ success
Desktop Shell (windows-2022) ✅ success
web-shell E2E Smoke (ubuntu-latest, Node 22.x) ✅ success
Classify PR ✅ success
precheck-pr / precheck ✅ success
review-pr ✅ success
Test (macos-latest, Node 22.x) ⏭️ skipped — merge queue only
Test (windows-latest, Node 22.x) ⏭️ skipped — merge queue only
Integration Tests (CLI, No Sandbox) ⏭️ skipped — merge queue only

The three skips are by design, not a gap: the macOS/Windows test legs and the CLI integration suite are merge_group-only jobs in ci.yml (named jobs rather than a matrix so the required check contexts resolve), so they never run on PR pushes and will execute when this PR enters the merge queue. Everything that runs on a PR push completed green on this SHA.

Not verified: live measurement of how model-authored scripts change under the new text — the PR declares that out of scope, and the merge-relevant claim (the description carries both the runtime facts and the policy) is exactly what the pinned tests substantiate. The maintainer's round-2/3 verification comments in this thread report live model script-authoring runs; that is the maintainer's own evidence and is recorded here as such.

中文说明

代码审查

干净通过——无阻塞项,本轮已将 diff 与运行时端到端重新核对。

形态正是 issue 所要的:描述从构造函数移入 WORKFLOW_TOOL_DESCRIPTION,补上策略层,新测试在两份模型可见文本(工具描述与 script 参数)中锚定承重论断。亮点在于上限如何保持诚实:agent 上限与两个 env 开关在两侧都从 orchestrator 导出的常量(DEFAULT_MAX_AGENTS_PER_RUNMAX_WORKFLOW_AGENTS_ENVMAX_WORKFLOW_CONCURRENCY_ENV)插值,抬高上限时所有模型可见副本同步移动。唯一保留的字面量——"30-minute wall-clock cap"——无法避免(DEFAULT_MAX_WALL_CLOCK_MSworkflow-sandbox.ts 私有),且代码与测试两侧都注明了需手工同步。

新文本中的每条能力事实都对照实现核实过:

  • 并发默认 max(1, min(16, cpus-2)) 与 1000 agent 上限和 resolveConcurrencyLimit / DEFAULT_MAX_AGENTS_PER_RUN 一致;env 开关位置正确。
  • 单层 workflow() 嵌套限制属实——嵌套沙箱创建时不带 workflow 实现,第二层调用会抛错。
  • 已保存 workflow 的解析(<projectRoot>/.qwen/workflows~/.qwen/workflows,项目层遮蔽用户层)与 workflow-saved.ts 一致;slash 命令入口是 SavedWorkflowLoader
  • budget.total === null = 无上限、预算拒绝语义(WorkflowBudgetExceededErrorparallel()/pipeline() 中被拒槽位变 null)、resumeFromRunId 前缀哈希日志缓存、run_in_background 仅限交互式 TUI、/workflows 对话框的协作式暂停/恢复、node:vm 沙箱声明,全部与代码一致。
  • issue 的六个策略章节在测试中各有至少一条断言,章节被删除即变红。

workflow.ts 之外没有任何代码依赖旧描述字符串。

CI 测试证据

此前的阻塞已解除:分支已带上 main(包含早前头部失败的 check:voice-guard-sync 步骤),套件在本提交上全绿。

三个 skipped 项是设计使然而非缺口:macOS/Windows 测试与 CLI 集成套件在 ci.yml 中是 merge_group 专属作业(用命名作业而非矩阵,以便必需检查上下文可解析),PR 推送时从不运行,PR 进入合并队列时执行。PR 推送时会跑的作业在本 SHA 上全部绿色完成。

未验证:新文本对模型实际产出脚本影响的在线测量——PR 已声明超出范围;与合并相关的论断(描述同时携带运行时事实与策略)正是锚定测试所证实的内容。maintainer 在本帖 round-2/3 验证评论中报告了在线模型脚本编写运行,那是 maintainer 自己的证据,此处如实标注。

Qwen Code · qwen3.8-max

Reviewed at 5a466444e7b1c18e667e1e17ce76f6d39089632f · re-run with @qwen-code /triage

@qwen-code-ci-bot

qwen-code-ci-bot commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Confidence: 5/5 — clean across every stage; the only earlier reservation (CI never ran the suite on the old head) is resolved by the main merge, and the suite is green on this commit.

Stepping back: this is what a good prompt-text PR looks like. The motivation reads straight off the current description, the scope is exactly the issue's six points and nothing else, every runtime fact in the new prose was re-checked against the code this round and holds, and the pinning tests guard the one realistic regression — a future edit quietly shrinking the description back to a capability list. Interpolating the caps from the orchestrator's exported constants rather than hard-coding them is the right call: it removes the hand-sync failure mode for everything except the one literal that cannot be exported, and that one is documented on both sides.

The author is upfront about the two tradeoffs — the longer description costs context on every workflow-enabled turn, and the effect on model-authored scripts is unmeasured by this PR. Both are fair to accept knowingly: the context cost is the price of the policy layer the issue asked for, and the maintainer's verification rounds in this thread (including live model script-authoring runs) cover the behavioral side.

No outstanding questions from this side. Approving, pinned to the reviewed commit. Thanks @qqqys — nice work seeing the branch staleness through. ✅

中文说明

置信度:5/5 —— 各阶段全部干净;此前唯一的保留意见(旧头部上 CI 从未跑到单测)已随合入 main 解除,套件在本提交上全绿。

退一步看:这是一份标准的提示词类 PR。动机在现有描述里一眼可见;范围恰好是 issue 的六点、别无其他;本轮已将新文本里每条运行时事实重新对照代码核实;锚定测试也恰好防住唯一现实的回归——日后某次编辑把描述悄悄退回成能力清单。上限从 orchestrator 导出的常量插值而非硬编码是正确做法:除唯一无法导出的字面量外,手工同步这一失败模式被彻底消除,而那个例外在两侧都有注明。

作者坦承两个取舍——更长的描述在每个启用 workflow 的轮次消耗上下文、本 PR 未测量对模型产出脚本的影响。两者都可以在知情前提下接受:上下文成本正是 issue 所要求策略层的代价,本帖中 maintainer 的多轮验证(含在线模型脚本编写运行)已覆盖行为面。

本侧无未决问题。批准,锚定在被审查的提交上。感谢 @qqqys 把分支过期问题一路跟到底。✅

Qwen Code · qwen3.8-max

Reviewed at 5a466444e7b1c18e667e1e17ce76f6d39089632f · re-run with @qwen-code /triage

The branch forked from 8fd0162, before 732f4d8 added the `check:voice-guard-sync` package.json script and its CI step. Because ci.yml checks out refs/pull/N/head while taking the workflow YAML from the merge ref, the step ran but the script did not exist, so Test (ubuntu-latest) failed with `npm error Missing script` and the unit suite never ran on this commit.

Merging current main brings the script in. No PR code changes: the diff against main is still only workflow.ts and workflow.test.ts.

Verified: npm run check:voice-guard-sync (passes), npm run build, npm run typecheck, and packages/core src/tools/workflow/workflow.test.ts (40/40).

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no blockers. Suggestions are inline. Test Plan (not a blocker): 40 passed — this review observed 1092, 18154, 1464, 475, 2868, 444 passed.

中文说明

已审查——无阻断问题。 建议见行内评论。 Test Plan(非阻断):40 passed — this review observed 1092, 18154, 1464, 475, 2868, 444 passed

— qwen3.8-max via Qwen Code /review (v0.21.7)

Comment on lines +56 to +57
it('description carries both the runtime facts and the orchestration policy', () => {
const { description } = new WorkflowTool(fakeConfig());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R1-1: The pinning test's anchors leave policy sections unguarded. The test exists — per its own comment — to catch "A refactor that drops the policy prose", but deleting the "What a workflow is for" or "Scout first, then orchestrate" section (including "Common single-phase shapes") from WORKFLOW_TOOL_DESCRIPTION keeps all 10 assertions green (verified by mutation probe). The linked issue's six-item scope includes purpose framing, hybrid scouting, and reusable shapes — none of them is anchored. — Failure scenario: a future refactor deletes those policy paragraphs → every anchor stays intact and the suite passes green, shipping exactly the regression class this test claims to catch for three of the issue's six content areas.

Add one anchor per uncovered section:

expect(description).toMatch(/Parallelism on its own is not a reason/);
expect(description).toMatch(/only before the orchestration step/);
expect(description).toMatch(/Common single-phase shapes/);
中文说明

[Suggestion] 锚定测试的锚点没有覆盖全部策略段落。该测试按其自身注释的说法,是为了捕获"删掉策略 prose 的重构",但经变异探针验证:从 WORKFLOW_TOOL_DESCRIPTION 中删除 "What a workflow is for" 或 "Scout first, then orchestrate"(含 "Common single-phase shapes")段落后,全部 10 条断言仍然为绿。关联 issue 的六点范围包含目的定位、混合侦察与可复用形状——这三项均未被锚定。失败场景:未来某次重构删掉这些策略段落 → 所有锚点原样存活、测试套件全绿,恰好把该测试声称要捕获的回归类型放行(涉及 issue 六点中的三点)。

— qwen3.8-max via Qwen Code /review (v0.21.7)


Reach for one to be comprehensive (decompose the work and cover every part in parallel), to be confident (independent perspectives and adversarial checks before an answer is committed to), or to take on scale a single context cannot hold — migrations, audits, broad sweeps. The script is where that structure is encoded: what fans out, what verifies, what synthesizes. Parallelism on its own is not a reason; work that is already one short sequence of edits belongs in the main loop.

**Runtime** — see the \`script\` parameter for the detailed authoring contract.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R1-2: The description newly advertises the workflow(nameOrRef, args?) and budget globals while pointing at the script parameter for "the detailed authoring contract" — but that contract documents neither global. The hard single-level nesting limit (the sandbox throws "workflow() nesting is limited to a single level") and budget's total/spent()/remaining() semantics appear nowhere model-visible. — Failure scenario: an author composing two levels of workflow() nesting — the natural reading of an advertised primitive — discovers the throwing stub only at runtime → the run has already burned agents/tokens on everything before the failing call.

For example, add to the Runtime paragraph: "workflow(nameOrRef, args?) runs a saved workflow sharing this run's caps; nesting is single-level (a workflow invoked via workflow() cannot call workflow() itself)", and/or document both globals in the script parameter description.

中文说明

[Suggestion] 描述新近宣传了 workflow(nameOrRef, args?)budget 全局,并指向 script 参数的"详细编写契约"——但该契约对这两个全局只字未提:单层嵌套的硬性上限(沙箱会抛出 "workflow() nesting is limited to a single level")以及 budgettotal/spent()/remaining() 语义在任何模型可见的文本中都不存在。失败场景:作者组合两层 workflow() 嵌套——对被宣传原语的自然读法——只在运行时才撞到 throwing stub → 此时本次运行已在失败调用之前烧掉了 agents/tokens。

— qwen3.8-max via Qwen Code /review (v0.21.7)

Comment on lines +518 to +519
* `script`'s own description carries the exact authoring contract (error
* strings, serialization rules, env overrides) and is not duplicated here.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R1-3: The unconditional 30-minute wall-clock cap per run (DEFAULT_MAX_WALL_CLOCK_MS; override QWEN_CODE_MAX_WORKFLOW_SECONDS) appears nowhere in the model-visible surface, while this description encourages runs that can exceed it ("up to 1000 agents total", migrations/audits/broad sweeps) — and this comment claims the script contract carries the env overrides, while only two of the three workflow knobs are enumerated in the Runtime sentence. The sandbox's own source comment computes that a legitimate 1000-agent fan-out already exceeds 30 minutes. — Failure scenario: a model-authored migration workflow legitimately needing more than 30 minutes → aborted mid-run by the wall-clock watchdog, in-flight subagents cancelled, partial result failed as execution_failed; recovery is an env var the model cannot set, discoverable only from the timeout error text after the run already died. (Mitigating: that error text does name the override.)

For example, extend the Runtime limits sentence: "…up to 1000 agents total (QWEN_CODE_MAX_WORKFLOW_CONCURRENCY / QWEN_CODE_MAX_WORKFLOW_AGENTS), under a 30-minute wall-clock cap per run (QWEN_CODE_MAX_WORKFLOW_SECONDS)".

中文说明

[Suggestion] 每次运行无条件 30 分钟的墙钟上限(DEFAULT_MAX_WALL_CLOCK_MS;覆盖变量 QWEN_CODE_MAX_WORKFLOW_SECONDS)在任何模型可见文本中都不存在,而本描述恰恰鼓励可能超过它的运行("最多 1000 个 agent"、迁移/审计/大面积排查)——且本注释声称 script 契约包含 env 覆盖,而 Runtime 句子里实际只列出了三个 workflow 旋钮中的两个。沙箱自身的源码注释已经算出:一次合理的 1000-agent 扇出就会超过 30 分钟。失败场景:模型编写的迁移 workflow 合理需要超过 30 分钟 → 被墙钟看门狗中途终止,在途 subagent 被取消,部分结果以 execution_failed 失败;恢复手段是一个模型无法设置的 env 变量,只能在运行死后从超时错误文本中发现。(缓解项:该错误文本确实写明了覆盖变量。)

— qwen3.8-max via Qwen Code /review (v0.21.7)

Comment on lines +58 to +60
for (const anchor of [
'min(16, cpus-2)',
'QWEN_CODE_MAX_WORKFLOW_AGENTS',

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R1-4: The anchor list pins only one of the two env-var names the Runtime paragraph advertises — QWEN_CODE_MAX_WORKFLOW_AGENTS is protected, but its sibling QWEN_CODE_MAX_WORKFLOW_CONCURRENCY appears exactly once in the description and is unprotected by any assertion (probe-verified: renaming the description occurrence escaped this test). — Failure scenario: a future edit corrupts or renames QWEN_CODE_MAX_WORKFLOW_CONCURRENCY in the description → all assertions stay green; the model instructs users to set a misspelled env var; the runtime reads MAX_WORKFLOW_CONCURRENCY_ENV correctly, so the override silently never applies and users conclude the concurrency knob is broken.

Suggested change
for (const anchor of [
'min(16, cpus-2)',
'QWEN_CODE_MAX_WORKFLOW_AGENTS',
for (const anchor of [
'min(16, cpus-2)',
'QWEN_CODE_MAX_WORKFLOW_AGENTS',
'QWEN_CODE_MAX_WORKFLOW_CONCURRENCY',
中文说明

[Suggestion] 锚点列表只钉住了 Runtime 段落宣传的两个环境变量名之一——QWEN_CODE_MAX_WORKFLOW_AGENTS 有保护,而其姊妹 QWEN_CODE_MAX_WORKFLOW_CONCURRENCY 在描述中恰好只出现一次且无任何断言保护(探针验证:重命名描述中的该变量可以逃过本测试)。失败场景:未来某次编辑损坏或重命名描述中的 QWEN_CODE_MAX_WORKFLOW_CONCURRENCY → 所有断言保持绿色;模型指导用户设置拼错的环境变量;运行时读取的是正确常量,覆盖静默失效,用户会以为并发旋钮坏了。

— qwen3.8-max via Qwen Code /review (v0.21.7)

Comment on lines +518 to +519
* `script`'s own description carries the exact authoring contract (error
* strings, serialization rules, env overrides) and is not duplicated here.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R1-5: This comment asserts the authoring contract "is not duplicated here", but the description below duplicates contract content verbatim — both env-override names, the max(1, min(16, cpus-2)) default and the 1000-agent cap, the per-call agent({…}) opts, resumeFromRunId cache semantics, and run_in_background behavior all appear both here and in the script/param descriptions. — Failure scenario: a future change to one of the duplicated facts (bumping DEFAULT_MAX_AGENTS_PER_RUN, renaming the concurrency env var) is applied to the runtime and the script param description but skips the tool description, because this comment tells the editor it carries no contract content → the drift ships silently for every duplicated fact the pinning test does not anchor, leaving the model-visible description contradicting the param contract.

Suggested change
* `script`'s own description carries the exact authoring contract (error
* strings, serialization rules, env overrides) and is not duplicated here.
* `script`'s own description carries the detailed authoring contract (error
* strings, serialization rules); the limits and env knobs restated above are
* a summary keep both in sync when they change.
中文说明

[Suggestion] 本注释断言编写契约"不会在此重复",但下方描述逐字重复了契约内容——两个 env 覆盖变量名、max(1, min(16, cpus-2)) 默认值与 1000-agent 上限、按调用的 agent({…}) 选项、resumeFromRunId 缓存语义、run_in_background 行为,在此处与 script/参数描述中均出现。失败场景:未来对某个重复事实的修改(如提高 DEFAULT_MAX_AGENTS_PER_RUN、重命名并发环境变量)应用到了运行时与 script 参数描述,却跳过了工具描述——因为本注释明确告诉编辑此处不含契约内容 → 对锚定测试未钉住的每一个重复事实,漂移都会静默上线,模型可见的描述将与参数契约相互矛盾。

— qwen3.8-max via Qwen Code /review (v0.21.7)


**Runtime** — see the \`script\` parameter for the detailed authoring contract.

\`phase(title)\`, \`log(msg)\`, \`agent(prompt, opts?)\`, \`parallel(thunks)\`, \`pipeline(items, ...stages)\`, \`workflow(nameOrRef, args?)\`, plus the \`args\` and \`budget\` globals. Default \`max(1, min(16, cpus-2))\` agents in flight per run, up to 1000 agents total (\`QWEN_CODE_MAX_WORKFLOW_CONCURRENCY\` / \`QWEN_CODE_MAX_WORKFLOW_AGENTS\`). Per-call \`agent({ schema, agentType, model, isolation: 'worktree' })\` covers structured-output contracts, declarative-agent selection, model override, and git-worktree-isolated subagents. \`resumeFromRunId\` resumes a prior run — agent() calls whose rolling prefix-hash matches the journal are served from cache for the longest unchanged prefix. Runs appear in the background-tasks view and the \`/workflows\` dialog (live phase tree, token usage, cancel); \`run_in_background: true\` returns a run handle immediately in the interactive TUI and delivers completion through the conversation. Scripts run in a node:vm sandbox with no filesystem or shell access — all I/O happens through the spawned agents.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R1-6: The per-run output-token cap QWEN_CODE_MAX_TOKENS_PER_WORKFLOW — which gates every agent() dispatch (countedDispatch throws WorkflowBudgetExceededError, cancelling in-flight agents and failing the run) — appears in neither model-visible half, even though this description newly advertises the budget global. It is the one cap operators are actively prompted to set (the first-success usage banner says exactly that), its error message deliberately names no knob (see the P5 R2 comment in workflow-budget.ts), and the banner is user-visible returnDisplay only. — Failure scenario: an operator follows the banner and sets a token cap; the model, primed by this description for large fan-outs ("migrations, audits, broad sweeps", "up to 1000 agents total"), never probes budget.total (whose null/cap semantics appear nowhere model-visible) → a mid-run budget-exceeded error cancels in-flight agents and fails the run with partial results; the first capped run dies un-warned.

Intent-compatible with the P5 R2 rationale (which deliberately keeps the env-var name out of model-reachable error text), add a probe clause rather than the knob name — e.g. "a per-run output-token cap may be in effect; read budget.total (null = uncapped) before large fan-outs" — to the Runtime paragraph and/or the script contract.

中文说明

[Suggestion] 每运行的输出 token 上限 QWEN_CODE_MAX_TOKENS_PER_WORKFLOW——它把关每一次 agent() 派发(countedDispatch 抛出 WorkflowBudgetExceededError,取消在途 agent 并使运行失败)——在两个模型可见 half 中均不存在,尽管本描述新近宣传了 budget 全局。它是产品主动提示操作者去设置的唯一上限(首次成功的 usage banner 原文如此),其错误消息刻意不写任何旋钮(见 workflow-budget.ts 的 P5 R2 注释),而 banner 仅存在于用户可见的 returnDisplay。失败场景:操作者照 banner 设置了 token 上限;模型被本描述鼓动去做大型扇出("迁移、审计、大面积排查"、"最多 1000 个 agent"),又从不探测 budget.total(其 null/上限语义在任何模型可见处都不存在)→ 运行中途的预算超限错误取消在途 agent、以部分结果使运行失败;第一次受限的运行毫无预警地死去。

与 P5 R2 的理由兼容(该理由刻意让 env 变量名远离模型可达的错误文本):添加探测子句而非旋钮名——例如 "a per-run output-token cap may be in effect; read budget.total (null = uncapped) before large fan-outs"——加到 Runtime 段落和/或 script 契约中。

— qwen3.8-max via Qwen Code /review (v0.21.7)

…icy section

Addresses the round-1 review on QwenLM#8694 (R1-1 through R1-6).

The description now names the three limits a model has to plan around
rather than discover from a mid-run failure: `workflow()` nests one level
only, the 30-minute wall-clock cap per run
(`QWEN_CODE_MAX_WORKFLOW_SECONDS`), and the per-run output-token cap —
surfaced as a `budget.total` probe rather than a knob name, keeping the
P5 R2 rule that the env var stays out of model-reachable text.

The pinning test grew anchors for the three policy sections a mutation
probe could delete while it stayed green, and for
`QWEN_CODE_MAX_WORKFLOW_CONCURRENCY`, which the description advertised
with nothing asserting its spelling.

The JSDoc claim that the authoring contract "is not duplicated here" was
false — the limits and env knobs appear in both halves — so it now says
they are a summary to keep in sync.
@wenshao

wenshao commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Local deep verification (maintainer round) — verdict: ✅ merge-ready · 68/68 scripted assertions passed · verified head 0ecc3160b3 (base 9e1b1eb49e)

中文摘要(Chinese summary)

结论:merge-ready(可以合并)。68/68 项脚本化断言通过,验证 head 0ecc3160b3

  • A/B 与变异矩阵:把描述换回 base 版本,新测试在预期断言处变红(缺 QWEN_CODE_MAX_WORKFLOW_AGENTS 锚点,workflow.test.ts:70);只删 "Verify before believing" 政策段(运行时事实全部保留)仍在 /refute/ 处变红(第 79 行);PR head 上 40/40 全绿。测试非空转,同时锚定了环境变量名与政策正文(见 02、03 截图)。
  • 事实核查 60/60:描述中每一条运行时声明——1000 agent 上限、默认并发 max(1, min(16, cpus-2))、三个环境变量名(QWEN_CODE_MAX_WORKFLOW_{AGENTS,CONCURRENCY,SECONDS})、30 分钟墙钟上限、budget.totalnull 即无上限、单层嵌套越级抛错、node:vm 沙箱无 fs/shell、resumeFromRunId 前缀哈希缓存、run_in_background 句柄语义、background-tasks 视图与 /workflows——与真实运行时常量逐一相符;旧描述 22 条能力事实全部保留("both env-overridable" 升级为三个具名环境变量);issue feat(workflows): add an orchestration policy layer to the Workflow tool description #8690 要求的六层编排策略齐备(见 01 截图)。
  • 上下文成本:描述 1000 → 5574 字符(约 250 → 约 1394 tokens,+约 1144 tokens/轮,仅在 workflows 启用时),系 PR 已声明的取舍。
  • 非阻塞小发现(2 条 suggestion):① "the run ends on partial results" 是实用简写——机制上超预算的 dispatch 会 reject,parallel()/pipeline() 槽位按 errors-as-data 变 null、运行继续;② 环境变量括注可被读作两个变量都覆盖总数,旧文本 "both env-overridable" 有同样歧义,非回归。
  • 未覆盖:模型行为是否真的改善的实测(PR 自述超出范围);macOS/Windows CI lane 因 fork PR 权限跳过(ubuntu 已绿,本轮补 macOS 本地证据);全量套件未本地重跑(CI 已覆盖)。

1. Central claim & A/B (mutation matrix)

Central claim: the description now carries both the runtime facts and an orchestration policy layer, and the new test pins both halves.

cell description under test new test outcome expected
base 9e1b1eb49e old description REDtoContain('QWEN_CODE_MAX_WORKFLOW_AGENTS') fails, workflow.test.ts:70
mutation runtime facts intact, "Verify before believing" section removed RED/refute/ fails, workflow.test.ts:79
PR head 0ecc3160b3 new description GREEN — 40/40

Both mutations failed on the intended assertion (quoted expected-vs-received, not an import/compile error), and the restored tree went green with 0 dirty entries — the test is load-bearing and pins the policy prose, not just the env-knob spellings.

vacuity: base description turns the new test red; restored 40/40 green

finer mutation: policy section removed, runtime facts kept, still red on /refute/

2. Fact-check: description ↔ real runtime (60/60)

Every capability claim in the new description was cross-checked against the runtime constants that read it (built dist/, imported through the same lazy dynamic-import path production uses): the 1000-agent ceiling, max(1, min(16, cpus-2)) default concurrency (13 on this box), all three env knobs honored at resolve time, the 30-minute wall-clock default, budget.total === null = uncapped, single-level workflow() nesting, node:vm sandbox with no fs/shell, resumeFromRunId prefix-hash semantics, run_in_background handle semantics, and the background-tasks//workflows surfaces. All 22 capability facts of the old description survive (plus the "both env-overridable" claim, upgraded to three named env vars), and all six policy layers requested by issue #8690 (purpose framing, pipeline-default + barrier test, hybrid scouting, reusable shapes, verification/convergence incl. the dedup-against-everything-seen trap, scale calibration + honest reporting) are present.

fact-check harness: 60/60 pass

3. Context cost (acknowledged tradeoff, measured)

metric old (base) new (PR) delta
chars 1,000 5,574 +4,574
words 134 848 +714
est. tokens (~4 chars/tok) ~250 ~1,394 ~+1,144

Paid on every turn with workflows enabled. The PR discloses this deliberately; the number is confirmed, not larger than stated.

4. Findings (non-blocking)

  1. Suggestion — "the run ends on partial results" is a practical shorthand, not the mechanical truth: countedDispatch rejects every over-cap dispatch with WorkflowBudgetExceededError, but in parallel()/pipeline() a rejected slot becomes null (errors-as-data) and the script continues; a sequential await agent() observes the rejection and may catch it. The model-facing advice ("read budget.total before committing to a large fan-out") is right either way. A future wording tweak could say "every further agent() call rejects (null slots in parallel()/pipeline())". Not a defect.
  2. Suggestion — the parenthetical (QWEN_CODE_MAX_WORKFLOW_CONCURRENCY/QWEN_CODE_MAX_WORKFLOW_AGENTS) follows "up to 1000 agents total" and could be read as both knobs overriding the total; the intended pairing is positional (concurrency↔in-flight window, agents↔total). The old text's "both env-overridable" carried the same ambiguity, so this is inherited, not a regression.

5. Not covered

  • No live measurement of whether models actually pick pipeline()/verify more with the new text — the PR itself declares this out of scope ("needs live runs across a range of prompts"). This round verifies facts and pinning, not behavioral uplift.
  • macOS/Windows CI lanes are skipped for this fork PR; ubuntu-latest Test passed (16m5s). This local round adds a macOS datapoint (darwin, Node v22.22.2).
  • The repo-wide suite was not re-run locally (CI covers it); the targeted workflow suite ran locally: 40/40.
  • The "/workflows … cancel" phrasing: cancellation lives in the background-tasks dialog; the phrase is inherited verbatim from the old description (pre-existing, unchanged by this PR).

6. Methodology

Scratch git worktree at the PR head (tmp/pr8694-head), npm ci (2,046 packages) + postinstall build; harnesses import the built dist/ via production's lazy dynamic-import path (config.ts:8232) to avoid the cold circular-import TDZ. Re-runnable harnesses: 01-factcheck.mjs (60 assertions), 02-vacuity.sh, 03-policy-mutation.sh, 04-gates.mjs; terminal captures rendered with scripts/verify-capture.mjs. Trial merge into current origin/main via git merge-tree --write-tree — conflict-free (base is 10 commits behind, none touching the workflow surface). No other code/snapshot/doc consumes the old description text. Raw logs, harnesses, and the wire declaration JSON live under tmp/pr8694-verify-20260808-120000/.

gates: head identity, clean tree, conflict-free trial merge, no stray consumers

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no blockers. Suggestions are inline. Test Plan (not a blocker): 40 passed — this review observed 19287, 1092, 18157, 1464, 475, 2868, 444 passed.

中文说明

已审查——无阻断问题。 建议见行内评论。 Test Plan(非阻断):40 passed — this review observed 19287, 1092, 18157, 1464, 475, 2868, 444 passed

— qwen3.8-max via Qwen Code /review (v0.21.7)

Comment on lines +63 to +65
'QWEN_CODE_MAX_WORKFLOW_AGENTS',
'QWEN_CODE_MAX_WORKFLOW_CONCURRENCY',
'QWEN_CODE_MAX_WORKFLOW_SECONDS',

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R2-1: The env-knob anchors are hardcoded string literals instead of the runtime's exported constants (MAX_WORKFLOW_AGENTS_ENV, MAX_WORKFLOW_CONCURRENCY_ENV in workflow-orchestrator.ts), so the guard only catches description-side typos — a rename on the runtime side drifts past it. — Failure scenario: a future PR renames MAX_WORKFLOW_AGENTS_ENV (or its string value) and updates workflow-orchestrator.test.ts's hardcoded literals, but misses WORKFLOW_TOOL_DESCRIPTION → this test stays green while the model tells users to set a variable nothing reads. Probe-verified at the reviewed commit: mutating the constant's value leaves all 40 tests green; anchoring through the imported constant makes the same mutation fail. (QWEN_CODE_MAX_WORKFLOW_SECONDS has no exported constant — workflow-sandbox.ts reads it inline — so that anchor stays a literal unless one is exported.)

Suggested change
'QWEN_CODE_MAX_WORKFLOW_AGENTS',
'QWEN_CODE_MAX_WORKFLOW_CONCURRENCY',
'QWEN_CODE_MAX_WORKFLOW_SECONDS',
MAX_WORKFLOW_AGENTS_ENV,
MAX_WORKFLOW_CONCURRENCY_ENV,
'QWEN_CODE_MAX_WORKFLOW_SECONDS',

plus the import: import { MAX_WORKFLOW_AGENTS_ENV, MAX_WORKFLOW_CONCURRENCY_ENV } from '../../agents/runtime/workflow-orchestrator.js';

中文说明

[Suggestion] env 旋钮锚点是硬编码字符串字面量,而非运行时导出的常量(workflow-orchestrator.ts 中的 MAX_WORKFLOW_AGENTS_ENVMAX_WORKFLOW_CONCURRENCY_ENV),因此该守卫只能捕获描述侧的拼写错误——运行时侧的重命名会从它眼皮底下溜过。失败场景:未来某个 PR 重命名 MAX_WORKFLOW_AGENTS_ENV(或其字符串值)并更新了 workflow-orchestrator.test.ts 中的硬编码字面量,却漏掉 WORKFLOW_TOOL_DESCRIPTION → 本测试保持绿色,而模型会指导用户设置一个没有任何代码读取的变量。已在被审 commit 上用探针验证:改变常量值后全部 40 个测试仍绿;把锚点换成导入的常量后,同样的变异会使测试失败。(QWEN_CODE_MAX_WORKFLOW_SECONDS 没有导出的常量——workflow-sandbox.ts 以行内方式读取——因此除非导出一个常量,该锚点保持字面量。)

— qwen3.8-max via Qwen Code /review (v0.21.7)

Comment on lines +82 to +84
// Limits the model has to plan around rather than discover from a
// mid-run failure.
expect(description).toMatch(/nests one level only/);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R2-2: The numeric caps the model plans around — "up to 1000 agents total" and the "30-minute wall-clock cap" — are not anchored by any assertion, even though this comment names exactly those planning limits and the sibling min(16, cpus-2) formula IS anchored. — Failure scenario: a future PR raises DEFAULT_MAX_AGENTS_PER_RUN or DEFAULT_MAX_WALL_CLOCK_MS and forgets WORKFLOW_TOOL_DESCRIPTION → all 17 anchors stay green while the model sizes fan-outs against stale caps, discovering the real one from a mid-run rejection. Probe-verified: mutating the description's 1000500 and 30-minute15-minute leaves the test green; adding the two anchors below flips the same mutation red.

Suggested change
// Limits the model has to plan around rather than discover from a
// mid-run failure.
expect(description).toMatch(/nests one level only/);
// Limits the model has to plan around rather than discover from a
// mid-run failure.
expect(description).toMatch(/up to 1000 agents total/);
expect(description).toMatch(/30-minute wall-clock cap/);
expect(description).toMatch(/nests one level only/);
中文说明

[Suggestion] 模型制定计划所依据的数值上限——"最多 1000 个 agent"与"30 分钟墙钟上限"——没有任何断言锚定,尽管这段注释指名的正是这些规划限制,且同段的 min(16, cpus-2) 公式已被锚定。失败场景:未来某个 PR 提高 DEFAULT_MAX_AGENTS_PER_RUNDEFAULT_MAX_WALL_CLOCK_MS 却忘记更新 WORKFLOW_TOOL_DESCRIPTION → 所有 17 个锚点保持绿色,而模型按过时的上限规划扇出,直到运行中途被拒绝才发现真实上限。探针验证:将描述中的 100050030-minute15-minute 变异后测试仍绿;加入上面两个锚点后同一变异即转红。

— qwen3.8-max via Qwen Code /review (v0.21.7)

Comment on lines +518 to +521
* `script`'s own description carries the exact authoring contract (error
* strings, serialization rules); the limits and env knobs restated below
* are a summary of it, not a separate source of truth — when one of them
* changes, change both.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R2-3: The sync claim in this comment is only partially true: the concurrency window / agent cap and their two env knobs do restate the script parameter description, but the wall-clock cap (QWEN_CODE_MAX_WORKFLOW_SECONDS), the budget.total semantics, and the one-level workflow() nesting limit appear ONLY in this tool description — the script description's globals list is phase/log/agent/args. — Failure scenario: a maintainer changes DEFAULT_MAX_WALL_CLOCK_MS and follows "change both" → looks for the cap in the script description (the declared source of truth), finds nothing, and since this text is declared "not a separate source of truth", the 30-minute sentence and its test anchor go stale silently — exactly the drift this PR's test exists to catch, and it then stays green (probe-verified, see R2-2).

Suggested change
* `script`'s own description carries the exact authoring contract (error
* strings, serialization rules); the limits and env knobs restated below
* are a summary of it, not a separate source of truth when one of them
* changes, change both.
* `script`'s own description carries the exact authoring contract (error
* strings, serialization rules). The concurrency window, agent cap, and
* their two env knobs restate it when one of them changes, change both.
* The wall-clock cap, output-token budget, and `workflow()` nesting limit
* live only in this description keep it in sync with the runtime
* constants (e.g. `DEFAULT_MAX_WALL_CLOCK_MS`, `DEFAULT_MAX_AGENTS_PER_RUN`).
中文说明

[Suggestion] 本注释的同步声明只部分为真:并发窗口/agent 上限及其两个 env 旋钮确实是 script 参数描述的复述,但墙钟上限(QWEN_CODE_MAX_WORKFLOW_SECONDS)、budget.total 语义与单层 workflow() 嵌套限制只出现在本工具描述中——script 描述的全局清单是 phase/log/agent/args。失败场景:维护者修改 DEFAULT_MAX_WALL_CLOCK_MS 后按"两处一起改"去找 script 描述(声明的权威来源),什么也找不到;而本段又被声明为"不是独立的权威来源",于是 30 分钟那句话及其测试锚点悄悄过期——这正是本 PR 的测试要捕获的漂移,而该测试届时保持绿色(探针验证见 R2-2)。

— qwen3.8-max via Qwen Code /review (v0.21.7)

… its limit

Round-2 review (R2-1/R2-2/R2-3) plus the two non-blocking suggestions from
the maintainer verification round.

- R2-1: anchor the two env knobs through the orchestrator's exported
  MAX_WORKFLOW_AGENTS_ENV / MAX_WORKFLOW_CONCURRENCY_ENV instead of
  hardcoded literals, so a rename on the runtime side fails the guard too.
  QWEN_CODE_MAX_WORKFLOW_SECONDS has no exported constant
  (workflow-sandbox.ts reads it inline), so it stays a literal.
- R2-2: pin the numbers the model plans around — "up to 1000 agents total"
  and the "30-minute wall-clock cap" — not just the knob names.
- R2-3: the sync note claimed the whole limits block restates the `script`
  contract; the wall-clock cap, token budget and one-level workflow()
  nesting limit live only in the tool description. Say which is which.
- W-1: budget exhaustion refuses each further agent() call; a bare
  sequential await sees the rejection, while parallel()/pipeline() turn the
  refused slot into null and keep running (errors-as-data, settleToNullArray).
- W-2: pair each env knob with the quantity it overrides — the shared
  parenthetical read as if both knobs covered the total.

Mutation-probed: 1000->500, 30-minute->15-minute, and renaming
MAX_WORKFLOW_AGENTS_ENV's value each turn the suite red on the intended
assertion; restored tree is 40/40 green.
wenshao added a commit that referenced this pull request Aug 7, 2026
wenshao added a commit that referenced this pull request Aug 7, 2026
wenshao added a commit that referenced this pull request Aug 7, 2026
@wenshao

wenshao commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Follow-up maintainer verification: current head + live model script-authoring A/B

Recommendation: merge-ready from the verification side. The current check suite is green; GitHub still reports BLOCKED, so the remaining non-test repository gate must also be satisfied.

Current PR head e4913243841ae55da437aa3bc898d48c44cb04d5 was rebuilt and rechecked against current origin/main 4ec0371e616decbe723cec250e19943b226d31e1. The live-model behavior samples were recorded on its direct parent 0ecc3160b31179d5eb17f4108305b1a2757cd437; the intervening commit changes only runtime-limit wording and test pinning, while all orchestration-policy sections used by the behavior analysis remain unchanged.

This follows the earlier 68/68 deterministic verification, which explicitly left live model behavior unmeasured.

1. Current-head deterministic verification

  • Built current head successfully.
  • Description/runtime factcheck: 60/60; the current model-visible description is 5,716 characters.
  • Targeted packages/core/src/tools/workflow/workflow.test.ts: 40/40 at current head.
  • Base-description mutation: expected RED on the environment-variable anchor; restored current head: GREEN 40/40.
  • Policy-only mutation: deleting only Verify before believing produced the expected RED on /refute/; restored current head: GREEN 40/40. This proves the prose is pinned, not that downstream agents were executed.
  • A no-commit trial merge of current PR head into current origin/main completed without conflicts. Only workflow.ts and workflow.test.ts were staged, and the targeted test passed 40/40 in that synthetic merged tree.
  • Preserved macOS sandbox probe: PR source readable; ~/.qwen/settings.json access denied with EPERM; outbound TCP connect denied with EPERM. The trusted installed CLI, not untrusted PR code, handled model authentication and traffic.
  • At publication time, the Ubuntu test and web-shell E2E Smoke had passed. macOS/Windows and integration lanes were skipped by the fork-PR workflow policy. GitHub still reported BLOCKED, indicating a remaining non-test repository gate outside this verification.

Current-head deterministic gates

2. Strategy-aligned script-authoring A/B

I ran gpt-5.6-sol three times per arm on three strategy-aligned natural-language prompts, for 18/18 successful script-authoring calls. The prompts did not name pipeline(), parallel(), or the Workflow policy sections, but they did state task properties such as independent files, global deduplication, and unknown-size discovery; this is directional evidence, not unbiased production sampling.

A zero-dependency MCP probe recorded the authored Workflow scripts and never executed their subagent fleets. Therefore the claims below concern authored orchestration structure, not final workflow output, runtime convergence, or end-to-end finding quality. Static adjudication was not blinded. The full rubric and per-sample evidence are preserved in manual-adjudication.md.

The behavior samples use parent head 0ecc3160. Comparing it with current head e4913243 shows that the policy sections (Default to pipeline, verification, convergence, honest reporting, and barrier guidance) are unchanged; the new commit only clarifies runtime-limit mechanics and strengthens tests.

Strategy-aligned authored-script A/B

directly observable authored-script metric base policy description
Review: per-item pipeline 0/3 3/3
Review: explicit adversarial/independent verification 3/3 3/3
Cross-package: required full-set clustering/dedup barrier retained 3/3 3/3
Cross-package: explicit attempt to refute duplicate hypothesis 0/3 2/3
Open audit: static rubric pass 2/3 3/3

Review scenario

All three base scripts waited for every initial file review before one global verification stage. All three policy-description scripts used a per-file pipeline and retained explicit verification. One authored workflow required two independent verifiers to support the same finding before retaining it. This is the clearest observed behavior change: pipeline structure improved without weakening the false-positive gate.

Cross-package scenario

All six authored scripts retained the genuinely necessary full-set barrier: inspect packages, build a complete semantic catalog, globally cluster/deduplicate, then launch expensive analysis only for distinct candidates. The policy description did not mechanically remove the barrier. Two of three policy-description scripts explicitly instructed an agent to try to refute the duplicate hypothesis; none of the base scripts used equally explicit refutation wording.

Open-ended audit scenario

Base sample 1 authored a reusable methodology instead of repository-directed audit stages; the other two base scripts satisfied the static rubric. All three policy-description scripts authored discovery stages, a permanent all-seen ledger including dismissed/rediscovered candidates, consecutive-zero-new stopping logic, independent refutation, closing gap challenges, and confidence downgrades at caps or unresolved gaps.

This 2/3 → 3/3 figure is a documented manual static judgment. A preliminary regex classifier did not confirm the head count because it missed Chinese and semantically equivalent code shapes; that raw 10/18 classifier result is excluded rather than presented as green evidence.

3. Paired request calibration and context cost

For the 21 recorded pairs (12 explicit positive-control pairs and 9 strategy-aligned pairs), the request that first carried the complete deferred Workflow declaration was compared after normalizing:

  • the intentionally different Workflow description wherever it appeared;
  • arm-specific runtime/work paths;
  • random tool-call IDs;
  • stochastic prior assistant preamble/reasoning already present in the second-turn request.

The remaining harness-controlled fields were identical in 21/21 pairs: model, user prompt, cwd semantics, tool set, tool parameters/schema, tool_search arguments, and request configuration. This is a normalized harness-calibration result, not a claim that the literal second-turn requests were byte-identical before normalization.

Provider-reported token usage was measured on the nine paired requests at parent head 0ecc3160, whose description was 5,574 characters. The current head description is 5,716 characters, so the exact current-head provider cost was not remeasured.

Measured parent-head Workflow-schema cost

  • Parent-head paired prompt-token delta: +1,920 to +1,993 on the request carrying the full deferred schema.
  • Cache-read input tokens on that request: 13,824 → 14,848.
  • Billing impact depends on provider cache pricing and the specific model/provider tokenizer.

4. Interpretation and limitations

  • n=3 per cell is bounded directional evidence, not a statistical-significance claim.
  • The probe measured script-authoring behavior and deliberately did not execute 18 large subagent fleets.
  • Adjudication was static, arm-visible, and performed after generation; the exact rubric and per-sample evidence are preserved for review.
  • The prompts were strategy-aligned and supplied the relevant task constraints, although they did not disclose API/function names.
  • The trusted installed Qwen Code CLI 0.21.7 supplied the model/runtime; the deferred Workflow description was injected through the probe.
  • No repo-wide local suite was rerun. Current-head and current-main synthetic-merge targeted tests passed; broader platform coverage remains CI's responsibility.
  • The current check suite completed successfully. GitHub still reported BLOCKED, so a non-test repository gate remains outside this verification.

5. Recommendation

No local correctness, compatibility, or authored-orchestration blocker was found. The current head passes the deterministic gates and current-main synthetic merge. The parent-head A/B, whose policy text is unchanged in the current head, provides directional evidence that the description:

  1. moved the three review samples from a full-set verification barrier to per-item pipelines;
  2. retained explicit adversarial verification in those review samples;
  3. preserved full-set barriers in all six cross-package samples where global context was genuinely required;
  4. authored more deliberate convergence and honest-coverage logic in the three open-audit samples.

The material tradeoff is context cost: approximately +1.9k prompt tokens in the measured parent-head deferred-schema request. From the verification side, I would merge this PR; the remaining GitHub non-test gate still needs to be satisfied.

中文报告(Chinese report)

建议:从验证角度看已 merge-ready。当前 check suite 已全绿;GitHub 仍显示 BLOCKED,还需要满足剩余的非测试仓库 gate。

当前 PR head 为 e4913243841ae55da437aa3bc898d48c44cb04d5,已基于当前 origin/main 4ec0371e616decbe723cec250e19943b226d31e1 重新构建和验证。真实模型脚本编排 A/B 采样于其直接父提交 0ecc3160b31179d5eb17f4108305b1a2757cd437;两者之间的新提交只澄清 runtime limit 文案并加强测试 pinning,A/B 所评估的 pipeline、barrier、verification、convergence、honest reporting 等策略段未改变。

当前 head 的确定性门禁

  • build 成功;描述/runtime 事实核查 60/60,当前描述长度 5,716 字符。
  • 当前 head 的 workflow.test.ts40/40
  • base-description mutation 在环境变量锚点按预期变红,恢复后 40/40
  • 仅删除 Verify before believing 段时在 /refute/ 按预期变红,恢复后 40/40。这证明 policy prose 被测试 pin 住,不代表子 agent 已实际执行。
  • 当前 PR head 无冲突地应用到当前 origin/main 的 no-commit synthetic merge;暂存范围仅两个 Workflow 文件,合并态目标测试 40/40
  • macOS sandbox 原始 probe 已落盘:PR 源码可读,访问 ~/.qwen/settings.json 得到 EPERM,外连 TCP 得到 EPERM。模型认证和流量由受信任的已安装 CLI 处理,而不是未合并 PR 代码。
  • 发布时 Ubuntu test 与 web-shell E2E Smoke 均已通过;macOS/Windows 与 integration lanes 按 fork-PR 策略跳过。GitHub 仍显示 BLOCKED,说明还有本验证范围外的非测试 gate。

真实模型 authored-script A/B

使用 gpt-5.6-sol,三个 strategy-aligned 自然语言任务、每个 arm 3 次,共 18/18 次脚本生成调用成功。prompt 没有直接写 pipeline()parallel() 或策略章节名,但给出了“文件独立”“需要全局去重”“未知数量发现”等任务属性,因此这是有方向的真实证据,不是无偏生产流量采样。

MCP probe 只记录模型写出的 Workflow script,不执行其中的 subagent fleet。因此以下结论评价的是模型编排出的脚本结构,不是最终业务输出、实际运行收敛或 finding 正确率。人工静态裁决未盲化,完整 rubric 和逐样本证据已保存。

  • Review:per-item pipeline 0/3 → 3/3;明确验证 3/3 → 3/3。base 都等待全部初审后再做一次全局验证;policy description 下三个样本都逐文件 pipeline,并保留反证/precision gate。
  • Cross-package:必要的 full-set barrier 3/3 → 3/3,没有被 pipeline default 机械删除;明确反驳 duplicate hypothesis 0/3 → 2/3
  • Open audit:按静态 rubric 为 2/3 → 3/3。policy description 下三个样本都编排了 all-seen ledger、连续零新增停止条件、独立反证、closing gap challenge 和 cap 时降置信度。早期 regex classifier 未确认 head 计数,因为漏判中文和语义等价结构;该原始 10/18 结果明确排除,不冒充绿色证据。

Wire 校准与成本

21 对样本中,选取首次携带完整 deferred Workflow schema 的请求。归一化受控变量 Workflow description、arm-specific 隔离路径、随机 tool-call ID、以及模型在前一轮生成的随机前言/思考后,剩余 harness-controlled 字段 21/21 一致:模型、用户 prompt、cwd 语义、工具集合/schema、tool_search 参数和请求配置。这里不声称未经归一化的实际第二轮请求逐字相同。

provider token 成本测量来自父提交 0ecc3160(描述 5,574 字符)的 9 对 strategy-aligned 请求:

  • paired prompt-token delta:+1,920 到 +1,993
  • 对应请求的 cache-read input tokens:13,824 → 14,848
  • 当前 head 描述为 5,716 字符,未重新测量精确 provider token 成本;实际计费取决于 provider cache 定价和 tokenizer。

限制与结论

  • 每格仅 n=3,不作统计显著性声明。
  • 只验证 script authoring,不验证 18 个大型 workflow 的实际业务输出。
  • 人工裁决可见 arm,且在生成后进行;prompt 也包含目标任务约束。
  • 未本地运行 repo-wide 全套测试;当前 head 与当前 main synthetic merge 的目标测试均通过,更广平台覆盖由 CI 负责。
  • 当前 check suite 已成功完成;GitHub 仍显示 BLOCKED,因此还存在本验证范围外的非测试仓库 gate。

综合当前 head 的确定性验证,以及策略段未变的父提交 authored-script A/B:本地未发现 correctness、compatibility 或 orchestration blocker。主要取舍是 deferred schema 请求中实测约 +1.9k prompt tokens。从验证角度看我建议合并;仍需满足 GitHub 剩余的非测试 gate。

wenshao
wenshao previously approved these changes Aug 8, 2026
@wenshao

wenshao commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

@qwen-code /resolve

Resolves the conflict in packages/core/src/tools/workflow/workflow.ts.
main's QwenLM#8320 edited the inline constructor description this branch is
replacing, so the two sides touched the same argument:

- Kept this branch's `WORKFLOW_TOOL_DESCRIPTION` constant — extracting
  that description is the whole point of the PR.
- Ported QwenLM#8320's fact into the constant. The extracted text still listed
  the `/workflows` dialog controls as "live phase tree, token usage,
  cancel"; taking our side verbatim would have dropped cooperative
  pause/resume from the description the model actually reads.
- Pinned that capability list in the existing description test. Nothing
  else asserted it, so the same silent drop could recur on the next base
  merge.

Verified: npm run build, npm run typecheck, eslint on both changed files,
and packages/core src/tools/workflow/workflow.test.ts (40 passed).
@qqqys

qqqys commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator Author

Merge conflict resolved and pushed as 2a00e36e2. mergeable is back to true.

Why it conflicted. #8320 landed in main and edited the inline description string in the WorkflowTool constructor — the exact string this branch removes in favour of the extracted WORKFLOW_TOOL_DESCRIPTION constant. Both sides rewrote the same super() argument.

How I resolved it. Kept this branch's constant (that extraction is the point of the PR), then ported #8320's fact into it: the extracted text still described the /workflows dialog as "live phase tree, token usage, cancel", so taking our side verbatim would have silently dropped cooperative pause/resume from the description the model actually reads. I also pinned that capability list in the existing description test — nothing else asserted it, so the same silent drop could recur on the next base merge.

Scope against main is unchanged: still exactly the same 2 files.

Verification

  • npm run build — clean
  • npm run typecheck — clean
  • npx eslint packages/core/src/tools/workflow/workflow.{ts,test.ts} — clean
  • npx vitest run src/tools/workflow/workflow.test.ts40 passed

Two things worth flagging, @wenshao:

  1. Your @qwen-code /resolve (05:06:09Z) never dispatched. Workflow 178983984 (qwen-code-pr-review.yml) has produced zero issue_comment runs since 2026-08-07T16:58:13Z, and /resolve is triggered only from that workflow. Qwen Triage and Qwen Autofix both received the same comment event normally (runs at 05:06:12Z / 05:06:21Z), so the event fired — it is that one workflow that is not loading. Its runs on other events render with the file path as the run name instead of 🧐 Qwen Pull Request Review and fail with zero jobs, which GitHub reports as "This run likely failed because of a workflow file issue." That is why nothing answered you; I resolved the conflict by hand instead.

  2. This push dismissed your approval (4887470734). Resolving the conflict required a push, and a push voids the approval, so the PR now needs a fresh code-owner approval before it can go in. Nothing else is outstanding: no CHANGES_REQUESTED, and CI is re-running on the new head.


合并冲突已解决并推送为 2a00e36e2mergeable 已恢复 true

冲突原因#8320 合入 main 后修改了 WorkflowTool 构造函数里的内联描述字符串,而本分支正是要把该字符串抽成 WORKFLOW_TOOL_DESCRIPTION 常量,双方改的是同一个 super() 实参。

解决方式:保留本分支的常量(这正是本 PR 的目的),并把 #8320 的信息补进常量 —— 抽出的文本仍把 /workflows 面板写作"live phase tree, token usage, cancel",若原样采用我方版本,会静默丢掉 cooperative pause/resume。同时在既有描述测试中钉住该能力列表,避免下次合并 base 时再次静默丢失。相对 main 的改动范围不变,仍是同样的 2 个文件。

另外两点:① 你在 05:06:09Z 发的 @qwen-code /resolve 没有触发任何 run —— qwen-code-pr-review.yml178983984)自 2026-08-07T16:58:13Z 起再未产生过 issue_comment run,而 /resolve 只由该 workflow 触发;同一条评论事件正常送达了 Qwen TriageQwen Autofix,所以是这个 workflow 本身加载失败。② 本次推送使你的 approval(4887470734)被 dismiss,解决冲突必须推送,因此需要重新给一次 code owner approval;除此之外没有其他阻塞项。

@wenshao

wenshao commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

@qwen-code /review

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor
_Qwen Code review request accepted. Review is queued in [workflow run](https://github.com/QwenLM/qwen-code/actions/runs/31266125026)._

@qwen-code-ci-bot

Copy link
Copy Markdown
Collaborator

Qwen Code review timed out. Qwen review timed out after 10800 seconds (of the 180-minute budget). For large PRs, retry with a longer timeout by commenting: @qwen-code /review --timeout=360. See workflow logs.

@qqqys

qqqys commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator Author

@qwen-code /review

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed. Suggestions are inline. Not reviewed: reverse audit — stopped before round 3 by the review time budget. Test Plan (not a blocker): 40 passed — this review observed 19434, 1101, 18427, 1466, 481, 2941, 454 passed.

中文说明

已审查。 建议见行内评论。 未审查:反向审计——评审时间预算不足,未能开始第 3 轮。 Test Plan(非阻断):40 passed — this review observed 19434, 1101, 18427, 1466, 481, 2941, 454 passed

— qwen3.8-max via Qwen Code /review (v0.21.7)

Comment on lines +100 to +101
expect(description).toMatch(/up to 1000 agents total/);
expect(description).toMatch(/30-minute wall-clock cap/);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R3-1: The numeric-cap anchors pin description-side literals, so the drift the adjacent comment claims to catch — raising a runtime constant without updating the description — runs green; the guard is one level shallower than the problem it describes. — Failure scenario: a maintainer raises DEFAULT_MAX_AGENTS_PER_RUN (exported, workflow-orchestrator.ts) from 1000 to 2000 and leaves the description untouched → /up to 1000 agents total/ still matches the unchanged text, the suite stays green, and the model sizes fan-outs against a stale cap. Probe-verified at this commit: the mutation leaves all 40 tests green; anchoring through the imported constant turns the same mutation red. DEFAULT_MAX_WALL_CLOCK_MS vs /30-minute wall-clock cap/ is the same shape, but that constant is module-private in workflow-sandbox.ts — either export it and derive the expected minutes, or narrow the comment's claim to what the literals actually catch.

Suggested change
expect(description).toMatch(/up to 1000 agents total/);
expect(description).toMatch(/30-minute wall-clock cap/);
expect(description).toContain(`up to ${DEFAULT_MAX_AGENTS_PER_RUN} agents total`);
expect(description).toMatch(/30-minute wall-clock cap/);

(add DEFAULT_MAX_AGENTS_PER_RUN to the existing workflow-orchestrator.js import)

中文说明

[Suggestion] R3-1:数值上限锚点锚定的是描述侧的字面量,因此相邻注释声称要捕获的漂移——提高运行时常量而不更新描述——会原样保持绿色;守卫比它描述的问题浅了一层。失败场景:维护者把 DEFAULT_MAX_AGENTS_PER_RUN(已导出,workflow-orchestrator.ts)从 1000 提到 2000 而不动描述 → /up to 1000 agents total/ 仍然匹配未变化的文本,整套测试保持绿色,模型却按过时的上限规划扇出。已在被审 commit 上用探针验证:该变异下全部 40 个测试仍绿;把锚点换成导入的常量后,同一变异转红。DEFAULT_MAX_WALL_CLOCK_MS/30-minute wall-clock cap/ 同形,但该常量在 workflow-sandbox.ts 中是模块私有的——要么导出它并按其推导分钟数,要么把注释的声明收窄到字面量实际能捕获的范围。(建议块替换被锚定的两行;导入需另行添加。)

— qwen3.8-max via Qwen Code /review (v0.21.7)

Comment on lines +519 to +520
* strings, serialization rules). The concurrency window, the agent cap, and
* their two env knobs restate it — when one of those changes, change both.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R3-2: The description restates three values that already exist as exported constants this file can import — DEFAULT_MAX_AGENTS_PER_RUN (workflow-orchestrator.ts), MAX_WORKFLOW_AGENTS_ENV, and MAX_WORKFLOW_CONCURRENCY_ENV — instead of interpolating them into the template literal. — Concrete cost: a permanent multi-site sync obligation the doc comment itself concedes ("change both", "keep it in sync"), with the pinning test's /up to 1000 agents total/ literal a third copy of the same value — any change to one of those constants requires a synchronized prose edit that only fails at test time. workflow.ts already imports from the runtime module, and buildUsageBanner already interpolates MAX_TOKENS_PER_WORKFLOW_ENV into model-visible prose in this same file, so the duplication is removable at zero structural cost:

up to ${DEFAULT_MAX_AGENTS_PER_RUN} agents total (`${MAX_WORKFLOW_AGENTS_ENV}`) ... (`${MAX_WORKFLOW_CONCURRENCY_ENV}`)

The 30-minute wall-clock cap and the max(1, min(16, cpus-2)) formula have no importable constant (module-private / computed inline) and correctly stay literals.

中文说明

[Suggestion] R3-2:描述复述了三个本文件可以直接导入的已导出常量——DEFAULT_MAX_AGENTS_PER_RUNworkflow-orchestrator.ts)、MAX_WORKFLOW_AGENTS_ENVMAX_WORKFLOW_CONCURRENCY_ENV——而没有把它们插值进模板字面量。具体代价:一份永久的多站点同步义务,文档注释自己也承认("两处一起改"、"保持同步"),而钉定测试里的 /up to 1000 agents total/ 字面量是同一数值的第三份拷贝——任何一个常量的变化都需要同步的文案编辑,且只在测试时才暴露。workflow.ts 已经从该运行时模块导入,且同文件的 buildUsageBanner 已经把 MAX_TOKENS_PER_WORKFLOW_ENV 插值进模型可见文案,因此这份重复可以零结构成本地消除(见上代码片段)。30 分钟墙钟上限与 max(1, min(16, cpus-2)) 公式没有可导入的常量(模块私有/行内计算),保持字面量是正确的。

— qwen3.8-max via Qwen Code /review (v0.21.7)

// so raising a runtime constant without updating the description here
// turns this red instead of leaving the model sizing fan-outs against
// a stale cap.
expect(description).toMatch(/up to 1000 agents total/);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R3-3: The cap numbers are hardcoded a second time in the script parameter description (workflow.tsmax(1, min(16, cpus-2)) and At most 1000 agent() calls per run), which nothing pins — this new test anchors only the tool description. — Failure scenario: a maintainer raises DEFAULT_MAX_AGENTS_PER_RUN to 2000 → this test goes red → they edit the tool-description constant, the test goes green, and they stop. The script parameter description still tells the model "At most 1000 agent() calls per run", so the model reads two contradictory caps from one tool call and under-sizes fan-outs against the stale one — exactly the "model sizing fan-outs against a stale cap" failure this test's own comment says it exists to prevent, and the red test actively directs the fixer to only one of the two sites. Suggested fix — extend the pinning to the parameter side via the schema accessor this file already uses:

const schema = tool.schema.parametersJsonSchema;
expect(schema.properties.script.description).toContain('At most 1000 agent() calls');
expect(schema.properties.script.description).toContain('max(1, min(16, cpus-2))');
中文说明

[Suggestion] R3-3:上限数值在 script 参数描述中被第二次硬编码(workflow.ts——max(1, min(16, cpus-2))At most 1000 agent() calls per run),而没有任何断言钉定它们——新增测试只锚定了工具描述。失败场景:维护者把 DEFAULT_MAX_AGENTS_PER_RUN 提到 2000 → 本测试转红 → 他们更新工具描述常量,测试转绿,随即停手。script 参数描述仍然告诉模型"At most 1000 agent() calls per run",于是模型从同一次工具调用里读到两个互相矛盾的上限,并按过时的哪一个缩小扇出——正是本测试注释声称要防止的"模型按过时上限规划扇出",而且变红的测试还会主动把修复者引向两处中的一处。建议修复——用本文件已在使用的 schema 访问器把钉定扩展到参数侧(见上代码片段)。

— qwen3.8-max via Qwen Code /review (v0.21.7)


**Runtime** — see the \`script\` parameter for the detailed authoring contract.

\`phase(title)\`, \`log(msg)\`, \`agent(prompt, opts?)\`, \`parallel(thunks)\`, \`pipeline(items, ...stages)\`, \`workflow(nameOrRef, args?)\`, plus the \`args\` and \`budget\` globals. \`workflow()\` runs a saved workflow inline under this run's caps and nests one level only — a workflow reached through \`workflow()\` cannot call \`workflow()\` itself, and doing so throws. Default \`max(1, min(16, cpus-2))\` agents in flight per run (\`QWEN_CODE_MAX_WORKFLOW_CONCURRENCY\`), up to 1000 agents total (\`QWEN_CODE_MAX_WORKFLOW_AGENTS\`), under a 30-minute wall-clock cap per run (\`QWEN_CODE_MAX_WORKFLOW_SECONDS\`) — a fan-out near the agent cap will not fit inside the default cap. A per-run output-token cap may also be in effect: read \`budget.total\` (\`null\` = uncapped) before committing to a large fan-out, because once the cap is reached every further \`agent()\` call is refused — a bare sequential \`await agent()\` sees the rejection, while inside \`parallel()\`/\`pipeline()\` the refused slot becomes \`null\` and the script keeps running on partial results. Per-call \`agent({ schema, agentType, model, isolation: 'worktree' })\` covers structured-output contracts, declarative-agent selection, model override, and git-worktree-isolated subagents. \`resumeFromRunId\` resumes a prior run — agent() calls whose rolling prefix-hash matches the journal are served from cache for the longest unchanged prefix. Runs appear in the background-tasks view and the \`/workflows\` dialog (live phase tree, token usage, cooperative pause/resume, cancel); \`run_in_background: true\` returns a run handle immediately in the interactive TUI and delivers completion through the conversation. Scripts run in a node:vm sandbox with no filesystem or shell access — all I/O happens through the spawned agents.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R3-4: The description never says where saved workflows live — issue #8690 explicitly asks the text to use this project's own vocabulary (".qwen/ paths, QWEN_CODE_* env overrides, the workflow keyword trigger, and the /workflows view"), and the new text carries the env knobs and /workflows but zero .qwen/ paths. — Failure scenario: a user says "run my saved triage workflow" → the model's only surfaces are "runs a saved workflow" (no location) and scriptPath (needs an absolute path it has no way to construct), so it guesses workflow('<name>') blindly or abandons reuse and authors a fresh inline script, leaving the save/reuse feature invisible from the model side. The adjacent script parameter description already uses this vocabulary for agent resolution (.qwen/agents/<name>.md, project then user then built-in), and workflow-saved.ts/storage.ts confirm names resolve against .qwen/workflows/ (project) / ~/.qwen/workflows/ (user). Suggested fix: name the location where saved workflows are introduced — e.g. "workflow() runs a saved workflow (.qwen/workflows/<name>.js, project or user) inline under this run's caps..." — and add a .qwen/workflows anchor to the pinning test.

中文说明

[Suggestion] R3-4:描述从未说明已保存的 workflow 存放在哪里——issue #8690 明确要求文本使用本项目自己的词汇(".qwen/ 路径、QWEN_CODE_* 环境覆盖、workflow 关键词触发,以及 /workflows 视图"),而新文本带了环境旋钮与 /workflows,却没有一处 .qwen/ 路径。失败场景:用户说"运行我保存的 triage workflow" → 模型仅有的两个表面是"runs a saved workflow"(没有位置信息)与 scriptPath(需要一个它无从构造的绝对路径),于是它要么盲目猜测 workflow('<name>'),要么放弃复用、重新现写一段内联脚本,令保存/复用特性在模型侧不可见。相邻的 script 参数描述已经在 agent 解析上使用了这一词汇(.qwen/agents/<name>.md,先项目后用户再内置),且 workflow-saved.ts/storage.ts 确认名称解析基于 .qwen/workflows/(项目)/~/.qwen/workflows/(用户)。建议修复:在引入 saved workflow 的位置写明路径——例如 "workflow() runs a saved workflow (.qwen/workflows/<name>.js, project or user) inline under this run's caps..."——并在钉定测试中加入 .qwen/workflows 锚点。

— qwen3.8-max via Qwen Code /review (v0.21.7)

Addresses the four Suggestions from review round 3 on QwenLM#8694.

R3-2 / R3-3: the agent cap and the two env-knob names were prose
literals in both model-visible halves — the tool description and the
`script` parameter description — a third copy sitting in the pinning
test. Interpolate `DEFAULT_MAX_AGENTS_PER_RUN`, `MAX_WORKFLOW_AGENTS_ENV`
and `MAX_WORKFLOW_CONCURRENCY_ENV` from `workflow-orchestrator.ts` into
both halves instead, so raising a cap moves every copy at once and the
model can no longer read two contradictory caps from one tool call.
`DEFAULT_MAX_WALL_CLOCK_MS` stays a literal: it is private to
`workflow-sandbox.ts`. The doc comment now says which values still need
hand-syncing rather than claiming all of them do.

R3-1: the numeric anchor pinned the description's own literal, so it
could not catch the drift its comment claimed to catch. Anchor it
through the exported constant, and add a test pinning the `script`
parameter description's copy of the same caps — nothing asserted it
before, so a maintainer could raise a cap, get the tool-description
test green again, and leave `script` advertising the old number.

R3-4: issue QwenLM#8690 asked the text to use this project's vocabulary. The
description advertised `workflow('<name>')` without ever saying where
saved workflows live, leaving the model to guess a name blindly or
construct an absolute `scriptPath` it has no way to know. Name both
scopes and their precedence.

Verification: npm run build, npm run typecheck, eslint on both changed
files, and vitest on workflow.test.ts + workflow-orchestrator.test.ts
(174 passed). Probed the new anchors both ways with the cap temporarily
raised to 2000: the suite stays green (the description tracks the
constant), and pasting the literal `1000` back into the description
turns both assertions red.

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no blockers. Suggestions are inline. Test Plan (not a blocker): 40 passed — this review observed 19435, 1101, 18430, 1466, 481, 2941, 454 passed.

中文说明

已审查——无阻断问题。 建议见行内评论。 Test Plan(非阻断):40 passed — this review observed 19435, 1101, 18430, 1466, 481, 2941, 454 passed

— qwen3.8-max via Qwen Code /review (v0.21.8)


**Runtime** — see the \`script\` parameter for the detailed authoring contract.

\`phase(title)\`, \`log(msg)\`, \`agent(prompt, opts?)\`, \`parallel(thunks)\`, \`pipeline(items, ...stages)\`, \`workflow(nameOrRef, args?)\`, plus the \`args\` and \`budget\` globals. \`workflow()\` runs a saved workflow inline under this run's caps and nests one level only — a workflow reached through \`workflow()\` cannot call \`workflow()\` itself, and doing so throws. Saved workflows are \`<name>.js\` files under \`<projectRoot>/.qwen/workflows\` (project scope, also surfaced as \`/<name>\` slash commands) or \`~/.qwen/workflows\` (user scope, lower precedence when both define the same name); \`workflow('<name>')\` resolves against those two directories, while \`scriptPath\` takes an absolute path to a script anywhere. Default \`max(1, min(16, cpus-2))\` agents in flight per run (\`${MAX_WORKFLOW_CONCURRENCY_ENV}\`), up to ${DEFAULT_MAX_AGENTS_PER_RUN} agents total (\`${MAX_WORKFLOW_AGENTS_ENV}\`), under a 30-minute wall-clock cap per run (\`QWEN_CODE_MAX_WORKFLOW_SECONDS\`) — a fan-out near the agent cap will not fit inside the default cap. A per-run output-token cap may also be in effect: read \`budget.total\` (\`null\` = uncapped) before committing to a large fan-out, because once the cap is reached every further \`agent()\` call is refused — a bare sequential \`await agent()\` sees the rejection, while inside \`parallel()\`/\`pipeline()\` the refused slot becomes \`null\` and the script keeps running on partial results. Per-call \`agent({ schema, agentType, model, isolation: 'worktree' })\` covers structured-output contracts, declarative-agent selection, model override, and git-worktree-isolated subagents. \`resumeFromRunId\` resumes a prior run — agent() calls whose rolling prefix-hash matches the journal are served from cache for the longest unchanged prefix. Runs appear in the background-tasks view and the \`/workflows\` dialog (live phase tree, token usage, cooperative pause/resume, cancel); \`run_in_background: true\` returns a run handle immediately in the interactive TUI and delivers completion through the conversation. Scripts run in a node:vm sandbox with no filesystem or shell access — all I/O happens through the spawned agents.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] The description states that scriptPath "takes an absolute path to a script anywhere", but the runtime refuses any scriptPath outside the two saved-workflow directories (<projectRoot>/.qwen/workflows and ~/.qwen/workflows). Probe-verified at this commit through the real tool path: a scriptPath pointing at a file elsewhere throws refusing to load a workflow file outside the saved-workflow directories (the realpath boundary check in workflow-saved.ts). Enforcement is stricter than advertised — fail-safe direction, so this is a description defect, not a missing guard — but the clause also contradicts the scriptPath parameter's own schema description ("Absolute path to a saved workflow .js file"), so the model reads two incompatible statements in one tool declaration. — Failure scenario: a model trusts the description and passes a path like ~/scratch/my-flow.js as scriptPath → the call fails every time, in a way the description says cannot happen → the model retries or improvises instead of telling the user to place the file under one of the two directories or inline it via script.

// in WORKFLOW_TOOL_DESCRIPTION (Runtime paragraph):
- while `scriptPath` takes an absolute path to a script anywhere.
+ while `scriptPath` takes an absolute path to a saved-workflow file
+ inside one of them (paths outside are refused).

(the matching new test comment in workflow.test.ts — "scriptPath wants an absolute path it cannot construct" — wants the same adjustment)

中文说明

描述声称 scriptPath "接受指向任意脚本的绝对路径",但运行时拒绝任何位于两个 saved-workflow 目录(<projectRoot>/.qwen/workflows~/.qwen/workflows)之外的 scriptPath。已在被审 commit 上通过真实工具路径用探针验证:指向其他位置的 scriptPath 会抛出 refusing to load a workflow file outside the saved-workflow directoriesworkflow-saved.ts 中的 realpath 边界检查)。实际执行比描述更严格——方向是 fail-safe 的,因此这是描述缺陷而非缺少防护——但该子句同时与 scriptPath 参数自身的 schema 描述("Absolute path to a saved workflow .js file")矛盾,模型会在同一次工具声明里读到两种互不相容的说法。失败场景:模型相信描述,把 ~/scratch/my-flow.js 这类路径作为 scriptPath 传入 → 调用每次都按描述声称"不可能发生"的方式失败 → 模型反复重试或改走野路子,而不是告诉用户把文件放到上述两个目录之一、或用 script 内联。(workflow.test.ts 中新增的那句注释——"scriptPath wants an absolute path it cannot construct"——也建议一并调整。)

— qwen3.8-max via Qwen Code /review (v0.21.8)

Comment on lines +529 to +530
* every model-visible copy at once — there is no prose to hand-sync.
* The wall-clock cap is the one exception: `DEFAULT_MAX_WALL_CLOCK_MS` is

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] The "there is no prose to hand-sync … the wall-clock cap is the one exception" claim is inaccurate: the default concurrency window max(1, min(16, cpus-2)) is also hand-synced prose in BOTH model-visible descriptions — resolveConcurrencyLimit computes it inline (workflow-orchestrator.ts) and no constant is exported, so the tool description and the script parameter description each carry the formula as prose. — Failure scenario: a maintainer widens the default window (16 → 32) → both descriptions keep advertising min(16, cpus-2) and nothing bridges the runtime to either description: the 'min(16, cpus-2)' anchor in the pinning test only trips if the tool-description prose is edited, and the parity test anchors the agent cap and env knobs in the script description but not the formula — while this comment actively assures the reader that the wall-clock literal is the only one needing a companion edit.

Suggested change
* every model-visible copy at once there is no prose to hand-sync.
* The wall-clock cap is the one exception: `DEFAULT_MAX_WALL_CLOCK_MS` is
* every model-visible copy at once there is no prose to hand-sync.
* The wall-clock cap and the `max(1, min(16, cpus-2))` default window are the exceptions: `DEFAULT_MAX_WALL_CLOCK_MS` is
中文说明

"无需手工同步的文案……墙钟上限是唯一的例外"这一说法不准确:默认并发窗口 max(1, min(16, cpus-2)) 同样是两份模型可见描述中的手工同步文案——resolveConcurrencyLimit 行内计算该值(workflow-orchestrator.ts),没有导出常量,因此工具描述与 script 参数描述各自以文案形式携带该公式。失败场景:维护者把默认窗口放宽(16 → 32)→ 两份描述继续宣称 min(16, cpus-2),且没有任何东西把运行时与这两份描述关联起来:钉定测试里的 'min(16, cpus-2)' 锚点只在工具描述文案被编辑时才会转红,而对等测试只钉定了 script 描述里的 agent 上限与环境旋钮、没有钉定该公式——而这条注释却在主动向读者保证墙钟字面量是唯一需要配套修改的地方。

— qwen3.8-max via Qwen Code /review (v0.21.8)


**Runtime** — see the \`script\` parameter for the detailed authoring contract.

\`phase(title)\`, \`log(msg)\`, \`agent(prompt, opts?)\`, \`parallel(thunks)\`, \`pipeline(items, ...stages)\`, \`workflow(nameOrRef, args?)\`, plus the \`args\` and \`budget\` globals. \`workflow()\` runs a saved workflow inline under this run's caps and nests one level only — a workflow reached through \`workflow()\` cannot call \`workflow()\` itself, and doing so throws. Saved workflows are \`<name>.js\` files under \`<projectRoot>/.qwen/workflows\` (project scope, also surfaced as \`/<name>\` slash commands) or \`~/.qwen/workflows\` (user scope, lower precedence when both define the same name); \`workflow('<name>')\` resolves against those two directories, while \`scriptPath\` takes an absolute path to a script anywhere. Default \`max(1, min(16, cpus-2))\` agents in flight per run (\`${MAX_WORKFLOW_CONCURRENCY_ENV}\`), up to ${DEFAULT_MAX_AGENTS_PER_RUN} agents total (\`${MAX_WORKFLOW_AGENTS_ENV}\`), under a 30-minute wall-clock cap per run (\`QWEN_CODE_MAX_WORKFLOW_SECONDS\`) — a fan-out near the agent cap will not fit inside the default cap. A per-run output-token cap may also be in effect: read \`budget.total\` (\`null\` = uncapped) before committing to a large fan-out, because once the cap is reached every further \`agent()\` call is refused — a bare sequential \`await agent()\` sees the rejection, while inside \`parallel()\`/\`pipeline()\` the refused slot becomes \`null\` and the script keeps running on partial results. Per-call \`agent({ schema, agentType, model, isolation: 'worktree' })\` covers structured-output contracts, declarative-agent selection, model override, and git-worktree-isolated subagents. \`resumeFromRunId\` resumes a prior run — agent() calls whose rolling prefix-hash matches the journal are served from cache for the longest unchanged prefix. Runs appear in the background-tasks view and the \`/workflows\` dialog (live phase tree, token usage, cooperative pause/resume, cancel); \`run_in_background: true\` returns a run handle immediately in the interactive TUI and delivers completion through the conversation. Scripts run in a node:vm sandbox with no filesystem or shell access — all I/O happens through the spawned agents.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] The parenthetical pins slash-command surfacing to project scope only — "(project scope, also surfaced as /<name> slash commands)" — but user-scope saved workflows are surfaced as /<name> slash commands too: SavedWorkflowLoader.loadCommands maps every entry from listSavedWorkflows (both scopes, project shadowing same-named user entries) to a command with no scope filter, and its own test feeds a user-scope entry and asserts it becomes a command. — Failure scenario: a model reading this description concludes a user-scope workflow is not reachable as /<name> → it avoids recommending that path, or tells the user their user-scope workflow has no slash command when it does. Impact is bounded because workflow('<name>') resolution against both directories is documented in the same sentence.

// in WORKFLOW_TOOL_DESCRIPTION (Runtime paragraph):
- `<projectRoot>/.qwen/workflows` (project scope, also surfaced as
- `/<name>` slash commands) or `~/.qwen/workflows` (user scope, ...)
+ `<projectRoot>/.qwen/workflows` (project scope) or `~/.qwen/workflows`
+ (user scope, lower precedence when both define the same name); both
+ are surfaced as `/<name>` slash commands
中文说明

这个括号注释把斜杠命令的暴露限定在了项目作用域——"(project scope, also surfaced as /<name> slash commands)"——但用户作用域的 saved workflow 同样会暴露为 /<name> 斜杠命令:SavedWorkflowLoader.loadCommands 会把 listSavedWorkflows(两个作用域,项目侧遮蔽同名的用户侧条目)返回的每个条目都映射为命令,没有任何作用域过滤,它自己的测试就喂入过一个用户作用域条目并断言其成为命令。失败场景:模型读到这段描述后断定用户作用域的 workflow 无法通过 /<name> 触达 → 于是避免推荐这条路径,或者在用户的 user-scope workflow 明明有斜杠命令时告诉他没有。影响有限,因为同一句话里也写了 workflow('<name>') 会在这两个目录里解析。

— qwen3.8-max via Qwen Code /review (v0.21.8)

// dialog while this branch was moving the description into a constant,
// and the base merge conflicted exactly here. Nothing else asserts the
// control set, so dropping one on the next merge would be silent.
expect(description).toMatch(/cooperative pause\/resume/);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] The /workflows dialog capability list is under-anchored in two probe-proven shapes: (1) the loop anchor toContain('/workflows') above is permanently masked by the saved-workflow path strings — both <projectRoot>/.qwen/workflows and ~/.qwen/workflows contain /workflows — and (2) of the four advertised capabilities, "(live phase tree, token usage, cooperative pause/resume, cancel)", only cooperative pause/resume is anchored. — Failure scenario: probed at this commit — deleting only the /workflows dialog reference from the sentence leaves 41/41 green (the path strings satisfy the masked anchor), and deleting , cancel from the parenthetical also leaves 41/41 green. Either silent drop removes the model's only description-side pointer to the dialog or its controls — exactly the #8320-style merge drop the comment above says this anchor exists to catch.

Suggested change
expect(description).toMatch(/cooperative pause\/resume/);
expect(description).toMatch(/the `\/workflows` dialog \(live phase tree, token usage, cooperative pause\/resume, cancel\)/);

(one unit anchor pins the dialog reference plus all four capabilities; the masked '/workflows' loop entry can then be dropped)

中文说明

/workflows 对话框能力清单的锚定不足,且两种形态都已用探针证实:(1) 上面循环里的 toContain('/workflows') 锚点被 saved-workflow 路径字符串永久遮蔽——<projectRoot>/.qwen/workflows~/.qwen/workflows 都包含 /workflows 子串;(2) 描述宣称的四项能力 "(live phase tree, token usage, cooperative pause/resume, cancel)" 中只有 cooperative pause/resume 被锚定。失败场景:已在被审 commit 上用探针验证——只删掉句中的 /workflows 对话框引用,41/41 依旧全绿(路径字符串能满足被遮蔽的锚点);删掉括号里的 , cancel 同样全绿。任何一种静默删除都会抹掉模型在描述侧指向该对话框或其控件的唯一线索——正是上方注释声称这个锚点要拦截的 #8320 式合并丢失。(建议块用一条整体锚点同时钉住对话框引用与全部四项能力;之后可以删掉被遮蔽的 '/workflows' 循环项。)

— qwen3.8-max via Qwen Code /review (v0.21.8)

// a location, "runs a saved workflow" leaves the model no way to reach
// one: `workflow('<name>')` is a blind guess and `scriptPath` wants an
// absolute path it cannot construct.
expect(description).toContain('.qwen/workflows');

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] This anchor is masked by the user-scope path: ~/.qwen/workflows contains the substring .qwen/workflows, so dropping or rewriting the project-scope clause <projectRoot>/.qwen/workflows while the user path remains leaves the anchor green — losing exactly the project-scope placement the comment above (issue #8690's "speak this project's own vocabulary" ask) says this anchor defends. — Failure scenario: a maintainer or the next base merge drops the project-scope mention while ~/.qwen/workflows remains → toContain('.qwen/workflows') still matches → the model loses the project-scope location (the primary one — project scope shadows user scope and is surfaced in the slash-command list first) and no test turns red.

Suggested change
expect(description).toContain('.qwen/workflows');
expect(description).toContain('<projectRoot>/.qwen/workflows');
expect(description).toContain('~/.qwen/workflows');

(the ~ form also pins the tilde, which .qwen/workflows never checks)

中文说明

这个锚点被用户作用域路径遮蔽:~/.qwen/workflows 本身就包含 .qwen/workflows 子串,因此只要用户侧路径还在,删掉或改写项目作用域的 <projectRoot>/.qwen/workflows 子句,锚点依旧为绿——丢掉的恰恰是上方注释(issue #8690 "使用本项目自己的词汇"的要求)声称这个锚点要守护的项目作用域位置。失败场景:维护者或下一次 base 合并删掉了项目作用域的提及而 ~/.qwen/workflows 仍在 → toContain('.qwen/workflows') 仍然匹配 → 模型失去项目作用域的位置信息(那是主要位置——项目作用域遮蔽用户作用域,且在斜杠命令列表中最先呈现),同时没有任何测试转红。(~ 的写法同时还钉住了波浪号,.qwen/workflows 永远检查不到它。)

— qwen3.8-max via Qwen Code /review (v0.21.8)

@wenshao

wenshao commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

Maintainer verification, round 3 — head 5a466444e7 · verdict: ✅ merge-ready from the verification side

Earlier rounds (68/68 deterministic, then the live-model follow-up) checked the description against the runtime constants. Since then the branch merged main and added 5a466444 ("derive description caps from the runtime constants"), which also introduced brand-new model-visible content — the saved-workflow location sentence and cooperative pause/resume. So this round does something the earlier ones did not: it executes every factual claim in the new text through the real built CLI, and reads the answers back off the wire.

Base for the diff is the merge-base fd76d4ddde. Built from PR source in a scratch worktree (npm ci + npm run bundledist/cli.js), driven headless against a recording OpenAI-compatible provider with an isolated QWEN_HOME and workspace per case.

中文摘要(Chinese summary)

结论:从验证角度看可以合并(merge-ready),验证 head 5a466444e7,diff 基线取 merge-base fd76d4ddde

前两轮是拿描述去比对运行时常量。此后分支合了 main,又加了 5a466444(把上限从运行时常量插值出来),同时带进了两段全新的模型可见内容——saved workflow 的存放位置那句,以及 cooperative pause/resume。所以这一轮做了前两轮没做的事:把新描述里每一条事实声明都在真实运行时里跑一遍,答案从网络请求里读回来。

  • 线上载荷(截图 1):真实 CLI 打到记录型 provider。workflow 工具描述 base 1026 字符 → head 6088 字符(5.9×);整个请求体 +3.16%/轮。同时确认 script 参数描述逐字节相同(两侧 2413 字符)、工具集合相同(65 个)、system prompt 相同——也就是说常量插值这次重构在模型可见层面是彻底的 no-op,只有工具描述本身变了。
  • 14 条运行时声明逐条实跑(截图 1)Date.now()/Math.random()/new Date() 抛错;沙箱里 require/process/fetch 全是 undefinedworkflow() 单层嵌套——项目级 workflow 跑起来了,它自己再调 workflow() 被拒;workflow('<name>') 分别从项目 .qwen/workflows~/.qwen/workflows 解析成功,同名时项目级胜出;budget.total 无上限时为 null;超预算后裸 await agent() reject,而 parallel()/pipeline() 槽位变 null 且脚本继续;默认并发 = max(1, min(16, cpus-2))——10 核机器上实测在飞峰值正好 8;三个环境变量(并发 3、agent 上限 3、墙钟 2 秒)逐个实测生效。14/14 全过。
  • 测试与变异(截图 2、4):head workflow.test.ts 41/41(PR 正文写的是 40,当前 head 已是 41);未被本 PR 改动的 5 个 workflow 运行时套件 349/349。把 head 的测试文件放到 base 源码上——描述测试如期变红。定点变异:删掉 “Verify before believing” 整段 → 在 /refute/ 变红;删掉 saved-workflow 位置那句 → 在 .qwen/workflows 变红;恢复后 41/41 且工作树干净。
  • 5a466444 这一版到底买到了什么(截图 4):把运行时常量 DEFAULT_MAX_AGENTS_PER_RUN 从 1000 改成 777。上一版 head e4913243:测试 40/40 全绿,但工具描述和 script 参数都还在告诉模型 1000——描述悄悄说了谎而没有任何信号。当前 head:两处都跟着变成 777。这正是该 commit 的价值所在。
  • 其他闸门:prettier/eslint 干净;对当前 origin/main1cbf2e8f,本 PR 落后 14 个 commit)试合并无冲突,合并后 workflow.tsworkflow.test.ts 两个 blob 与 PR head 完全一致;仓库里没有别的测试、文档或快照引用旧描述文本。
  • 非阻塞小发现(3 条):① “also surfaced as /<name> slash commands” 这个括注挂在 project scope 上,但两个 scope 都会被暴露成斜杠命令(packages/cli/src/services/saved-workflow-loader.ts:9),建议把括注移到分号之后覆盖两者;② <projectRoot> 实际是 <targetDir>Storage.getQwenDir(),即 CLI 的工作目录,未必是 git 根);③ PR 正文说本地 40 项通过,当前 head 是 41。
  • 本轮未覆盖:模型行为提升的实测本轮没重跑(公司模型网关此刻从这台机器不通);上一轮在 parent head 0ecc3160 上采的样本仍然适用——5a466444 只动了上限插值、新增位置说明与 pause/resume 措辞,A/B 依赖的策略段落一字未改。macOS/Windows CI lane 对 fork PR 依旧 skip,本轮补的是 macOS 本地数据点。仓库全量套件未在本地重跑。

1 · What actually reaches the model, and what it costs

The workflow tool is opt-in (QWEN_CODE_ENABLE_WORKFLOWS=1), so the harness enables it and captures the request body the CLI sends.

base fd76d4dd head 5a466444
workflow tool description 1,026 chars, 1 line 6,088 chars, 35 lines (5.9×)
script parameter description 2,413 chars 2,413 chars — byte-identical
tool set 65 tools 65 tools, identical names
system prompt unchanged unchanged
whole request payload 161,973 B 167,099 B (+3.16% per turn)

The byte-identical script parameter is the useful part: 5a466444 replaced hand-written env-var names and the 1000 literal with interpolations of the exported constants, and on the wire that refactor is a provable no-op. Only the tool description text moved.

wire capture and runtime claim checks

2 · Every factual claim in the new text, executed (14/14)

Each row is a tool result read off the wire after a probe workflow ran inside the real CLI. The runtime was never mocked — only the model was.

claim in the description how it was exercised result
Date.now() / Math.random() throw called them in a script both rejected, plus new Date()
node:vm sandbox, no filesystem or shell typeof require / process / fetch all undefined
workflow() nests one level only, "doing so throws" project workflow that itself calls workflow() outer ran, inner call rejected with the nesting error
saved workflows under <projectRoot>/.qwen/workflows workflow('outer') resolved from the project dir
… or ~/.qwen/workflows (user scope) workflow('useronly') resolved from QWEN_HOME
user scope is lower precedence same name in both scopes project copy won
budget.total (null = uncapped) no cap set null
once the cap is reached every further agent() is refused cap 200, agents burned 60 tokens each stopped at 240/200 after 4 agents
a bare sequential await agent() sees the rejection one after exhaustion rejected: … exceeded the token budget (240 / 200 output tokens spent)
inside parallel()/pipeline() the refused slot becomes null one of each after exhaustion both null, script kept running
default max(1, min(16, cpus-2)) in flight 24 agents fanned out, responses held 800 ms 10 cores → formula 8; measured peak in flight = 8, in waves of 8/8/8
QWEN_CODE_MAX_WORKFLOW_CONCURRENCY set to 3 measured peak in flight = 3
QWEN_CODE_MAX_WORKFLOW_AGENTS (1000 default) set to 3 rejected at call 3: Workflow exceeded the maximum of 3 agent() calls per run
30-minute wall-clock cap, QWEN_CODE_MAX_WORKFLOW_SECONDS set to 2 s Workflow execution exceeded 2000 ms of active time

Every capability fact the old one-paragraph description stated survives in the new Runtime section, which additionally states the saved-workflow location, the nesting limit, the wall-clock cap and the output-token budget — none of which the old text mentioned.

description shape, before and after, as captured from the wire

3 · Tests and mutations

  • packages/core/src/tools/workflow/workflow.test.ts41/41 at head. (The PR body says 40; the current head has 41.)
  • The five workflow runtime suites this PR does not touch → 349/349, confirming no behavioural drift.
  • Head's test file copied onto the base source tree → the description test fails on its intended assertion (toContain('QWEN_CODE_MAX_WORKFLOW_AGENTS')), the other 40 still pass.
  • Targeted prose mutations on head: removing only "Verify before believing" goes red on /refute/; removing only the saved-workflow location sentence goes red on .qwen/workflows. Restored → 41/41, clean tree.
  • Note: the second new test (script parameter description states the same caps) also passes on base — it is a forward guard for a future cap change, not a regression catcher.

tests and mutation check

4 · What 5a466444 buys, isolated

Mutate the runtime constant DEFAULT_MAX_AGENTS_PER_RUN from 1000 to 777 and ask what the model is told. Both arms keep their own suite green.

arm runtime cap tool description says script param says tests
e4913243 (previous head) 777 1000 1000 40/40 green
5a466444 (this head) 777 777 777 41/41 green

Before this commit, raising a cap would have left the model sizing fan-outs against a stale number with every test still passing. That is a real, silent failure mode, and it is now closed on both model-visible copies.

cap-drift A/B and gates

5 · Gates

  • prettier --check and eslint clean on both changed files.
  • Trial merge into current origin/main (1cbf2e8f; the PR is 14 commits behind) is conflict-free, and in the merged tree workflow.ts and workflow.test.ts are byte-identical to the PR head.
  • No other test, doc, or snapshot consumes the old description string.

6 · Findings (all non-blocking, wording only)

  1. The slash-command parenthetical sits on the wrong side. The text reads <projectRoot>/.qwen/workflows (project scope, also surfaced as /<name> slash commands) or ~/.qwen/workflows (user scope, …), which reads as if only project-scope workflows become slash commands. Both scopes do — saved-workflow-loader.ts:9loads.qwen/workflows/**and**~/.qwen/workflows/"and exposes each as a/`". Moving the parenthetical after the semicolon would cover both.
  2. <projectRoot> is really <targetDir>. Storage.getProjectWorkflowsDir() joins getQwenDir(), i.e. the directory the CLI was launched in, not the git root. Harmless shorthand, but the model may construct paths from it.
  3. PR body test count is stale — it says 40 passed locally; head is at 41.

7 · Not covered

  • No live-model behavioural sampling this round. The model gateway used for the previous round's authored-script A/B is not reachable from this machine right now. Those samples were taken on parent head 0ecc3160; 5a466444 changed only the cap interpolation, the saved-workflow location sentence and the pause/resume wording, leaving every policy section the A/B relied on untouched — so that evidence still applies, but it was not re-collected here.
  • macOS and Windows CI lanes remain skipped for this fork PR. This round is a macOS datapoint (darwin 25.6, Node v24.18.1).
  • The repo-wide suite was not re-run locally; only the workflow surfaces above.

8 · Methodology

Two scratch worktrees (base = merge-base fd76d4dd, head = 5a466444), each npm ci + npm run bundle, driven as node dist/cli.js --yolo -p … with QWEN_HOME, workspace, and saved-workflow fixtures isolated per case. The provider is a ~130-line recording OpenAI-compatible SSE server that writes every request body to disk, scripts the assistant's replies (so the "model" deterministically emits the probe workflow), reports usage so the output-token budget actually accrues, and can hold responses open so in-flight overlap is measurable — that last part is how the default concurrency window was measured rather than asserted. Harnesses, raw request logs, timelines and ANSI captures live under tmp/pr8694-verify/.

@wenshao

wenshao commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

@qwen-code /triage

@wenshao
wenshao enabled auto-merge August 9, 2026 07:50
@qwen-code-ci-bot

qwen-code-ci-bot commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

Sandboxed verification: ⚠️ incomplete — the run timed out with partial evidence - workflow run

The verification run did not complete, so the phases below may be partial or missing entirely. Advisory evidence for human reviewers — not a review, an approval, or a CI check.

中文 — 判定:⚠️ 未完成 · 运行超时,证据不完整

本次验证运行未正常结束,下列内容可能不完整甚至缺失。仅作为评审证据,不构成评审、批准或 CI 检查

No report.md was found in the run artifacts, so the report section is omitted — see the workflow run output.

Evidence images

01-ab-description-base-vs-head

Harness scripts and raw logs are in the workflow run artifacts (7-day retention).

Qwen Code · sandboxed verification

@wenshao

wenshao commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

@qwen-code /triage

@qwen-code-ci-bot

qwen-code-ci-bot commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

Sandboxed verification: ✅ passed — merge-ready (agent verdict) - workflow run

Ran the PR in an isolated, token-free container: A/B against the base build, mock-free harness assertions, targeted gates. Advisory evidence for human reviewers — not a review, an approval, or a CI check.

Scripted assertions: 260 passed · 0 failed · 260 total

中文 — 判定:✅ 通过 · 可合入(agent 判定)

沙箱验证在隔离、无凭证的容器中执行了该 PR 的代码(与 base 构建 A/B 对照、无 mock harness 断言、定向门禁)。仅作为评审证据,不构成评审、批准或 CI 检查

脚本断言:260 通过 · 0 失败 · 260 总计

Verification report

Verify PR #8694 — feat(workflows): add an orchestration policy layer to the Workflow tool description

Verdict: merge-ready — 260/260 scripted assertions passed, 0 unexpected failures.
Verified head: 5a466444e7b1c18e667e1e17ce76f6d39089632f (merge-ref base 10621b3).
First round (no previous-report.md in the context dir).

中文摘要
  • 结论merge-ready。260 项脚本化断言全部通过,0 项意外失败。
  • A/B 结论:承重机制成立。把 DEFAULT_MAX_AGENTS_PER_RUN 从 1000 提到 2000 后,head 的工具描述与 script 参数描述两半都跟随变成 2000;base 的两半保持字面 1000 不变(31/31,见 01-ab-constant-tracking-base-vs-head.png)。能力事实普查:base 14/14、head 17/17,旧描述的全部能力事实在新描述中保留。
  • 测试非空:9 个单点突变全部按设计表现(9/9,见 02-mutation-matrix-pinning-tests.png)。整体回退描述会命中预期断言(expected … to contain 'QWEN_CODE_MAX_WORKFLOW_AGENTS');把字面 1000 粘回描述而常量提到 2000 时,两个新测试同时变红——与作者声称一致。
  • 描述准确性:15/15。描述里手工同步的字面量(30 分钟墙钟、min(16, cpus-2)、单层嵌套、budget.total)与运行时源码逐一核对一致;描述点名的三个环境变量均为运行时真实读取的变量。
  • 未覆盖:未做真实模型编排质量测量(PR 自认需要实机运行);6 个 commit 的逐 commit 归因(浅克隆仅 3 个 commit,只验证聚合 diff);未跑全仓测试(PR 自身 CI 覆盖)。
  • 代价:工具描述 1026 → 6088 字符(+5062,约 1.3k tokens/轮,按 4 字符/token 估);script 参数描述长度不变。PR 已显式承认该取舍。

Central claim + A/B

Central claim: the tool description now carries an orchestration policy layer on top of the runtime facts, and the model-visible caps/env-knobs in both description halves are derived from the orchestrator's exported constants, so raising a cap moves every copy at once (no hand-synced prose).

The load-bearing proof is the tracking flip: raise DEFAULT_MAX_AGENTS_PER_RUN 1000→2000 in identical scratch trees of head and base; head's descriptions must follow, base's must stay literal. Witness: 01-ab-constant-tracking-base-vs-head.png.

cell tree constant tool-desc cap script-param cap result
A head, as-is 1000 1000 1000 knobs named, 8/8 policy anchors present
C base, as-is 1000 1000 (literal) 1000 (literal) knobs NOT named in tool desc, 0/8 policy anchors
B head + cap=2000 2000 2000 (follows) 2000 (follows) tracking ✓
D base + cap=2000 2000 1000 (stays) 1000 (stays) literal control ✓

Flip: 2 cells change from stale-literal (base) to tracking (head). 31/31 A/B assertions passed. The same harness also measured the accepted tradeoff: tool description 1026 → 6088 chars (+5062 ≈ 1.3k tokens at ~4 chars/token); the script param description is byte-length unchanged (2413 = 2413) because the interpolation reproduces the old literals exactly.

Capability preservation (reviewer test plan). Symmetric census of 12 phrasing-robust capability facts (parallel/pipeline signatures, concurrency formula, agent cap, per-call options, resume/journal cache, background-tasks + /workflows surfaces, run_in_background, node:vm sandbox, I/O boundary): base 14/14 (12 common + 2 base-wording), head 17/17 (12 common + 5 head upgrades incl. the three named env knobs). Nothing regressed on the capability side.

Corrections

  • PR body says "40 passed locally"; the shipped head has 41 tests in workflow.test.ts (base 39 + the two new pinning tests). The body was written before the final commit added the script-param test. Gate run confirms 41/41 (and 133/133 orchestrator). Not a defect; the number in the body is stale.
  • The round-2 commit message claims "1000->500 … turn the suite red". At the shipped head that no longer holds by design: round 3 moved the anchors through the exported constants, so a cap change now tracks green (M4/M7) and only a pasted literal goes red (M5/M6). The commit messages describe intermediate states; the final semantics are the ones verified here.

Findings

No blocking findings. Two observations, both accepted/pinned by the PR itself:

  1. Context cost is real and deliberate. +5062 chars (~1.3k tokens) per turn the Workflow tool is exposed. The description states this tradeoff; the census shows it buys the policy layer plus named knobs. Accepted, not blocking.
  2. Two hand-synced literals remain ("30-minute wall-clock cap", and QWEN_CODE_MAX_WORKFLOW_SECONDS which has no exported constant). Both are pinned by the new test and verified accurate against workflow-sandbox.ts (DEFAULT_MAX_WALL_CLOCK_MS = 30 * 60 * 1000, process.env['QWEN_CODE_MAX_WORKFLOW_SECONDS'] read at :589). The JSDoc names them as the remaining hand-sync items. Noted, not blocking.

Mutation matrix + vacuity (9/9 as predicted)

Witness: 02-mutation-matrix-pinning-tests.png. Each mutant = one single-point edit in a scratch head tree, run against workflow.test.ts; expectation encoded independently and adjudicated by script.

mutant edit expected actual
M0 none (control) green 41 passed ✓
M1 revert description to base text red (vacuity) 1 failed — expected … to contain 'QWEN_CODE_MAX_WORKFLOW_AGENTS'
M2 drop "Verify before believing" section red 1 failed ✓
M3 30-minute → 15-minute red 1 failed ✓
M4 cap 1000→2000 green (tracks) 41 passed ✓
M5 cap=2000 + paste literal 1000 in tool desc red (both new tests) 2 failed ✓
M6 cap=2000 + paste literal 1000 in script param red (script test) 1 failed ✓
M7 rename MAX_WORKFLOW_AGENTS_ENV value green (tracks) 41 passed ✓
M8 run_in_background default false→true (positive control) red 1 failed ✓

M1 reached the intended assertion (expected-vs-actual quoted above), satisfying the vacuity rule; M8 proves the harness can red an unrelated pre-existing test. No survivor went unclassified.

Description accuracy (15/15)

Every hand-synced literal and named knob checked against the real runtime source: cap=1000, both env knobs read by resolveConcurrencyLimit/agent-cap resolution, concurrency default max(1, min(16, cpus-2)), wall-clock 30 min + QWEN_CODE_MAX_WORKFLOW_SECONDS read, single-level nesting enforced, budget.total present with null = uncapped, budget gate refuses dispatch with settleToNullArray nulling refused slots in parallel()/pipeline(). The description names no variable the runtime fails to read.

Not covered

  • Live orchestration-quality measurement (does the policy text actually change model-authored scripts?) — the PR itself declares this out of scope; it needs live model runs. This round verifies the text is present, pinned, accurate, and derived — not its behavioral effect on models.
  • Per-commit attribution across the 6 PR commits: the checkout is depth-2 (merge, base tip, PR head only); git rev-list HEAD^1..HEAD^2 returns 1 at the shallow boundary. Only the aggregate HEAD^1..HEAD diff was exercised.
  • Repo-wide gates beyond the two affected suites (174 tests). The PR's own CI covers the rest; I did not re-run it.
  • Compiled-dist A/B: harnesses load source via the repo's own vitest transform (per-tree scratch worktrees), not dist/. The changed surface is a pure string constant, so source vs dist is immaterial; the prebuilt head dist/ exists and the gate ran on source.
  • TUI rendering of the description: it is model-facing tool-schema text, not rendered in the TUI; verified via the instantiated WorkflowTool, which is the surface the model reads.

Methodology

Environment: CI node:22-bookworm container, merge-ref checkout (HEAD=merge, HEAD^1=base 10621b3, HEAD^2=head 5a46644), npm ci+npm run build pre-run at head. Harnesses live in harness/ and raw logs in logs/. A/B and matrix ran in scratch git worktrees (tmp/base-tree, tmp/head-mut-tree, removed after); per-package packages/core/node_modules was symlinked into the worktrees (the PR touches no package.json/lockfile, so this is a clean control). The loaded module graph (workflow.ts, workflow-orchestrator.ts) has no @qwen-code/* imports, so each cell resolves only its own tree's source (realpath-checked). Fact collectors run under the repo's vitest with --silent=false to surface the ABFACTS/CAPFACTS lines; the base capability census additionally extracts the base description textually from the super() string-literal concatenation. Targeted gate: npx vitest run src/tools/workflow/workflow.test.ts src/agents/runtime/workflow-orchestrator.test.ts → 174/174.

Evidence images

01-ab-constant-tracking-base-vs-head

02-mutation-matrix-pinning-tests

Harness scripts and raw logs are in the workflow run artifacts (7-day retention).

Qwen Code · sandboxed verification

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, looks ready to ship. ✅

@wenshao
wenshao added this pull request to the merge queue Aug 9, 2026
Merged via the queue into QwenLM:main with commit 55e20db Aug 9, 2026
109 checks passed
qqqys added a commit to qqqys/qwen-code that referenced this pull request Aug 9, 2026
Resolves the one conflict, packages/core/src/tools/workflow/workflow.test.ts,
by keeping both sides: main's QwenLM#8694 description/policy anchors and this
branch's manifest-seeded preflight-cancellation test.
@qwen-code-ci-bot

Copy link
Copy Markdown
Collaborator

Released in v0.21.9.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/self-reported The linked issue was opened by the PR author (self-reported)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(workflows): add an orchestration policy layer to the Workflow tool description

3 participants