feat: chain Terminal-Bench release evaluation - #9120
Merged
Merged
Conversation
DennisYu07
had a problem deploying
to
production-release
August 14, 2026 07:55 — with 
GitHub Actions
Failure
GitHub Actions
Failure
Contributor
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
DennisYu07
had a problem deploying
to
production-release
August 15, 2026 07:09 — with
GitHub Actions
Failure
GitHub Actions
Failure
DennisYu07
had a problem deploying
to
production-release
August 15, 2026 11:56 — with
GitHub Actions
Failure
GitHub Actions
Failure
DennisYu07
had a problem deploying
to
production-release
August 15, 2026 12:02 — with
GitHub Actions
Failure
GitHub Actions
Failure
DennisYu07
had a problem deploying
to
production-release
August 15, 2026 13:14 — with
GitHub Actions
Failure
GitHub Actions
Failure
DennisYu07
had a problem deploying
to
production-release
August 15, 2026 13:20 — with
GitHub Actions
Failure
GitHub Actions
Failure
DennisYu07
had a problem deploying
to
production-release
August 15, 2026 13:51 — with
GitHub Actions
Failure
GitHub Actions
Failure
DennisYu07
had a problem deploying
to
production-release
August 16, 2026 01:43 — with
GitHub Actions
Failure
GitHub Actions
Failure
DennisYu07
had a problem deploying
to
production-release
August 16, 2026 02:12 — with
GitHub Actions
Failure
GitHub Actions
Failure
DennisYu07
had a problem deploying
to
production-release
August 16, 2026 02:35 — with
GitHub Actions
Failure
GitHub Actions
Failure
DennisYu07
marked this pull request as ready for review
August 17, 2026 01:56
Collaborator
|
✅ Qwen Triage finished — CI landed green on ✅ Qwen Triage 已完成 —— |
Collaborator
|
Thanks for the PR! Re-run after the follow-up commit — both blocking items from the last pass are addressed, so the gate result stands with updates: Template looks good ✓ Problem: Unchanged — a real infrastructure need, not theory: the release benchmark must record the Terminal-Bench follow-up without holding a GitHub Action open for hours (the 15-minute job timeout makes the durable-chain design a forced move). Extends the already-merged SWE release pipeline (#7656). Direction: Still aligned — a natural continuation of the DSW release pipeline (#7656 merged the SWE-only chain; this adds the frozen Terminal-Bench 2.0 leg and the EAS Harbor backends). CHANGELOG: not applicable (internal benchmark infrastructure). Size: Not applicable — no core package paths touched (~337 production lines of shell/Python/workflow plus 58 lines of new helper test). Approach: The two questions that were open at the gate are now answered by the code itself. The tag semantics are explicit in the gate — Risk: No elevated risk signals (no high-risk path matches). Moving on to code review. 🔍 中文说明感谢贡献！ 针对后续 commit 的复查——上一轮的两个阻塞项均已解决，gate 结论更新如下： 模板完整 ✓ 问题： 不变——真实的基础设施需求，不是理论问题：发布评测需要在不长时间占用 GitHub Action 的情况下登记 Terminal-Bench 后续任务（15 分钟 job 超时决定了必须采用持久化链路设计）。是对已合入的 SWE 发布流水线（#7656）的扩展。 方向： 依然对齐——DSW 发布流水线的自然延续（#7656 合入了 SWE 单链路，本 PR 增加冻结的 Terminal-Bench 2.0 环节与 EAS Harbor 后端）。CHANGELOG：不适用（内部评测基础设施）。 规模： 不适用——未触及核心包路径（约 337 行 shell/Python/工作流生产代码，另有 58 行新 helper 测试）。 方案： gate 阶段的两个疑问已由代码本身回答：标签语义在 gate 中明确写出—— 风险： 无升级风险信号（未命中高风险路径）。 进入代码审查 🔍 — Qwen Code · qwen3.8-max Reviewed at |
Collaborator
Code reviewRe-reviewed at the new head. Both blocking findings from the previous pass are fixed, verified in the diff:
The scaffolding still reads well: fail-fast validation throughout, the TB limit constrained to {1, 89} on both the shell and the Python side, deterministic sorted task selection, and the manifest reader only enumerates tar member names (no extraction). One question from the last pass is settled by the code itself: the smoke-tag inversion is gone — One honest caveat: the The runtime flow (unchanged from the last pass): sequenceDiagram
participant P1 as Release event
participant P2 as release_gate job
participant P3 as dispatch script
participant P4 as pool CLI
participant P5 as DSW Publisher
participant P6 as GitHub Release
P1->>P2: release published or manual dispatch
P2->>P3: backend, limits and tag outputs
P3->>P4: submit SWE run
P3->>P3: build frozen TB manifest of 89 tasks
P3->>P4: create-release-chain with TB pending
P3-->>P2: dispatch receipt, Action exits
Note over P4,P5: SWE reaches a terminal state
P5->>P6: publish SWE JSON, trajectories, score section
P5->>P4: mark SWE PUBLISHED, dispatch TB idempotently
P5->>P6: append TB JSON, trajectories, score section
Test evidence (this PR's own CI, read via API — PR code never executed here)The ubuntu suite — the leg that carries actionlint/shellcheck and the Final CI results for
One row per check name (latest run); skipped checks omitted; failures sort first. / 每个检查名一行（取最新一次运行），省略 skipped，失败项排在最前。 Not verified: the end-to-end chain behavior (SWE publication → idempotent TB dispatch → both sections in one Release). That behavior lives in the companion benchmark-service Publisher and the self-hosted DSW environment, which this repo's CI cannot reach — and the CLI-scoped sandboxed lanes don't apply to 中文说明代码审查已按新 head 重新审查。上一轮的两个阻塞项均已修复，并经 diff 核实：
脚手架依旧扎实：处处快速失败校验、shell 与 Python 两侧都把 TB limit 限制在 {1, 89}、确定性排序选取任务、manifest 读取器只枚举 tar 成员名（不解压）。 上轮的一个疑问已由代码本身解决：smoke 标签反转不复存在—— 一点诚实的保留：8 月 15–16 日的 测试证据此处仅通过 API 读取该 PR 自身的 CI 结果，未执行任何 PR 代码。ubuntu 套件（承载 actionlint/shellcheck 与 未验证：端到端链路行为（SWE 发布 → 幂等 TB 派发 → 同一 Release 双区块）。该行为位于配套 benchmark-service 的 Publisher 与自托管 DSW 环境，本仓库 CI 无法触达——CLI 侧沙箱通道也不适用于 — Qwen Code · qwen3.8-max Reviewed at |
Collaborator
|
Confidence: 4/5 — both requested fixes are in and verified in the diff; the only remaining caveat is operational, not code: the final commit has passed nothing but CI static checks so far, and the ubuntu suite carrying them is still running. Stepping back: the approach still matches what I'd have proposed independently — submit SWE, persist the frozen TB task set immediately, and let the Publisher dispatch TB only after the SWE Release write succeeds. Both items from the previous The one reservation, stated plainly: the chain behavior at the final commit is backed by static checks and the author's Aug 15–16 lane runs on earlier revisions of this branch, not by a live dispatch of this exact code. That residual gap closes with the first Verdict: approve once CI lands green. The ubuntu suite (actionlint, shellcheck, and the newly wired manifest test) is still in progress on the reviewed commit, so approval is deferred until CI lands green on 中文说明置信度：4/5 —— 所要求的两项修复均已落地并经 diff 核实；唯一保留是运维层面的而非代码层面的：最终 commit 目前只通过了 CI 静态检查，且承载这些检查的 ubuntu 套件仍在运行。 退一步看：方案依旧与我的独立设想一致——提交 SWE、立即持久化冻结的 TB 任务集、由 Publisher 在 SWE Release 回写成功后才派发 TB。上一轮 唯一保留，直说：最终 commit 上的链路行为，依据是静态检查加上作者 8 月 15–16 日在此分支早期版本上的链路运行记录，而非这份代码本身的实弹 dispatch。该缺口由首次 结论：CI 全绿后批准。ubuntu 套件（actionlint、shellcheck 与新接入的 manifest 测试）在受审 commit 上仍在进行，批准将推迟到 CI 在该 commit 上全绿之后。 — Qwen Code · qwen3.8-max Reviewed at |
qwen-code-ci-bot
requested changes
Aug 17, 2026
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
Two concrete items before this can merge — details in my review comments above:
- Add
.github/scripts/dsw-swe-verified/make-terminal-bench-manifest.test.mjstoHELPER_TESTSin.github/workflows/ci.yml— as-is the new test never runs in CI. - The durable-chain behavior depends on the companion benchmark-service, and the PR body says the full 500 + 89 E2E evidence is still pending while the PR is marked ready — please land the
dsw-eas-tb-smoke-*/dsw-eas-full-*evidence or switch back to draft until it exists.
@DennisYu07 also worth answering in the same pass: the dsw-eas-smoke-* tag chaining the full 89-task TB suite (vs 1 for dsw-eas-tb-smoke-*), chainless-run handling when create-release-chain fails after submit, and restoring the env: pattern for the two free-text dispatch inputs.
Collaborator
Author
|
@qwen-code /triage |
Collaborator
|
Triage re-run completed without a new review.
The stage comments above were updated with the latest result. View workflow run. 上方各阶段评论已更新为最新结果。查看工作流运行。 |
Collaborator
|
Sandboxed verification: ❌ not passed — findings reported (agent verdict) - workflow run Ran the PR in an isolated, token-free container: A/B against the base build, mock-free harness assertions, targeted gates. Advisory evidence for human reviewers — not a review, an approval, or a CI check. Scripted assertions: 130 passed · 0 failed · 130 total 中文 — 判定：❌ 不通过 · 报告了发现（agent 判定）沙箱验证在隔离、无凭证的容器中执行了该 PR 的代码（与 base 构建 A/B 对照、无 mock harness 断言、定向门禁）。仅作为评审证据，不构成评审、批准或 CI 检查。 脚本断言：130 通过 · 0 失败 · 130 总计 Verification reportPR #9120 — Deep verification reportVerdict: 中文摘要
Central claim + A/BCentral claim: the release chain records a frozen Terminal-Bench 2.0 follow-up exactly once at dispatch time (and the Action exits), with per-tag routing and per-backend wiring exactly as described.
Witnesses: The load-bearing proof is the base/head contrast: base never calls FindingsF1 — Frozen-manifest count gate silently absorbs duplicate task directory names (Low)
Likelihood is low — the archive is a frozen, versioned ACR artifact and task names are presumably globally unique by construction — but this is exactly the boundary the "validates exactly 89 tasks" claim should decide. Reproduce: Minimal suggested fix (measured, preserves intent)Count filtered members instead of unique names: task_names: set[str] = set()
instruction_count = 0
...
instruction_count += 1
task_names.add(parts[-2])
if instruction_count != 89:
raise SystemExit(f"expected 89 Terminal-Bench 2.0 tasks, found {instruction_count}")Measured in F2 — New test suite does not pin the
|
qwen-code-ci-bot
approved these changes
Aug 17, 2026
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
LGTM, looks ready to ship — CI landed green after the review. ✅
yiliang114
approved these changes
Aug 17, 2026
Collaborator
|
Released in v0.21.14. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Adds a durable Harbor benchmark release chain. A Release event starts one short GitHub Action that submits the 500-case SWE-bench Verified run and records the frozen 89-task Terminal-Bench 2.0 follow-up in the same PostgreSQL release chain. The Action then exits; it does not wait for either benchmark.
After SWE reaches a terminal state, the persistent DSW Publisher uploads the SWE JSON result and trajectory bundle and updates the SWE Release-note section. Only after that GitHub Release update succeeds does the Publisher mark the SWE publication
PUBLISHEDand dispatch the recorded TB run. When TB finishes, the Publisher appends a distinct TB JSON result, trajectory bundle, and### Terminal-Bench 2.0section to the same Release.Why it's needed
A release benchmark needs an end-to-end, auditable result without holding a GitHub Action open for hours. The durable chain preserves the exact TB task set and makes dispatch idempotent. SWE and TB scores remain independent: a successfully published quarantined SWE result still starts TB, while a failed SWE Release update keeps TB pending until publication retry succeeds.
End-to-end sequence
pool_release_chainsrow containing the frozen TB 2.0 89-task follow-up.PUBLISHEDand dispatches TB idempotently.Reviewer Test Plan
tb_dispatch_state=PENDING.publication_status=PUBLISHED, creates exactly one 89-case TB run, and setstb_dispatch_state=DISPATCHEDwithtb_run_id.QUARANTINEDSWE result also dispatches TB.resolved / (resolved + unresolved)from final verifier results.Evidence
Tested on
Risk & Scope
Linked Issues
None.
中文说明
该 PR 新增一条持久化 Harbor 发布评测链路：Release 事件只触发一次短时 Action。Action 提交 SWE-bench Verified 500 case，并在同一条 PG release chain 中登记冻结的 Terminal-Bench 2.0 89-task follow-up，然后立即退出，不等待评测完成。
SWE 终态后，DSW Publisher 先把 SWE JSON、trajectory bundle 和分数区块写入同一个 Release。只有 GitHub Release 回写成功后，Publisher 才把 SWE publication 标为
PUBLISHED，并幂等创建 TB run、写入tb_run_id、把tb_dispatch_state改为DISPATCHED。TB 完成后，再向同一个 Release 追加独立的 TB JSON、trajectory bundle 和### Terminal-Bench 2.0区块。SWE 与 TB 的评分相互独立：成功发布的
QUARANTINEDSWE 结果仍会启动 TB；如果 SWE Release 回写失败，TB 保持 PENDING，等 Publisher 重试发布成功后再启动。