ci: add isolated DSW SWE-bench release pipeline - #7656
Merged
Conversation
DennisYu07
had a problem deploying
to
production-release
July 24, 2026 06:43 — with 
GitHub Actions
Failure
GitHub Actions
Failure
DennisYu07
had a problem deploying
to
production-release
July 24, 2026 06:51 — with
GitHub Actions
Error
GitHub Actions
Error
DennisYu07
had a problem deploying
to
production-release
July 24, 2026 06:58 — with
GitHub Actions
Error
GitHub Actions
Error
DennisYu07
had a problem deploying
to
production-release
July 24, 2026 07:02 — with
GitHub Actions
Error
GitHub Actions
Error
qwen-code-ci-bot
had a problem deploying
to
production-release
July 24, 2026 07:38 — with
GitHub Actions
Error
GitHub Actions
Error
qwen-code-ci-bot
had a problem deploying
to
production-release
July 24, 2026 07:40 — with
GitHub Actions
Error
GitHub Actions
Error
DennisYu07
had a problem deploying
to
production-release
July 24, 2026 08:34 — with
GitHub Actions
Failure
GitHub Actions
Failure
Collaborator
Author
|
Full 500-case DSW release-chain run completed and was written back to the test prerelease.
Public JSON validation passed: 500 records, 500 unique instance IDs, no duplicate IDs, and state totals match the Release summary. DSW SSH is currently unreachable, so final PostgreSQL-to-OSS alignment and unified root-cause analysis of the 56 execution errors / 5 infrastructure failures remain pending host access restoration. This was a non-production prerelease validation; do not merge based on the quarantined score. |
Collaborator
Review:
|
This was referenced Jul 25, 2026
DennisYu07
had a problem deploying
to
production-release
July 27, 2026 09:22 — with
GitHub Actions
Failure
GitHub Actions
Failure
Collaborator
Author
|
Updated the release benchmark gate in
Verified locally with the repository-pinned |
DennisYu07
marked this pull request as ready for review
July 28, 2026 09:09
Collaborator
|
✅ Qwen Triage finished — CI landed green on ✅ Qwen Triage 已完成 —— |
Collaborator
|
Thanks for the PR! Template looks good ✓ — all major sections are present with extensive detail. Problem: this is a Direction: aligned. Automated release benchmarking is a legitimate CI/CD need, and the isolation from #7584 (separate workflow, runner label, suite, database, OSS prefix, and Release markers) is thorough. No core product code is touched — this is purely Size: not applicable — no core paths touched. 441 additions across 4 new files, all production CI/docs (no test or generated files). Approach: the scope feels right. Four files, each with a clear purpose: workflow trigger, dispatch script, manifest generator, design doc. No drive-by changes or unrelated edits. The decomposition (gate job on GitHub-hosted runner → benchmark job on self-hosted runner) is clean. wenshao's three blocking findings (unfiltered release trigger, yamllint violations, instance_id forwarded for multi-instance runs) are all addressed in the current diff — the version gate regex Risk: no elevated risk signals — no high-risk paths matched. Moving on to code review. 🔍 中文说明感谢贡献！ 模板完整 ✓ — 所有主要章节均存在，内容详尽。 问题：这是一个 方向：对齐。自动化发布基准测试是合理的 CI/CD 需求，与 #7584 的隔离（独立 workflow、runner 标签、suite、数据库、OSS 前缀和 Release 标记）非常彻底。未触及核心产品代码。 规模：不适用 — 未触及核心路径。4 个新文件共 441 行添加。 方案：范围合理。四个文件各有明确用途。wenshao 此前提出的三个阻塞问题（未过滤的 release 触发器、yamllint 违规、多实例运行时转发 instance_id）均已在当前 diff 中修复。 风险：无升级风险信号。 进入代码审查 🔍 — Qwen Code · qwen3.8-max-preview Reviewed at |
Collaborator
Code ReviewRead the PR title and "Why it is needed" section first, then sketched my own approach: a release-triggered workflow with a version gate, a short dispatch script that freezes a manifest and submits to the pool, and a design doc. The PR matches this almost exactly — same decomposition, same boundaries. Went through all four files line by line. No critical blockers, no AGENTS.md violations. dispatch-release-benchmark.sh — defensive and well-structured. make-manifest.py — minimal, stdlib-only, does exactly one thing. The hard assertion on exactly 500 instances is the right call for SWE-bench Verified. Single-instance mode with the dsw-swe-verified-release.yml — the two-job structure (gate on GitHub-hosted → benchmark on self-hosted) is the right split. Design doc — comprehensive without being bloated. Isolation boundaries, component contract, and validation evidence are all documented. Good reference for future maintainers. All three of wenshao's prior blocking findings are addressed in the current diff. No new concerns surfaced. CI Test EvidenceCI is still running on the reviewed commit — the ubuntu-latest test job is in progress. This PR only touches Final CI results for
One row per check name (latest run); skipped checks omitted; failures sort first. / 每个检查名一行（取最新一次运行），省略 skipped，失败项排在最前。 Real-scenario testing: N/A — this PR adds CI/CD infrastructure (workflow, scripts, design doc) with no user-visible behavioral changes to the CLI product. 中文说明代码审查先读了 PR 标题和"Why it is needed"部分，独立构思了方案：一个 release 触发的 workflow（带版本门控）、一个短小的分发脚本（冻结 manifest 并提交到池）、以及一份设计文档。PR 与此几乎完全一致。 逐行审查了全部四个文件。无关键阻塞问题，无 AGENTS.md 违规。 dispatch-release-benchmark.sh — 防御性良好，结构清晰。所有必需环境变量预先断言，数值输入有范围检查，路径和缓存目录在工作开始前验证。无注入向量。 make-manifest.py — 最小化，仅用标准库。对恰好 500 个实例的硬断言是正确的。 dsw-swe-verified-release.yml — 双 job 结构（GitHub 托管上的门控 → 自托管上的基准测试）划分合理。权限最小化。版本门控正则 设计文档 — 全面而不冗余。 CI 测试证据CI 仍在运行中 — ubuntu-latest 测试 job 进行中。此 PR 仅触及 真实场景测试：N/A — 此 PR 添加 CI/CD 基础设施，无用户可见的行为变更。 — Qwen Code · qwen3.8-max-preview Reviewed at |
Collaborator
|
Confidence: 4/5 — clean, focused infrastructure PR; all prior review feedback addressed; validation evidence is concrete and public. Only reservation is CI still in flight. Stepping back: this PR does one thing — wire a GitHub Release event to a persistent DSW benchmark pool — and does it cleanly. The four-file decomposition is exactly what I'd propose independently. No scope creep, no drive-by changes, no over-engineering. The shell script is defensive without being paranoid, the Python is minimal, the workflow is properly gated, and the design doc gives future maintainers the context they need. The async dispatch pattern (Actions submits and exits in <15 minutes, persistent pool owns the 12+ hour run) is the right architectural call — it decouples CI worker lifetime from benchmark duration. The isolation from #7584 is thorough: separate workflow, runner label, suite, database, runtime root, OSS prefix, and Release markers. Nothing leaks. wenshao's three blocking findings (unfiltered trigger, yamllint, instance_id forwarding) are all properly fixed in the current diff. The version gate regex If I had to maintain this in six months, I'd thank the author — the design doc, isolation boundaries, and idempotency key make it straightforward to reason about. Approval deferred until CI lands green on 中文说明置信度：4/5 — 干净、聚焦的基础设施 PR；所有先前 review 反馈已处理；验证证据具体且公开。唯一保留是 CI 仍在运行。 退一步看：这个 PR 只做一件事——将 GitHub Release 事件连接到持久 DSW 基准测试池——而且做得很干净。四文件分解与我的独立方案完全一致。无范围蔓延，无顺手改动，无过度工程。 异步分发模式（Actions 在 <15 分钟内提交并退出，持久池拥有 12+ 小时运行）是正确的架构选择。与 #7584 的隔离非常彻底。 wenshao 此前提出的三个阻塞问题均已在当前 diff 中正确修复。已完成的 500 案例验证运行及公开 JSON 证据让我确信管道端到端可用。 批准推迟到 CI 在 — Qwen Code · qwen3.8-max-preview Reviewed at |
Contributor
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
qwen-code-ci-bot
previously approved these changes
Jul 28, 2026
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
LGTM, looks ready to ship — CI landed green after the review. ✅
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed — no blockers. Suggestions are inline.
— qwen3.7-max via Qwen Code /review
| { | ||
| echo "### DSW SWE-bench Verified dispatch" | ||
| echo | ||
| echo "- Pool run: \`${{ steps.dispatch.outputs.run_id }}\`" |
Collaborator
There was a problem hiding this comment.
[Suggestion] ${{ steps.dispatch.outputs.run_id }} is interpolated directly into a run: shell script, making the value shell syntax rather than shell data. Every other step output in this workflow is passed through env: and referenced as a shell variable — this is the sole exception.
Failure scenario: If the pool tool ever returned a run_id containing shell metacharacters (backticks, $(...), semicolons), they would be interpreted by the shell. The practical risk is low — the pool is an internal tool on a self-hosted runner — but defense-in-depth consistency favors routing through env:.
- name: 'Record dispatch receipt'
env:
POOL_RUN_ID: '${{ steps.dispatch.outputs.run_id }}'
run: |-
{
echo "### DSW SWE-bench Verified dispatch"
echo
echo "- Pool run: \`${POOL_RUN_ID}\`"
echo "- State: queued"
echo "- Cases: ${INSTANCE_LIMIT}"
echo "- The persistent DSW publisher will update the Release after the run reaches a validated terminal state."
} >> "${GITHUB_STEP_SUMMARY}"— qwen3.7-max via Qwen Code /review
Comment on lines
+17
to
+21
| instance_ids = sorted( | ||
| path.name | ||
| for path in args.dataset_root.iterdir() | ||
| if path.is_dir() and "__" in path.name | ||
| ) |
Collaborator
There was a problem hiding this comment.
[Suggestion] The manifest-generation logic has no tests, despite the project maintaining 8 co-located .test.mjs files for other .github/scripts/ entries. This is also the first Python script in this directory, so no test runner is wired up for it.
Failure scenario: If the "__" in path.name filter is inadvertently changed or the dataset structure evolves (e.g., directories renamed, extra non-instance directories added), the 500-count assertion could either fail on a valid dataset or pass with wrong instance IDs. A test creating a temporary directory with mock instance folders would catch such regressions — and would have caught the --instance-id/--limit interaction that was blocking issue (3) in the prior review.
— qwen3.7-max via Qwen Code /review
Collaborator
Local runtime verification of #7656I rebuilt the GitHub half of this pipeline on a local machine and ran the PR's actual scripts end to end, then drove a matrix of trigger and guard scenarios against them. Result: the mechanics are solid — 13 trigger scenarios and 13 guard cases each produced exactly what the code specifies, including idempotent submission and the frozen-manifest checks. Three issues should be resolved before merge:
How this was verifiedNothing in the PR was modified for the test:
Environment: Linux, bash 5.2, jq 1.7, Python 3.13, Node 22, PR head 1. Blocking — the repo's own
|
| Area | Checks |
|---|---|
| Input validation | INSTANCE_LIMIT ∈ {0, 501, -1, abc} rejected with exit 2 before anything is submitted; BENCHMARK_MAX_ATTEMPTS=9 and a non-integer backoff rejected |
| DSW preflight | Missing pool binary, missing dataset root, missing cache subdirectory, and a non-writable cache each stop with exit 2 and a ::error:: annotation |
| Manifest freeze | A 499-instance dataset is refused; an unknown --instance-id is refused; the emitted manifest always carries 500/5/1 unique IDs matching expected_instances |
| Idempotency | Two dispatches with the same BENCHMARK_IDEMPOTENCY_KEY (an Actions re-run) return the same pool run_id and do not double-queue |
| Release resolution | Draft releases rejected; prerelease=true + marker honoured with CRLF bodies; marker in a stable release body correctly ignored; tag resolved to an immutable commit through a depth-1 fetch |
| Submission payload | Suite, dataset, revision, task prefix, model, retry budget, release id/tag and run URL all arrive as documented |
| Isolation from #7584 | Distinct workflow file, suite name, database, runtime root and OSS prefix; no path overlaps with benchmark-dispatch.yml |
| Repo conventions | actionlint, shellcheck clean; actions/checkout pinned to the same SHA ci.yml uses |
One informational note: the new cache-writability preflight uses [[ -w ]], which is always true for root. Confirmed with an unprivileged uid (correctly rejected) versus root (passes through). The design doc implies the runner is not root (model.key is root:github-runner 0640), so this is a caveat rather than a defect.
Verdict
The GitHub-side contract is well built: it validates early, freezes the manifest, never partially submits, and is genuinely idempotent. I would like 1–3 fixed before merge (1 is already failing CI, 2 has a daily cost, 3 breaks two of the three dropdown options), and 4–7 are worth a follow-up.
中文版本（点击展开）
对 #7656 的本地真实运行验证
我在本地把这条流水线的 GitHub 侧完整搭了出来，直接跑 PR 里未经修改的脚本，并用一组触发场景与防护场景做了矩阵测试。
结论：机制本身是扎实的 —— 13 个触发场景与 13 个防护用例的结果都与代码规格完全一致（包含幂等提交与 manifest 冻结校验）。合并前建议先解决 3 个问题：
| # | 级别 | 问题 |
|---|---|---|
| 1 | 阻塞 | 仓库自带的 yamllint 门禁在新 workflow 上失败，本 PR 的 CI 已经是红的 |
| 2 | 高 | release: published 没有 prerelease 守卫，每天的 nightly 发布都会触发一次 500 用例（约 12.5 小时）的完整跑 |
| 3 | 高 | workflow_dispatch 选 5 或 500 时，如果不手动清空 instance_id 会直接失败 |
| 4 | 中 | executor_count 从未传给 pool，但 receipt 和 job summary 却按已生效汇报 |
| 5 | 低 | release 正文里的 marker 会静默覆盖显式传入的 qwen_release_tag |
| 6 | 低 | 两条失败路径没有可读的错误信息 |
| 7 | 细节 | permissions: contents: write 超出该 job 的实际需要 |
验证方式
dispatch-release-benchmark.sh与make-manifest.py直接使用 PR worktree 中的原始文件运行。Resolve release步骤的脚本体从 workflow YAML 中逐字提取，用bash -e执行；api.github.com由 PATH 上的curlshim 指向本地 mock（只改主机名），git 远端是真实的本地仓库并带有真实 tag；release 正文 fixture 使用 CRLF，与 REST API 的真实返回一致。env:的取值不是手抄的：用一个小型 GitHub 表达式求值器（已用 9 条&&/||/ falsy 语义用例校准）读取本 PR 的 YAML，为每种事件负载计算RELEASE_TAG、INSTANCE_LIMIT、EXECUTOR_COUNT、BENCHMARK_INSTANCE_ID等。- 内部仓库里的
qwen-benchmark-pool与prepare-agent-cache.py用基于 SQLite 的替身实现，参数面与脚本调用完全一致，因此"幂等性"和"最终提交给 pool 的到底是什么"都可观测。 - 数据集 fixture 使用真实的 500 个 SWE-bench Verified instance ID（由公开的
swe-bench/experiments结果清单还原）。
环境：Linux、bash 5.2、jq 1.7、Python 3.13、Node 22，PR head 6dc5a249a。
1（阻塞）仓库自带的 yamllint 门禁失败
node scripts/lint.js --yamllint 在 main 上干净退出 0，在本 PR head 上退出 1，报 40 条 quoted-strings 错误，全部集中在 .github/workflows/dsw-swe-verified-release.yml。GitHub Actions 上同一 head 的结论一致：job 89939747042 第 20 步 Run yamllint 失败，之后所有步骤（包括全部测试）被跳过。actionlint、shellcheck、eslint 均通过，仅违反 YAML 引号规则；仓库里其它 workflow（含 #7584）都用单引号。修复：给标量加单引号，再用 node scripts/lint.js --yamllint 复核。
2（高）每个 nightly 发布都会拉起一次 500 用例全量跑
job 没有任何 if:，且 release 分支的表达式把 INSTANCE_LIMIT 写死为 '500'。用当前 nightly tag 的真实 release.published 负载驱动，队列里就是 500 个任务，与正式版发布完全相同。
依据：.github/workflows/release.yml 每天 cron: '0 0 * * *' 发布一个 nightly Release（7-25 / 7-26 / 7-27 均有）；该 Release 由 secrets.CI_BOT_PAT 创建，release.yml 里明确注释"必须用 PAT，因为 GITHUB_TOKEN 事件无法触发下游 release 事件的 workflow"，所以 release.published 确实会触发。nightly 是 prerelease 且发布说明为自动生成，不含 Benchmark-Qwen-Ref: marker，因此会直接对该 nightly 跑满 500 用例。按本 PR 自己的数据，一次 500 用例约 12 小时 27 分。另外，concurrency 只串行化 Actions 侧的短任务，pool 侧的 run 仍会照常提交并叠加。
#7584 恰好挡住了这种情况：if: github.event_name != 'release' || github.event.release.prerelease == false。修复：要么照搬该守卫（只跑正式版，POC 用 workflow_dispatch），要么仅在正文包含 Benchmark-Qwen-Ref: marker 时才接受 prerelease —— 后者能保留文档里的 POC 流程并排除 nightly。
3（高）workflow_dispatch 选 5 / 500 时会因默认 instance_id 直接失败
instance_id 带非空默认值 sympy__sympy-20590，除非手动清空，GitHub 都会提交它；脚本只要该变量非空就会传 --instance-id，而 make-manifest.py 拒绝这种组合：D1（limit 1）通过；D2（limit 500 + 预填 instance_id）报 --instance-id requires --limit 1、exit 1、什么都没入队；D3（手动清空）正常入队 500。评审测试计划第 5 步只覆盖 limit=1，所以问题被掩盖了。修复：仅当 INSTANCE_LIMIT 为 1 时才传 --instance-id，或者去掉该输入的默认值。
4（中）executor_count 实际未生效却按生效汇报
EXECUTOR_COUNT 从未传给 qwen-benchmark-pool submit，只出现在 receipt JSON 和 step summary（- DSW executors: ${EXECUTOR_COUNT}）。D4 用 executor_count=1 跑出来的 receipt 写着 "executor_count": 1，而提交给 pool 的 payload 里根本没有这个字段，常驻服务仍是 10 个 Executor。修复：要么真的传下去，要么删掉这个输入并停止汇报。
5（低）release 正文 marker 静默覆盖显式输入
Resolve release 里 qwen_ref 先取 QWEN_REF（手动触发时即 inputs.qwen_release_tag），随后只要是 prerelease 就被正文里的 Benchmark-Qwen-Ref: 无条件替换。以 qwen_release_tag=v0.21.0 触发一个正文仍写着 v0.20.0-nightly.20260722.b98306b7e 的 POC prerelease，最终跑的是那个 nightly，日志里没有任何提示（场景 D6）。修复：让显式输入优先，或至少在覆盖时打一条 ::notice::。
6（低）两条失败路径没有可读信息
草稿 release：test ... = "false" 失败时 stderr 为空，操作者只能看到 Process completed with exit code 1（场景 R6）；marker 指向不存在的 tag：fatal: couldn't find remote ref ...、exit 128，看不出根因在 release 正文（场景 R7）。修复：各加一条 ::error::。
7（细节）最小权限
job 声明了 permissions: contents: write，但它只通过 REST API 读取 release、并 fetch 一个 tag；写回由 DSW 侧的 Publisher 用自己的凭据完成。contents: read 足够，也能避免在长期存活的 self-hosted runner 上留下带写权限的 token。另外 instance_limit=5 恒定取排序后的前 5 个 astropy 用例（instance_ids[:limit]），作为冒烟没问题，但不是抽样。
验证通过的部分
输入校验（INSTANCE_LIMIT 为 0/501/-1/abc 均在提交前 exit 2；BENCHMARK_MAX_ATTEMPTS=9 与非整数 backoff 均被拒）；DSW 前置检查（缺 pool 二进制、缺数据集根目录、缺 cache 子目录、cache 不可写，均 exit 2 且带 ::error::）；manifest 冻结（499 个实例的数据集被拒、未知 --instance-id 被拒、产出的 manifest ID 数量与 expected_instances 始终一致且唯一）；幂等性（同一 BENCHMARK_IDEMPOTENCY_KEY 重复触发返回同一个 run_id，不会重复入队）；release 解析（草稿被拒、CRLF 正文下 prerelease + marker 生效、正式版正文里的 marker 被正确忽略、depth-1 fetch 后 tag 解析为不可变 commit）；提交负载（suite、dataset、revision、task prefix、model、重试预算、release id/tag、run URL 均符合文档）；与 #7584 的隔离（workflow 文件、suite、数据库、运行根目录、OSS 前缀均不同，与 benchmark-dispatch.yml 无路径重叠）；仓库规范（actionlint、shellcheck 干净，actions/checkout 与 ci.yml 用同一 SHA）。
一条补充说明：新增的 cache 可写性检查用 [[ -w ]]，对 root 恒为真。我用非特权 uid（正确拒绝）与 root（直接放行）各验证了一次。设计文档中 model.key 为 root:github-runner 0640，说明 runner 并非 root，因此这只是个注意事项而非缺陷。
总体意见
GitHub 侧的契约做得不错：校验前置、manifest 冻结、不会部分提交、幂等性是真的。建议合并前先处理 1–3（1 已经导致 CI 失败，2 有每天的真实成本，3 让下拉框三个选项中的两个不可用），4–7 可以后续跟进。
Collaborator
Author
|
Publisher gate and artifact contract updated:
Implementation and validation:
|
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed — no blockers. Suggestions are inline.
— qwen3.7-max via Qwen Code /review
| INSTANCE_LIMIT: '${{ github.event_name == ''release'' && ''500'' || inputs.instance_limit }}' | ||
| BENCHMARK_INSTANCE_ID: '${{ (github.event_name != ''release'' && inputs.instance_limit == ''1'') && inputs.instance_id || '''' }}' | ||
| RELEASE_ID: '${{ github.event_name == ''release'' && github.event.release.id || '''' }}' | ||
| BENCHMARK_TRIGGER: '${{ github.event_name }}' |
Collaborator
There was a problem hiding this comment.
[Suggestion] BENCHMARK_TRIGGER is declared in the benchmark job's env: block but is never read by dispatch-release-benchmark.sh, make-manifest.py, or any other step in this workflow.
Failure scenario: A future maintainer greps for BENCHMARK_TRIGGER, finds it defined here, and assumes changing it affects dispatch logic — or spends time tracing why a change has no effect.
Suggested fix: Remove the line, or if it is intended for a future consumer in the DSW pool binary, add a comment naming what will read it.
— qwen3.7-max via Qwen Code /review
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed — no blockers. Suggestions are inline.
— qwen3.7-max via Qwen Code /review
Comment on lines
+91
to
+96
| curl --fail --silent --show-error --location \ | ||
| --header "Accept: application/vnd.github+json" \ | ||
| --header "Authorization: Bearer ${GH_TOKEN}" \ | ||
| --header "X-GitHub-Api-Version: 2022-11-28" \ | ||
| "https://api.github.com/repos/${GITHUB_REPOSITORY}/releases/tags/${RELEASE_TAG}" \ | ||
| > release.json |
Collaborator
There was a problem hiding this comment.
[Suggestion] This is the only curl call against api.github.com in the entire repository — every other workflow (140+ call sites) uses gh api, which handles authentication headers, API versioning, and content negotiation automatically.
Concrete cost: maintenance divergence. Any future change to GitHub API auth conventions must be applied here manually, and the GH_TOKEN env var is only needed because of curl — gh api reads GITHUB_TOKEN automatically on Actions runners.
Suggested change
| curl --fail --silent --show-error --location \ | |
| --header "Accept: application/vnd.github+json" \ | |
| --header "Authorization: Bearer ${GH_TOKEN}" \ | |
| --header "X-GitHub-Api-Version: 2022-11-28" \ | |
| "https://api.github.com/repos/${GITHUB_REPOSITORY}/releases/tags/${RELEASE_TAG}" \ | |
| > release.json | |
| gh api "repos/${GITHUB_REPOSITORY}/releases/tags/${RELEASE_TAG}" > release.json |
If gh is unavailable on the qwen-benchmark-dsw runner, a comment on this step explaining why curl is used would help future maintainers.
— qwen3.7-max via Qwen Code /review
Comment on lines
+108
to
+112
| run_id="$( | ||
| "${python_bin}" -c ' | ||
| import json | ||
| import re | ||
| import sys |
Collaborator
There was a problem hiding this comment.
[Suggestion] This inline Python run_id validator (JSON parse + type check + regex validation) is the most logic-dense piece of the dispatch script but has no test coverage.
Failure scenario: If pool submit returns a response where run_id is an integer, an empty string, nested in a list, or contains shell-special characters, the validator's accept/reject behaviour is untested. A regex change or a pool output format change could silently accept a malformed run_id or reject a valid one.
Suggested fix: extract this inline Python into a standalone script (e.g., validate-run-id.py) and add a .test.mjs that feeds it known-good and known-bad JSON fixtures.
— qwen3.7-max via Qwen Code /review
…-pipeline # Conflicts: # .github/workflows/ci.yml
qwen-code-ci-bot
approved these changes
Jul 29, 2026
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
No issues found. LGTM! ✅
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/ci.yml: actionlint embedded-shell source mapping is not yet supported — not linted; the executable-script lint — .github/workflows/dsw-swe-verified-release.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
— qwen3.8-max-preview via Qwen Code /review
yiliang114
self-requested a review
July 29, 2026 06:25
yiliang114
approved these changes
Jul 29, 2026
yiliang114
left a comment
yiliang114
left a comment
Collaborator
There was a problem hiding this comment.
LGTM — well-engineered CI pipeline with strong shell hygiene and proper GitHub Actions security patterns. No blockers.
Non-blocking suggestions:
- Add semver format validation for
qwen_refon theworkflow_dispatchpath (release-event path already has it) - Validate
RELEASE_TAGformat before using it in the curl URL path segment - Guard
make-manifest.py --limitagainst 0/negative values - Add boundary tests (limit=0, limit>500, empty dataset)
Collaborator
|
Released in v0.21.2. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Adds the GitHub-facing half of an isolated, asynchronous Qwen Code release benchmark pipeline:
GitHub Release -> DSW self-hosted runner -> one-shot submission -> PostgreSQL -> Coordinator + Prewarmer + 10 Executors -> Harbor/Qwen Code/SWE grader -> OSS -> Publisher -> originating ReleaseThe Actions job resolves an immutable Qwen Code tag/commit, freezes the SWE-bench Verified manifest, prepares the version-pinned local cache, submits an idempotent pool run, records the
run_id, and exits. It does not wait for the long-running benchmark.Release trigger policy
Automatic execution is limited to stable
vX.Y.0releases:v0.21.0,v1.0.0: run the full 500-case benchmark.v0.21.1and other patch releases: skip.workflow_dispatch: remains available for explicit branch diagnostics and prerelease validation.A lightweight GitHub-hosted gate evaluates the tag before the DSW self-hosted job is scheduled, so skipped releases do not occupy the DSW runner.
Execution model
Retryable infrastructure failures receive four total attempts with PostgreSQL-scheduled
60 -> 120 -> 240second backoff. Valid unresolved results and normal model/test failures are not retried. An isolated terminal failure does not cancel the remaining cases.Publication gate and public assets
Publication waits until all 500 frozen cases reach a unique terminal state.
A score is published only when:
CANCELED; andEXECUTION_ERROR + INFRA_FAILED < 10.The score uses valid grader results as its denominator:
RESOLVED / (RESOLVED + UNRESOLVED)Terminal errors participate in the publication gate but are not included in the score denominator. Ten or more terminal errors, a canceled task, or incomplete case state makes the run
QUARANTINEDand suppresses the score.The Publisher uploads:
swe-bench-verified-<release>.json: aggregate and instance-level public result;swe-bench-verified-<release>-trajectories.tar.gz: final-attempt per-case trajectories plus a manifest containing state, attempt number, size, and SHA-256.A missing final-attempt trajectory for a
RESOLVEDorUNRESOLVEDcase blocks publication and leaves Publisher inRETRY, preventing an incomplete public bundle.Repository boundary
This PR contains only the GitHub integration contract:
.github/workflows/dsw-swe-verified-release.yml.github/scripts/dsw-swe-verified/dispatch-release-benchmark.sh.github/scripts/dsw-swe-verified/make-manifest.pyand its colocated regression test.github/workflows/ci.ymlhelper-test registrationdocs/design/dsw-swe-verified-release-pipeline.mdThe DSW implementation is maintained in the separate internal
qwen-code-benchmark-dswrepository. Deployed internalmainis46e36ee(followingcdc2f75). Credentials, PostgreSQL data, Docker images, caches, and benchmark artifacts are not committed to this PR.The implementation remains isolated from #7584 through its workflow, runner label
qwen-benchmark-dsw, suite, PostgreSQL database, runtime root, OSS prefix, and Release markers.Historical full-suite validation
The first isolated 500-case prerelease run completed on 2026-07-25:
dsw-swe-full-async-poc-20260724-2c5ad4a5d0-r330079405895/pool-31a24bc8acca49d2swe-bench/swe-bench-verified@2, 500 casesv0.20.0-nightly.20260722.b98306b7e/qwen3.7-maxQUARANTINED,score_percent=nullunder the then-strict gateEvidence:
Historical r3 state and assets are intentionally unchanged.
Validation
48 passed.SUCCEEDED; 10 ->QUARANTINED.Reviewer test plan
vX.Y.0release events schedule the DSW benchmark job.instance_idfor 5/500 runs.RETRY.Scope
Out of scope: PAI-Sandbox, Kubernetes, multi-node scaling, and RDS migration. External operational dependencies still include model capacity, npm/PyPI/network availability on cold cache, Docker registry availability, and local disk pressure.
Related to #7584, with intentionally separate workflow, state, data, and experiments.