feat(review): script-lint — run linters over a diff's executable scripts, as a required step - #7749
feat(review): script-lint — run linters over a diff's executable scripts, as a required step#7749wenshao wants to merge 3 commits into
Conversation
…xecutable scripts
A diff's shell — a `.sh` file, a Makefile recipe, a Dockerfile `RUN`, a GitHub
Actions `run:` block — is code, and its bugs (an unquoted `$x` that word-splits,
a `${PIPESTATUS[1]}` read after the array was reset) are the class a reviewer
misses by reading a long YAML and catches by running the checker. Measured: a
model told in prose to "run the workflow scripts" reads instead (0/4 executed).
So the execution is a command, not a request. `qwen review script-lint` reads the
plan, dispatches shellcheck / actionlint / hadolint by file type over the changed
executable files, filters every finding to whether its line is one the diff
changed (`inDiff`), and reports JSON. A linter that is not installed is disclosed
as skipped, never a clean bill. It is not GitHub-specific — shellcheck applies to
shell wherever it appears; actionlint/hadolint are front-ends for two embeds.
This is the command only; the agent, roster requirement and coverage gate that
make it a non-skippable step follow.
The script-lint command exists; nothing ran it. This wires it into the review the way build-test is wired, so a diff that changes an executable script cannot be certified without its linters having run. - A `script-lint` agent role (agent-briefs): reads no diff, runs the command, reports from its JSON. `inDiff` findings are the PR's; `skipped` (a linter not installed) is disclosed as unreviewed, never clean. Rules are not injected into it, same as Build & Test — it reports a tool's verdict, not a read. - agent-prompt welds the exact `qwen review script-lint --plan/--worktree/--out` into that agent's brief with absolute paths, guarding an absent PR number out of the --out name — the same treatment, and the same traps avoided, as the build-test block it sits beside. - The roster requires the agent whenever the diff carries a file a linter owns by path (a `.sh`/`.bash`, a `.github/workflows/*`, a Dockerfile) and the review has a worktree to lint in. Detected by the command's own `pathTool`, so the roster and the command cannot disagree about what counts. A pure-TS diff does not require it; a diff-only review (no tree) cannot run it, so does not. - Coverage needs no new code: it derives missing roles from the roster generically (`BRIEFS[role].label`), so a required script-lint agent that did not run exit-3s check-coverage like any other. The role carries its three labels for that. End-to-end on a crafted diff (a workflow plus a `deploy.sh` with `rm -rf $TARGET`): the roster requires script-lint, and the command blocks on the SC2086 on the changed line while disclosing the workflow as skipped where actionlint is absent. SKILL.md documents the step; tests cover the roster requirement, the brief weld, and the subcommand registration.
Independent local verification report (Linux, real bundled CLI + real shellcheck)Verdict: verified working end-to-end. The blocking/disclosure/skipped triage behaves exactly as designed, including the shebang-detected extensionless script. Two non-blocking observations: unparseable linter output silently reads as clean, and MethodBuilt this branch ( 1. Scenario matrix (real binary, real linter)
Unit tests: the three PR-touched files ( 2. Design points that check out in code
3. Observations (non-blocking)
ConclusionThesis: "running the checker" became a required, coverage-gated command precisely because prose instructions to run things don't get executed — and the command's three-way triage (changed-line finding blocks / pre-existing discloses / missing tool is unreviewed) is the right severity calibration, now verified against a real linter on real files. Evidence: the four-scenario matrix with a real shellcheck (§1), the structural single-detector and fail-disclosed properties confirmed in code (§2), and a subcommand-absence baseline. LGTM with §3 as cheap follow-ups. 中文摘要Linux 真机验证(构建本分支 + 真实 shellcheck 0.10.0,actionlint/hadolint 刻意缺席):四场景矩阵全过——改动行 — independent review loop, real-linter verification on Linux |
Review —
|
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Reviewed. Suggestions are inline.
中文说明
已审查。 建议见行内评论。
— qwen3.7-max via Qwen Code /review
wenshao
left a comment
There was a problem hiding this comment.
— Codex GPT-5 via Qwen Code /review
…osed, symlinks, quoting Addresses the review findings on this PR: - inDiff was keyed off the plan's hunk ranges, which include git's three context lines, so a pre-existing diagnostic near a real change was marked this PR's and could block it. Classify off the diff's added-line ranges (context excluded), parsed from the diff; fall back to the plan hunks only when the diff is absent. - The command wrote the report to --out and printed only "Wrote ...", while the agent's brief (and the roster's generated command, which passes --out) says to read the JSON it prints. Match build-test: write the file AND always print JSON. - runTool failed open — every non-ENOENT failure (EACCES, a signal, maxBuffer, an unexpected status) fell through as empty stdout and became ok:true. Fail closed: such a run is `errored`, which forces ok:false; ENOENT alone stays "not installed". - The shebang read slurped the whole file and followed symlinks — a changed `hang.sh` -> /dev/zero would hang the reviewer. Read only regular files (lstat, no follow) and only the first block. - The welded command interpolated plan/worktree/out as bare words; a worktree path with a space would split. Quote them with shellQuotePath. - Harden the checker environment: shellcheck --norc + drop SHELLCHECK_OPTS, so a PR-controlled .shellcheckrc or inherited opts cannot suppress SC2086. - The roster required the agent for a pure-deletion .sh (a mandatory no-op); gate on added lines. Drop the Makefile-recipe claim (no detector backs it). Fix the `ok` JSDoc (info blocks too, not just error/warning). - Tests: inject the tool runner (no binary needed) to cover actionlint/hadolint normalisation, the three fail-closed paths, and the context-line classification.
Why this feature exists — a measurementI dogfooded I took PR #7724 at the exact commit where an earlier review round had verified three defects by running probes (a Result: 0 of the 3 were caught. Not for lack of looking — the agents examined the exact code and reasoned past it:
That is the whole argument for It also sharpens the review feedback on this PR: the deterministic-gate follow-up (have 🤖 Claude Code · Claude Opus 4.8 |
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Reviewed. Suggestions are inline. Unresolved, please confirm: [Critical] C1 (actionlint inDiff misattribution for workflow run: blocks) — author acknowledged as real gap, deferred to follow-up; cannot determine if the deferred timeline is acceptable Not reviewed: reverse audit — an auditor ran and opened its brief, but no agent was launched with the prompt the CLI built — the launch was written by hand, and what the agent was actually asked is not what this skill certifies.
中文说明
已审查。 建议见行内评论。 未决,请确认:[Critical] C1 (actionlint inDiff misattribution for workflow run: blocks) — author acknowledged as real gap, deferred to follow-up; cannot determine if the deferred timeline is acceptable 未审查:反向审计——有审计 agent 运行并打开了自己的 brief,但没有 agent 是用 CLI 构建的 prompt 启动的——启动 prompt 是手写的,agent 实际被要求做的并不是本 skill 所认证的内容。
— qwen3.7-max via Qwen Code /review
|
Superseded by #7751 and closing in its favour. Per this review's architectural findings (#10/#11/#12), the executable-script lint is better as a deterministic gate than an agent: #7751 has the orchestrator run Thanks for the thorough review across both rounds — it's what drove the redesign. 🤖 Claude Code · Claude Opus 4.8 |
Review SummaryThis PR introduces a well-designed feature: running deterministic linters ( What's Done Well
Outstanding Items (Acknowledged for Follow-up)The author has transparently noted these gaps for the fast-follow:
Minor Observations
ConclusionApprove for the design direction and implementation quality. The PR correctly addresses the core premise: prose instructions to "run the workflow scripts" don't get executed (0/4 measured), so running must be a command. The follow-up in #7751 to make this a deterministic gate (orchestrator → artifact → compose-review) is the right architectural evolution. Good work on the thorough iteration through review feedback—each critical finding was addressed in cd8702c with appropriate tests. 中文摘要本 PR 设计方向正确,实现质量高。核心价值:把 "运行检查器" 从请求变成命令,解决了模型在 prose 指令下不执行的问题(实测 0/4)。已修复的关键问题包括:hunk 上下文行误判、非 ENOENT 失败当干净、符号链接/管道挂起风险、PR 控制 shellcheck 配置等。遗留项(actionlint 行号归属、无扩展名 shebang 不触发要求、hadolint 配置隔离)已在 PR 中透明说明并跟踪为后续优化。架构演进至 #7751 的确定性门控(编排器运行 → compose-review 读结果)是正确的方向。 |
…ads the report, no agent (QwenLM#7751) * feat(review): add script-lint — deterministic linters over a diff's executable scripts A diff's shell — a `.sh` file, a Makefile recipe, a Dockerfile `RUN`, a GitHub Actions `run:` block — is code, and its bugs (an unquoted `$x` that word-splits, a `${PIPESTATUS[1]}` read after the array was reset) are the class a reviewer misses by reading a long YAML and catches by running the checker. Measured: a model told in prose to "run the workflow scripts" reads instead (0/4 executed). So the execution is a command, not a request. `qwen review script-lint` reads the plan, dispatches shellcheck / actionlint / hadolint by file type over the changed executable files, filters every finding to whether its line is one the diff changed (`inDiff`), and reports JSON. A linter that is not installed is disclosed as skipped, never a clean bill. It is not GitHub-specific — shellcheck applies to shell wherever it appears; actionlint/hadolint are front-ends for two embeds. This is the command only; the agent, roster requirement and coverage gate that make it a non-skippable step follow. * feat(review): make script-lint a required, coverage-gated review step The script-lint command exists; nothing ran it. This wires it into the review the way build-test is wired, so a diff that changes an executable script cannot be certified without its linters having run. - A `script-lint` agent role (agent-briefs): reads no diff, runs the command, reports from its JSON. `inDiff` findings are the PR's; `skipped` (a linter not installed) is disclosed as unreviewed, never clean. Rules are not injected into it, same as Build & Test — it reports a tool's verdict, not a read. - agent-prompt welds the exact `qwen review script-lint --plan/--worktree/--out` into that agent's brief with absolute paths, guarding an absent PR number out of the --out name — the same treatment, and the same traps avoided, as the build-test block it sits beside. - The roster requires the agent whenever the diff carries a file a linter owns by path (a `.sh`/`.bash`, a `.github/workflows/*`, a Dockerfile) and the review has a worktree to lint in. Detected by the command's own `pathTool`, so the roster and the command cannot disagree about what counts. A pure-TS diff does not require it; a diff-only review (no tree) cannot run it, so does not. - Coverage needs no new code: it derives missing roles from the roster generically (`BRIEFS[role].label`), so a required script-lint agent that did not run exit-3s check-coverage like any other. The role carries its three labels for that. End-to-end on a crafted diff (a workflow plus a `deploy.sh` with `rm -rf $TARGET`): the roster requires script-lint, and the command blocks on the SC2086 on the changed line while disclosing the workflow as skipped where actionlint is absent. SKILL.md documents the step; tests cover the roster requirement, the brief weld, and the subcommand registration. * fix(review): harden script-lint after review — context lines, fail-closed, symlinks, quoting Addresses the review findings on this PR: - inDiff was keyed off the plan's hunk ranges, which include git's three context lines, so a pre-existing diagnostic near a real change was marked this PR's and could block it. Classify off the diff's added-line ranges (context excluded), parsed from the diff; fall back to the plan hunks only when the diff is absent. - The command wrote the report to --out and printed only "Wrote ...", while the agent's brief (and the roster's generated command, which passes --out) says to read the JSON it prints. Match build-test: write the file AND always print JSON. - runTool failed open — every non-ENOENT failure (EACCES, a signal, maxBuffer, an unexpected status) fell through as empty stdout and became ok:true. Fail closed: such a run is `errored`, which forces ok:false; ENOENT alone stays "not installed". - The shebang read slurped the whole file and followed symlinks — a changed `hang.sh` -> /dev/zero would hang the reviewer. Read only regular files (lstat, no follow) and only the first block. - The welded command interpolated plan/worktree/out as bare words; a worktree path with a space would split. Quote them with shellQuotePath. - Harden the checker environment: shellcheck --norc + drop SHELLCHECK_OPTS, so a PR-controlled .shellcheckrc or inherited opts cannot suppress SC2086. - The roster required the agent for a pure-deletion .sh (a mandatory no-op); gate on added lines. Drop the Makefile-recipe claim (no detector backs it). Fix the `ok` JSDoc (info blocks too, not just error/warning). - Tests: inject the tool runner (no binary needed) to cover actionlint/hadolint normalisation, the three fail-closed paths, and the context-line classification. * feat(review): make script-lint a deterministic gate — orchestrator runs it, compose-review is the authority The review of QwenLM#7749 landed three architectural findings (#10/#11/#12): the executable-script lint was run by an AGENT, so its execution rested on the model's honor system, the model decided each finding's severity, and an uninstalled checker was only disclosed in prose. All three are the exact "a rule a model is asked to remember will eventually not be remembered" trap the feature exists to close — and a measurement on PR QwenLM#7724 confirmed it: the strongest model's Step-3 agents, given the diff and the worktree, missed all three execution-confirmed bugs, one attacker persona walking into a double-execute and declaring it correct. So take the model out of the gate entirely: - The orchestrator runs `qwen review script-lint` as a deterministic step (like presubmit) and writes the report next to the plan. No agent. - `compose-review` derives the report path from the plan and reads it as the SOLE authority: a finding on a changed line above `style` is a pre-confirmed `[lint]` Critical (needs no verifier — the tool already ran); an uninstalled or crashed checker is unreviewed scope that caps a would-be Approve; and — proof it ran — a diff that carries an executable script but produced no readable report is itself unreviewed (fail closed). A diff-only review, which has no worktree to run it, is exempt. Nothing here comes from the input JSON a model wrote. Removed the `script-lint` agent role (agent-briefs, agent-prompt weld, roster requirement) and its SKILL.md agent entry; the roster's `hasExecutableScript` is now exported as the shared predicate the gate reads. The `script-lint` command itself is unchanged — it was always the deterministic engine; only its trigger and consumer moved. * fix(review): address the second review pass on script-lint - parseFindings failed OPEN on non-empty unparseable output (a version skew or a deprecation line before the JSON) → `[]` → recorded as a clean `checked` file. Return null on that path and push it to `errored`, fail-closed like a bad exit. - hadolint had no config isolation while shellcheck gets `--norc`; a PR-controlled `.hadolint.yaml` could `ignored:` its findings away. Add `--no-config`. - pathTool recognised only `.sh`/`.bash`, but the shebang regex matches four shells — a `.ksh`/`.dash` file never required the lint. Add those extensions so the gate's owed-predicate and the command cannot disagree. - Pin `r.ok` in the not-installed test, and add a mock test for the unparseable path. * fix(review): address the deterministic-gate re-review — real correctness bugs on the write path The second review pass on the gate found real bugs, including one this PR introduced. Fixed: - hadolint has no `--no-config` flag — a prior round added it, so hadolint exited 2 (usage error) on EVERY Dockerfile and reported it `errored`. Reverted to the plain working invocation; hadolint config isolation needs a verified mechanism and is tracked separately. - actionlint's JSON anchors each diagnostic at the `run:` key line (not the changed shell line) and flattens ShellCheck severity, so a style nit read as a blocking `error` and a real finding read as pre-existing. Until that source mapping is parsed and verified, a workflow is deferred to `skipped` (unreviewed) and actionlint is never run — shellcheck still covers standalone shell. - A recognised path that `firstLineOf` refused (a `hook.sh` symlink, a fifo) was silently dropped from the report, so an empty report read as clean. It is now recorded as `skipped`. `firstLineOf` distinguishes a true deletion (skip) from an irregular file (record). - The gate's owed-predicate excluded any file with zero added lines, so a deletion-only edit that breaks a surviving `.sh` was treated as not-owed. Keyed on the post-image (`fileLines`) now: a surviving script is owed, only a true deletion is exempt. - compose-review trusted any body-Critical string containing `[lint]` as deterministic — a model-written or injected claim could launder itself past verification. Provenance decides now: only `scriptLintGate`'s own findings are deterministic (tracked by count); `[lint]` is no longer a trusted tag. - A stale report from an earlier review of an older commit could certify the current one if the lint step were skipped. The report records its `headSha` (`git rev-parse HEAD`); compose-review rejects it as stale when it disagrees with the plan's `fetchedSha`. * fix(review): third gate re-review — read the report before the owed-predicate, fail closed on all paths - The gate returned early on `hasExecutableScript` (path-only), so a shebang script the command DID lint (`.husky/pre-commit`, detected by `#!`) had its findings dropped — the gate never read the report. Read the report first; the path-predicate now only gates the no-report fail-closed case. Findings from any script the report names are processed. - The staleness guard was a no-op when `report.headSha` was absent (git unreadable): `planSha && report.headSha && …` short-circuited, so an unverifiable report certified new code. It now fails closed on a missing headSha when the plan names a commit. - The plan-parse catch failed OPEN (returned "nothing owed") while every other path fails closed. It now discloses "could not read the plan" as unreviewed. - hadolint had no config isolation. Point `HADOLINT_CONFIG` at an empty file so a PR-added `.hadolint.yaml` cannot suppress findings — an env var, benign if the tool ignores it (unlike the invalid `--no-config` flag from the last round). - `buildNote` hardcoded "not installed" for every skipped file; `skipped` now mixes reasons (missing tool, irregular file, deferred checker), so it summarises by tool and leaves the specific reason on each entry. * fix(review): R3 — deferred (non-capping) actionlint, local-flow staleness, disclosure - actionlint deferral capped EVERY workflow-touching PR (≈15% of merges): a deferred workflow went to `skipped` → unreviewed → no Approve, on a checker present but deliberately not run, which "install the tool" cannot fix. Split a third `deferred` state — disclosed in the note, but the gate never reads it, so it does not cap. `skipped`/`errored` still cap. - The staleness guard was PR-only: `fetchedSha` is written by fetch-pr, not capture-local, so a local review short-circuited the check and a stale local report could certify new code — the exact fail-open, in the local flow. capture-local now records the local HEAD as `fetchedSha`. - The local path was armed (reviewMode `local`) but SKILL's `--worktree` was unfillable there. SKILL now covers a local review (worktree = the project root) and the derived report name. - Nits: the skipped/irregular reasons no longer lead with the path (the gate prefixes it — was printing it twice); merged the two `./lib/roster.js` imports. - Tests: deferred does not cap; the staleness guard protects a local review too. * fix(review): R4 — disclose deferred lint without capping; bind report freshness to diff content B1 — the deferred checker was silent. R3 split a third `deferred` state so a workflow's embedded shell (actionlint, whose source-mapping this env can't verify) no longer caps the verdict — but the disclosure half was never wired: `scriptLintGate` returned only `{criticals, unreviewed}`, so a workflow-only PR went quiet on a file no checker examined. That is #10's original complaint ("only disclosed in prose") returning as disclosed nowhere. Add a third channel: the gate now returns `disclosed`, populated from `report.deferred`, and `composeReview` renders it in the body on every verdict — including Approve — without pushing it into `cappedBy`. SKILL.md's contract paragraph now documents the deferred state alongside unreviewed scope. B2 — the staleness guard keyed on HEAD. A local review is defined by uncommitted work, so `git rev-parse HEAD` is not a content identity for what it reviews: two reviews at the same HEAD with different working-tree content shared a `fetchedSha` and a stale clean report could certify broken code. Bind freshness to the diff's content instead: `runScriptLint` stamps the report with a sha256 of the captured diff (`diffHashOf`), and the gate re-hashes the plan's current diff and rejects a mismatch. Correct for a PR (a later commit → a different diff) and for local uncommitted work alike. `capture-local`'s `fetchedSha` plumbing is removed. Tests: the DEFERRED-only case now asserts the disclosure is present and non-capping (it previously pinned the silence). New composeReview gate tests kill the surviving mutants — the gate critical stands with no verifier (provenance, not the `[lint]` marker), and an errored checker caps a would-be Approve to Comment. New runScriptLint tests pin the diff-hash stamp. 972 review tests, ESLint --max-warnings 0, tsc, Prettier clean. * fix(review): R5 — freshness fails closed when neither side has a hash; pin config isolation The staleness guard was `report.diffHash !== planDiffHash`. When the plan names no readable diff AND the report carries no hash, both are `undefined` and `undefined !== undefined` is false — so the guard did not fire and an arbitrary hashless report was accepted as this review's, its findings promoted to `[lint]` Criticals. Every other branch of this gate fails closed; this one inverted, on the exact unverifiable case its own comment claimed to reject. Fixed with `!planDiffHash || …`, and the stale `headShaOf` JSDoc left stacked above `diffHashOf` is removed. The gate's happy-path tests were green *because* of that hole: `writePlan`/ `writeReport` (and the composeReview `gateReadyPlan`/`writeGateReport`) wrote no diff and no hash, so every test that didn't override them ran through the fail-open branch — proving the gate works on an *unverifiable* report, not a fresh one. The fixtures are now FRESH by default: a captured diff exists (`DIFF` is a real file; coverage only string-matches it, so this is transparent to it) and both plan and report bind to its hash. A freshness test overrides one side to model staleness, and a new test pins the both-undefined case directly. Also closes the last mutation gap R5 flagged three rounds running: the config isolation (`--norc`, `SHELLCHECK_OPTS` scrub, `HADOLINT_CONFIG` → empty file) is a security property — a PR-added linter config must not suppress the finding the gate blocks on — but it lived inside `runTool`, behind the spawn an injected runner bypasses. Extracted `buildToolInvocation` (pure argv + env) so all three defences are asserted without a binary; each was hand-mutated to confirm its test fails when the defence is deleted. 976 review tests, ESLint --max-warnings 0, tsc, Prettier clean. * test(review): pin the plan-parse disclosure — the last surviving gate mutant `scriptLintGate` on an unreadable plan fails closed and pushes its own reason into `unreviewed`. The coverage machinery already caps an unreadable plan, so the verdict and its cap are identical with or without this line — what it loses when deleted is the specific "could not read the plan" sentence in the body. That made it a disclosure guarantee with no test, and the one gate mutant left standing after five rounds. Pin it directly on the gate. 977 review tests, ESLint/tsc/Prettier clean. * fix(review): harden script-lint tmp-file + spawn, leak-proof the tests (R8 suggestions) Four suggestions from the re-review, all on script-lint: - Symlink race on the hadolint empty-config: it was written to a fixed `tmpdir()/qwen-review-hadolint-empty.yaml`, so on a shared runner a pre-planted symlink there would have `writeFileSync` follow it and truncate the target. Write it inside a fresh `mkdtempSync` directory instead — a 0700 dir with a random suffix that cannot pre-exist, so the write is safe and the path unpredictable. - `runTool`'s `spawnSync` had no `timeout`, unlike the sibling runners in build-test.ts and test-efficacy.ts: a crafted script that hangs a linter would block the review until the outer CI job timeout reclaimed the runner. Added a 120s bound; a timeout kills with SIGTERM, which the existing `r.signal` branch already turns into a fail-closed error. - Two test-cleanup leaks: script-lint.test.ts's symlink test made a second temp dir cleaned only by an inline `rmSync` a failing assertion would skip, and script-lint.mock.test.ts used inline `fresh()`/`clean()` with no hook. Both now tear down in `afterEach`, which runs even when a test throws. 977 review tests, ESLint --max-warnings 0, tsc, Prettier clean. * fix(review): hadolint isolation fails closed, not to a plantable path (R8) R8 caught that the previous commit's fallback reopened the vector it closed. When `mkdtempSync` failed, `emptyHadolintConfig` returned a FIXED `tmpdir()/qwen-review-hadolint-none/x`. That path is a config hadolint *reads*, so an attacker who plants an `ignored:` file there gets those ignores honoured — the opposite threat direction from the write-truncation the mkdtemp move fixed. There is now no predictable fallback: on failure `emptyHadolintConfig` returns `undefined`, the env var is set only for a hadolint run and only when a private config exists, and `runTool` fails the hadolint run CLOSED (errored) rather than lint against a config it cannot vouch for. Also from R8: - The timeout comment claimed the `r.signal` branch handles a timeout. On Node a timeout sets BOTH `r.error` (ETIMEDOUT) and `r.signal` (SIGTERM), and `r.error` is checked first — so it is reported through the error branch. Comment corrected; still fail-closed either way. - Both hardening changes were untested. `buildToolInvocation` now returns `timeoutMs` so the bound is one asserted value, and two tests pin it: HADOLINT_CONFIG is a fresh 0700 mkdtemp path (not the old fixed name), and the timeout is 120s. Each fails under the corresponding mutation. - The mkdtemp dir is swept at process exit, so the fixed-file-to-per-run-dir change does not leak into the OS tmpdir. 979 review tests, ESLint --max-warnings 0, tsc, Prettier clean. * test(review): pin the hadolint fail-closed guard in its own file (R9) The `if (tool === 'hadolint' && !emptyHadolintConfig())` guard is what the previous commit exists to add, yet deleting it left the suite green: `emptyHadolintConfig` caches at module scope, so by the time any test in script-lint.mock.test.ts runs the cache is warm and the failure path is unreachable from that file. A dedicated file gets a fresh module registry — it points TMPDIR at a path that does not exist before importing, so the first `emptyHadolintConfig()` call fails to mkdtemp and the guard fires. Two tests, split so a mutation attributes cleanly: (1) buildToolInvocation sets no HADOLINT_CONFIG (the env logic, holds either way); (2) runScriptLint errors a Dockerfile closed rather than lint it unisolated (the guard). Deleting the guard reddens (2) and leaves (1) green. Also from R9: register the process-exit cleanup right after mkdtempSync, before the write, so a writeFileSync that throws still leaves the temp dir swept, not leaked. 981 review tests, ESLint --max-warnings 0, tsc, Prettier clean. * fix(review): isolate hadolint via --config, not the ignored HADOLINT_CONFIG env (R10) Verified against real hadolint 2.14.0: the binary does not read `HADOLINT_CONFIG` (its `strings` list it among no `HADOLINT_*` config vars; `-V` reports "No configuration was specified"). It reads `--config`, then a `.hadolint.yaml` in the process CWD, then XDG. Because a local review runs with `--worktree .`, the linter ran inside the reviewed tree and honoured the diff's OWN `.hadolint.yaml` — so the env-based "isolation" was a silent no-op and a PR could suppress its own Dockerfile findings (a DL3018 `ignored:` made the finding vanish with nothing disclosed). Isolate on the channel hadolint actually reads: pass `--config <private neutral file>`, which overrides both the cwd and XDG configs. The config content is now `ignored: []` rather than empty — `--config` rejects an empty file ("empty YAML stream"). The mkdtemp 0700 dir and the fail-closed guard are kept (the config is a path hadolint reads, so both still matter); the no-op `HADOLINT_CONFIG` / `HADOLINT_NO_COLOR` env vars are dropped. Tests updated to assert the real channel: hadolint's argv carries `--config` at a fresh 0700 mkdtemp path holding `ignored: []`, and the isolation-unavailable path adds no `--config` (and still fails closed). shellcheck's `--norc` / `SHELLCHECK_OPTS` isolation is unchanged — it was verified sound against the real binary. 981 review tests, ESLint/tsc/Prettier clean. * fix(review): address maintainer review — worktree containment, hunk-fallback false positive, zsh/shebang docs From @yiliang114's review of the script-lint gate: - P1 shebang/zsh: `toolFor` intentionally omits zsh (and fish) — shellcheck refuses both (`SC1071: ShellCheck only supports sh/bash/dash/ksh`), so routing a `#!/usr/bin/env zsh` hook to it would make every zsh file a bogus SC1071 `[lint]` Critical on its shebang. Documented the deliberate exclusion rather than adding zsh. - P1 hunk fallback: `inDiff` fell back to the plan's context-inclusive hunks whenever `addedRanges.get(path)` missed — including when the diff parsed fine but a path was unmatched, which could promote a pre-existing finding on a context line to a blocker. Now a parsed diff that does not mention a path yields `[]` (nothing added → nothing blocks); the context-inclusive `hunksOf` fallback is used only when NO diff parsed at all (a report the freshness guard already rejects as stale), and that degraded path now logs a warning. - P2 worktree containment: `resolve(join(worktree, path))` now refuses a path that escapes the worktree (`../../etc/passwd`), disclosing it as skipped rather than stat-ing/linting a file outside the reviewed tree. - P2 roster/gate shebang gap: made explicit in the gate's no-report branch that a shebang-only script (which `hasExecutableScript` cannot see) is covered by the always-run contract, not the `owed` predicate. Two items left as tracked follow-ups per the reviewer's own framing (both flagged low-priority / design): an aggregate cross-file time budget (the per-file 120s bound is the main defense), and per-severity finding classification (the v1 all-or-nothing stance is documented and defensible). 983 review tests, ESLint/tsc/Prettier clean. * fix(review): harden the script-lint gate against the adversarial review round From the Codex GPT-5 /review pass. Five fixed: - Provenance by IDENTITY, not count (compose-review): the gate's `[lint]` criticals are now tracked as a separate list rather than pushed into `bodyCriticals` and removed by a count subtraction. The old `(filtered) − gateCount` misfired when a model claim carried a `[build]/[test]/[probe]` tag (filtered out before the subtract) or a gate finding's own text contained one — erasing an unrelated claim's verification requirement. Model criticals alone decide the verify count. - Distinguish unknown size from deletion (roster): `fileLines: 0` is a real post-image count only in pr-worktree; in local/diff-only the report builder writes 0 for EVERY file, so keying deletion on it read a surviving `deploy.sh` as deleted and let a missing report pass uncapped. `hasExecutableScript` now trusts `fileLines` only in pr-worktree and owes any path-detected script otherwise. - Hash the bytes used for range mapping (script-lint): the diff was read twice — ranges before the linters, hash after — a TOCTOU window where a concurrent recapture could pair snapshot A's ranges with snapshot B's hash. Read the diff once into one buffer and derive both from it; an unparseable diff now yields no `diffHash` (gate fails closed) instead of context-inclusive hunks. - Escape PR-controlled paths in the body (compose-review): a diff filename can carry a newline, `@mention`, HTML or Markdown; the gate's criticals/unreviewed/disclosed strings rendered it verbatim into the posted body. Paths and linter messages now render in an inline code span with backticks and newlines stripped (`mdField`). - Classify lstat failures (script-lint): only ENOENT/ENOTDIR is "missing" (deleted); EACCES/EIO/ELOOP is now `irregular` (skipped, fail closed) rather than silently read as a deletion into an `ok: true` "nothing changed". Three left as reasoned follow-ups: verifier-provenance for `[build]/[test]/[probe]` tags (a pre-existing verification-model concern, not this gate); actionlint's native workflow diagnostics (deferring the whole tool loses them, but separating native from embedded-shell output needs an actionlint-output parser this env can't verify); and cryptographic authentication of the report file (the orchestrator runs the deterministic command and overwrites any pre-planted file; full anti-forgery is a broader harness-trust change). 985 review tests, ESLint/tsc/Prettier clean. * fix(review): clean error from the script-lint handler on a bad plan Wrap `runScriptLint` in the command handler in try/catch, matching sibling build-test: a missing or invalid plan makes it throw, and yargs' default handler would print a stack trace the orchestrator has to parse. Emit the one-line message and a non-zero exit instead; the gate still fails closed on the absent report. * fix(review): scrub HADOLINT_* env, canonicalize worktree paths, portable isolation tests Follow-up adversarial round on the previous fixes: - Scrub HADOLINT_* from the child env (script-lint): hadolint 2.14 MERGES config from HADOLINT_IGNORE / HADOLINT_OVERRIDE_* / HADOLINT_CONFIG with the explicit --config, so an inherited one could suppress the findings the neutral config forces. Drop every HADOLINT_* var, mirroring the SHELLCHECK_OPTS scrub — configuration comes from our --config alone. Hostile-env regression test added (this also makes the HADOLINT_CONFIG assertion hermetic under an inherited value). - Canonicalize the worktree-containment check (script-lint): the lexical startsWith guard is defeated by a symlinked ANCESTOR — a directory replaced with a symlink leaves a child path lexically inside but resolving outside, and lstat/the linter follow it. Add a realpathSync check against the canonical worktree; a path that cannot be canonicalised is handled as missing downstream. Symlink-escape test added. - Portable isolation tests: override TEMP/TMP as well as TMPDIR (os.tmpdir() ignores TMPDIR on Windows, leaving the fail-closed path unexercised there), and guard the 0o700 mode-bit assertion with process.platform !== 'win32'. Left tracked: fail-closed on a PARTIALLY parsed diff — narrow (capture writes the diff in one writeFileSync, not incrementally) and already mitigated (a truncation that later completes changes the hash, so the freshness guard rejects the report). 987 review tests, ESLint/tsc/Prettier clean. --------- Co-authored-by: verify <verify@local>
What & why
A diff's shell is code, and its bugs are a class code review reliably misses. An unquoted
$xthat word-splits on a path with a space, a${PIPESTATUS[1]}read after the array was already reset, a[ ]where[[ ]]was meant — these hide from a human (or a model) reading a long workflow YAML, and are caught in a second by running the checker. Measured against this skill's own agents: a reviewer told in prose to "run the changed step scripts" reads and reasons instead — it does not run them.So the running is a command, not a request.
/reviewnow has ascript-lintstep that dispatches the deterministic linters over the executable files a diff changes, and — like Build & Test — it is a required, coverage-gated step, not a lens an agent has to remember to apply.It is not GitHub-specific.
shellcheckis the workhorse and applies to shell wherever it appears;actionlintandhadolintare format front-ends for the two embeds worth special-casing (a workflow'srun:blocks, a Dockerfile). A.shhelper, a git hook — all in scope.How it works
qwen review script-lint(new deterministic command) reads the plan, dispatchesshellcheck/actionlint/hadolintby file type over each changed executable file, and marks every finding with whether its line is one the diff changed (inDiff). A finding on a changed line is the PR's to answer for; one on an unchanged line is pre-existing and disclosed as such — the same "changed file vs. not" calibration Build & Test already uses. A linter that is not installed on the runner is reported as skipped (unreviewed), never treated as a clean bill of health.script-lintagent role reads no diff — it runs the command and reports from its JSON.agent-promptwelds the exact invocation (absolute--plan/--worktree/--out) into the agent's brief, beside the build-test block and with the same PATH-skew and absent-PR-number guards..sh/.bash, a.github/workflows/*, a Dockerfile) and the review has a worktree to lint in. The requirement uses the command's ownpathTool, so the roster and the command cannot disagree about what counts. A pure-TypeScript diff does not require it; a diff-only (cross-repo lightweight) review has no tree to lint, so it does not either.script-lintagent that did not run exit-3scheck-coverageexactly like any other missing dimension. The role carries itslabel/publicLabel/publicLabelZhfor that machinery.Validation
toolFor/pathTooldispatch,runScriptLint(anSC2086on a changed line blocks; the same finding on an unchanged line is disclosed but does not; a clean script passes; an uninstalled linter is skipped, not clean), the roster requirement (required for a.sh/workflow/Dockerfile diff, not for pure TS, not for diff-only), and the brief weld.scripts/deploy.shcontainingrm -rf $TARGET): the roster requiresscript-lint, and the command blocks on theSC2086(word-splitting on a destructive command) on the changed line, while disclosing the workflow as skipped whereactionlintis absent./reviewcommand suite green (932 tests); ESLint clean at--max-warnings 0.中文说明
背景与动机
diff 里的 shell 也是代码,而它的 bug 恰恰是代码评审最容易漏掉的一类:一个没加引号、会在带空格的路径上发生词拆分的
$x,一个在数组已被重置之后才读取的${PIPESTATUS[1]},一个本该用[[ ]]却写成[ ]的判断——这些在人(或模型)阅读一大段 workflow YAML 时几乎看不出来,而运行一下检查器就能瞬间发现。针对本 skill 自己的 agent 做过测量:仅在提示词里告诉评审者"去运行改动的 step 脚本",它只会去读、去推理,并不会真的运行。所以"运行"应当是一条命令,而不是一句请求。
/review现在有了一个script-lint步骤:对 diff 改动到的可执行文件跑确定性的 linter;并且和 Build & Test 一样,它是一个必需的、由 coverage 把关的步骤,而不是一个需要 agent 记得去用的"视角"。它不局限于 GitHub。
shellcheck是主力,适用于任何地方出现的 shell;actionlint和hadolint只是针对两类值得特殊处理的嵌入格式(workflow 的run:块、Dockerfile)的前端。.sh辅助脚本、git hook——都在覆盖范围内。实现方式
qwen review script-lint(新的确定性命令)读取 plan,按文件类型对每个改动的可执行文件分发shellcheck/actionlint/hadolint,并为每条发现标注它所在的行是否被本次 diff 改动(inDiff)。落在改动行上的发现由本 PR 负责;落在未改动行上的是既有问题,会如实披露——这与 Build & Test 已有的"改动文件 vs. 未改动文件"的定级方式一致。运行机器上未安装的 linter 会被报告为 skipped(未评审),绝不当作"干净"。script-lintagent 角色不读 diff——它运行命令,并根据其 JSON 汇报。agent-prompt会把确切的调用(绝对路径的--plan/--worktree/--out)焊进该 agent 的 brief,紧挨 build-test 块,并带有同样的 PATH 偏移与"PR 号缺失"防护。.sh/.bash、.github/workflows/*、Dockerfile)且评审有可供检查的 worktree 时。该要求复用命令自己的pathTool,因此 roster 与命令对"什么算可执行脚本"不会产生分歧。纯 TypeScript 的 diff 不会触发;diff-only(跨仓轻量)评审没有可检查的代码树,因此也不会触发。script-lintagent 会像任何其他缺失维度一样让check-coverage以 exit 3 失败。该角色为此机制带上了label/publicLabel/publicLabelZh。验证
toolFor/pathTool分发、runScriptLint(改动行上的SC2086会拦截;同样的发现落在未改动行则披露但不拦截;干净脚本通过;未安装的 linter 记为 skipped 而非干净)、roster 要求(对.sh/workflow/Dockerfile 的 diff 要求、对纯 TS 不要求、对 diff-only 不要求)、以及 brief 焊接。rm -rf $TARGET的scripts/deploy.sh):roster 要求script-lint,命令对改动行上的SC2086(在破坏性命令上的词拆分)拦截,同时在actionlint缺失处将 workflow 披露为 skipped。/review命令全量测试通过(932 项);ESLint 在--max-warnings 0下无告警。