Repository navigation
feat(audit): add legacy code audit workflow #8403
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 17 commits
Commits
Show all changes
50 commits
Select commit
Hold shift + click to select a range
afa3b4d
docs: add legacy code audit (/audit) design doc
wenshao 522e23c
docs: revise legacy audit design with round-2 replication evidence
wenshao 509e79a
docs: note cross-file tracer cost and budget rule in legacy audit design
wenshao 7fd87c6
feat(audit): add legacy code audit workflow
wenshao 2992d53
Merge remote-tracking branch 'origin/main' into feat/legacy-code-audit
wenshao 2b9cf83
feat(audit): align the audit workflow with the merged design doc
wenshao e4386ea
Merge remote-tracking branch 'origin/main' into feat/legacy-code-audit
wenshao 0ad3dd9
fix(audit): lift tokenizeArgs and anchor the guard probes at the invo…
wenshao ff9c53b
fix(audit): make check-anchors --callers conditional in the skill
wenshao 94b0b95
Merge branch 'main' into feat/legacy-code-audit
qwen-code-dev-bot 6d633b9
fix(audit): name-record unreadable sidecar callers
qwen-code-dev-bot 03c1519
Merge remote-tracking branch 'origin/main' into feat/legacy-code-audit
qwen-code-dev-bot 5eb156e
fix(audit): close fail-open gates found in review across the pipeline
qwen-code-dev-bot 0642802
Merge branch 'main' into feat/legacy-code-audit
qwen-code-dev-bot d883ddc
fix(cli): harden audit helpers per review feedback (#8403)
qwen-code-dev-bot e1e6fcf
fix(cli): address second review round for audit helpers (#8403)
qwen-code-dev-bot 67b1320
fix(external-context): read the response body with a reader, not for-…
wenshao 7c33b0b
fix(audit): make fallback relocation paths explicit in the skill comm…
qwen-code-dev-bot 8209913
Merge branch 'main' into feat/legacy-code-audit
qwen-code-dev-bot 319c0cf
fix(cli): apply review-round fixes to audit anchors, walk, and exit f…
qwen-code-dev-bot 72a4d89
Merge branch 'main' into feat/legacy-code-audit
qwen-code-dev-bot 029231e
Merge branch 'main' into feat/legacy-code-audit
qwen-code-dev-bot 2dc868d
Merge branch 'main' into feat/legacy-code-audit
qwen-code-dev-bot 5df644a
Merge remote-tracking branch 'origin/main' into feat/legacy-code-audit
qwen-code-ci-bot 2581412
fix(audit): address round-3 review findings for the audit module (#8403)
qwen-code-ci-bot 54c5cbe
Merge branch 'main' into feat/legacy-code-audit
qwen-code-dev-bot 722b06d
Merge branch 'main' into feat/legacy-code-audit
qwen-code-dev-bot c714b90
Merge remote-tracking branch 'origin/main' into feat/legacy-code-audit
qwen-code-dev-bot 216937d
fix(cli): apply round-4 review fixes to audit guard, anchors, and sid…
qwen-code-dev-bot 538905d
fix(cli): apply round-6 review fixes to audit guard, anchors, and sid…
qwen-code-dev-bot d5c48e3
Merge branch 'main' into feat/legacy-code-audit
wenshao e528918
fix(cli): close round-6 audit guard, anchor, and hash holes (#8403)
qwen-code-dev-bot 789b6e4
fix(cli): close round-8 audit anchor and guard criticals (#8403)
qwen-code-dev-bot 25da0ff
Merge branch 'main' into feat/legacy-code-audit
qwen-code-dev-bot cd1276a
Merge branch 'main' into feat/legacy-code-audit
qwen-code-dev-bot 92642bd
Merge branch 'main' into feat/legacy-code-audit
qwen-code-dev-bot 2afc1fb
Merge branch 'main' into feat/legacy-code-audit
qwen-code-dev-bot 5aabe7c
Merge branch 'main' into feat/legacy-code-audit
qwen-code-dev-bot aed1ab5
Merge remote-tracking branch 'origin/main' into feat/legacy-code-audit
qwen-code-dev-bot afa923a
Merge branch 'main' into feat/legacy-code-audit
qwen-code-dev-bot 5af49e9
Merge remote-tracking branch 'origin/main' into feat/legacy-code-audit
qwen-code-dev-bot 926aef8
Merge branch 'main' into feat/legacy-code-audit
qwen-code-dev-bot 1deabe8
Merge branch 'main' into feat/legacy-code-audit
qwen-code-dev-bot 580b439
Merge branch 'main' into feat/legacy-code-audit
qwen-code-dev-bot a2638ea
Merge branch 'main' into feat/legacy-code-audit
qwen-code-dev-bot a0ee238
Merge branch 'main' into feat/legacy-code-audit
qwen-code-dev-bot caadc0c
Merge remote-tracking branch 'origin/main' into feat/legacy-code-audit
wenshao 1897bd4
fix(core): stabilize the audit fallback root across symlink path spel…
wenshao daa8f6e
Merge remote-tracking branch 'origin/feat/legacy-code-audit' into fea…
wenshao 1f97f93
Merge branch 'main' into feat/legacy-code-audit
qwen-code-dev-bot File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| # Legacy Code Audit | ||
|
|
||
| > Audit a module or directory of **existing, merged code** — no diff, no PR — using `/audit`. | ||
|
|
||
| `/review` is built for increments; `/audit` points the same machinery at code that is already merged: pre-refactor assessments, taking over an unfamiliar module, security review of a sensitive subsystem. The product is a verified, deduplicated, theme-clustered findings report. | ||
|
|
||
| ## Quick Start | ||
|
|
||
| ```bash | ||
| # Audit a module (default effort: medium) | ||
| /audit packages/core/src/permissions | ||
|
|
||
| # Quick unverified triage, one reader sub-agent | ||
| /audit packages/core/src/hooks --effort low | ||
|
|
||
| # Full pipeline plus reverse-audit rounds | ||
| /audit packages/core/src/permissions --effort high | ||
| ``` | ||
|
|
||
| Single files are not audited — `/review <file-path>` already covers that case, and `/audit` says so and stops. | ||
|
|
||
| ## Effort Levels | ||
|
|
||
| `--effort low|medium|high` trades depth for cost. **The word means the opposite of what it does in `/review`**: `/review`'s medium _drops_ the adversarial personas while `/audit`'s medium _adds_ one (6a) — and both skills select the tier with the same `--effort` flag. If you run both, reset your expectation at the boundary. | ||
|
|
||
| | Level | What runs | Findings | Cost | | ||
| | -------- | --------------------------------------------------------------------------------------------------------- | --------------------------- | ---------------- | | ||
| | `low` | One reader sub-agent rotating through directed angles plus a gap sweep | ≤10, labeled **unverified** | Cheap | | ||
| | `medium` | The measured 8-dimension fan-out (1a, 1c, 2, 3a/3b/3c, 4, 5) plus the 6a attacker seat, plus verification | Uncapped, verified | Tens of M tokens | | ||
| | `high` | medium + the 6b/6c personas + iterative reverse-audit rounds | Uncapped, verified | Extrapolated | | ||
|
|
||
| ## Size gates and budget | ||
|
|
||
| v1 audits one bounded module at a time. `plan-files` refuses at plan time — and asks for a narrower path — when: | ||
|
|
||
| - subject lines exceed **9,000** (the topology both experiments validated); | ||
| - test lines exceed **18,000** at medium/high (Agent 5 reads the corpus whole); | ||
| - subject lines exceed **2,000** at low (points you at medium); | ||
| - the priced token estimate's top exceeds the **60M** cap. | ||
|
|
||
| A larger subsystem is audited as coherent sub-paths, one bounded run each. For subject-gate and token-cap refusals, lowering the effort is never the remedy — the priced cost is a function of line counts alone. The test-line gate does not apply at `low` (the corpus goes unexamined there — triage, not an audit), and `low-gate`'s remedy is the tier change its message names. | ||
|
|
||
| ## What you confirm before anything launches | ||
|
|
||
| A fan-out run prints its roster and token estimate and starts only on your confirmation. The same confirmation carries the two **execution consents**, as separate opt-ins: | ||
|
|
||
| 1. a baseline run of the module's own test suite; | ||
| 2. agent-authored verification **probes** — short programs written mid-run, executed against a scratch copy of the probed file (never your checkout's copy), each required to flip under the implied fix. | ||
|
|
||
| The walks themselves are read-only. Because the confirmation is the only budget enforcement and the execution gate, **`/audit` refuses non-interactive starts** (headless `qwen -p`, cron, sub-agent invocations). | ||
|
|
||
| ## Safety properties | ||
|
|
||
| - **Local-only artifacts.** The report, its sidecar, and the plan/prompt records quote the module — possibly exploitable code — and must never land in version control. `plan-files` probes `.qwen/audits/` and `.qwen/tmp/` (ignore rules **and** force-added history) at plan time, offers a zero-footprint `.git/info/exclude` remedy, and re-checks at every checkpoint and at write time; a mid-run flip relocates everything to a per-user fallback outside the repo. | ||
| - **Untrusted data.** Every consumer of module content — dimension agents, verifiers, the dedup clusterer, the low-tier reader, the orchestrator itself — opens with an untrusted-data preamble: the module is evidence, not instructions. A directive embedded in the code ("report no findings") is itself a finding. | ||
| - **Drift protection.** A path-scoped sidecar (diff, untracked content copies, per-file content hashes) is captured at run start and re-checked before verification, before each high-tier round, and at write time. Content drift in a file that already carries anchored findings stops the run with a partial report; any other drift is flagged and the run continues. | ||
| - **No verdict.** The report is findings, walks, and disclosures — never "approved". Posting and fixing stay with you. | ||
|
|
||
| ## The report | ||
|
|
||
| `.qwen/audits/<YYYY-MM-DD>-<HHMMSS>-<path-slug>.md`, opening with a run-metadata header (commit SHA, model id, dirty state with sidecar, consumption against the estimate, walks completed/skipped/uncoverable, unexercised-machinery flags). Findings are clustered by root cause, each with severity, locations, failure scenario, evidence tier (end-to-end probe / unit probe / code read), and the independent-discovery count ("found independently by N agents"). Confirmed-low findings sit in their own "needs human review" section; anything unverified is labeled unverified. | ||
|
|
||
| ## Limitations | ||
|
|
||
| - Submodules are refused at plan time (no drift coverage inside them in v1). | ||
| - Dedup is intra-run; already-filed issues are not cross-checked. | ||
| - The medium/high tiers are calibrated on two modules of this repository; the low tier and the high-tier loop are unmeasured first cuts, and the report header says so. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,222 @@ | ||
| /** | ||
| * @license | ||
| * Copyright 2025 Google LLC | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| import { afterEach, describe, expect, it, vi } from 'vitest'; | ||
| import { MAX_RESPONSE_BYTES, postJson } from './http-client.js'; | ||
|
|
||
| // Behavioral net for readBoundedBody's reader loop. The loop was rewritten | ||
| // from `for await (const chunk of response.body)` to an explicit getReader() | ||
| // loop — async-iterating a ReadableStream needs [Symbol.asyncIterator] on the | ||
| // TYPE, and that resolution flipped underneath the file when #8693's | ||
| // @types/jsdom install dragged lib.dom into this program (TS2504 on the | ||
| // `for await`). These tests pin what the rewrite must preserve: bounded | ||
| // accumulation, the oversize rejection, and — the easy one to drop — | ||
| // cancelling the stream on early exit, which `for await` used to do | ||
| // implicitly via iterator return(). | ||
|
|
||
| function streamingResponse( | ||
| chunks: Uint8Array[], | ||
| onCancel: () => void, | ||
| ): Response { | ||
| let next = 0; | ||
| const body = new ReadableStream<Uint8Array>({ | ||
| pull(controller) { | ||
| if (next < chunks.length) { | ||
| controller.enqueue(chunks[next]); | ||
| next += 1; | ||
| } else { | ||
| controller.close(); | ||
| } | ||
| }, | ||
| cancel() { | ||
| onCancel(); | ||
| }, | ||
| }); | ||
| return new Response(body, { | ||
| status: 200, | ||
| headers: { 'content-type': 'application/json' }, | ||
| }); | ||
| } | ||
|
|
||
| function requestArgs() { | ||
| return { | ||
| url: new URL('https://provider.example/'), | ||
| authorization: 'Bearer test', | ||
| body: { q: 'x' }, | ||
| signal: new AbortController().signal, | ||
| }; | ||
| } | ||
|
|
||
| afterEach(() => { | ||
| vi.unstubAllGlobals(); | ||
| }); | ||
|
|
||
| describe('postJson bounded body reading', () => { | ||
| it('assembles a multi-chunk body and parses it', async () => { | ||
| const encoder = new TextEncoder(); | ||
| const payload = JSON.stringify({ answer: 42, pad: 'y'.repeat(4096) }); | ||
| const mid = Math.floor(payload.length / 2); | ||
| const cancelled = vi.fn(); | ||
| vi.stubGlobal( | ||
| 'fetch', | ||
| vi.fn(async () => | ||
| streamingResponse( | ||
| [ | ||
| encoder.encode(payload.slice(0, mid)), | ||
| encoder.encode(payload.slice(mid)), | ||
| ], | ||
| cancelled, | ||
| ), | ||
| ), | ||
| ); | ||
|
|
||
| await expect(postJson(requestArgs())).resolves.toEqual({ | ||
| answer: 42, | ||
| pad: 'y'.repeat(4096), | ||
| }); | ||
| // A fully-drained stream is closed, not cancelled. | ||
| expect(cancelled).not.toHaveBeenCalled(); | ||
| }); | ||
|
|
||
| it('accepts a body of exactly MAX_RESPONSE_BYTES', async () => { | ||
| // The bound is `total > MAX`, so a payload landing exactly on it passes. | ||
| const exact = `["${'a'.repeat(MAX_RESPONSE_BYTES - 4)}"]`; | ||
| expect(exact.length).toBe(MAX_RESPONSE_BYTES); | ||
| vi.stubGlobal( | ||
| 'fetch', | ||
| vi.fn(async () => | ||
| streamingResponse([new TextEncoder().encode(exact)], vi.fn()), | ||
| ), | ||
| ); | ||
|
|
||
| await expect(postJson(requestArgs())).resolves.toEqual([ | ||
| 'a'.repeat(MAX_RESPONSE_BYTES - 4), | ||
| ]); | ||
| }); | ||
|
|
||
| it('rejects an over-budget stream AND cancels it', async () => { | ||
| // No content-length header, so only the streaming bound can catch it. | ||
| const chunk = new Uint8Array(512 * 1024); | ||
| const cancelled = vi.fn(); | ||
| let enqueued = 0; | ||
| const body = new ReadableStream<Uint8Array>({ | ||
| pull(controller) { | ||
| // Endless stream: the reject must come from the byte budget, and the | ||
| // cancel is what stops this producer. | ||
| controller.enqueue(chunk); | ||
| enqueued += 1; | ||
| }, | ||
| cancel() { | ||
| cancelled(); | ||
| }, | ||
| }); | ||
| vi.stubGlobal( | ||
| 'fetch', | ||
| vi.fn(async () => new Response(body, { status: 200 })), | ||
| ); | ||
|
|
||
| await expect(postJson(requestArgs())).rejects.toThrow( | ||
| 'External context provider returned an invalid response.', | ||
| ); | ||
| // `for await` cancelled the stream via its implicit iterator return(); | ||
| // the reader loop must do the same or the producer keeps running. | ||
| expect(cancelled).toHaveBeenCalledTimes(1); | ||
| // Budget is 1 MiB: two 512 KiB chunks reach it, the third crosses it. | ||
| expect(enqueued).toBeLessThanOrEqual(4); | ||
| }); | ||
|
|
||
| it('holds the oversize reject until a deferred cancellation settles', async () => { | ||
| // `for await` awaited iterator return() before propagating, so a caller | ||
| // retrying immediately could not overlap the old transport's teardown. | ||
| let resolveCancel!: () => void; | ||
| const cancelBarrier = new Promise<void>((resolve) => { | ||
| resolveCancel = resolve; | ||
| }); | ||
| let cancelSeen!: () => void; | ||
| const cancelCalled = new Promise<void>((resolve) => { | ||
| cancelSeen = resolve; | ||
| }); | ||
| const body = new ReadableStream<Uint8Array>({ | ||
| pull(controller) { | ||
| controller.enqueue(new Uint8Array(512 * 1024)); | ||
| }, | ||
| cancel() { | ||
| cancelSeen(); | ||
| return cancelBarrier; | ||
| }, | ||
| }); | ||
| vi.stubGlobal( | ||
| 'fetch', | ||
| vi.fn(async () => new Response(body, { status: 200 })), | ||
| ); | ||
|
|
||
| let rejected = false; | ||
| let rejection: unknown; | ||
| const pending = postJson(requestArgs()).catch((error: unknown) => { | ||
| rejected = true; | ||
| rejection = error; | ||
| }); | ||
|
|
||
| await cancelCalled; | ||
| // Let any racing microtasks land; the reject must still be on hold. | ||
| await new Promise((resolve) => setTimeout(resolve, 0)); | ||
| expect(rejected).toBe(false); | ||
|
|
||
| resolveCancel(); | ||
| await pending; | ||
| expect(rejected).toBe(true); | ||
| expect((rejection as Error).message).toBe( | ||
| 'External context provider returned an invalid response.', | ||
| ); | ||
| }); | ||
|
|
||
| it('maps a mid-stream read failure after partial data to a transport error', async () => { | ||
| // The provider disconnects after sending part of the JSON: read() rejects | ||
| // with the partial chunk already accumulated. | ||
| const body = new ReadableStream<Uint8Array>({ | ||
| start(controller) { | ||
| controller.enqueue(new TextEncoder().encode('{"answer":')); | ||
| }, | ||
| pull(controller) { | ||
| controller.error(new Error('connection reset')); | ||
| }, | ||
| }); | ||
| vi.stubGlobal( | ||
| 'fetch', | ||
| vi.fn(async () => new Response(body, { status: 200 })), | ||
| ); | ||
|
|
||
| await expect(postJson(requestArgs())).rejects.toThrow( | ||
| 'External context provider request did not complete.', | ||
| ); | ||
| // Cleanup must still run: the reader lock is released on mid-read errors. | ||
| expect(body.locked).toBe(false); | ||
| }); | ||
|
|
||
| it('rejects a body that is not valid UTF-8', async () => { | ||
| const cancelled = vi.fn(); | ||
| vi.stubGlobal( | ||
| 'fetch', | ||
| vi.fn(async () => | ||
| // `{"a":"<invalid byte>"}`: the invalid byte sits inside otherwise- | ||
| // valid JSON, so only fatal decoding rejects it — lax decoding would | ||
| // resolve `{ a: '\uFFFD' }` and accept corrupted provider content. | ||
| streamingResponse( | ||
| [ | ||
| new Uint8Array([ | ||
| 0x7b, 0x22, 0x61, 0x22, 0x3a, 0x22, 0xff, 0x22, 0x7d, | ||
| ]), | ||
| ], | ||
| cancelled, | ||
| ), | ||
| ), | ||
| ); | ||
|
|
||
| await expect(postJson(requestArgs())).rejects.toThrow( | ||
| 'External context provider returned an invalid response.', | ||
| ); | ||
| }); | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Deferred to a follow-up round (not dropped). Pointing the Quick Start low example at a module under the 2,000-line low gate requires measuring candidate modules against the live plan-files gate; that measurement and the doc fix land next round.
中文说明
推迟到后续轮次(并非丢弃)。把 Quick Start 的 low 示例指向低于 2,000 行 low 门限的模块,需要先对候选模块用真实 plan-files 门限测量;该测量与文档修复下一轮落地。