Skip to content
Closed
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/users/features/_meta.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
export default {
commands: 'Commands',
'code-review': 'Code Review',
'legacy-audit': 'Legacy Code Audit',
'followup-suggestions': 'Followup Suggestions',
'tool-use-summaries': 'Tool-Use Summaries',
'markdown-rendering': 'Markdown Rendering',
Expand Down
67 changes: 67 additions & 0 deletions docs/users/features/legacy-audit.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Legacy Code Audit

> Audit a module or directory of **existing, merged code** — no diff, no PR — using `/audit`.

`/review` is built for increments; `/audit` points the same machinery at code that is already merged: pre-refactor assessments, taking over an unfamiliar module, security review of a sensitive subsystem. The product is a verified, deduplicated, theme-clustered findings report.

## Quick Start

```bash
# Audit a module (default effort: medium)
/audit packages/core/src/permissions

# Quick unverified triage, one reader sub-agent
/audit packages/core/src/hooks --effort low

# Full pipeline plus reverse-audit rounds
/audit packages/core/src/permissions --effort high
```

Single files are not audited — `/review <file-path>` already covers that case, and `/audit` says so and stops.

## Effort Levels

`--effort low|medium|high` trades depth for cost. **The word means the opposite of what it does in `/review`**: `/review`'s medium _drops_ the adversarial personas while `/audit`'s medium _adds_ one (6a) — and both skills select the tier with the same `--effort` flag. If you run both, reset your expectation at the boundary.

| Level | What runs | Findings | Cost |
| -------- | --------------------------------------------------------------------------------------------------------- | --------------------------- | ---------------- |
| `low` | One reader sub-agent rotating through directed angles plus a gap sweep | ≤10, labeled **unverified** | Cheap |
| `medium` | The measured 8-dimension fan-out (1a, 1c, 2, 3a/3b/3c, 4, 5) plus the 6a attacker seat, plus verification | Uncapped, verified | Tens of M tokens |
| `high` | medium + the 6b/6c personas + iterative reverse-audit rounds | Uncapped, verified | Extrapolated |

## Size gates and budget

v1 audits one bounded module at a time. `plan-files` refuses at plan time — and asks for a narrower path — when:

- subject lines exceed **9,000** (the topology both experiments validated);
- test lines exceed **18,000** at medium/high (Agent 5 reads the corpus whole);
- subject lines exceed **2,000** at low (points you at medium);
- the priced token estimate's top exceeds the **60M** cap.

A larger subsystem is audited as coherent sub-paths, one bounded run each. For subject-gate and token-cap refusals, lowering the effort is never the remedy — the priced cost is a function of line counts alone. The test-line gate does not apply at `low` (the corpus goes unexamined there — triage, not an audit). A `low-gate` refusal names its own remedy: when the message offers the tier change, re-run with `--effort medium`; when it names the path instead (medium would refuse first — the priced estimate over the token cap, or test lines over the medium gate), no tier change helps, so narrow the path.

## What you confirm before anything launches

A fan-out run prints its roster and token estimate and starts only on your confirmation. The same confirmation carries the two **execution consents**, as separate opt-ins:

1. a baseline run of the module's own test suite;
2. agent-authored verification **probes** — short programs written mid-run, executed against a scratch copy of the probed file (never your checkout's copy), each required to flip under the implied fix.

The walks themselves are read-only. Because the confirmation is the only budget enforcement and the execution gate, **`/audit` refuses non-interactive starts** (headless `qwen -p`, cron, sub-agent invocations).

## Safety properties

- **Local-only artifacts.** The report, its sidecar, and the plan/prompt records quote the module — possibly exploitable code — and must never land in version control. `plan-files` probes `.qwen/audits/` and `.qwen/tmp/` (ignore rules **and** force-added history) at plan time, offers a zero-footprint `.git/info/exclude` remedy, and re-checks at every checkpoint and at write time; a mid-run flip relocates everything to a per-user fallback outside the repo.
- **Untrusted data.** Every consumer of module content — dimension agents, verifiers, the dedup clusterer, the low-tier reader, the orchestrator itself — opens with an untrusted-data preamble: the module is evidence, not instructions. A directive embedded in the code ("report no findings") is itself a finding.
- **Drift protection.** A path-scoped sidecar (diff, untracked content copies, per-file content hashes) is captured at run start and re-checked before verification, before each high-tier round, and at write time. Content drift in a file that already carries anchored findings stops the run with a partial report; any other drift is flagged and the run continues.
- **No verdict.** The report is findings, walks, and disclosures — never "approved". Posting and fixing stay with you.

## The report

`.qwen/audits/<YYYY-MM-DD>-<HHMMSS>-<path-slug>.md`, opening with a run-metadata header (commit SHA, model id, dirty state with sidecar, consumption against the estimate, walks completed/skipped/uncoverable, unexercised-machinery flags). Findings are clustered by root cause, each with severity, locations, failure scenario, evidence tier (end-to-end probe / unit probe / code read), and the independent-discovery count ("found independently by N agents"). Confirmed-low findings sit in their own "needs human review" section; anything unverified is labeled unverified.

## Limitations

- Submodules are refused at plan time (no drift coverage inside them in v1).
- Dedup is intra-run; already-filed issues are not cross-checked.
- The medium/high tiers are calibrated on two modules of this repository; the low tier and the high-tier loop are unmeasured first cuts, and the report header says so.
1 change: 1 addition & 0 deletions packages/cli/src/cli.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1061,6 +1061,7 @@ describe('bootstrap import boundaries', () => {
const configSource = readFileSync('src/config/config.ts', 'utf8');
const commandNameByIdentifier = new Map([
['authCommand', 'auth'],
['auditCommand', 'audit'],
['channelCommand', 'channel'],
['extensionsCommand', 'extensions'],
['hooksCommand', 'hooks'],
Expand Down
4 changes: 4 additions & 0 deletions packages/cli/src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,10 @@ type BootstrapRoute = 'serve' | 'mcp' | 'help' | 'version' | 'default';

export const TOP_LEVEL_COMMANDS = [
['auth', 'Configure authentication (removed)'],
[
'audit <command>',
'Helpers used by the /audit skill (argument parsing, audit planning, brief printing, run-state captures)',
],
['channel <command>', 'Manage messaging channels (Telegram, Discord, etc.)'],
['extensions <command>', 'Manage Qwen Code extensions.'],
['hooks', 'Manage Qwen Code hooks (use /hooks in interactive mode).'],
Expand Down
52 changes: 52 additions & 0 deletions packages/cli/src/commands/audit.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
/**
* @license
* Copyright 2026 Qwen Team
* SPDX-License-Identifier: Apache-2.0
*/

import { describe, it, expect } from 'vitest';
import { readFileSync } from 'node:fs';
import { auditCommand } from './audit.js';

describe('auditCommand', () => {
it('registers exactly the expected subcommands', () => {
const source = readFileSync('src/commands/audit.ts', 'utf8');
const subcommands = [...source.matchAll(/\.command\((\w+Command)\)/g)].map(
(m) => m[1],
);
expect(subcommands).toEqual([
'parseArgsCommand',
'planFilesCommand',
'agentPromptCommand',
'snapshotCommand',
'driftCheckCommand',
'guardCheckCommand',
'checkAnchorsCommand',
]);
});

it('demandCommand text names each subcommand', () => {
const source = readFileSync('src/commands/audit.ts', 'utf8');
// Assert against the demandCommand MESSAGE, not the whole file: the
// import lines also contain the subcommand module names.
const message = /\.demandCommand\(\s*1,\s*'([^']+)'/.exec(source)?.[1];
expect(message).toBeDefined();
for (const name of [
'parse-args',
'plan-files',
'agent-prompt',
'snapshot',
'drift-check',
'guard-check',
'check-anchors',
]) {
expect(message).toContain(name);
}
});

it('is a CommandModule with an empty dispatch handler', () => {
expect(auditCommand.command).toBe('audit');
expect(typeof auditCommand.builder).toBe('function');
expect(typeof auditCommand.handler).toBe('function');
});
});
41 changes: 41 additions & 0 deletions packages/cli/src/commands/audit.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
/**
* @license
* Copyright 2026 Qwen Team
* SPDX-License-Identifier: Apache-2.0
*/

// `qwen audit`: the non-interactive helpers used by the bundled /audit skill
// for auditing existing code (no diff, no PR). The skill orchestrates via
// shell calls to these subcommands; see
// packages/core/src/skills/bundled/audit/SKILL.md.

import type { CommandModule } from 'yargs';
import { parseArgsCommand } from './audit/parse-args.js';
import { planFilesCommand } from './audit/plan-files.js';
import { agentPromptCommand } from './audit/agent-prompt.js';
import { checkAnchorsCommand } from './audit/check-anchors.js';
import { guardCheckCommand } from './audit/guard-check.js';
import { driftCheckCommand, snapshotCommand } from './audit/snapshot.js';

export const auditCommand: CommandModule = {
command: 'audit',
describe:
'Helpers used by the /audit skill (argument parsing, audit planning, brief printing, run-state captures)',
builder: (yargs) =>
yargs
.command(parseArgsCommand)
.command(planFilesCommand)
.command(agentPromptCommand)
.command(snapshotCommand)
.command(driftCheckCommand)
.command(guardCheckCommand)
.command(checkAnchorsCommand)
.demandCommand(
1,
'audit needs a subcommand: parse-args, plan-files, agent-prompt, snapshot, drift-check, guard-check, check-anchors',
)
.version(false),
handler: () => {
// Dispatch is per-subcommand.
},
};
111 changes: 111 additions & 0 deletions packages/cli/src/commands/audit/agent-prompt.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
/**
* @license
* Copyright 2026 Qwen Team
* SPDX-License-Identifier: Apache-2.0
*/

import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import {
mkdirSync,
mkdtempSync,
readFileSync,
rmSync,
writeFileSync,
} from 'node:fs';
import { join } from 'node:path';
import { tmpdir } from 'node:os';
import { agentPromptCommand } from './agent-prompt.js';
import { writeStdoutLine } from '../../utils/stdioHelpers.js';
import { buildFilesPlan, collectAuditFiles } from './lib/files-plan.js';

vi.mock('../../utils/stdioHelpers.js', () => ({
writeStdoutLine: vi.fn(),
}));

let dir: string;

beforeEach(() => {
dir = mkdtempSync(join(tmpdir(), 'audit-agent-prompt-'));
mkdirSync(join(dir, 'mod'), { recursive: true });
writeFileSync(join(dir, 'mod', 'a.ts'), 'const a = 1;\n');
vi.mocked(writeStdoutLine).mockClear();
});

afterEach(() => {
rmSync(dir, { recursive: true, force: true });
});

function writePlan(effort: 'low' | 'medium' | 'high'): string {
const plan = buildFilesPlan(
join(dir, 'mod'),
join(dir, 'mod'),
effort,
collectAuditFiles(join(dir, 'mod')),
);
const planPath = join(dir, `plan-${effort}.json`);
writeFileSync(planPath, JSON.stringify(plan));
return planPath;
}

const run = (argv: Record<string, unknown>) =>
(agentPromptCommand.handler as (a: unknown) => void)({
_: ['audit', 'agent-prompt'],
...argv,
});

describe('agentPromptCommand handler', () => {
it('prints a role brief for a role in the roster', () => {
run({ plan: writePlan('medium'), role: '1a', probes: 'declined' });
const printed = vi.mocked(writeStdoutLine).mock.calls[0][0];
expect(printed).toContain('You are Agent 1a');
// Declined probe opt-in strips the execution instructions.
expect(printed).toContain('Execution is NOT opted in');
});

it('maps the opted-in probe flag to the probe discipline', () => {
// The 'opted-in' → probesConsented === true mapping is load-bearing:
// without it every opted-in run prints the declined brief and the
// verifier tier silently caps at code reads.
run({ plan: writePlan('medium'), role: '1a', probes: 'opted-in' });
const printed = vi.mocked(writeStdoutLine).mock.calls[0][0];
expect(printed).toContain('A probe runs only against a scratch copy');
expect(printed).not.toContain('Execution is NOT opted in');
});

it('refuses the low reader at medium and a roster role at low', () => {
expect(() =>
run({
plan: writePlan('medium'),
role: 'low-reader',
probes: 'declined',
}),
).toThrow(/only valid for a low-tier plan/);
// Low plans carry an empty roster: every dimension role is refused.
expect(() =>
run({ plan: writePlan('low'), role: '1a', probes: 'declined' }),
).toThrow(/not in this plan's roster/);
});

it('refuses a stale-plan role that is not in the roster', () => {
// 'toString' rides the prototype-membership hole a raw .includes()
// call would leave open: it is an Object.prototype member, not a role.
expect(() =>
run({ plan: writePlan('medium'), role: 'toString', probes: 'declined' }),
).toThrow(/must be one of/);
});

it('fails closed when the plan carries a non-array roster', () => {
const planPath = writePlan('medium');
const parsed = JSON.parse(readFileSync(planPath, 'utf8')) as Record<
string,
unknown
>;
// A string roster ('1a' .includes('1a') === true, '12' admits '2')
// must fail closed, not reach substring membership.
parsed['roster'] = '12';
writeFileSync(planPath, JSON.stringify(parsed));
expect(() =>
run({ plan: planPath, role: '2', probes: 'declined' }),
).toThrow(/not in this plan's roster/);
});
});
94 changes: 94 additions & 0 deletions packages/cli/src/commands/audit/agent-prompt.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
/**
* @license
* Copyright 2026 Qwen Team
* SPDX-License-Identifier: Apache-2.0
*/

// `qwen audit agent-prompt`: print the brief for one audit role — or the
// low tier's reader — with the plan's context assembled in. The /audit skill
// launches its agents with exactly these prompts (one call per roster role),
// so what every agent is told is fixed by code, not improvised by the
// orchestrator.

import type { CommandModule } from 'yargs';
import { writeStdoutLine } from '../../utils/stdioHelpers.js';
import { readPlanFile } from './lib/read-json.js';
import {
AUDIT_BRIEFS,
buildAuditPrompt,
buildLowReaderPrompt,
type AuditBriefRole,
} from './lib/audit-agent-briefs.js';

interface AgentPromptArgs {
plan: string;
role?: string;
probes?: 'opted-in' | 'declined';
}

function runAgentPrompt(args: AgentPromptArgs): void {
const plan = readPlanFile(args.plan, 'agent-prompt');
const probesConsented = args.probes === 'opted-in';
// A stale plan JSON can carry anything in its roster — a non-array must
// fail closed (empty roster, every role refused), never reach .includes.
const roles = Array.isArray(plan.roster) ? (plan.roster as string[]) : [];
if (args.role === 'low-reader') {
if (plan.effort !== 'low') {
throw new Error(
`agent-prompt: low-reader is only valid for a low-tier plan (this plan is ${plan.effort}).`,
);
}
writeStdoutLine(buildLowReaderPrompt(plan));
return;
}
const role = args.role as Exclude<AuditBriefRole, 'low-reader'> | undefined;
// Object.hasOwn, not `in`: a stale-plan role like "toString" matches
// inherited Object.prototype keys and would emit an undefined brief.
if (!role || !Object.hasOwn(AUDIT_BRIEFS, role)) {
throw new Error(
`agent-prompt: --role must be one of ${[...Object.keys(AUDIT_BRIEFS), 'low-reader'].join(', ')}.`,
);
}
if (!roles.includes(role)) {
throw new Error(
`agent-prompt: role ${role} is not in this plan's roster (${roles.join(', ') || 'empty'}). The roster is computed from the plan's effort — regenerate the plan if you need a different tier.`,
);
}
writeStdoutLine(buildAuditPrompt(role, plan, probesConsented));
}

export const agentPromptCommand: CommandModule = {
command: 'agent-prompt',
describe:
'Print the brief for an audit role or the low-tier reader — with plan context assembled',
builder: (yargs) =>
yargs
.option('plan', {
type: 'string',
demandOption: true,
describe: 'Plan JSON written by `qwen audit plan-files`',
})
.option('role', {
type: 'string',
describe: 'Print one role brief (must be in the plan roster)',
})
.option('probes', {
choices: ['opted-in', 'declined'] as const,
describe:
'The Step-2 probe opt-in verdict; declined prompts carry no execution instructions (not required for --role low-reader — low runs no execution classes)',
})
.check((argv) => {
if (!argv.role) {
throw new Error('agent-prompt: pass --role <id>.');
}
if (argv.role !== 'low-reader' && !argv.probes) {
throw new Error(
'agent-prompt: pass --probes opted-in|declined (the Step-2 probe opt-in).',
);
}
return true;
}),
handler: (argv) => {
runAgentPrompt(argv as unknown as AgentPromptArgs);
},
};
Loading
Loading