feat(core): tell "found nothing" from "never ran" in the JSON report

[oekazuma]

Why

--reporter json could not answer "did this rule run?"

issues is filtered to penalized results, so a rule that found nothing contributes no entry. summary counts severities project-wide, so passed cannot be attributed to a rule. A rule reporting zero therefore had two indistinguishable meanings — every declaration matched and passed, or nothing matched at all — and zero is the output nobody thinks to question.

Not hypothetical. A field test configured architecture/unit-entry-file, saw nothing, and could not tell whether the tree conformed or the globs were dead. Proving the rule had executed required planting a deliberately non-conforming unit; the same probe was needed for two sibling rules.

The console reporter never had this gap — it lists every passing result under Passed (N). The gap was specific to the JSON channel, which is what the field test used and what CI uses.

What

JsonReport gains a top-level rules map:

"rules": {
  "architecture/unit-entry-file": { "findings": 0, "passed": 12 },
  "seo/single-h1": { "findings": 9, "passed": 342 }
}

Presence is the answer; the counts are detail. An entry with findings: 0 ran and reported nothing; a rule missing from the map was not selected.

That only works because the map is seeded from the ids of the rules that ran, passed as an optional fourth argument. Seeding from results alone would leave both cases empty — the original bug in a new place.

The distinction that made this harder than it looks

The list is not what selectRules returns. In the CLI, --category narrows the set after selection:

const selected = selectRules(allRules, config);
const rules = opts.categories ? selected.filter((r) => opts.categories!.includes(r.category)) : selected;

Passing selected would report rules excluded by --category as having run — precisely the confusion this change exists to remove. The Vite plugin has no --category equivalent, so there selectRules's output is what ran. The two channels are wired asymmetrically on purpose, and AnalyzeResult gained ruleIds because the CLI computes the set in analyzeProject and formats the report in run.

Design decisions worth stating, so they are not re-litigated

• Not inside summary. That type is shared with the console reporter, the markdown reporter, the CLI and the Vite plugin; a per-rule map would grow a type four consumers read and none of them want.
• findings is deliberately redundant with routes[].issues[] + siteIssues[]. Included so "did it run and find nothing" is a local question — forcing a full scan to answer the second half would defeat the change.
• No severity breakdown. Every issue already carries id and severity, so that grouping is derivable locally — unlike passed.
• Counts describe the report, not the tree. Baseline, suppression and --diff filtering run before the report is built, so a rule whose findings were all suppressed shows findings: 0 and stays present. Pinned by a test, and documented, because it is the surprising half.
• A rule disabled through an overrides entry still appears, since selectRules reads only top-level config.rules. Verified by running the CLI both ways, and now stated in the guide — the earlier draft claimed presence always means the rule ran, which was wrong for that one path.

Verification

core 1156, cli 805, vite 206 tests pass; typecheck clean in all three after rebuilding core; lint and format clean.

Both load-bearing mechanisms were confirmed by mutation, not assumed: deleting the seed loop fails 5 tests across the three packages, and swapping rules.map for selected.map fails exactly the --category test. An end-to-end CLI run shows 45 default-on rules present with 0/0, Σfindings equals the report's issue count and Σpassed equals summary.passed under both treatDynamicAs modes.

Known limitation, recorded rather than fixed

The HTML report's embedded snapshot and the dev dashboard's /data.json call buildJsonReport on the three-argument form, so their rules map is results-only — presence there does not imply selection. Nothing renders it today. Threading the list would cascade through three signatures on the dashboard path and collide with a separately-computed rule selection in the live hook, and doing only the easy half achieves none of the payoff, so both call sites carry a comment and the design doc records the difference.

Design: docs/superpowers/specs/2026-08-03-json-rule-evidence-design.md. Plan: docs/superpowers/plans/2026-08-03-json-rule-evidence.md.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • JSON reports now include per-rule evidence with finding and passed-result counts.
  • Selected rules with no findings remain visible with zero counts.
  • Reports reflect exclusions, suppressions, baselines, and diff filtering.
  • Rules disabled or excluded from selection are omitted from the report.
• Documentation
  • Updated English and Japanese reporter guides to explain per-rule evidence and counting behavior.
• Tests
  • Added coverage for selected, excluded, suppressed, and zero-finding rules across CLI, Vite, and core reporting.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c60aff37-0886-4325-94a6-3b2c822b0664

📥 Commits

Reviewing files that changed from the base of the PR and between d1e0a58 and 293ad5d.

📒 Files selected for processing (3)

• docs/superpowers/plans/2026-08-03-json-rule-evidence.md
• docs/superpowers/specs/2026-08-03-json-rule-evidence-design.md
• packages/core/src/reporter/json.ts

🚧 Files skipped from review as they are similar to previous changes (3)

• docs/superpowers/plans/2026-08-03-json-rule-evidence.md
• packages/core/src/reporter/json.ts
• docs/superpowers/specs/2026-08-03-json-rule-evidence-design.md

---

📝 Walkthrough

Walkthrough

The JSON reporter now includes a top-level rules map with findings and passed counts. CLI and Vite callers pass selected rule IDs so rules with zero results remain represented. Tests and documentation cover filtering, suppression, overrides, and compatibility.

Changes

JSON Rule Evidence

Layer / File(s)	Summary
Report contract and counting docs/superpowers/specs/..., packages/core/src/reporter/json.ts, packages/core/src/index.ts, packages/core/test/*	Adds RuleEvidence, extends JsonReport, counts findings and passed results, seeds selected rules with zero counts, and preserves fallback behavior when rule IDs are omitted.
Selected-rule propagation and validation packages/cli/src/index.ts, packages/vite/src/analyze.ts, packages/cli/test/*, packages/vite/test/*	Passes selected rule IDs from CLI and Vite analysis into JSON reporting. Tests cover empty, disabled, filtered, and suppressed rules.
Documentation and release records docs/src/content/docs/**/reporters.md, docs/superpowers/{plans,specs}/*, .changeset/*	Documents the rules map and its filtering semantics. Adds the implementation plan, design record, and release metadata.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CLI as CLI analyzeProject
  participant Vite as Vite analyze
  participant Reporter as formatJsonReport
  participant Builder as buildJsonReport
  CLI->>Reporter: pass analysis.ruleIds
  Vite->>Reporter: pass selected rule IDs
  Reporter->>Builder: pass results, config, meta, ruleIds
  Builder->>Builder: seed rules and count findings and passed results
  Builder-->>Reporter: return JsonReport with rules

Loading

Possibly related PRs

• oekazuma/svelte-vitals#144: Shares JSON reporting and buildJsonReport infrastructure.
• oekazuma/svelte-vitals#112: Also modifies the Vite analysis and JSON reporting flow.
• oekazuma/svelte-vitals#196: Shares suppression test coverage for JSON reporting.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: distinguishing rules with no findings from rules that did not run in JSON reports.
Docstring Coverage	✅ Passed	Docstring coverage is 87.50% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🧹 Nitpick comments (1)

docs/superpowers/plans/2026-08-03-json-rule-evidence.md (1)

408-429: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Record the shipped semver decision in the historical plan.

The plan specifies patch, but .changeset/json-rule-evidence.md ships minor releases for all three packages.

Do not rewrite the historical steps. Add a dated supersession note that explains the decision and links to the shipped changeset.

Based on learnings, plans under docs/superpowers/plans/ must preserve historical wording and record shipped divergences with a dated, scoped supersession note.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/superpowers/plans/2026-08-03-json-rule-evidence.md` around lines 408 -
429, Add a dated, scoped supersession note near the changeset step in the
historical plan, preserving all existing wording. State that the planned patch
releases were superseded by the shipped minor releases for all three packages,
and link to `.changeset/json-rule-evidence.md` as the authoritative shipped
decision.

Source: Learnings

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/superpowers/plans/2026-08-03-json-rule-evidence.md`:
- Around line 229-233: Update Step 5 in the seeding verification plan to remove
the invalid expectation that git diff on json.ts is empty after restoring the
loop. Instead, instruct the agent to compare the restored file with a copy saved
before deletion, or verify that the for (const id of ruleIds ?? []) seed loop is
present, while preserving the expected test failures when the loop is removed.

In `@docs/superpowers/specs/2026-08-03-json-rule-evidence-design.md`:
- Around line 93-95: Update the `formatHtmlReport` source reference in the
documentation to `packages/core/src/reporter/html.ts`; keep `renderAppShell`
associated with `packages/core/src/reporter/app-shell.ts`.

In `@packages/core/src/reporter/json.ts`:
- Around line 22-23: Update the RuleEvidence documentation in
packages/core/src/reporter/json.ts (lines 22-23) to state that zero-count
entries prove selection only when ruleIds is provided; otherwise, compatibility
fallback entries are derived from results and missing keys do not prove
non-selection. Update
docs/superpowers/specs/2026-08-03-json-rule-evidence-design.md (lines 32-35) to
qualify the rules-presence table accordingly and explicitly state that
override-disabled rules remain represented.

---

Nitpick comments:
In `@docs/superpowers/plans/2026-08-03-json-rule-evidence.md`:
- Around line 408-429: Add a dated, scoped supersession note near the changeset
step in the historical plan, preserving all existing wording. State that the
planned patch releases were superseded by the shipped minor releases for all
three packages, and link to `.changeset/json-rule-evidence.md` as the
authoritative shipped decision.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 252d0358-c131-4fba-a095-460ef26b3382

📥 Commits

Reviewing files that changed from the base of the PR and between b3dc3de and d1e0a58.

📒 Files selected for processing (18)

• .changeset/json-rule-evidence.md
• docs/src/content/docs/guides/(reporting)/reporters.md
• docs/src/content/docs/ja/guides/(reporting)/reporters.md
• docs/superpowers/plans/2026-08-03-json-rule-evidence.md
• docs/superpowers/specs/2026-08-03-json-rule-evidence-design.md
• packages/cli/src/index.ts
• packages/cli/test/run-suppressions.test.ts
• packages/cli/test/run.test.ts
• packages/core/src/index.ts
• packages/core/src/reporter/app-shell.ts
• packages/core/src/reporter/json.ts
• packages/core/test/html-report.test.ts
• packages/core/test/json-report.test.ts
• packages/vite/src/analyze.ts
• packages/vite/src/ui/snapshot.ts
• packages/vite/test/analyze.test.ts
• packages/vite/test/app-shell-static.test.ts
• packages/vite/test/ui-dashboard.test.ts