Skip to content

chore: full calibration run + add-rule pipeline test#18

Merged
let-sunny merged 7 commits intomainfrom
chore/full-calibration-run
Mar 24, 2026
Merged

chore: full calibration run + add-rule pipeline test#18
let-sunny merged 7 commits intomainfrom
chore/full-calibration-run

Conversation

@let-sunny
Copy link
Copy Markdown
Owner

@let-sunny let-sunny commented Mar 24, 2026

Summary

  • 6개 픽스처 전체 캘리브레이션 실행 및 파이프라인 테스트
  • /add-rule 파이프라인으로 repeated-frame-structure 룰 구현 및 디버깅
  • discovery-prune-evidence CLI 버그 수정

캘리브레이션 결과

Fixture Grade Similarity Applied
figma-ui3-kit B (75%) 81% 0
material3-kit-1 B (79%) 93% 0
material3-kit-2 B (77%) 97% 3
simple-ds-card-grid C+ (71%) 95% 1
simple-ds-page-sections B+ (80%) 98% 2
simple-ds-panel-sections B+ (80%) 98% 0

적용된 rule-config.ts 변경:

  • nested-instance-override: -5 → -2, severity risk → missing-info
  • fixed-width-in-responsive-context: -4 → -2, severity risk → missing-info
  • empty-frame: -2 → -3

⚠️ 현재 픽스처가 단순 섹션(20~96 노드)이라 점수 변경이 편향되었을 수 있음. 복잡한 픽스처 추가 후 재검증 필요. 머지하지 않고 참고용으로 유지.

add-rule 파이프라인 테스트

repeated-frame-structure 룰 구현:

  • 형제 FRAME 노드의 구조적 fingerprint(type:layoutMode:children to depth 3) 비교
  • Copy-paste 패턴 감지 (이름이 달라도 구조가 같으면 flag)
  • severity: suggestion, score: -3, minRepetitions: 2
  • 0% false positive, 1/6 fixture hit, 13 unit tests

파이프라인 발견 사항:

버그 수정

  • discovery-prune-evidence CLI: cac이 단일 variadic arg를 string으로 전달하는 문제 수정

생성된 이슈

Test plan

  • pnpm test:run — 261 tests passed
  • pnpm lint — clean
  • pnpm build — clean

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Added a "repeated-frame-structure" rule to detect frames with repeated structural patterns among siblings.
  • Changes

    • Adjusted severity and scoring for "fixed-width-in-responsive-context", "nested-instance-override", and "empty-frame".
    • The discovery-prune-evidence CLI now accepts a single category argument and always proceeds to pruning/logging.
  • Tests

    • Added tests covering the new rule's behavior and edge cases.

let-sunny and others added 5 commits March 24, 2026 20:55
- nested-instance-override: -5 → -3 (risk) — Critic revised; 2-case medium-confidence evidence, 60% reduction capped at midpoint
- fixed-width-in-responsive-context: -4 → -2 (risk) — Approved; medium confidence, 2 cases, 50% reduction meets threshold, severity unchanged
- empty-frame: -2 → -3 (missing-info) — Critic revised; direction supported, conservative midpoint, severity unchanged

Source: calibration against fixtures

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- nested-instance-override: -3 → -2 (Critic revised; high confidence, 3 cases, severity kept as risk)

Source: calibration agent debate session
- fixed-width-in-responsive-context: severity risk → missing-info (score -2 unchanged). Rule flags a missing design decision (no explicit breakpoint intent declared), not an imminent implementation breakage; reclassified to match information-gap semantics. 3 confirmed cases, high confidence.
- nested-instance-override: severity risk → missing-info (score -2 unchanged). Overrides inside nested instances represent absent handoff context that AI must guess, not structural breakage risk; reclassified accordingly. 3 confirmed cases, high confidence.

Source: calibration against fixture (cross-run evidence)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Detects sibling FRAME nodes with identical structural fingerprints
(type:layoutMode:children to depth 3) that aren't componentized.
Catches copy-paste patterns that missing-component misses (different
names, same structure). Value is design quality signal for AI token
efficiency on large pages, not rendering accuracy.

- severity: suggestion, score: -3
- minRepetitions: 2, maxFingerprintDepth: 3
- Skips INSTANCE subtrees and COMPONENT_SET parents
- 0% false positive rate across 6 fixtures
- 13 unit tests

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
cac library passes single variadic arg as string, not array.
Changed from <categories...> to <category> with Array.isArray guard.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Mar 24, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 5140eca9-85c1-4116-ac6c-dd608cb7e8f6

📥 Commits

Reviewing files that changed from the base of the PR and between d31c930 and cebfebd.

📒 Files selected for processing (1)
  • src/cli/index.ts

📝 Walkthrough

Walkthrough

Added a new component rule repeated-frame-structure detecting sibling FRAMEs with matching structural fingerprints; changed the discovery-prune-evidence CLI argument to a single <category> (normalized to an array); and updated several rule configurations and scores.

Changes

Cohort / File(s) Summary
CLI Command Update
src/cli/index.ts
Modified discovery-prune-evidence command to accept a single <category> argument instead of variadic <categories...>, normalizing the action callback parameter to an array and updating help text.
Type Definition
src/core/contracts/rule.ts
Added "repeated-frame-structure" to the RuleId union type and updated the component section comment from (7) to (8).
Rule Implementation
src/core/rules/component/index.ts
Added repeated-frame-structure rule with helpers (buildFingerprint, isInsideInstance) to compute recursive structural fingerprints, deduplicate reports, and respect minRepetitions / maxFingerprintDepth options.
Rule Test Suite
src/core/rules/component/repeated-frame-structure.test.ts
Added Jest tests covering metadata, non-FRAME/empty children cases, repetition thresholds, parent exclusions (INSTANCE, COMPONENT_SET), message/path formatting, options override, and ignoring non-FRAME siblings.
Rule Configuration
src/core/rules/rule-config.ts
Added config for repeated-frame-structure (severity: "suggestion", score: -3, enabled: true, options), adjusted severities/scores for existing rules and updated component rule count comment.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 A new rule hops into view,
Finding frames that match—how true!
Fingerprints whisper in sibling rows,
Suggesting components where repetition grows,
Tests and configs tidy—off we chew! 🥕

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Title check ⚠️ Warning The title 'chore: full calibration run + add-rule pipeline test' is only partially related to the changeset. While calibration adjustments are present (rule config changes), the primary substantive changes involve implementing a new rule (repeated-frame-structure) and fixing a CLI bug. The title emphasizes procedural/testing aspects rather than the actual code changes. Consider a more specific title that highlights the main implementation changes, such as 'feat: add repeated-frame-structure rule + calibrate rule configs' or 'feat: implement repeated-frame-structure detection rule'.
Docstring Coverage ⚠️ Warning Docstring coverage is 71.43% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (1 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch chore/full-calibration-run

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/core/rules/rule-config.ts (1)

123-125: ⚠️ Potential issue | 🟡 Minor

Update the component rules count comment.

The comment says "(7 rules)" but with the addition of "repeated-frame-structure", there are now 8 component rules in this section.

📝 Proposed fix
   // ============================================
-  // Component (7 rules)
+  // Component (8 rules)
   // ============================================
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/core/rules/rule-config.ts` around lines 123 - 125, Update the component
rules count comment to reflect the new rule: change the comment block "//
Component (7 rules)" to indicate 8 rules since "repeated-frame-structure" was
added; locate the comment near the component rules header and update the count
to "(8 rules)" so it matches the actual rules in the section.
🧹 Nitpick comments (1)
src/cli/index.ts (1)

664-667: Type annotation doesn't reflect runtime behavior.

The parameter is typed as string, but the Array.isArray guard and cast to string[] suggest cac might pass either type at runtime. Consider using a union type to accurately reflect this behavior and improve type safety.

♻️ Proposed improvement
-  .action((category: string) => {
-    const categories = Array.isArray(category)
-      ? (category as string[])
-      : [category];
+  .action((category: string | string[]) => {
+    const categories = Array.isArray(category) ? category : [category];
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/cli/index.ts` around lines 664 - 667, The action callback's parameter is
annotated as string but runtime uses Array.isArray, so change the parameter type
to a union (category: string | string[]) in the .action handler to match actual
runtime values; then remove the forced cast to string[] and narrow with the
existing Array.isArray check to produce const categories =
Array.isArray(category) ? category : [category] (refer to the .action callback
and the category parameter in src/cli/index.ts).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@src/core/rules/rule-config.ts`:
- Around line 123-125: Update the component rules count comment to reflect the
new rule: change the comment block "// Component (7 rules)" to indicate 8 rules
since "repeated-frame-structure" was added; locate the comment near the
component rules header and update the count to "(8 rules)" so it matches the
actual rules in the section.

---

Nitpick comments:
In `@src/cli/index.ts`:
- Around line 664-667: The action callback's parameter is annotated as string
but runtime uses Array.isArray, so change the parameter type to a union
(category: string | string[]) in the .action handler to match actual runtime
values; then remove the forced cast to string[] and narrow with the existing
Array.isArray check to produce const categories = Array.isArray(category) ?
category : [category] (refer to the .action callback and the category parameter
in src/cli/index.ts).

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 69bdeb8f-8e1a-4fb7-badf-796fd121d06d

📥 Commits

Reviewing files that changed from the base of the PR and between 0906cf1 and a2ea330.

📒 Files selected for processing (5)
  • src/cli/index.ts
  • src/core/contracts/rule.ts
  • src/core/rules/component/index.ts
  • src/core/rules/component/repeated-frame-structure.test.ts
  • src/core/rules/rule-config.ts

- Update component rules count comment from 7 to 8
- Fix discovery-prune-evidence action param type to string | string[]

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/cli/index.ts (1)

661-672: Empty check is unreachable with the new command signature.

With <category> being a required (non-optional) positional argument, cac will not invoke the action without a value. After normalization on line 665, categories will always have at least one element, making the categories.length === 0 check on line 666 dead code.

Additionally, changing from variadic <categories...> to singular <category> means users can no longer prune multiple categories in one command invocation. Verify this is the intended behavior.

🔧 Suggested fix: remove unreachable code
   .action((category: string | string[]) => {
     const categories = Array.isArray(category) ? category : [category];
-    if (categories.length === 0) {
-      console.log("No categories specified — nothing to prune.");
-      return;
-    }

     pruneDiscoveryEvidence(categories);
     console.log(`Pruned discovery evidence for categories: ${categories.join(", ")}`);
   });
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/cli/index.ts` around lines 661 - 672, The empty-length guard is dead
because the CLI uses a required positional "<category>" so after normalizing
into const categories = Array.isArray(category) ? category : [category] you'll
always have at least one element; remove the unreachable if (categories.length
=== 0) { ... } branch and its console.log, and either (A) keep the current
singular behavior and call pruneDiscoveryEvidence(categories) as-is, or (B) if
you want to support pruning multiple categories in one invocation, change the
command signature back to "discovery-prune-evidence <categories...>" so
categories can be an array—update the command declaration accordingly and ensure
pruneDiscoveryEvidence accepts an array.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@src/cli/index.ts`:
- Around line 661-672: The empty-length guard is dead because the CLI uses a
required positional "<category>" so after normalizing into const categories =
Array.isArray(category) ? category : [category] you'll always have at least one
element; remove the unreachable if (categories.length === 0) { ... } branch and
its console.log, and either (A) keep the current singular behavior and call
pruneDiscoveryEvidence(categories) as-is, or (B) if you want to support pruning
multiple categories in one invocation, change the command signature back to
"discovery-prune-evidence <categories...>" so categories can be an array—update
the command declaration accordingly and ensure pruneDiscoveryEvidence accepts an
array.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 546c0d7e-1fba-46d9-9ccf-1ca195d242a5

📥 Commits

Reviewing files that changed from the base of the PR and between a2ea330 and d31c930.

📒 Files selected for processing (2)
  • src/cli/index.ts
  • src/core/rules/rule-config.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/core/rules/rule-config.ts

Required positional arg guarantees categories is never empty.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant