fix: reduce perf comparator false positives

[Astro-Han]

Summary

• Raise the default interaction_ms_median floor from 10ms to 20ms so single-frame-ish deltas do not block the gate.
• Confirm perf-probe failures by full profile:scenario:metric keys instead of any failure in the same scenario.
• Keep requested confirm scenarios as hard input-integrity failures and expose confirmation diagnostics in JSON/stdout.

Why

Issue #736 showed two structural false positives in the perf comparator: the default interaction median threshold treated below-frame timing noise as a blocking regression, and the confirm pass could fail when a different metric regressed in the same scenario. This keeps the gate blocking only when the same metric failure repeats, while still failing corrupted or incomplete confirm artifacts.

Related Issue

Closes #736.

Human Review Status

Pending

Review Focus

Please check the boundary between metric-key filtering and hard missing-scenario failures, especially the requested scenario case where base and head are both absent.

Risk Notes

Behavior risk: this intentionally stops blocking on default interaction_ms_median deltas at or below 20ms, and on confirm-stage failures for different metrics in the same scenario. Catastrophic thresholds, low-end warning behavior, the perf scenario list, and the #746/#853 probe-window guard are unchanged.

Skipped conditional checklist items:

• Visible UI/copy check: not applicable, no visible UI or copy changed.
• macOS/Windows platform check: not applicable, no platform, packaging, updater, signing, shell, or permissions surface changed.
• Docs/release/dependencies/permissions check: not applicable, no docs, release notes, dependencies, permissions, credentials, deletion, generated content, or local-file surfaces changed.

Pre-fix workflow context only: the most recent 50 perf-probe-baseline runs sampled before this PR showed 38 success, 11 failure, and 1 run with no conclusion. This PR still relies on post-merge/PR CI to measure whether the false-positive rate drops in practice.

How To Verify

bun --cwd packages/app test
Result: 1541 pass, 0 fail, 3635 expect() calls.

bun --cwd packages/app typecheck
Result: tsgo -b completed with exit code 0.

git diff --check origin/dev...HEAD
Result: no whitespace errors.

Screenshots or Recordings

Not applicable: no visible UI changes.

Checklist

How to use this checklist:

• Tick a box by replacing [ ] with [x]. Do not edit, add, or remove items.
• The bot-applied label items can only be honestly ticked AFTER the PR is opened and the labeler / priority-triage bots have run — return to the PR description and tick them then.
• Most items are required. The few that are conditional are explicitly marked (conditional); for those, leave unticked if they truly do not apply and explain why in Risk Notes. All other items must be ticked before requesting human review.

• Type label — this PR carries exactly one of bug, enhancement, task, documentation. Type labels are author-added; the labeler bot does NOT assign them. Add the label in the GitHub UI, then tick this.
• Routing labels — this PR carries at least one of app, ui, platform, harness, ci. The labeler bot assigns these on PR open based on changed paths. Confirm the bot's choice (or override if wrong), then tick this.
• Priority label — this PR carries exactly one of P0, P1, P2, P3. The priority-triage bot suggests one on PR open. Confirm or override, then tick this.
• Human Review Status above is set to Pending, Approved by @<reviewer>, or Not required: <reason> (default is Pending; "not required" is restricted to bot-authored low-risk PRs).
• I linked the related issue, or stated in Summary why there is no issue.
• I described the review focus and any meaningful risks.
• I replaced the example block in How To Verify with the real verification steps and the key result for each.
• I did not introduce unrelated refactors, dependencies, generated files, or file changes beyond the stated scope.
• (conditional) I manually checked visible UI or copy changes when needed, with screenshots or recordings. Leave unticked only if no visible UI or copy changed.
• (conditional) I considered macOS and Windows impact for platform, packaging, updater, signing, paths, shell, or permissions changes. Leave unticked only if no platform/packaging surface was touched.
• (conditional) I called out docs, release notes, dependencies, permissions, credentials, deletion behavior, generated content, or local file changes when relevant. Leave unticked only if none of those surfaces was touched.
• I reviewed the final diff for unrelated changes and suspicious dependency changes.
• I am targeting dev, and my PR title and commit messages use Conventional Commits in English.

[coderabbitai]

Warning

Review limit reached

@Astro-Han, we couldn't start this review because you've used your available PR reviews for now.

Your plan currently allows 1 review/hour. Refill in 26 minutes and 48 seconds.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more review capacity refills, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than trial, open-source, and free plans. In all cases, review capacity refills continuously over time.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: fddd02f3-3cb8-4125-a54a-0cfac2552272

📥 Commits

Reviewing files that changed from the base of the PR and between 694a390 and b53932e.

📒 Files selected for processing (4)

• packages/app/script/compare-perf.ts
• packages/app/src/testing/compare-perf-script.test.ts
• packages/app/src/testing/perf-metrics.test.ts
• packages/app/src/testing/perf-metrics.ts

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/i736-perf-comparator

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

Suggested priority: P2 (includes user-path files (packages/app/src/testing/compare-perf-script.test.ts, packages/app/src/testing/perf-metrics.test.ts, packages/app/src/testing/perf-metrics.ts)).

P1/P0 are reserved for maintainer confirmation. Please relabel manually if this is a release blocker, security issue, data-loss risk, or updater/runtime failure.

[gemini-code-assist]

Code Review

This pull request enhances the performance comparison logic by introducing a confirmation mechanism that filters failures based on previous run results. Key changes include updating the compare-perf.ts script to extract specific failure keys, increasing the interaction median threshold from 10ms to 20ms, and adding comprehensive tests for the new confirmation workflow. Feedback from the reviewer highlights an opportunity to improve readFailureScope by capturing 'hard failures' (such as missing scenarios) from the top-level failures array to prevent incorrect passes. Additionally, it was recommended to deduplicate comparisonKeys in comparePerfBaselines to ensure the output summary remains concise and free of duplicate entries.

[github-actions]

Perf delta summary

Comparator: pass

Profile / Scenario	interaction median	interaction worst	long task max	tbt	frame gap p95	frame gap max	jank count	cls	status
default / homepage-cold	24 -> 24 (0)	32 -> 32 (0)	0 -> 0 (0)	0 -> 0 (0)	16.8 -> 16.8 (0)	100 -> 100 (0)	3 -> 2 (-1)	0 -> 0 (0)	pass
default / long-session-input-lag	40 -> 40 (0)	40 -> 40 (0)	0 -> 0 (0)	0 -> 0 (0)	16.8 -> 16.7 (-0.1)	16.8 -> 16.7 (-0.1)	0 -> 0 (0)	0 -> 0 (0)	pass
default / session-streaming-long	40 -> 40 (0)	48 -> 48 (0)	0 -> 0 (0)	0 -> 0 (0)	16.8 -> 16.8 (0)	16.8 -> 16.8 (0)	0 -> 0 (0)	0 -> 0 (0)	pass
default / tool-call-expand	24 -> 24 (0)	24 -> 24 (0)	0 -> 0 (0)	0 -> 0 (0)	16.8 -> 16.7 (-0.1)	16.8 -> 16.7 (-0.1)	0 -> 0 (0)	0 -> 0 (0)	pass
default / tool-default-open-heavy-bash	16 -> 16 (0)	16 -> 16 (0)	51 -> 0 (-51)	1 -> 0 (-1)	33.4 -> 33.3 (-0.1)	83.4 -> 83.4 (0)	3 -> 2 (-1)	0 -> 0 (0)	pass
default / terminal-side-panel-open	32 -> 32 (0)	40 -> 48 (+8)	0 -> 0 (0)	0 -> 0 (0)	16.8 -> 16.8 (0)	16.8 -> 16.8 (0)	0 -> 0 (0)	0 -> 0 (0)	pass
default / session-scroll-reading	0 -> 0 (0)	16 -> 16 (0)	0 -> 0 (0)	0 -> 0 (0)	16.7 -> 16.8 (+0.1)	16.7 -> 16.8 (+0.1)	0 -> 0 (0)	0 -> 0 (0)	pass
low-end / session-scroll-reading-long	0 -> 0 (0)	0 -> 0 (0)	61 -> 60 (-1)	12 -> 15 (+3)	33.3 -> 33.3 (0)	66.8 -> 83.4 (+16.6)	2 -> 2 (0)	0.011 -> 0.011 (0)	pass
low-end / session-timeline-recompute	24 -> 24 (0)	24 -> 24 (0)	0 -> 0 (0)	0 -> 0 (0)	16.8 -> 16.7 (-0.1)	16.8 -> 16.8 (0)	0 -> 0 (0)	1.075 -> 1.075 (0)	pass
low-end / concurrent-shimmer-extreme	0 -> 0 (0)	0 -> 0 (0)	0 -> 0 (0)	0 -> 0 (0)	16.7 -> 16.7 (0)	16.8 -> 16.8 (0)	0 -> 0 (0)	0 -> 0 (0)	pass