Skip to content

fix: cap benchmark confidence to measured evidence - #51

Merged
edhor1608 merged 1 commit into
mainfrom
repo-refresh-benchmark-confidence-evidence
May 2, 2026
Merged

edhor1608 merged 1 commit into
mainfrom
repo-refresh-benchmark-confidence-evidence

Conversation

@edhor1608

@edhor1608 edhor1608 commented May 1, 2026 •

Copy link
Copy Markdown
Owner

Cap benchmark confidence to 0.75 and restrict 'high' label to ≥0.9

  • computeConfidence in benchmark.ts now derives confidence solely from objective metrics: PSNR (0.35), SSIM (0.35), bitrate delta (0.2), and output colorspace (0.1); contributions from codec fields and output_matches_target are removed.
  • The computed value is clamped to a maximum of 0.75, preventing automated runs from ever producing a 'high' label.
  • confidenceLabel raises the 'high' threshold from ≥0.8 to ≥0.9, so only manually-supplied values above the cap could reach that label.
  • Behavioral Change: all existing benchmarks that previously reported {label:'high', value:1} now report {label:'medium', value:0.75}; snapshots and integration test bounds updated accordingly.

Macroscope summarized 5ee7fdd.

@coderabbitai

coderabbitai Bot commented May 1, 2026 •

Copy link
Copy Markdown

Warning

Rate limit exceeded

@edhor1608 has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 3 minutes and 48 seconds before requesting another review.

To keep reviews running without waiting, you can enable usage-based add-on for your organization. This allows additional reviews beyond the hourly cap. Account admins can enable it under billing.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 2c13b724-e620-443f-b1d0-64532e064a37

📥 Commits

Reviewing files that changed from the base of the PR and between 3038201 and 5ee7fdd.

📒 Files selected for processing (12)
  • docs/repo_refresh_audit.md
  • src/domain/benchmark.ts
  • tests/fixtures/e2e/snapshots/benchmark/benchmark-reliable-feed-landscape-white-classic.json
  • tests/fixtures/e2e/snapshots/benchmark/benchmark-reliable-feed-portrait-png.json
  • tests/fixtures/e2e/snapshots/benchmark/benchmark-reliable-reel-portrait-mp4-audio.json
  • tests/fixtures/e2e/snapshots/benchmark/benchmark-reliable-reel-white-canvas.json
  • tests/fixtures/e2e/snapshots/validate_matrix/validate-matrix-basic.json
  • tests/fixtures/e2e/snapshots/validate_matrix/validate-matrix-max-cases-1.json
  • tests/fixtures/e2e/snapshots/validate_matrix/validate-matrix-only-file-portrait.json
  • tests/fixtures/e2e/snapshots/validate_matrix/validate-matrix-only-portrait.json
  • tests/fixtures/e2e/snapshots/validate_matrix/validate-matrix-with-failure-fail-on-error.json
  • tests/integration/benchmark.integration.test.ts
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch repo-refresh-benchmark-confidence-evidence
✨ Simplify code
  • Create PR with simplified code
  • Commit simplified code in branch repo-refresh-benchmark-confidence-evidence

Review rate limit: 0/5 reviews remaining, refill in 3 minutes and 48 seconds.

Comment @coderabbitai help to get the list of available commands and usage tips.

@edhor1608
edhor1608 force-pushed the repo-refresh-upload-report-checks branch from a126d31 to e1b4987 Compare May 2, 2026 19:03
@edhor1608
edhor1608 force-pushed the repo-refresh-benchmark-confidence-evidence branch from c9a212e to 7d5031e Compare May 2, 2026 19:03
@edhor1608
edhor1608 marked this pull request as ready for review May 2, 2026 19:03
@edhor1608
edhor1608 force-pushed the repo-refresh-benchmark-confidence-evidence branch from 7d5031e to a7c008d Compare May 2, 2026 21:25
@edhor1608
edhor1608 force-pushed the repo-refresh-upload-report-checks branch 2 times, most recently from 6ca3a20 to 4dcb6ce Compare May 2, 2026 21:26
@edhor1608
edhor1608 force-pushed the repo-refresh-benchmark-confidence-evidence branch 2 times, most recently from e677481 to a7c008d Compare May 2, 2026 21:29
@edhor1608
edhor1608 force-pushed the repo-refresh-upload-report-checks branch from 4dcb6ce to 6ca3a20 Compare May 2, 2026 21:29
@edhor1608
edhor1608 force-pushed the repo-refresh-benchmark-confidence-evidence branch from a7c008d to d1947b5 Compare May 2, 2026 21:32
@edhor1608
edhor1608 force-pushed the repo-refresh-upload-report-checks branch from 6ca3a20 to 095b5ff Compare May 2, 2026 21:32
@graphite-app
graphite-app Bot changed the base branch from repo-refresh-upload-report-checks to graphite-base/51 May 2, 2026 21:40
@edhor1608
edhor1608 force-pushed the graphite-base/51 branch from 095b5ff to 3038201 Compare May 2, 2026 21:42
@edhor1608
edhor1608 force-pushed the repo-refresh-benchmark-confidence-evidence branch from d1947b5 to 5ee7fdd Compare May 2, 2026 21:42
@edhor1608
edhor1608 changed the base branch from graphite-base/51 to main May 2, 2026 21:42
@edhor1608
edhor1608 merged commit 6230088 into main May 2, 2026
4 checks passed
@edhor1608
edhor1608 deleted the repo-refresh-benchmark-confidence-evidence branch May 15, 2026 18:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant