ci(benchmark): Add performance benchmark workflow for PRs

[yamadashy]

Add a GitHub Actions workflow that benchmarks repomix by packing its own repository, comparing PR branch vs main branch performance.

How it works

1. Checks out PR and main into separate directories for measurement isolation
2. Builds both branches independently
3. Runs 2 warmup + 5 measured runs per branch, reports the median
4. Calculates time difference (seconds and %)
5. Posts a single comment on the PR (updates in-place on subsequent pushes)

Comment example

	Time
PR branch	4.52s
main branch	4.48s
Diff	+0.04s (+0.9%)

Design decisions

• Separate directory checkout: Avoids OS page cache bias between measurements (prior lesson)
• Warmup runs: Stabilizes JIT and page cache before measurement
• Concurrency group: Cancels outdated benchmark runs on the same PR
• Single comment per PR: Uses HTML comment marker (<!-- repomix-perf-benchmark -->) to find and update existing comments

Checklist

• Run npm run test
• Run npm run lint

🤖 Generated with Claude Code

---

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 58baac53-920a-414e-8288-db9a4fa4db4f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

A new GitHub Actions workflow for performance benchmarking was added. It checks out both the PR and main branches, builds each, runs the repomix CLI tool 5 times per branch with 2 warmup runs, calculates median runtimes, computes timing differences, and posts results as a PR comment.

Changes

Cohort / File(s)	Summary
Performance Benchmark Workflow .github/workflows/perf-benchmark.yml	New workflow that benchmarks repomix performance on each PR by comparing median runtime against main branch, with dual checkouts, builds, warmup runs, measured test runs, and automated PR comment posting.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related PRs

• ci(workflow): Add zizmor linter and harden GitHub Actions security #1212: Modifies GitHub Actions workflow security and checkout behavior with persist-credentials: false configuration, similar hardening approach used in the new benchmark workflow.
• ci(cache): Use setup-node npm cache #220: Adds npm caching to actions/setup-node in existing CI workflows, same caching strategy applied in the new benchmark workflow.

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main change: adding a performance benchmark workflow for PRs.
Description check	✅ Passed	The description is comprehensive and well-structured, covering how the workflow works, design decisions, and includes the required checklist, though checklist items are unchecked.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch ci/perf-benchmark

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

⚡ Performance Benchmark

Packing the repomix repository with node bin/repomix.cjs

Runner	PR	main	Diff
Ubuntu	2.48s (±0.07s)	2.50s (±0.01s)	-0.01s (-0.6%)
macOS	1.91s (±0.46s)	1.75s (±0.48s)	+0.16s (+8.9%)
Windows	3.23s (±0.30s)	3.04s (±0.03s)	+0.20s (+6.4%)

Details

• Warmup: 2 runs (discarded)
• Measurement: 10 runs (median)
• ±: IQR (Interquartile Range) — middle 50% of measurements spread
• Workflow run

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.18%. Comparing base (2be4744) to head (1d73cec).
⚠️ Report is 8 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1252   +/-   ##
=======================================
  Coverage   87.18%   87.18%           
=======================================
  Files         115      115           
  Lines        4324     4324           
  Branches     1002     1002           
=======================================
  Hits         3770     3770           
  Misses        554      554           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[cloudflare-workers-and-pages]

Deploying repomix with    Cloudflare Pages

Latest commit:	1d73cec
Status:	✅  Deploy successful!
Preview URL:	https://33779137.repomix.pages.dev
Branch Preview URL:	https://ci-perf-benchmark.repomix.pages.dev

View logs

[claude]

Code Review (Final): Previous reviews tracked 4 issues. 3/4 addressed. Still open: diff_pct N/A formatting (Line 163) - when main_ms=0, diff_pct is N/A but line 163 appends percent unconditionally. Resolved: download-artifact pinned to SHA, always() to !cancelled(), fork PR handling. LGTM with minor fix.

[claude]

Code Review (Follow-up): Still 1 open issue. diff_pct N/A formatting (line 163) - when main_ms=0, diff_pct is N/A but line 163 appends percent unconditionally producing N/A percent. Fix by using a diff_pct_fmt variable. Minor note: PR description says 5 measured runs but code does 10. LGTM with that fix.

[claude]

Code Review (Updated)

Previous reviews tracked 4 issues. 3 of 4 addressed — nice work on the iterations. The workflow is well-designed overall.

Still open

diff_pct N/A% formatting (Line 163)
When main_ms=0, diff_pct is set to N/A but the echo on line 163 unconditionally appends %, rendering N/A%. Fix by including % inside the if-block and using diff_pct directly in the output without a trailing %.

Minor suggestion (non-blocking)

Details

Multiple node -e invocations to parse the same JSON (Lines 141-147)

Five separate node -e calls parse the same JSON. A single invocation or using jq (pre-installed on all GitHub-hosted runners) would be cleaner and faster.

Premortem

Scenario	Likelihood	Mitigation
Shared-runner noise causes misleading diff	Medium	Median + IQR helps. Consider flagging only >5% regressions
PR adds many files, naturally increases pack time	Low	Document that benchmark packs the repo directory itself
!cancelled() runs comment job even if all benchmarks failed	Low	The file existence guard handles missing results gracefully

Verdict: LGTM with the minor N/A% fix.

Generated with Claude Code