Repository navigation
chore: refresh benchmarks for v1.24.0 - #937
Conversation
Greptile SummaryRefreshes the benchmark data from aube 1.23.0 to 1.24.0 and regenerates the
Confidence Score: 3/5The README ratio math is correct, but the aube cold-cache result represents a 2.24× regression that should be explained or ruled out before this lands in main. The warm-cache and repeat-test numbers look clean and consistent across all tools. The cold-cache aube figure, however, moved from 7.6 s to 17.0 s — a jump structurally different from any other measurement in the run (all other tools changed by ≤14%). The PR's own language asks reviewers to investigate results that look wildly off before merging; this one does. benchmarks/results.json — specifically the gvs-cold aube entry; README.md is fine. Important Files Changed
Reviews (1): Last reviewed commit: "chore: refresh benchmarks" | Re-trigger Greptile |
| "key": "gvs-cold", | ||
| "label": "Fresh install (cold cache)", | ||
| "values": { | ||
| "aube": 7592, | ||
| "bun": 5702, | ||
| "deno": 8036, | ||
| "pnpm": 12623, | ||
| "npm": 10331, | ||
| "yarn": 15713 | ||
| "aube": 17028, | ||
| "bun": 6525, | ||
| "deno": 8818, | ||
| "pnpm": 10921, | ||
| "npm": 11452, | ||
| "yarn": 15834 | ||
| }, | ||
| "stats": { |
There was a problem hiding this comment.
Aube cold-cache regression (+124%) far exceeds normal jitter
The gvs-cold aube result jumped from 7,592 ms to 17,028 ms — a 2.24× slowdown. Every other tool in the same run moved by ≤14% in either direction, and the new aube stddev (±728 ms, min 15.9 s / max 18.3 s) rules out a single-outlier fluke: the whole distribution shifted. Aube now trails every competitor in cold-cache installs except yarn. The PR description itself says "if anything looks wildly off vs. the previous release, investigate before landing" — this appears to qualify.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Central YAML (base), Organization UI (inherited) Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughBenchmark results are refreshed for aube 1.24.0: ChangesBenchmark Refresh for aube 1.24.0
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
🤖 Refreshed benchmarks
benchmarks/results.jsonwas pinned to aube1.23.0; the workspace is now1.24.0. Re-ranmise run bench:bumpon the hermetic Verdaccio registry (500mbit / 50ms per the mise task) and regeneratedbenchmarks/results.jsonplus the READMEBENCH_RATIOSblock. The benchmark matrix pins aube's GVS mode vianpm_config_enable_global_virtual_store=true|false(the auto-synthesized env alias for theenableGlobalVirtualStoresetting), so GitHub Actions' inheritedCI=trueenvironment does not change whether aube runs with GVS enabled or disabled.Benchmark changes
Versions:
Public ratios: warm installs vs Bun 2x -> 2x; warm installs vs pnpm 5x -> 3x; repeat test vs Bun 4x -> 5x; repeat test vs pnpm 34x -> 31x.
Review the numbers before merging — if anything looks wildly off vs. the previous release, investigate before landing. Hermetic proxy jitter or an npmjs uplink hiccup can occasionally skew results.
Once merged to main, the updated bench results flow into the next
release-plz-prrun automatically.Generated by the
bench-refreshworkflow.Note
Low Risk
Documentation and benchmark artifact updates only; no runtime or install logic changes.
Overview
Re-runs the hermetic benchmark suite and pins
benchmarks/results.jsonto aube 1.24.0 (from 1.23.0), updating timings for warm/cold fresh installs and repeatinstall && testacross all package managers.Regenerates the README
BENCH_RATIOSmarketing copy to match the new numbers: warm install vs pnpm 5x → 3x, repeat test vs pnpm 34x → 31x, repeat test vs Bun 4x → 5x (warm vs Bun stays ~2x). Notable in the raw data: aube warm install is slower than before (+36%) and cold install is much slower (+124%), which drives the softer pnpm ratio.Reviewed by Cursor Bugbot for commit 6f1fa14. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by CodeRabbit
Documentation
Chores