Skip to content

chore: refresh benchmarks for v1.24.0 - #937

Merged
jdx merged 1 commit into
mainfrom
bench-refresh
Jun 24, 2026
Merged

jdx merged 1 commit into
mainfrom
bench-refresh

Conversation

@jdx

@jdx jdx commented Jun 24, 2026 •

Copy link
Copy Markdown
Collaborator

🤖 Refreshed benchmarks

benchmarks/results.json was pinned to aube 1.23.0; the workspace is now 1.24.0. Re-ran mise run bench:bump on the hermetic Verdaccio registry (500mbit / 50ms per the mise task) and regenerated benchmarks/results.json plus the README BENCH_RATIOS block. The benchmark matrix pins aube's GVS mode via npm_config_enable_global_virtual_store=true|false (the auto-synthesized env alias for the enableGlobalVirtualStore setting), so GitHub Actions' inherited CI=true environment does not change whether aube runs with GVS enabled or disabled.

Benchmark changes

Versions:

  • aube: 1.23.0 -> 1.24.0

Public ratios: warm installs vs Bun 2x -> 2x; warm installs vs pnpm 5x -> 3x; repeat test vs Bun 4x -> 5x; repeat test vs pnpm 34x -> 31x.

Benchmark aube bun deno pnpm npm yarn
Fresh install (warm cache) 669ms -> 912ms (+36%) 1669ms -> 1876ms (+12%) 1439ms -> 1691ms (+18%) 3236ms -> 3015ms (-7%) 9027ms -> 8159ms (-10%) 10051ms -> 11498ms (+14%)
Fresh install (cold cache) 7592ms -> 17028ms (+124%) 5702ms -> 6525ms (+14%) 8036ms -> 8818ms (+10%) 12623ms -> 10921ms (-13%) 10331ms -> 11452ms (+11%) 15713ms -> 15834ms (+1%)
npm install && npm run test 12ms -> 13ms (+8%) 47ms -> 70ms (+49%) 80ms -> 65ms (-19%) 406ms -> 399ms (-2%) 836ms -> 859ms (+3%) 1967ms -> 1886ms (-4%)

Review the numbers before merging — if anything looks wildly off vs. the previous release, investigate before landing. Hermetic proxy jitter or an npmjs uplink hiccup can occasionally skew results.

Once merged to main, the updated bench results flow into the next release-plz-pr run automatically.


Generated by the bench-refresh workflow.


Note

Low Risk
Documentation and benchmark artifact updates only; no runtime or install logic changes.

Overview
Re-runs the hermetic benchmark suite and pins benchmarks/results.json to aube 1.24.0 (from 1.23.0), updating timings for warm/cold fresh installs and repeat install && test across all package managers.

Regenerates the README BENCH_RATIOS marketing copy to match the new numbers: warm install vs pnpm 5x → 3x, repeat test vs pnpm 34x → 31x, repeat test vs Bun 4x → 5x (warm vs Bun stays ~2x). Notable in the raw data: aube warm install is slower than before (+36%) and cold install is much slower (+124%), which drives the softer pnpm ratio.

Reviewed by Cursor Bugbot for commit 6f1fa14. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by CodeRabbit

  • Documentation

    • Updated benchmark claims in the README to reflect the latest measured install speedups.
  • Chores

    • Refreshed benchmark results data and timestamps.
    • Updated reported version information and performance metrics across the latest install test runs.

@greptile-apps

greptile-apps Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Refreshes the benchmark data from aube 1.23.0 to 1.24.0 and regenerates the README.md BENCH_RATIOS block to match. The README ratios (3×/2× warm installs, 31×/5× repeat test vs pnpm/Bun) all check out against the new results.json values.

  • benchmarks/results.json: all six tools updated; warm-cache and repeat-test numbers move within the ±14% range seen in normal runs across tools, but the cold-cache aube result jumps from 7,592 ms to 17,028 ms (+124%) — a 2.24× slowdown — while every other tool in the same run stays within ±14%.
  • README.md: four BENCH_RATIOS figures updated correctly and match the new data.

Confidence Score: 3/5

The README ratio math is correct, but the aube cold-cache result represents a 2.24× regression that should be explained or ruled out before this lands in main.

The warm-cache and repeat-test numbers look clean and consistent across all tools. The cold-cache aube figure, however, moved from 7.6 s to 17.0 s — a jump structurally different from any other measurement in the run (all other tools changed by ≤14%). The PR's own language asks reviewers to investigate results that look wildly off before merging; this one does.

benchmarks/results.json — specifically the gvs-cold aube entry; README.md is fine.

Important Files Changed

Filename Overview
benchmarks/results.json Benchmark data updated to aube v1.24.0; all metrics consistent except aube cold-cache which jumped +124% (7,592ms → 17,028ms), well outside jitter range and now slower than all competitors except yarn.
README.md BENCH_RATIOS block updated; all four stated ratios (3x/2x warm, 31x/5x repeat) match the new results.json values to within rounding.

Fix All in Claude Code

Reviews (1): Last reviewed commit: "chore: refresh benchmarks" | Re-trigger Greptile

Comment thread benchmarks/results.json
Comment on lines 79 to 89
"key": "gvs-cold",
"label": "Fresh install (cold cache)",
"values": {
"aube": 7592,
"bun": 5702,
"deno": 8036,
"pnpm": 12623,
"npm": 10331,
"yarn": 15713
"aube": 17028,
"bun": 6525,
"deno": 8818,
"pnpm": 10921,
"npm": 11452,
"yarn": 15834
},
"stats": {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Aube cold-cache regression (+124%) far exceeds normal jitter

The gvs-cold aube result jumped from 7,592 ms to 17,028 ms — a 2.24× slowdown. Every other tool in the same run moved by ≤14% in either direction, and the new aube stddev (±728 ms, min 15.9 s / max 18.3 s) rules out a single-outlier fluke: the whole distribution shifted. Aube now trails every competitor in cold-cache installs except yarn. The PR description itself says "if anything looks wildly off vs. the previous release, investigate before landing" — this appears to qualify.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Claude Code

@coderabbitai

coderabbitai Bot commented Jun 24, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 5c184c78-17f2-4da1-a96f-3da91d5b1db3

📥 Commits

Reviewing files that changed from the base of the PR and between 54f5cd9 and 6f1fa14.

📒 Files selected for processing (2)
  • README.md
  • benchmarks/results.json

📝 Walkthrough

Walkthrough

Benchmark results are refreshed for aube 1.24.0: benchmarks/results.json receives a new timestamp, the aube version entry, and updated per-manager stats for gvs-warm, gvs-cold, and install-test. The README "Fast installs" speedup ratios versus pnpm and Bun are adjusted to match.

Changes

Benchmark Refresh for aube 1.24.0

Layer / File(s) Summary
Benchmark results data refresh
benchmarks/results.json
updated timestamp set to 2026-06-24T06:52:41.785Z, aube version bumped from 1.23.0 to 1.24.0, and all per-manager values/stats fields (text, mean, stddev, min, max) replaced for gvs-warm, gvs-cold, and install-test across all six package managers.
README speedup ratio adjustments
README.md
"Fast installs" benchmark text updated: warm-install speedup vs pnpm changed from ~5x to ~3x, repeat-test vs pnpm from up to 34x to up to 31x, and repeat-test vs Bun from up to 4x to up to 5x.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Possibly related PRs

  • jdx/aube#922: Refreshes the same benchmark artifacts — README "Fast installs" ratios and benchmarks/results.json timestamps, version entries, and rows statistics.
  • jdx/aube#907: Regenerates the same benchmark data and README ratio claims with newly measured stats, identical in scope to this PR.
  • jdx/aube#890: Updates the same README fast-install figures and re-runs benchmarks/results.json rows for gvs-warm, gvs-cold, and install-test.

Poem

🐇 Hop hop, the numbers race anew,
Warm installs shimmer — a snappier hue!
Three times past pnpm, thirty-one strong,
Bun gets five-times'd in the benchmark song.
The JSON refreshed, the README gleams bright,
A new version bunny bounding to light! 🌟

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: refreshing benchmark data for version 1.24.0.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@jdx
jdx merged commit 239accf into main Jun 24, 2026
18 checks passed
@jdx
jdx deleted the bench-refresh branch June 24, 2026 12:11
@cursor cursor Bot mentioned this pull request Jun 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant