Skip to content

chore: refresh benchmarks for v1.37.0 - #1206

Merged
jdx merged 1 commit into
mainfrom
bench-refresh
Aug 1, 2026
Merged

jdx merged 1 commit into
mainfrom
bench-refresh

Conversation

@jdx

@jdx jdx commented Aug 1, 2026 •

Copy link
Copy Markdown
Collaborator

🤖 Refreshed benchmarks

benchmarks/results.json was pinned to aube 1.36.0; the workspace is now 1.37.0. Re-ran mise run bench:bump on the hermetic Verdaccio registry (500mbit / 50ms per the mise task) and regenerated benchmarks/results.json plus the README BENCH_RATIOS block. The benchmark matrix pins aube's GVS mode via npm_config_enable_global_virtual_store=true|false (the auto-synthesized env alias for the enableGlobalVirtualStore setting), so GitHub Actions' inherited CI=true environment does not change whether aube runs with GVS enabled or disabled.

Benchmark changes

Versions:

  • aube: 1.36.0 -> 1.37.0
  • pnpm: 11.17.0 -> 11.18.0
  • yarn: 4.17.1 -> 4.18.0

Public ratios: warm installs vs Bun 1.9x -> 3x; warm installs vs pnpm 3x -> 8x; repeat test vs Bun 4x -> 5x; repeat test vs pnpm 33x -> 44x.

Benchmark aube bun deno pnpm npm yarn
Fresh install (warm cache) 728ms -> 299ms (-59%) 1366ms -> 1032ms (-24%) 1692ms -> 1424ms (-16%) 2442ms -> 2341ms (-4%) 6875ms -> 7230ms (+5%) 8624ms -> 8595ms (0%)
Fresh install (cold cache) 16854ms -> 14490ms (-14%) 5147ms -> 5064ms (-2%) 8248ms -> 7531ms (-9%) 13174ms -> 10670ms (-19%) 9034ms -> 9019ms (0%) 15877ms -> 11826ms (-26%)
npm install && npm run test 12ms -> 8ms (-33%) 45ms -> 43ms (-4%) 45ms -> 40ms (-11%) 398ms -> 351ms (-12%) 757ms -> 702ms (-7%) 1666ms -> 1270ms (-24%)

Review the numbers before merging — if anything looks wildly off vs. the previous release, investigate before landing. Hermetic proxy jitter or an npmjs uplink hiccup can occasionally skew results.

Once merged to main, the updated bench results flow into the next release-plz-pr run automatically.


Generated by the bench-refresh workflow.


Note

Low Risk
Documentation and benchmark data only; no product code or configuration behavior changes.

Overview
Re-runs the benchmark suite and updates published numbers for aube 1.37.0 (from 1.36.0), along with pnpm 11.18.0 and yarn 4.18.0 in benchmarks/results.json.

The README BENCH_RATIOS block is regenerated to match: warm installs are now cited as ~8x vs pnpm and ~3x vs Bun (up from ~3x and ~1.9x); repeat install && test is up to 44x vs pnpm and 5x vs Bun. Notable timing shifts include aube warm install 728ms → 299ms and repeat test 12ms → 8ms; other managers’ rows move modestly in the same file.

No runtime or install logic changes—only benchmark artifacts and marketing copy tied to the hermetic bench run.

Reviewed by Cursor Bugbot for commit 99b3a59. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by CodeRabbit

  • Documentation
    • Updated benchmark performance claims for warm installs and repeated test commands.
  • Benchmarks
    • Refreshed benchmark results with current tool versions, timestamps, and measurements across package managers.

@coderabbitai

coderabbitai Bot commented Aug 1, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: faba55b5-e26a-4468-9802-db717830111f

📥 Commits

Reviewing files that changed from the base of the PR and between 856fef1 and 99b3a59.

📒 Files selected for processing (2)
  • README.md
  • benchmarks/results.json

📝 Walkthrough

Walkthrough

The benchmark dataset now contains updated metadata, tool versions, and measurements. The README reports revised speedup ratios for warm installs and repeated test commands.

Changes

Benchmark refresh

Layer / File(s) Summary
Refresh benchmark datasets
benchmarks/results.json
Updated the timestamp, selected package-manager versions, and warm-cache, cold-cache, and install/test measurements.
Update documented ratios
README.md
Updated the documented speedup ratios for warm installs and repeated test commands.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • jdx/aube#937: Refreshes benchmark data and corresponding README performance claims.
  • jdx/aube#1080: Updates benchmark measurements, versions, timestamps, and README claims.
  • jdx/aube#1172: Refreshes the same benchmark data and README performance claims.

Suggested reviewers: mise-en-dev

Poem

A rabbit checked the numbers twice,
Warm caches hopped to newer heights.
Cold runs followed, neat and clear,
Test commands raced from ear to ear.
The README now tells the tale.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the benchmark refresh for version 1.37.0.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@greptile-apps

greptile-apps Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Refreshes benchmark artifacts for aube 1.37.0.

  • Updates benchmarked aube, pnpm, and yarn versions and regenerated timing statistics.
  • Updates README performance ratios to match the refreshed results.

Confidence Score: 5/5

The PR appears safe to merge because the refreshed benchmark data and README ratios are internally consistent.

The benchmark matrix remains complete, every displayed millisecond value and statistic matches the generation format, and the README ratios correctly reflect the committed results.

Important Files Changed

Filename Overview
README.md Updates generated benchmark-ratio prose; all four ratios agree with the refreshed benchmark values and repository rounding rules.
benchmarks/results.json Refreshes benchmark versions and measurements; rows are complete and values, means, deviations, and display text are internally consistent.

Reviews (1): Last reviewed commit: "chore: refresh benchmarks" | Re-trigger Greptile

@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown

Instruction counts

benchmark trend instructions Δ wall (min) Δ
graph ▅▅▅▃▃▆▃▅▅▄▂▆▆▆▂▁█▃▃▄ 16,714,929 → 16,715,279 +0.00% 4.13 → 3.20ms -22.66%
install ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁████ 114,223,965 → 114,201,883 -0.02% 36.17 → 27.86ms -22.97%
startup ▂▆▆▅▁█▅▂▂▂▁▂▆▃▁▂▇▅▆▂ 7,676,287 → 7,667,173 -0.12% 2.55 → 2.26ms -11.60%
tree ▁▁▁▇▇█▇███▇███▇▇▇▇▇▇ 17,797,255 → 17,797,777 +0.00% 4.09 → 3.37ms -17.60%

No instruction-count regression above 1%.

Only instruction counts gate. Wall clock is shown for context — on identical hardware it moves 4-20% run to run.

Measured by tak — instruction-counted CLI benchmarks, stored in this repository's git notes.

99b3a59eb727 vs 856fef1c70e1 · measured on this runner, not pushed to the history.

@jdx
jdx merged commit abe4c1c into main Aug 1, 2026
20 checks passed
@jdx
jdx deleted the bench-refresh branch August 1, 2026 16:28
@cursor cursor Bot mentioned this pull request Aug 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant