Repository navigation
chore: refresh benchmarks for v1.28.0 - #1050
Conversation
![release-plz[bot]](https://github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
📝 WalkthroughWalkthroughBenchmark metadata and results for three scenarios were refreshed, and the README’s fast-install performance claims were updated to match the new figures. ChangesBenchmark refresh
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Greptile SummaryThis PR refreshes the benchmark data from aube 1.27.0 to 1.28.0 (and deno 2.9.2 → 2.9.3) and updates the README headline ratios to match.
Confidence Score: 3/5The README and benchmark JSON are internally consistent, but the aube warm-cache regression (+133%) and the widened stddev suggest the numbers may not be stable enough to publish as the definitive 1.28.0 baseline. The aube warm-cache install time more than doubled (263 ms → 612 ms) with a stddev that grew from ±18 ms to ±132 ms, while Bun's number almost halved in the same run. The swing is large enough that the benchmark harness may have experienced interference during the aube runs — the PR's own note says to investigate before landing if numbers look wildly off, and this qualifies. benchmarks/results.json — specifically the gvs-warm block for aube, where both the mean and stddev changed substantially enough to warrant re-running before treating these as the canonical 1.28.0 numbers. Important Files Changed
Reviews (1): Last reviewed commit: "chore: refresh benchmarks" | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| "bun": 1391, | ||
| "deno": 2375, | ||
| "pnpm": 3321, | ||
| "npm": 7798, |
There was a problem hiding this comment.
The gvs-warm value for aube jumped from 263 ms to 612 ms (+133%) while Bun's number almost halved (2924 ms → 1391 ms). The combined effect flips the headline ratio from ~11x to ~2x — a dramatic swing that the PR description acknowledges but doesn't explain. The stddev for aube also widened from ±18 ms to ±132 ms, suggesting high variance rather than a stable measurement. The PR itself asks reviewers to investigate before merging if numbers look wildly off — this seems like a case where that investigation should happen: was GVS forced on/off consistently? Were all runs on the same fixture? Is there a known regression in aube 1.28.0's warm-cache path?
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@benchmarks/results.json`:
- Around line 26-77: Investigate the warm-cache benchmark regression represented
in benchmarks/results.json lines 26-77 and compare it with the cold-cache
results in lines 78-133; identify why Aube’s warm-cache time increased from
263ms to 612ms while cold-cache performance did not show the same change, then
correct the underlying benchmark or implementation issue and regenerate the
affected results. The sibling range requires no direct change unless the
investigation identifies a shared root cause.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Central YAML (base), Organization UI (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 3fefd6f7-cfec-405e-b4c4-4041ba5a8cfc
📒 Files selected for processing (2)
README.mdbenchmarks/results.json
| "aube": 612, | ||
| "bun": 1391, | ||
| "deno": 2375, | ||
| "pnpm": 3321, | ||
| "npm": 7798, | ||
| "yarn": 11457 | ||
| }, | ||
| "stats": { | ||
| "aube": { | ||
| "text": "0.263s ± 0.018s", | ||
| "mean": 0.26340452584, | ||
| "stddev": 0.017752895232022173, | ||
| "min": 0.24065752724, | ||
| "max": 0.29142403224 | ||
| "text": "0.612s ± 0.132s", | ||
| "mean": 0.61201968926, | ||
| "stddev": 0.13248000991220288, | ||
| "min": 0.45671489126000003, | ||
| "max": 0.8189740602600001 | ||
| }, | ||
| "bun": { | ||
| "text": "2.924s ± 0.261s", | ||
| "mean": 2.9239087655, | ||
| "stddev": 0.26094548627684094, | ||
| "min": 2.6482493493, | ||
| "max": 3.4492324233 | ||
| "text": "1.391s ± 0.104s", | ||
| "mean": 1.3907503655000002, | ||
| "stddev": 0.10385844872313059, | ||
| "min": 1.1841856580999999, | ||
| "max": 1.5300708681 | ||
| }, | ||
| "deno": { | ||
| "text": "1.292s ± 0.158s", | ||
| "mean": 1.29186679204, | ||
| "stddev": 0.15795715667232396, | ||
| "min": 1.0006102911399999, | ||
| "max": 1.5238904391399999 | ||
| "text": "2.375s ± 0.088s", | ||
| "mean": 2.3749851078799997, | ||
| "stddev": 0.08809400631893297, | ||
| "min": 2.24374544148, | ||
| "max": 2.5185086664800003 | ||
| }, | ||
| "pnpm": { | ||
| "text": "3.419s ± 0.627s", | ||
| "mean": 3.4189730159, | ||
| "stddev": 0.6274974055754965, | ||
| "min": 2.6780925231, | ||
| "max": 4.058610031100001 | ||
| "text": "3.321s ± 0.017s", | ||
| "mean": 3.3206561146399998, | ||
| "stddev": 0.017153938253707438, | ||
| "min": 3.29070552804, | ||
| "max": 3.33398594204 | ||
| }, | ||
| "npm": { | ||
| "text": "7.825s ± 0.387s", | ||
| "mean": 7.825020734210001, | ||
| "stddev": 0.3868661734265498, | ||
| "min": 7.451966108460001, | ||
| "max": 8.30574446846 | ||
| "text": "7.798s ± 0.747s", | ||
| "mean": 7.797686414699999, | ||
| "stddev": 0.7465282042047758, | ||
| "min": 7.1093607907, | ||
| "max": 8.6362445597 | ||
| }, | ||
| "yarn": { | ||
| "text": "9.920s ± 0.214s", | ||
| "mean": 9.91973897257, | ||
| "stddev": 0.21406660036591305, | ||
| "min": 9.70148654982, | ||
| "max": 10.17380474682 | ||
| "text": "11.457s ± 0.646s", | ||
| "mean": 11.457067808840002, | ||
| "stddev": 0.6458587587427252, | ||
| "min": 10.78923128234, | ||
| "max": 12.29368329134 | ||
| } | ||
| } | ||
| }, |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map the benchmark file and nearby context first.
wc -l benchmarks/results.json
sed -n '1,220p' benchmarks/results.json
# Look for any references to these benchmark keys or expected baselines.
rg -n '"gvs-warm"|"gvs-cold"|enable_global_virtual_store|1\.28\.0|warm cache|cold cache' .Repository: jdx/aube
Length of output: 13555
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Show the focused diff for the benchmark file.
git diff -- benchmarks/results.json
# Inspect the previous version of the benchmark file if available.
if git rev-parse --verify HEAD^ >/dev/null 2>&1; then
echo '--- PREVIOUS VERSION ---'
git show HEAD^:benchmarks/results.json | sed -n '1,220p'
fiRepository: jdx/aube
Length of output: 5040
Investigate the warm-cache regression in benchmarks/results.json#L23-L77. Aube’s fresh-install warm-cache time more than doubled (263ms → 612ms); the cold-cache row doesn’t show the same drop.
🧰 Tools
🪛 OpenGrep (1.25.0)
[ERROR] 53-53: Possible credit card number (PAN) detected in source code. Credit card numbers should never be hardcoded or stored in source files. Use a secrets manager or tokenization service instead.
(coderabbit.pii.credit-card-number)
📍 Affects 1 file
benchmarks/results.json#L26-L77(this comment)benchmarks/results.json#L78-L133
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@benchmarks/results.json` around lines 26 - 77, Investigate the warm-cache
benchmark regression represented in benchmarks/results.json lines 26-77 and
compare it with the cold-cache results in lines 78-133; identify why Aube’s
warm-cache time increased from 263ms to 612ms while cold-cache performance did
not show the same change, then correct the underlying benchmark or
implementation issue and regenerate the affected results. The sibling range
requires no direct change unless the investigation identifies a shared root
cause.
🤖 Refreshed benchmarks
benchmarks/results.jsonwas pinned to aube1.27.0; the workspace is now1.28.0. Re-ranmise run bench:bumpon the hermetic Verdaccio registry (500mbit / 50ms per the mise task) and regeneratedbenchmarks/results.jsonplus the READMEBENCH_RATIOSblock. The benchmark matrix pins aube's GVS mode vianpm_config_enable_global_virtual_store=true|false(the auto-synthesized env alias for theenableGlobalVirtualStoresetting), so GitHub Actions' inheritedCI=trueenvironment does not change whether aube runs with GVS enabled or disabled.Benchmark changes
Versions:
Public ratios: warm installs vs Bun 11x -> 2x; warm installs vs pnpm 13x -> 5x; repeat test vs Bun 6x -> 6x; repeat test vs pnpm 43x -> 39x.
Review the numbers before merging — if anything looks wildly off vs. the previous release, investigate before landing. Hermetic proxy jitter or an npmjs uplink hiccup can occasionally skew results.
Once merged to main, the updated bench results flow into the next
release-plz-prrun automatically.Generated by the
bench-refreshworkflow.Note
Low Risk
Documentation and benchmark artifact updates only; no runtime or install logic changes.
Overview
Refreshes published benchmark numbers after re-running the hermetic
bench:bumpmatrix for aube 1.28.0 (was 1.27.0) and deno 2.9.3.benchmarks/results.jsongets new timings, stats, andupdatedmetadata for warm/cold fresh installs and repeatinstall && test. The README BENCH_RATIOS block is regenerated so marketing copy matches the JSON—warm-install claims shift from ~13×/11× vs pnpm/Bun to ~5× vs pnpm and ~2× vs Bun, and repeat-test vs pnpm moves from 43× to 39×.Reviewed by Cursor Bugbot for commit 48ed254. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by CodeRabbit
Documentation
Chores