Repository navigation
chore: refresh benchmarks for v1.37.0 - #1211
Conversation
![release-plz[bot]](https://github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
📝 WalkthroughWalkthroughThe benchmark timestamp and measurements were refreshed in ChangesBenchmark refresh
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Greptile SummaryRefreshes the public benchmark data for aube 1.37.0.
Confidence Score: 4/5The PR appears safe to merge after confirming or rerunning the unusually noisy Deno repeat-test measurement. The generated values, displayed statistics, and README ratios are internally consistent, but Deno’s repeat-test sample has substantial variance and reports a large counter-trend regression that may make the published comparison misleading. Files Needing Attention: benchmarks/results.json Important Files Changed
Reviews (1): Last reviewed commit: "chore: refresh benchmarks" | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| "deno": { | ||
| "text": "0.040s ± 0.003s", | ||
| "mean": 0.0398718457, | ||
| "stddev": 0.002657353545266187, | ||
| "min": 0.0366187298, | ||
| "max": 0.0461166818 | ||
| "text": "0.104s ± 0.031s", | ||
| "mean": 0.10425082904, | ||
| "stddev": 0.031295927026236105, | ||
| "min": 0.06208616854, | ||
| "max": 0.17296669954000002 |
There was a problem hiding this comment.
The refreshed Deno repeat-test result reports a 104ms mean with a 31ms standard deviation and samples spanning 62–173ms, while the previous mean was 40ms. Confirm or rerun this outlier before publishing it so readers receive a stable comparison.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@benchmarks/results.json`:
- Around line 82-133: Reproduce the unexplained aube timings in the gvs-cold and
install-test rows using the exact shared cache-reset procedure, GVS mode,
Verdaccio registry, workload, and measurement command used for all tools. Verify
the results and include the raw samples or command output before publishing the
refreshed benchmarks/results.json data.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Central YAML (base), Organization UI (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: d7dbc254-3efe-4dce-a61b-9c53906da7f5
📒 Files selected for processing (2)
README.mdbenchmarks/results.json
| "aube": 6712, | ||
| "bun": 1448, | ||
| "deno": 1658, | ||
| "pnpm": 6649, | ||
| "npm": 5689, | ||
| "yarn": 6229 | ||
| }, | ||
| "stats": { | ||
| "aube": { | ||
| "text": "14.490s ± 0.149s", | ||
| "mean": 14.4895393858, | ||
| "stddev": 0.14858330659677607, | ||
| "min": 14.2098521182, | ||
| "max": 14.704034914200001 | ||
| "text": "6.712s ± 0.115s", | ||
| "mean": 6.711541839260001, | ||
| "stddev": 0.11531064403065654, | ||
| "min": 6.598416060060001, | ||
| "max": 6.952401146060001 | ||
| }, | ||
| "bun": { | ||
| "text": "5.064s ± 0.442s", | ||
| "mean": 5.063803260559999, | ||
| "stddev": 0.44244186163601684, | ||
| "min": 4.59301794356, | ||
| "max": 5.90023339456 | ||
| "text": "1.448s ± 0.032s", | ||
| "mean": 1.4482807701399998, | ||
| "stddev": 0.03173685604461604, | ||
| "min": 1.40050465664, | ||
| "max": 1.4858291936399999 | ||
| }, | ||
| "deno": { | ||
| "text": "7.531s ± 0.345s", | ||
| "mean": 7.53053008942, | ||
| "stddev": 0.3454735031899585, | ||
| "min": 6.99150810152, | ||
| "max": 8.31398546552 | ||
| "text": "1.658s ± 0.080s", | ||
| "mean": 1.657637708, | ||
| "stddev": 0.08042016722296035, | ||
| "min": 1.5599847371000002, | ||
| "max": 1.8078081451 | ||
| }, | ||
| "pnpm": { | ||
| "text": "10.670s ± 0.508s", | ||
| "mean": 10.670484293399998, | ||
| "stddev": 0.5080510394960686, | ||
| "min": 10.025206554199999, | ||
| "max": 11.3454253272 | ||
| "text": "6.649s ± 0.120s", | ||
| "mean": 6.6492609505199995, | ||
| "stddev": 0.11954274297116929, | ||
| "min": 6.47819847112, | ||
| "max": 6.77868802112 | ||
| }, | ||
| "npm": { | ||
| "text": "9.019s ± 0.099s", | ||
| "mean": 9.01888841329, | ||
| "stddev": 0.09946545632005284, | ||
| "min": 8.92484662154, | ||
| "max": 9.127589460540001 | ||
| "text": "5.689s ± 0.020s", | ||
| "mean": 5.688687332250001, | ||
| "stddev": 0.01955056282605294, | ||
| "min": 5.6712272415000005, | ||
| "max": 5.716088296500001 | ||
| }, | ||
| "yarn": { | ||
| "text": "11.826s ± 0.142s", | ||
| "mean": 11.82639240329, | ||
| "stddev": 0.14217192590030683, | ||
| "min": 11.64412466704, | ||
| "max": 11.99152539804 | ||
| "text": "6.229s ± 0.158s", | ||
| "mean": 6.22947097494, | ||
| "stddev": 0.1578732250564702, | ||
| "min": 6.131162020940001, | ||
| "max": 6.46445025694 | ||
| } | ||
| } | ||
| }, |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map benchmark sources before inspecting the benchmark implementation.
ast-grep outline benchmarks --items all
# Locate cache handling, registry setup, GVS configuration, commands, and sampling.
rg -n -C 5 \
-g '*.{mts,ts,js,json}' \
'gvs-cold|install-test|Verdaccio|GVS|gvs|npm install|npm run test|cache|warmup|iterations|samples' \
benchmarks
# Check that displayed millisecond values match the recorded means.
python - <<'PY'
import json
with open("benchmarks/results.json", encoding="utf-8") as f:
results = json.load(f)
for row in results["rows"]:
print(f'\n{row["key"]}')
for tool, value_ms in row["values"].items():
mean_ms = row["stats"][tool]["mean"] * 1000
print(f" {tool}: value={value_ms}ms, mean={mean_ms:.3f}ms")
PYRepository: jdx/aube
Length of output: 20351
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the benchmark generator and related input files/workflow metadata to identify cache/reset and row provenance.
wc -l benchmarks/generate-results.js benchmarks/results.json benchmarks/validate-results.mts
sed -n '1,220p' benchmarks/generate-results.js
sed -n '1,120p' benchmarks/validate-results.mts
# Search repository-wide for gvs benchmark commands and cache reset mechanisms.
rg -n -C 3 \
'gvs-cold|gvs-warm|VXZ|cache|cache:|cache.*aube|aube.*cache|npm run .*gvs|pnpm .*gvs|bun .*gvs|deno .*gvs|NODE\(\)|GVS|install-test|generate-phase-results|phase.*jsonl|refresh' \
. \
-g '!benchmarks/results.json' \
-g '!node_modules' \
-g '!dist' \
-g '!build'Repository: jdx/aube
Length of output: 50365
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== repo tracked relevant files =="
git ls-files | rg '(^|/)(benchmarks|package.json|pnpm-workspace.yaml|README.md|\.github/workflows/.*ya?ml)$' || true
echo
echo "== package manifests around scripts =="
for f in package.json benchmarks/package.json pnpm-workspace.yaml; do
if [ -f "$f" ]; then
echo "--- $f ---"
sed -n '1,220p' "$f"
fi
done
echo
echo "== precise benchmark command/cache references =="
rg -n -C 3 \
--glob '!node_modules/**' \
--glob '!test/registry/storage/**' \
--glob '!benchmarks/results.json' \
'gvs-cold|gvs-warm|VXZ|aube.*cache|cache.*aube|npm run .*bench|node .*generate-phase|node .*summarize|Verdaccio|verdaccio|proxy|install-test|npm install.*test|npm run test' \
.Repository: jdx/aube
Length of output: 50365
Don’t publish the unexplained refresh results without proving the row setup.
The gvs-cold refresh shows aube at 6712ms while the warm row is 203ms, and install-test shows aube at 7ms. Publish only after these rows are reproduced with the exact same cache-reset procedure, GVS mode, Verdaccio registry, test workload, and measurement command used by every tool; include the raw samples or command output with the refresh.
🧰 Tools
🪛 OpenGrep (1.26.0)
[ERROR] 99-99: Possible credit card number (PAN) detected in source code. Credit card numbers should never be hardcoded or stored in source files. Use a secrets manager or tokenization service instead.
(coderabbit.pii.credit-card-number)
[ERROR] 102-102: Possible credit card number (PAN) detected in source code. Credit card numbers should never be hardcoded or stored in source files. Use a secrets manager or tokenization service instead.
(coderabbit.pii.credit-card-number)
[ERROR] 108-108: Possible credit card number (PAN) detected in source code. Credit card numbers should never be hardcoded or stored in source files. Use a secrets manager or tokenization service instead.
(coderabbit.pii.credit-card-number)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@benchmarks/results.json` around lines 82 - 133, Reproduce the unexplained
aube timings in the gvs-cold and install-test rows using the exact shared
cache-reset procedure, GVS mode, Verdaccio registry, workload, and measurement
command used for all tools. Verify the results and include the raw samples or
command output before publishing the refreshed benchmarks/results.json data.
🤖 Refreshed benchmarks
benchmarks/results.jsonwas pinned to aube1.37.0; the workspace is now1.37.0. Re-ranmise run bench:bumpon the hermetic Verdaccio registry (500mbit / 50ms per the mise task) and regeneratedbenchmarks/results.jsonplus the READMEBENCH_RATIOSblock. The benchmark matrix pins aube's GVS mode vianpm_config_enable_global_virtual_store=true|false(the auto-synthesized env alias for theenableGlobalVirtualStoresetting), so GitHub Actions' inheritedCI=trueenvironment does not change whether aube runs with GVS enabled or disabled.Benchmark changes
Public ratios: warm installs vs Bun 3x -> 3x; warm installs vs pnpm 8x -> 7x; repeat test vs Bun 5x -> 4x; repeat test vs pnpm 44x -> 48x.
Review the numbers before merging — if anything looks wildly off vs. the previous release, investigate before landing. Hermetic proxy jitter or an npmjs uplink hiccup can occasionally skew results.
Once merged to main, the updated bench results flow into the next
release-plz-prrun automatically.Generated by the
bench-refreshworkflow.Note
Low Risk
Documentation and benchmark artifacts only; no runtime or package-manager code changes.
Overview
Re-runs the hermetic benchmark suite for aube 1.37.0 and commits the new numbers in
benchmarks/results.json, including an updatedupdatedtimestamp and pinned tool versions.The README BENCH_RATIOS block is regenerated from those results: warm install vs pnpm 8x → 7x, repeat test vs pnpm 44x → 48x, and repeat test vs Bun 5x → 4x (warm vs Bun stays ~3x). Timings across warm/cold install and
install && testscenarios shift for all managers in the JSON (e.g. aube warm install 299ms → 203ms).Reviewed by Cursor Bugbot for commit 60712de. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by CodeRabbit
Documentation
Benchmarks