Skip to content

chore: refresh benchmarks for v1.37.0 #1211

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jdx merged 1 commit into from
Aug 2, 2026
Merged

chore: refresh benchmarks for v1.37.0 #1211

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
## Why Try It

<!-- BENCH_RATIOS:START -->
**[Fast installs](https://aube.jdx.dev/benchmarks).** Warm installs are about 8x faster than pnpm and about 3x faster than Bun in the current benchmarks. Repeat test commands run up to 44x faster than pnpm and up to 5x faster than Bun.
**[Fast installs](https://aube.jdx.dev/benchmarks).** Warm installs are about 7x faster than pnpm and about 3x faster than Bun in the current benchmarks. Repeat test commands run up to 48x faster than pnpm and up to 4x faster than Bun.
<!-- BENCH_RATIOS:END -->

**[Existing lockfiles](https://aube.jdx.dev/package-manager/lockfiles).** Reads and writes `pnpm-lock.yaml`, `package-lock.json`, `npm-shrinkwrap.json`, `yarn.lock`, and `bun.lock` in place.
Expand Down
218 changes: 109 additions & 109 deletions benchmarks/results.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"updated": "2026-08-01T06:28:32.819Z",
"updated": "2026-08-02T00:37:53.969Z",
"unit": "ms",
"managers": [
"aube",
Expand All @@ -23,167 +23,167 @@
"key": "gvs-warm",
"label": "Fresh install (warm cache)",
"values": {
"aube": 299,
"bun": 1032,
"deno": 1424,
"pnpm": 2341,
"npm": 7230,
"yarn": 8595
"aube": 203,
"bun": 519,
"deno": 461,
"pnpm": 1417,
"npm": 3590,
"yarn": 2847
},
"stats": {
"aube": {
"text": "0.299s ± 0.021s",
"mean": 0.29919213854,
"stddev": 0.021158920989155044,
"min": 0.28591308724000003,
"max": 0.35758727124
"text": "0.203s ± 0.003s",
"mean": 0.20293146162000003,
"stddev": 0.00285646178191689,
"min": 0.19694160122,
"max": 0.20665504322
},
"bun": {
"text": "1.032s ± 0.152s",
"mean": 1.0315839468,
"stddev": 0.15205145407346204,
"min": 0.8932390507000001,
"max": 1.2738837557
"text": "0.519s ± 0.040s",
"mean": 0.51922345586,
"stddev": 0.039580410213259316,
"min": 0.49339830166000004,
"max": 0.60066409666
},
"deno": {
"text": "1.424s ± 0.095s",
"mean": 1.4240965527,
"stddev": 0.09527042264321739,
"min": 1.2927187841,
"max": 1.5847718310999999
"text": "0.461s ± 0.040s",
"mean": 0.46102898332,
"stddev": 0.03959394998491349,
"min": 0.40314492462,
"max": 0.5295995426200001
},
"pnpm": {
"text": "2.341s ± 0.064s",
"mean": 2.3412827357,
"stddev": 0.06368309500279636,
"min": 2.2379883707,
"max": 2.3968555717
"text": "1.417s ± 0.032s",
"mean": 1.41698886006,
"stddev": 0.032270623684346335,
"min": 1.3850337526599998,
"max": 1.45109257366
},
"npm": {
"text": "7.230s ± 0.084s",
"mean": 7.229782048690001,
"stddev": 0.08429430549286024,
"min": 7.13401214644,
"max": 7.31834338244
"text": "3.590s ± 0.022s",
"mean": 3.59044892206,
"stddev": 0.022112033218122645,
"min": 3.56835632806,
"max": 3.6201401740600003
},
"yarn": {
"text": "8.595s ± 0.464s",
"mean": 8.59480242745,
"stddev": 0.46385134712748094,
"min": 8.0858468537,
"max": 9.066324119699999
"text": "2.847s ± 0.032s",
"mean": 2.8473375586899996,
"stddev": 0.03222145420253788,
"min": 2.80170297994,
"max": 2.87283727294
}
}
},
{
"key": "gvs-cold",
"label": "Fresh install (cold cache)",
"values": {
"aube": 14490,
"bun": 5064,
"deno": 7531,
"pnpm": 10670,
"npm": 9019,
"yarn": 11826
"aube": 6712,
"bun": 1448,
"deno": 1658,
"pnpm": 6649,
"npm": 5689,
"yarn": 6229
},
"stats": {
"aube": {
"text": "14.490s ± 0.149s",
"mean": 14.4895393858,
"stddev": 0.14858330659677607,
"min": 14.2098521182,
"max": 14.704034914200001
"text": "6.712s ± 0.115s",
"mean": 6.711541839260001,
"stddev": 0.11531064403065654,
"min": 6.598416060060001,
"max": 6.952401146060001
},
"bun": {
"text": "5.064s ± 0.442s",
"mean": 5.063803260559999,
"stddev": 0.44244186163601684,
"min": 4.59301794356,
"max": 5.90023339456
"text": "1.448s ± 0.032s",
"mean": 1.4482807701399998,
"stddev": 0.03173685604461604,
"min": 1.40050465664,
"max": 1.4858291936399999
},
"deno": {
"text": "7.531s ± 0.345s",
"mean": 7.53053008942,
"stddev": 0.3454735031899585,
"min": 6.99150810152,
"max": 8.31398546552
"text": "1.658s ± 0.080s",
"mean": 1.657637708,
"stddev": 0.08042016722296035,
"min": 1.5599847371000002,
"max": 1.8078081451
},
"pnpm": {
"text": "10.670s ± 0.508s",
"mean": 10.670484293399998,
"stddev": 0.5080510394960686,
"min": 10.025206554199999,
"max": 11.3454253272
"text": "6.649s ± 0.120s",
"mean": 6.6492609505199995,
"stddev": 0.11954274297116929,
"min": 6.47819847112,
"max": 6.77868802112
},
"npm": {
"text": "9.019s ± 0.099s",
"mean": 9.01888841329,
"stddev": 0.09946545632005284,
"min": 8.92484662154,
"max": 9.127589460540001
"text": "5.689s ± 0.020s",
"mean": 5.688687332250001,
"stddev": 0.01955056282605294,
"min": 5.6712272415000005,
"max": 5.716088296500001
},
"yarn": {
"text": "11.826s ± 0.142s",
"mean": 11.82639240329,
"stddev": 0.14217192590030683,
"min": 11.64412466704,
"max": 11.99152539804
"text": "6.229s ± 0.158s",
"mean": 6.22947097494,
"stddev": 0.1578732250564702,
"min": 6.131162020940001,
"max": 6.46445025694
}
}
},
Comment on lines +82 to 133

@coderabbitai coderabbitai Bot Aug 2, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Map benchmark sources before inspecting the benchmark implementation.
ast-grep outline benchmarks --items all

# Locate cache handling, registry setup, GVS configuration, commands, and sampling.
rg -n -C 5 \
  -g '*.{mts,ts,js,json}' \
  'gvs-cold|install-test|Verdaccio|GVS|gvs|npm install|npm run test|cache|warmup|iterations|samples' \
  benchmarks

# Check that displayed millisecond values match the recorded means.
python - <<'PY'
import json

with open("benchmarks/results.json", encoding="utf-8") as f:
    results = json.load(f)

for row in results["rows"]:
    print(f'\n{row["key"]}')
    for tool, value_ms in row["values"].items():
        mean_ms = row["stats"][tool]["mean"] * 1000
        print(f"  {tool}: value={value_ms}ms, mean={mean_ms:.3f}ms")
PY

Repository: jdx/aube

Length of output: 20351

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the benchmark generator and related input files/workflow metadata to identify cache/reset and row provenance.
wc -l benchmarks/generate-results.js benchmarks/results.json benchmarks/validate-results.mts
sed -n '1,220p' benchmarks/generate-results.js
sed -n '1,120p' benchmarks/validate-results.mts

# Search repository-wide for gvs benchmark commands and cache reset mechanisms.
rg -n -C 3 \
  'gvs-cold|gvs-warm|VXZ|cache|cache:|cache.*aube|aube.*cache|npm run .*gvs|pnpm .*gvs|bun .*gvs|deno .*gvs|NODE\(\)|GVS|install-test|generate-phase-results|phase.*jsonl|refresh' \
  . \
  -g '!benchmarks/results.json' \
  -g '!node_modules' \
  -g '!dist' \
  -g '!build'

Repository: jdx/aube

Length of output: 50365

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== repo tracked relevant files =="
git ls-files | rg '(^|/)(benchmarks|package.json|pnpm-workspace.yaml|README.md|\.github/workflows/.*ya?ml)$' || true

echo
echo "== package manifests around scripts =="
for f in package.json benchmarks/package.json pnpm-workspace.yaml; do
  if [ -f "$f" ]; then
    echo "--- $f ---"
    sed -n '1,220p' "$f"
  fi
done

echo
echo "== precise benchmark command/cache references =="
rg -n -C 3 \
  --glob '!node_modules/**' \
  --glob '!test/registry/storage/**' \
  --glob '!benchmarks/results.json' \
  'gvs-cold|gvs-warm|VXZ|aube.*cache|cache.*aube|npm run .*bench|node .*generate-phase|node .*summarize|Verdaccio|verdaccio|proxy|install-test|npm install.*test|npm run test' \
  .

Repository: jdx/aube

Length of output: 50365

Don’t publish the unexplained refresh results without proving the row setup.

The gvs-cold refresh shows aube at 6712ms while the warm row is 203ms, and install-test shows aube at 7ms. Publish only after these rows are reproduced with the exact same cache-reset procedure, GVS mode, Verdaccio registry, test workload, and measurement command used by every tool; include the raw samples or command output with the refresh.

🧰 Tools
🪛 OpenGrep (1.26.0)

[ERROR] 99-99: Possible credit card number (PAN) detected in source code. Credit card numbers should never be hardcoded or stored in source files. Use a secrets manager or tokenization service instead.

(coderabbit.pii.credit-card-number)

[ERROR] 102-102: Possible credit card number (PAN) detected in source code. Credit card numbers should never be hardcoded or stored in source files. Use a secrets manager or tokenization service instead.

(coderabbit.pii.credit-card-number)

[ERROR] 108-108: Possible credit card number (PAN) detected in source code. Credit card numbers should never be hardcoded or stored in source files. Use a secrets manager or tokenization service instead.

(coderabbit.pii.credit-card-number)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@benchmarks/results.json` around lines 82 - 133, Reproduce the unexplained
aube timings in the gvs-cold and install-test rows using the exact shared
cache-reset procedure, GVS mode, Verdaccio registry, workload, and measurement
command used for all tools. Verify the results and include the raw samples or
command output before publishing the refreshed benchmarks/results.json data.

{
"key": "install-test",
"label": "npm install && npm run test",
"values": {
"aube": 8,
"bun": 43,
"deno": 40,
"pnpm": 351,
"npm": 702,
"yarn": 1270
"aube": 7,
"bun": 27,
"deno": 104,
"pnpm": 337,
"npm": 629,
"yarn": 1194
},
"stats": {
"aube": {
"text": "0.008s ± 0.001s",
"mean": 0.007989321800000001,
"stddev": 0.0010427972993753336,
"min": 0.007070051400000001,
"max": 0.010637869400000001
"text": "0.007s ± 0.000s",
"mean": 0.007040900500000001,
"stddev": 0.0004346772015380942,
"min": 0.0064894864,
"max": 0.0079875514
},
"bun": {
"text": "0.043s ± 0.001s",
"mean": 0.0430879775,
"stddev": 0.0012301007168983985,
"min": 0.04072197130000001,
"max": 0.044783846300000006
"text": "0.027s ± 0.001s",
"mean": 0.0271550692,
"stddev": 0.0006072728754385282,
"min": 0.0262839888,
"max": 0.027895719800000003
},
"deno": {
"text": "0.040s ± 0.003s",
"mean": 0.0398718457,
"stddev": 0.002657353545266187,
"min": 0.0366187298,
"max": 0.0461166818
"text": "0.104s ± 0.031s",
"mean": 0.10425082904,
"stddev": 0.031295927026236105,
"min": 0.06208616854,
"max": 0.17296669954000002
Comment on lines 160 to +165

@greptile-apps greptile-apps Bot Aug 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Noisy Deno repeat benchmark

The refreshed Deno repeat-test result reports a 104ms mean with a 31ms standard deviation and samples spanning 62–173ms, while the previous mean was 40ms. Confirm or rerun this outlier before publishing it so readers receive a stable comparison.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Claude Code

},
"pnpm": {
"text": "0.351s ± 0.010s",
"mean": 0.35095162136,
"stddev": 0.009614769523489471,
"min": 0.34033806656000004,
"max": 0.36589875556000007
"text": "0.337s ± 0.010s",
"mean": 0.33720814418,
"stddev": 0.0102765069643388,
"min": 0.32644017098,
"max": 0.35025715398
},
"npm": {
"text": "0.702s ± 0.023s",
"mean": 0.7021093705900001,
"stddev": 0.022891461326229576,
"min": 0.68024090334,
"max": 0.73339330934
"text": "0.629s ± 0.003s",
"mean": 0.62910446298,
"stddev": 0.003045106612038214,
"min": 0.62500962148,
"max": 0.63236864748
},
"yarn": {
"text": "1.270s ± 0.042s",
"mean": 1.2703657977499998,
"stddev": 0.04242801591506702,
"min": 1.2332348665,
"max": 1.3311796635
"text": "1.194s ± 0.008s",
"mean": 1.1940985661999999,
"stddev": 0.007656738184830295,
"min": 1.1872299956999999,
"max": 1.2009591467
}
}
}
Expand Down
Loading