Skip to content
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
4818f3d
Add Foundry multi-version benchmarking suite
yash-atreya Jun 10, 2025
f43b5d9
Fix benchmark script JSON data extraction and table formatting
yash-atreya Jun 11, 2025
a8e28b1
parallel benchmarking
yash-atreya Jun 12, 2025
0a569d7
refac: mv to benches/ dir
yash-atreya Jun 12, 2025
09bfb57
feat: criterion benches
yash-atreya Jun 18, 2025
8a673c9
fix: install foundry versions at once
yash-atreya Jun 18, 2025
c1e52dd
nit
yash-atreya Jun 18, 2025
01f1af6
Merge branch 'master' into yash/foundry-benchmarking-suite
yash-atreya Jun 19, 2025
0dc8187
- setup benchmark repos in parallel
yash-atreya Jun 25, 2025
9f13124
feat: shell script to run benches
yash-atreya Jun 25, 2025
7d1d85a
feat: ci workflow, fix script
yash-atreya Jun 25, 2025
1a1587a
Merge branch 'yash/foundry-benchmarking-suite' into yash/criterion-be…
yash-atreya Jun 25, 2025
b667106
update readme
yash-atreya Jun 26, 2025
a396202
feat: enhance benchmarking suite with version flexibility
yash-atreya Jun 27, 2025
fcba242
latest bench
yash-atreya Jun 27, 2025
858d8d9
rm notes
yash-atreya Jun 27, 2025
44835cf
remove shell based bench suite
yash-atreya Jun 27, 2025
64baae2
feat: benches using criterion (#10805)
yash-atreya Jun 27, 2025
792c592
unified benchmarker -
yash-atreya Jun 30, 2025
b18141d
parallel bench
yash-atreya Jun 30, 2025
35d9861
refac
yash-atreya Jul 1, 2025
2392590
refac benchmark results table generation
yash-atreya Jul 1, 2025
4f896ae
cleanup main.rs
yash-atreya Jul 1, 2025
7edb40e
rm dep
yash-atreya Jul 1, 2025
d0a1525
cleanup main.rs
yash-atreya Jul 1, 2025
03b54fd
deser estimate
yash-atreya Jul 1, 2025
6c82d2f
nit
yash-atreya Jul 1, 2025
ca5f86c
cleanup CriterionResult type
yash-atreya Jul 1, 2025
6a010de
feat: specify repos via flag
yash-atreya Jul 1, 2025
c453bca
Merge branch 'yash/unified-benchmark-runner' into yash/foundry-benchm…
yash-atreya Jul 1, 2025
0a58ecc
nits
yash-atreya Jul 1, 2025
c395527
update bench ci and README
yash-atreya Jul 1, 2025
fdd948a
Merge branch 'master' into yash/foundry-benchmarking-suite
yash-atreya Jul 1, 2025
ad5ef6e
bench fuzz tests
yash-atreya Jul 1, 2025
afcf3ed
fmt
yash-atreya Jul 1, 2025
ed82d8a
license
yash-atreya Jul 1, 2025
4dac5c3
coverage bench
yash-atreya Jul 1, 2025
3887370
nits
yash-atreya Jul 1, 2025
cbd17d8
clippy
yash-atreya Jul 1, 2025
4fa2315
clippy
yash-atreya Jul 1, 2025
afc236b
separate benches into different jobs in CI
yash-atreya Jul 2, 2025
3c1c8cb
remove criterion
yash-atreya Jul 3, 2025
51d10b5
feat: hyperfine setup in foundry-bench
yash-atreya Jul 3, 2025
a79409d
forge version details: hash and date
yash-atreya Jul 3, 2025
354a8fe
run benches again - run cov with --ir-min
yash-atreya Jul 4, 2025
3666090
del
yash-atreya Jul 4, 2025
fcd2d82
bench in separate ci jobs
yash-atreya Jul 4, 2025
51225d6
move combine bench results logic to scripts
yash-atreya Jul 4, 2025
f23b52b
setup foundryup in ci
yash-atreya Jul 4, 2025
09a174d
setup foundryup fix
yash-atreya Jul 4, 2025
3bbc762
clippy
yash-atreya Jul 4, 2025
7335a6e
ci: run on foundry-runner
yash-atreya Jul 7, 2025
fcbaddc
ci: don't use wget
yash-atreya Jul 7, 2025
34e0c7e
ci: add build essential
yash-atreya Jul 7, 2025
0d3aa3b
ci: nodejs and npm
yash-atreya Jul 7, 2025
8228529
install hyperfine for each job
yash-atreya Jul 7, 2025
eef0a5f
fix
yash-atreya Jul 7, 2025
3a7fab7
install deps script
yash-atreya Jul 8, 2025
2b327cd
add benchmark-setup, using setup-node action, remove redundant files
yash-atreya Jul 8, 2025
541cd48
fix
yash-atreya Jul 8, 2025
834be01
fix
yash-atreya Jul 8, 2025
f314721
checkout repo
yash-atreya Jul 8, 2025
51efad3
nits
yash-atreya Jul 9, 2025
cc7977f
Merge branch 'master' into yash/foundry-benchmarking-suite
yash-atreya Jul 9, 2025
bdca555
nit
yash-atreya Jul 9, 2025
236da6b
fix
yash-atreya Jul 11, 2025
88e45c7
show forge test result in top comment
yash-atreya Jul 11, 2025
24ce0a2
force foundry install
yash-atreya Jul 11, 2025
1f4cfdf
fix bench comment aggregation
yash-atreya Jul 11, 2025
178f3c5
nit
yash-atreya Jul 11, 2025
c47af23
fix
yash-atreya Jul 11, 2025
a2252fc
feat: create PR for manual runs, else commit in the PR itself.
yash-atreya Jul 11, 2025
609ba81
fix
yash-atreya Jul 11, 2025
42a553a
fetch and pull
yash-atreya Jul 11, 2025
c4b707f
chore(`benches`): update benchmark results
actions-user Jul 11, 2025
7472110
fix
yash-atreya Jul 11, 2025
4acd4f7
chore(`benches`): update benchmark results
actions-user Jul 11, 2025
7b49712
Merge branch 'master' into yash/foundry-benchmarking-suite
yash-atreya Jul 11, 2025
81cef23
chore(`benches`): update benchmark results
actions-user Jul 11, 2025
8de8549
don't run on PRs
yash-atreya Jul 11, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,6 @@ out.json
.vscode
.claude
CLAUDE.md
# Benchmark directories
benches/benchmark_repos/
benches/benchmark_results/
84 changes: 84 additions & 0 deletions benches/LATEST.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# Forge Benchmarking Results

**Generated on:** Wed 18 Jun 2025 17:46:19 BST
**Hyperfine Version:** hyperfine 1.19.0
**Foundry Versions Tested:** stable nightly
**Repositories Tested:** ithacaxyz-account solady

## Summary

This report contains comprehensive benchmarking results comparing different Foundry versions across multiple projects.
The following benchmarks were performed:

1. **forge test - Running the test suite (5 runs, 1 warmup)**
2. **forge build (no cache) - Clean build without cache (5 runs, cache cleaned after each run)**
3. **forge build (with cache) - Build with warm cache (5 runs, 1 warmup)**

---

## Performance Comparison Tables

### forge test

Mean execution time in seconds (lower is better):

| Project | stable (s) | nightly (s) |
|------|--------:|--------:|
| **ithacaxyz-account** | 5.791 | 3.875 |
| **solady** | 3.578 | 2.966 |


### forge build no cache

Mean execution time in seconds (lower is better):

| Project | stable (s) | nightly (s) |
|------|--------:|--------:|
| **ithacaxyz-account** | 19.079 | 16.177 |
| **solady** | 27.408 | 22.745 |


### forge build with cache

Mean execution time in seconds (lower is better):

| Project | stable (s) | nightly (s) |
|------|--------:|--------:|
| **ithacaxyz-account** | 0.181 | 0.158 |
| **solady** | 0.091 | 0.103 |


## Foundry Version Details

### stable

```
forge Version: 1.2.3-stable
```

### nightly

```
forge Version: 1.2.3-nightly
```


## Notes

- All benchmarks were run with hyperfine in parallel mode
- **forge test - Running the test suite (5 runs, 1 warmup)**
- **forge build (no cache) - Clean build without cache (5 runs, cache cleaned after each run)**
- **forge build (with cache) - Build with warm cache (5 runs, 1 warmup)**
- Results show mean execution time in seconds
- N/A indicates benchmark failed or data unavailable

## System Information

- **OS:** Darwin
- **Architecture:** arm64
- **Date:** Wed 18 Jun 2025 17:46:19 BST

## Raw Data

Raw JSON benchmark data is available in: `/Users/yash/dev/paradigm/foundry-rs/foundry/benches/benchmark_results/json_20250618_174101`

Loading