Skip to content

feat: Add CPU scaling benchmark script for remote execution#16918

Merged
johnathan79717 merged 3 commits intomerge-train/barretenbergfrom
jh/bench-cpu-scaling-remote
Sep 11, 2025
Merged

feat: Add CPU scaling benchmark script for remote execution#16918
johnathan79717 merged 3 commits intomerge-train/barretenbergfrom
jh/bench-cpu-scaling-remote

Conversation

@johnathan79717
Copy link
Contributor

@johnathan79717 johnathan79717 commented Sep 10, 2025 •
edited
Loading

Summary

Adds a new benchmarking script bench_cpu_scaling_remote.sh that measures CPU scaling performance of BB_BENCH entries on remote EC2 instances.

Features

  • Remote execution: Uses benchmark_remote.sh infrastructure with proper locking mechanism
  • CPU scaling analysis: Runs benchmarks with different HARDWARE_CONCURRENCY values (1, 2, 4, 8, etc.)
  • JSON output: Uses --bench_out flag to generate JSON output for accurate timing extraction
  • Performance metrics: Calculates speedup and parallel efficiency for each CPU count
  • Real-time output: Shows benchmark progress during execution with tee for transparency

Implementation Details

  • Automatically appends --bench_out flag to commands to generate JSON output
  • Retrieves JSON files from remote machine after each benchmark run
  • Extracts timing data from JSON format for more reliable parsing
  • Shows JSON content for debugging when extraction encounters issues

Usage

# Set required environment variables
export BB_SSH_KEY='-i /path/to/key.pem'
export BB_SSH_INSTANCE='user@ec2-instance.amazonaws.com'
export BB_SSH_CPP_PATH='/path/to/barretenberg/cpp'

# Run CPU scaling benchmark
./bench_cpu_scaling_remote.sh "ClientIvcProve" "./bb prove --scheme client_ivc --output_path /tmp --ivc_inputs_path ~/input.msgpack" "1,2,4,8"

# Note: --bench_out flag is automatically added to the command

Example Output

ClientIVC::prove CPU Scaling Results:
- 1 CPU: 27.96s (baseline)
- 2 CPUs: 15.77s (1.77x speedup, 88.5% efficiency)
- 4 CPUs: 9.23s (3.03x speedup, 75.8% efficiency)
- 8 CPUs: 5.85s (4.78x speedup, 59.8% efficiency)

@johnathan79717
feat: Add CPU scaling benchmark script for remote execution
034c492 
Adds bench_cpu_scaling_remote.sh script that:
- Runs benchmarks with different HARDWARE_CONCURRENCY values
- Uses benchmark_remote.sh for proper remote execution with locking
- Extracts timing data from hierarchical BB_BENCH output
- Calculates speedup and parallel efficiency metrics
- Generates visual ASCII charts and CSV results
- Shows real-time output during benchmark execution
@johnathan79717 johnathan79717 self-assigned this Sep 10, 2025
@johnathan79717
feat: Update bench_cpu_scaling_remote.sh to use --bench_out for JSON …
11ea714 
…timing extraction

Changes:
- Use --bench_out flag to generate JSON output from bb
- Extract timing data from JSON file instead of parsing hierarchical output
- Retrieve JSON file from remote after benchmark execution
- Add fallback to hierarchical output if JSON extraction fails
- Show JSON content for debugging when extraction fails

This provides more reliable and accurate timing extraction as JSON format
is more stable and easier to parse than the hierarchical text output.
@johnathan79717
refactor: Move bench_cpu_scaling_remote.sh to scripts directory
76c44b7 
Move the benchmark script to barretenberg/cpp/scripts/ to keep it with
other benchmark and utility scripts in the project.
@johnathan79717 johnathan79717 merged commit 4d85947 into merge-train/barretenberg Sep 11, 2025
6 checks passed
@johnathan79717 johnathan79717 deleted the jh/bench-cpu-scaling-remote branch September 11, 2025 10:31
@AztecBot AztecBot mentioned this pull request Sep 11, 2025
Merged
github-merge-queue bot pushed a commit that referenced this pull request Sep 11, 2025
@ludamad
feat: merge-train/barretenberg (#16928)
486c3c7 
BEGIN_COMMIT_OVERRIDE
fix: Origin Tags edgecase (#16921)
chore: cycle group cleanup #2 (#16876)
chore: civc tidy 3 (#16671)
refactor(bb): optimize batch_mul_with_endomorphism (#16905)
feat: check op queue wires are zero past minicircuit in Translator
(#16858)
feat: Add CPU scaling benchmark script for remote execution (#16918)
fix: Add free witness tag to field constructor (#16827)
fix(bb): darwin build (#16957)
END_COMMIT_OVERRIDE
Umarb97 pushed a commit to Umarb97/aztec-packages that referenced this pull request Sep 16, 2025
@johnathan79717 @Umarb97
feat: Add CPU scaling benchmark script for remote execution (AztecPro…
90137ff 
…tocol#16918)

## Summary

Adds a new benchmarking script `bench_cpu_scaling_remote.sh` that
measures CPU scaling performance of BB_BENCH entries on remote EC2
instances.

## Features

- **Remote execution**: Uses `benchmark_remote.sh` infrastructure with
proper locking mechanism
- **CPU scaling analysis**: Runs benchmarks with different
`HARDWARE_CONCURRENCY` values (1, 2, 4, 8, etc.)
- **JSON output**: Uses `--bench_out` flag to generate JSON output for
accurate timing extraction
- **Performance metrics**: Calculates speedup and parallel efficiency
for each CPU count
- **Real-time output**: Shows benchmark progress during execution with
`tee` for transparency

## Implementation Details

- Automatically appends `--bench_out` flag to commands to generate JSON
output
- Retrieves JSON files from remote machine after each benchmark run
- Extracts timing data from JSON format for more reliable parsing
- Shows JSON content for debugging when extraction encounters issues

## Usage

```bash
# Set required environment variables
export BB_SSH_KEY='-i /path/to/key.pem'
export BB_SSH_INSTANCE='user@ec2-instance.amazonaws.com'
export BB_SSH_CPP_PATH='/path/to/barretenberg/cpp'

# Run CPU scaling benchmark
./bench_cpu_scaling_remote.sh "ClientIvcProve" "./bb prove --scheme client_ivc --output_path /tmp --ivc_inputs_path ~/input.msgpack" "1,2,4,8"

# Note: --bench_out flag is automatically added to the command
```

## Example Output

```
ClientIVC::prove CPU Scaling Results:
- 1 CPU: 27.96s (baseline)
- 2 CPUs: 15.77s (1.77x speedup, 88.5% efficiency)
- 4 CPUs: 9.23s (3.03x speedup, 75.8% efficiency)
- 8 CPUs: 5.85s (4.78x speedup, 59.8% efficiency)
```
mralj pushed a commit that referenced this pull request Oct 13, 2025
@johnathan79717 @mralj
feat: Add CPU scaling benchmark script for remote execution (#16918)
9047cae 
## Summary

Adds a new benchmarking script `bench_cpu_scaling_remote.sh` that
measures CPU scaling performance of BB_BENCH entries on remote EC2
instances.

## Features

- **Remote execution**: Uses `benchmark_remote.sh` infrastructure with
proper locking mechanism
- **CPU scaling analysis**: Runs benchmarks with different
`HARDWARE_CONCURRENCY` values (1, 2, 4, 8, etc.)
- **JSON output**: Uses `--bench_out` flag to generate JSON output for
accurate timing extraction
- **Performance metrics**: Calculates speedup and parallel efficiency
for each CPU count
- **Real-time output**: Shows benchmark progress during execution with
`tee` for transparency

## Implementation Details

- Automatically appends `--bench_out` flag to commands to generate JSON
output
- Retrieves JSON files from remote machine after each benchmark run
- Extracts timing data from JSON format for more reliable parsing
- Shows JSON content for debugging when extraction encounters issues

## Usage

```bash
# Set required environment variables
export BB_SSH_KEY='-i /path/to/key.pem'
export BB_SSH_INSTANCE='user@ec2-instance.amazonaws.com'
export BB_SSH_CPP_PATH='/path/to/barretenberg/cpp'

# Run CPU scaling benchmark
./bench_cpu_scaling_remote.sh "ClientIvcProve" "./bb prove --scheme client_ivc --output_path /tmp --ivc_inputs_path ~/input.msgpack" "1,2,4,8"

# Note: --bench_out flag is automatically added to the command
```

## Example Output

```
ClientIVC::prove CPU Scaling Results:
- 1 CPU: 27.96s (baseline)
- 2 CPUs: 15.77s (1.77x speedup, 88.5% efficiency)
- 4 CPUs: 9.23s (3.03x speedup, 75.8% efficiency)
- 8 CPUs: 5.85s (4.78x speedup, 59.8% efficiency)
```
ludamad pushed a commit that referenced this pull request Dec 16, 2025
@johnathan79717 @ludamad
feat: Add CPU scaling benchmark script for remote execution (#16918)
01ea60d 
## Summary

Adds a new benchmarking script `bench_cpu_scaling_remote.sh` that
measures CPU scaling performance of BB_BENCH entries on remote EC2
instances.

## Features

- **Remote execution**: Uses `benchmark_remote.sh` infrastructure with
proper locking mechanism
- **CPU scaling analysis**: Runs benchmarks with different
`HARDWARE_CONCURRENCY` values (1, 2, 4, 8, etc.)
- **JSON output**: Uses `--bench_out` flag to generate JSON output for
accurate timing extraction
- **Performance metrics**: Calculates speedup and parallel efficiency
for each CPU count
- **Real-time output**: Shows benchmark progress during execution with
`tee` for transparency

## Implementation Details

- Automatically appends `--bench_out` flag to commands to generate JSON
output
- Retrieves JSON files from remote machine after each benchmark run
- Extracts timing data from JSON format for more reliable parsing
- Shows JSON content for debugging when extraction encounters issues

## Usage

```bash
# Set required environment variables
export BB_SSH_KEY='-i /path/to/key.pem'
export BB_SSH_INSTANCE='user@ec2-instance.amazonaws.com'
export BB_SSH_CPP_PATH='/path/to/barretenberg/cpp'

# Run CPU scaling benchmark
./bench_cpu_scaling_remote.sh "ClientIvcProve" "./bb prove --scheme client_ivc --output_path /tmp --ivc_inputs_path ~/input.msgpack" "1,2,4,8"

# Note: --bench_out flag is automatically added to the command
```

## Example Output

```
ClientIVC::prove CPU Scaling Results:
- 1 CPU: 27.96s (baseline)
- 2 CPUs: 15.77s (1.77x speedup, 88.5% efficiency)
- 4 CPUs: 9.23s (3.03x speedup, 75.8% efficiency)
- 8 CPUs: 5.85s (4.78x speedup, 59.8% efficiency)
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ludamad ludamad ludamad approved these changes

Assignees

@johnathan79717 johnathan79717

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@johnathan79717 @ludamad