Skip to content

Conversation

@RodrigoVillar
Copy link
Contributor

@RodrigoVillar RodrigoVillar commented Dec 2, 2025

Why this should be merged

As part of #4508, this PR removes usage of go bench from the reexecution test. This offers the following benefits:

  • Although the reexecution test is housed within a go benchmark, we only run it once, which is not idiomatic as go benchmarks are meant to be iterated over multiple times.
  • This sets up the foundation to reintroduce multiple metrics (e.g. block acceptance time, block verification time, etc.) to our dashboard as adding additional metrics is as simple as calling addResult() on the new benchmark tool.

How this works

  • Removes usage of *testing.B
  • Output of the reexecution statistics file is handled by the test itself rather than ./scripts/benchmark_cchain_range.sh
  • Updates GAB (github-action-benchmark) action to use custom benchmarks rather than go bench

How this was tested

CI + ran task reexecute-cchain-range START_BLOCK=1 END_BLOCK=100_000 BENCHMARK_OUTPUT_FILE=<INSERT_CUSTOM_PATH_HERE> locally and got the following:

[
  {
    "name": "BenchmarkReexecuteRange/[1,100000]-Config-default-Runner-dev - mgas/s",
    "value": "254.90228109633716",
    "unit": "mgas/s"
  }
]

Furthermore, results are logged to the terminal as follows:

[12-09|13:06:35.049] INFO c-chain-reexecution c/vm_reexecute.go:637 BenchmarkReexecuteRange/[1,100000]-Config-default-Runner-dev {"result": "262.43886323313035 mgas/s"}

To test that benchmark results display correctly on the GAB dashboard, I created a sample benchmark (https://github.com/RodrigoVillar/sample-reexecution-custom-benchmark/blob/main/main.go) which is a copy-and-paste of the benchmarking code in this PR and plotted some points into my own dashboard. The results of the sample benchmark can be seen below:

Screenshot 2025-12-04 at 09 35 04

Attached is a link to the sample benchmark dashboard: https://rodrigovillar.github.io/sample-reexecution-custom-benchmark/dev/bench/

Need to be documented in RELEASES.md?

No

TODOs after merging

  • Prune gh-pages branch of data points which use go bench

@RodrigoVillar RodrigoVillar force-pushed the rodrigo/custom-benchmarks branch from bd29254 to 1935adf Compare December 3, 2025 23:36
@RodrigoVillar RodrigoVillar self-assigned this Dec 3, 2025
Base automatically changed from rodrigo/block-export to master December 9, 2025 16:49
@RodrigoVillar RodrigoVillar marked this pull request as ready for review December 9, 2025 18:19
Copilot AI review requested due to automatic review settings December 9, 2025 18:19
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR removes the use of go bench from the C-Chain reexecution test, transitioning it from a Go benchmark to a standalone executable. The change enables more flexible metric reporting and aligns the test with its actual usage pattern (single execution rather than iterative benchmarking).

Key Changes:

  • Replaced *testing.B with a custom benchmarkTool for metric collection and reporting
  • Changed the package from vm to main and converted the benchmark function to a standalone executable
  • Introduced tests.TestContext throughout to manage test lifecycle and cleanup operations

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/reexecute/db.go Updated CreateBlockChanFromLevelDB signature to use tests.TestContext instead of testing.TB and removed cleanup parameter
tests/reexecute/c/vm_reexecute.go Converted from benchmark test to main executable; added custom benchmarkTool for metric reporting and JSON output
tests/reexecute/blockexport/main.go Removed unused cleanup parameter from CreateBlockChanFromLevelDB call
scripts/benchmark_cchain_range.sh Changed from go test to go run and updated output handling to use JSON file directly
.github/actions/c-chain-reexecution-benchmark/action.yml Updated benchmark tool from 'go' to 'customBiggerIsBetter' and changed output file extension to .json

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

else
eval "$cmd"
fi
eval "$cmd"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we remove this and just run the command now?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, removed eval here: 955273e


cmd="go test -timeout=0 -v -benchtime=1x -bench=BenchmarkReexecuteRange -run=^$ github.com/ava-labs/avalanchego/tests/reexecute/c \
cmd="go run github.com/ava-labs/avalanchego/tests/reexecute/c \
--benchmark-output-file=\"${BENCHMARK_OUTPUT_FILE}\" \
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How should this be handled if the benchmark output file is not defined?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In vm_reexecute.go, we don't write to a file if the benchmark output filepath is empty, but you're right that if a user currently doesn't set BENCHMARK_OUTPUT_FILE, that we'll have:

go run github.com/ava-labs/avalanchego/tests/reexecute/c --benchmark-output-file= --current-state-dir=...`

Changed it so that if a user doesn't define BENCHMARK_OUTPUT_FILE, then --benchmark-output-file isn't set: f682b20

@github-project-automation github-project-automation bot moved this to In Progress 🏗️ in avalanchego Dec 11, 2025
@aaronbuchwald aaronbuchwald added this pull request to the merge queue Dec 16, 2025
Merged via the queue into master with commit c609692 Dec 16, 2025
63 of 65 checks passed
@aaronbuchwald aaronbuchwald deleted the rodrigo/custom-benchmarks branch December 16, 2025 08:44
@github-project-automation github-project-automation bot moved this from In Progress 🏗️ to Done 🎉 in avalanchego Dec 16, 2025
RodrigoVillar added a commit that referenced this pull request Dec 16, 2025
ci: add chaos test job

chore: nits

chore: add nix installation step

chore: configure AWS credentials

chore: remove inputs

chore: add perms

chore: nit

chore: nit

chore: extend wait time

chore: nits

chore: nit

chore: nit

chore: Create shared `evm` module (#4690)

chore(reexecute/c): remove go bench from benchmark (#4640)

chore: nits

fix: MAX_WAIT_TIME

chore: nit

chore: extend wait times

chore: log errs

chore: stdout tail

chore: nit

chore: nits

chore: nits

ci: improve workflow

chore: nits

chore: nits

chore: nits

chore: lint
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done 🎉

Development

Successfully merging this pull request may close these issues.

4 participants