Skip to content

Conversation

@teor2345
Copy link
Member

This PR improves the runtime-benchmark.sh, making it more portable and easier to use, and automatically fetching the list of benchmarks from the runtime lib.rs files.

Bug fixes

Script Improvements

  • uses typical shell script syntax: capitals for variables, lowercase for functions and commands
  • avoids echoing long runs of commands by using set -x instead of echo
  • allows builds to be skipped
  • checks that the script is run from the correct directory
  • logs what it is doing
  • makes future updates easier by matching the construct_runtime! pallet order
  • automatically picks up new benchmarks from the runtime lib.rs files

Generated file improvements

  • the script automatically fixes incorrectly generated module names
  • this PR updates comments in generated Rust weights files with the command-line used by the script

Close #3582
Parts of #3587

Future Work

We could move some extension benchmark weights into the runtime, I've left TODOs in the script.

Code contributor checklist:

@teor2345 teor2345 self-assigned this Jun 17, 2025
@teor2345 teor2345 added the improvement it is already working, but can be better label Jun 17, 2025
@teor2345 teor2345 requested a review from nazar-pc as a code owner June 17, 2025 06:19
@teor2345 teor2345 added the execution Subspace execution label Jun 17, 2025
@teor2345 teor2345 enabled auto-merge June 17, 2025 06:20
Copy link
Contributor

@NingLin-P NingLin-P left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work!!! Thank you!

@teor2345 teor2345 added this pull request to the merge queue Jun 17, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 17, 2025
@teor2345 teor2345 added this pull request to the merge queue Jun 17, 2025
@teor2345
Copy link
Member Author

Failed in the merge queue, possibly due to some GitHub internal infra issues:

Windows clippy:

Run arduino/setup-protoc@c65c819552d16ad3c9b72d9dfd5ba5237b9c906b
  with:
    repo-token: ***
    version: 23.x
    include-pre-releases: false
  env:
    CARGO_INCREMENTAL: 0
    CARGO_TERM_COLOR: always
    RUSTFLAGS: -C strip=symbols
    RUST_BACKTRACE: full
Error: Error: No server is currently available to service your request. Sorry about that. Please try resubmitting your request and contact us if the problem persists

https://github.com/autonomys/subspace/actions/runs/15716424422/job/44287153916#step:7:28

macOS tests:

Run bash --noprofile --norc "${GITHUB_ACTION_PATH:?}/main.sh"
##[debug]/opt/homebrew/bin/bash --noprofile --norc -e -o pipefail /Users/hetzner/actions-runner/_work/_temp/ae36c7c0-081d-4568-8e87-3bb1ae071144.sh
info: host platform: aarch64_macos
info: installing cargo-nextest@latest
info: downloading https://github.com/nextest-rs/nextest/releases/download/cargo-nextest-0.9.72/cargo-nextest-0.9.72-universal-apple-darwin.tar.gz
curl: (56) The requested URL returned error: 500
curl: (56) The requested URL returned error: 500
curl: (56) The requested URL returned error: 500
curl: (56) The requested URL returned error: 500
curl: (56) The requested URL returned error: 500
curl: (56) The requested URL returned error: 500
curl: (56) The requested URL returned error: 500
curl: (56) The requested URL returned error: 500
curl: (56) The requested URL returned error: 500
curl: (56) The requested URL returned error: 500
curl: (56) The requested URL returned error: 500
Error: Process completed with exit code 56.

https://github.com/autonomys/subspace/actions/runs/15716424422/job/44287153941#step:9:91

@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 18, 2025
@teor2345
Copy link
Member Author

Failed again due to "the operation was canceled", I'm following up on Slack.

@teor2345 teor2345 added this pull request to the merge queue Jun 18, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 18, 2025
@teor2345 teor2345 added this pull request to the merge queue Jun 18, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 18, 2025
@teor2345 teor2345 added this pull request to the merge queue Jun 18, 2025
Merged via the queue into main with commit 6609138 Jun 18, 2025
10 checks passed
@teor2345 teor2345 deleted the auto-benchmark-script-pallets branch June 18, 2025 03:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

execution Subspace execution improvement it is already working, but can be better

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Keep benchmark script pallet lists in sync with the runtime

3 participants