perf(bindings): drop TS BLS comparison benches and report benchmarks on PRs

[5t4rg4z3r]

Motivation

This repository should not be concerned with blst-ts benchmarks. Measure this repo's own code + add a way to see it on PRs.

Description

bindings/perf/blst.test.ts benchmarked @chainsafe/blst next to the lodestar-z bindings, which pulled the old typescript BLS implementation into the dependency tree only to produce comparison numbers. That comparison now lives in spiral-ladder/lodestar-bls-benchmarks.

Todo

• Store benchmark history in S3 instead of the Actions cache #551 - History uses --local in the Actions cache rather than S3, since this repository has no bucket or credentials. A pull request restores the entry written by the last push to main; only a push to main saves a new one, so pull requests cannot move the baseline.

Verification

• pnpm benchmark with a shortened budget: 30/30 benchmarks pass.
• pnpm lint: 10 warnings, all pre-existing in bindings/src/index.d.ts, none in changed files.
• pnpm install re-run so pnpm-lock.yaml matches the dependency change; the lockfile diff is large for that reason.
• The workflow itself was not executed locally. Its if guard skips forked pull requests, so it will not run on this pull request either; it needs a push to main or workflow_dispatch after merge.

This PR was written by an AI agent (stargazer), supervised by a human.

[spiral-ladder]

for proper benchmark reporting this seems blocked by #551 , otherwise lgtm

[spiral-ladder]

please change this to match upstream lodestar's comment

[5t4rg4z3r]

Done in 0c5dff3. The comment now matches upstream lodestar.

This reply was written by an AI agent (stargazer), supervised by a human.

[5t4rg4z3r]

Thanks for the review. Both points noted.

The workflow comment will be changed to match upstream lodestar's wording. That is queued and will go up as a follow-up commit on this branch.

On #551: understood, and I agree that is the right sequencing for durable reporting. That issue is open and waiting on a maintainer to provision the bucket and secrets; I cannot add them from outside. Once they exist, the workflow change is small and I am happy to open it. Until then this branch keeps the Actions cache provider so the comparison comment still works for the common case.

I have not run the workflow itself, here or anywhere. Its fork guard means it does not execute on this pull request, so its behaviour remains unverified by me and needs a push to main or a workflow_dispatch after merge.

This comment was written by an AI agent (stargazer), supervised by a human.

[GrapeBaBa]

The sample output shows lodestar-z is worse than chainsafe blst https://github.com/spiral-ladder/lodestar-bls-benchmarks, is it out of date?

[5t4rg4z3r]

Correction: I focused on the package versions and raw benchmark rows, but I missed the key question: which paths matter to Lodestar. I checked upstream Lodestar. I found no production calls to aggregateVerify or the synchronous aggregateWithRandomness. Lodestar does use verifyMultipleAggregateSignatures, aggregatePublicKeys, aggregateSignatures, and asyncAggregateWithRandomness.

The slower unused paths therefore do not show a current Lodestar performance concern. @spiral-ladder’s reply gives the correct interpretation. My prior comment was misguided. Sorry for the noise.

This comment was written by an AI agent (stargazer), supervised by a human.

[spiral-ladder]

@GrapeBaBa ignore stargazer's comment, he isn't very smart still 😅

i assume you're referring to aggregateVerify and aggregateWithRandomness - those paths are slower but they aren't used anywhere in upstream lodestar so i'm not too concerned tbh, probably still worth looking into why they're slower but low prior imo?

[GrapeBaBa]

@GrapeBaBa ignore stargazer's comment, he isn't very smart still 😅

i assume you're referring to aggregateVerify and aggregateWithRandomness - those paths are slower but they aren't used anywhere in upstream lodestar so i'm not too concerned tbh, probably still worth looking into why they're slower but low prior imo?

I saw blst have some good results with different sets in each category

[spiral-ladder]

@GrapeBaBa now that #562 is in, some of those benchmarks might be redundant (since we no longer) use them

@5t4rg4z3r can you confirm

[5t4rg4z3r]

Confirmed, with one timing caveat: Lodestar-Z #562 is merged, but the Lodestar switch in #9820 is still open.

On #9820 head b4020c06:

• aggregateWithRandomness and asyncAggregateWithRandomness have no references under packages/.
• aggregateVerify remains only in BLS spec tests.
• aggregatePublicKeys and aggregateSignatures still have production callers.
• verifyMultipleAggregateSignatures remains in processDeposit.

For Lodestar-facing performance coverage, the aggregateVerify, aggregateWithRandomness, and asyncAggregateWithRandomness groups can be replaced with benchmarks for verifySignatureSets and verifySignatureSetsSameMessage. Current Lodestar unstable has not made that switch yet.

This comment was written by an AI agent (stargazer), supervised by a human.

[wemeetagain]

@5t4rg4z3r fix the merge conflict here

[spiral-ladder]

@wemeetagain i set it to only reply to me for now 😁

@5t4rg4z3r fix the merge conflict

[spiral-ladder]

but we might want to get #551 sorted before merging this?

[5t4rg4z3r]

Confirmed. GitHub currently reports this PR as CONFLICTING / DIRTY. Resolving the merge conflict is queued for an execution-capable run.

I agree that durable benchmark reporting should wait on #551. That issue is still open and has no maintainer response yet. It needs repository S3 secrets before I can replace the Actions-cache provider, so I will keep #552 blocked on it rather than treat the current cache workflow as merge-ready.

This comment was written by an AI agent (stargazer), supervised by a human.

[5t4rg4z3r]

Resolved in 85f24fa4. GitHub now reports this PR as mergeable. The merge keeps the lodestar-z-only benchmark scope and removes the aggregateWithRandomness and asyncAggregateWithRandomness groups deleted from current main.

Verification: scoped Zig formatting passed, pnpm lint exited 0 with 10 pre-existing warnings, the BLS binding tests passed 143/143, and the focused benchmark passed 20/20. #551 remains the separate reporting blocker.

This comment was written by an AI agent (stargazer), supervised by a human.