fix: restore 255 missing benchmarks — jest-mock-extended crash + parallelize_strict error swallowing#21868
Draft
fix: restore 255 missing benchmarks — jest-mock-extended crash + parallelize_strict error swallowing#21868
Conversation
…llelize_strict error swallowing Two interacting bugs caused 255 benchmarks (bb-micro-bench, l1-contracts, sol verifiers, noir circuit gate counts, p2p, prover-client) to stop being published since 2026-03-18: 1. PR #21619 changed p2p bench workers to import `jest-mock-extended`, which crashes when the workers run outside Jest (as Node worker processes). This caused the p2p bench to fail immediately. 2. `parallelize_strict` ran `run_tests | tee` which put `run_tests` in a subshell. When the first benchmark failed, the subshell exited but the main script saw no jobs and exited 0 — silently swallowing the failure and preventing all subsequent benchmarks from running. Fixes: - Replace `mock<EpochCache>()` with `createMockEpochCache()` in both bench worker files (no Jest dependency needed) - Use process substitution `> >(tee $output)` instead of pipe so `run_tests` stays in the main shell and background jobs remain trackable - Continue scheduling remaining benchmarks on failure instead of aborting - Report failures at exit but still collect partial results via bench_engine
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Since 2026-03-18 (PR #21627), 255 benchmark metrics stopped being published to the benchmark dashboard. This includes bb-micro-bench, l1-contracts gas, sol verifier gas, noir circuit gate/opcode counts, p2p benchmarks, prover-client benchmarks, and AVM test benchmarks.
Root Cause
Two interacting bugs:
jest-mock-extended crash in bench workers: PR fix: (A-649) tx collector bench test #21619 changed
p2p_client_testbench_worker.tsandproposal_tx_collector_worker.tsto usemock<EpochCache>()fromjest-mock-extended. These files run as Node worker processes outside Jest, so the import crashes immediately.parallelize_strict silently swallows failures:
run_tests | tee $outputputsrun_testsin a subshell (left side of pipe). When the p2p bench fails,wait_for_jobcallsexit 1— but this only exits the subshell. The main script sees no background jobs (they were children of the subshell) and exits 0. This prevents ALL subsequent benchmarks from being scheduled, while silently reporting success.Fix
mock<EpochCache>()withcreateMockEpochCache()in both bench worker files — this helper is already defined intestbench-utils.tsand has no Jest dependencyparallelize_strictto use process substitution (> >(tee $output)) instead of a pipe, keepingrun_testsin the main shell so background jobs remain trackablewait_for_jobto set afailedflag instead ofexit 1, allowing remaining benchmarks to continue runningbench_engineto tolerate failures fromparallelize_strictso partial results are still collected and uploadedClaudeBox log: https://claudebox.work/s/f0f8be7210f49dcb?run=2