Skip to content

fix: restore 255 missing benchmarks — jest-mock-extended crash + parallelize_strict error swallowing#21868

Draft
AztecBot wants to merge 1 commit intonextfrom
claudebox/f0f8be7210f49dcb-2
Draft

fix: restore 255 missing benchmarks — jest-mock-extended crash + parallelize_strict error swallowing#21868
AztecBot wants to merge 1 commit intonextfrom
claudebox/f0f8be7210f49dcb-2

Conversation

@AztecBot
Copy link
Collaborator

Summary

Since 2026-03-18 (PR #21627), 255 benchmark metrics stopped being published to the benchmark dashboard. This includes bb-micro-bench, l1-contracts gas, sol verifier gas, noir circuit gate/opcode counts, p2p benchmarks, prover-client benchmarks, and AVM test benchmarks.

Root Cause

Two interacting bugs:

  1. jest-mock-extended crash in bench workers: PR fix: (A-649) tx collector bench test #21619 changed p2p_client_testbench_worker.ts and proposal_tx_collector_worker.ts to use mock<EpochCache>() from jest-mock-extended. These files run as Node worker processes outside Jest, so the import crashes immediately.

  2. parallelize_strict silently swallows failures: run_tests | tee $output puts run_tests in a subshell (left side of pipe). When the p2p bench fails, wait_for_job calls exit 1 — but this only exits the subshell. The main script sees no background jobs (they were children of the subshell) and exits 0. This prevents ALL subsequent benchmarks from being scheduled, while silently reporting success.

Fix

  • Replace mock<EpochCache>() with createMockEpochCache() in both bench worker files — this helper is already defined in testbench-utils.ts and has no Jest dependency
  • Fix parallelize_strict to use process substitution (> >(tee $output)) instead of a pipe, keeping run_tests in the main shell so background jobs remain trackable
  • Change wait_for_job to set a failed flag instead of exit 1, allowing remaining benchmarks to continue running
  • Update bench_engine to tolerate failures from parallelize_strict so partial results are still collected and uploaded

ClaudeBox log: https://claudebox.work/s/f0f8be7210f49dcb?run=2

…llelize_strict error swallowing

Two interacting bugs caused 255 benchmarks (bb-micro-bench, l1-contracts,
sol verifiers, noir circuit gate counts, p2p, prover-client) to stop being
published since 2026-03-18:

1. PR #21619 changed p2p bench workers to import `jest-mock-extended`, which
   crashes when the workers run outside Jest (as Node worker processes). This
   caused the p2p bench to fail immediately.

2. `parallelize_strict` ran `run_tests | tee` which put `run_tests` in a
   subshell. When the first benchmark failed, the subshell exited but the main
   script saw no jobs and exited 0 — silently swallowing the failure and
   preventing all subsequent benchmarks from running.

Fixes:
- Replace `mock<EpochCache>()` with `createMockEpochCache()` in both bench
  worker files (no Jest dependency needed)
- Use process substitution `> >(tee $output)` instead of pipe so `run_tests`
  stays in the main shell and background jobs remain trackable
- Continue scheduling remaining benchmarks on failure instead of aborting
- Report failures at exit but still collect partial results via bench_engine
@AztecBot AztecBot added ci-draft Run CI on draft PRs. claudebox Owned by claudebox. it can push to this PR. labels Mar 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on draft PRs. claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant