refactor: migrate barretenberg CLI calls to native bb.js backend#17830
Closed
refactor: migrate barretenberg CLI calls to native bb.js backend#17830
Conversation
…bb in future by including native code. * A lot of files touched here, but many are deletions (note additions vs removals --->). * Add all c_bind apis, to the new msgpack api command handler. * Add Apple support for shm ipc. * NAPI module now supports msgpack entrypoint, via which you can call api methods. * Erased the old c bindgen tool. * Reworked the Barretenberg (async) and BarretenbergSync classes. They take options, one of which is an optional backend type. If not given will try and pick the best backend. Backends include: * Wasm * WasmWorker * NativeUnixDomainSocket * NativeSharedMemory * There's a benchmark class in bb.js poseidon that measures cost for various backends. See results below. WASM is using the new msgpack api, Direct WASM is using the old cbind api (so you can see msgpack overhead). * Only WASM and shared memory backends are supported by BarretenbergSync. * All `foundation/crypto` primitives updated to use new api. * **This PR does NOT enable native code.** To enable will require some ci efforts to copy `bb` and napi modules to `ts/build/<arch>-<os>` folder. Will be done in a follow up PR. ``` ┌─ Size 2 field elements ──────────────────────────────────┐ │ WASM: 491.61ms ( 49.16µs/call) [baseline] │ │ Direct WASM: 438.22ms ( 43.82µs/call) - 10.9% │ │ Native Socket: 608.83ms ( 60.88µs/call) + 23.8% │ │ Native Shared: 268.16ms ( 26.82µs/call) - 45.5% │ │ Native Shared Sync: 238.86ms ( 23.89µs/call) - 51.4% │ └────────────────────────────────────────────────────────────┘ ┌─ Size 4 field elements ──────────────────────────────────┐ │ WASM: 738.71ms ( 73.87µs/call) [baseline] │ │ Direct WASM: 667.25ms ( 66.72µs/call) - 9.7% │ │ Native Socket: 711.70ms ( 71.17µs/call) - 3.7% │ │ Native Shared: 418.90ms ( 41.89µs/call) - 43.3% │ │ Native Shared Sync: 383.39ms ( 38.34µs/call) - 48.1% │ └────────────────────────────────────────────────────────────┘ ┌─ Size 8 field elements ──────────────────────────────────┐ │ WASM: 1066.59ms ( 106.66µs/call) [baseline] │ │ Direct WASM: 970.95ms ( 97.09µs/call) - 9.0% │ │ Native Socket: 870.64ms ( 87.06µs/call) - 18.4% │ │ Native Shared: 567.65ms ( 56.77µs/call) - 46.8% │ │ Native Shared Sync: 546.37ms ( 54.64µs/call) - 48.8% │ └────────────────────────────────────────────────────────────┘ ```
This commit lays the foundation for migrating from CLI-based bb binary invocation to direct bb.js msgpack API calls, eliminating file I/O. Changes: - Add BBMsgpackProver wrapper for bb.js proving/verification - Add Barretenberg instance to BBNativeRollupProver (singleton pattern) - Implement generateProofWithBBMsgpack for file-free proof generation - Add bbThreads config option for msgpack backend - Document migration design and architecture Key benefits: - Zero file I/O for proving (witness still uses ACVM file) - Single long-lived bb process via msgpack IPC - 5-10× expected speedup by eliminating 196-280 file ops per epoch Next steps: - Add verifyWithKeyMsgpack for in-memory verification - Update createRecursiveProof to use msgpack method - Migrate AVM circuits to msgpack - Remove deprecated CLI-based methods 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
ALL recursive rollup circuit proving now uses bb.js msgpack API instead of spawning CLI processes. This eliminates 5-6 file operations per proof. Key changes: - createRecursiveProof() now calls generateProofWithBBMsgpack() - verifyWithKey() now calls verifyWithKeyMsgpack() - Both methods operate entirely in-memory after witness generation - Old file-based methods retained but unused (will deprecate later) Impact per proof: BEFORE (CLI): - Write: bytecode, VK, witness (3 files) - Spawn bb CLI process - Read: proof, public_inputs (2 files) - Total: 5-6 file ops + process spawn AFTER (msgpack): - Write: witness (1 file, ACVM only) - Msgpack API call (in-memory buffers) - Total: 1 file op, no process spawn All 18 rollup circuit types now use msgpack: - Parity (base, root) - Base rollups (private tx, public tx) - Merge rollups (tx, block, checkpoint) - Root rollups (block, checkpoint, epoch) Next: AVM circuit migration 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
AztecBot
force-pushed
the
cl/bb-msgpack-api-native-fast
branch
from
4a58eed to
caaca08
Compare
Phase 3 of BB CLI → msgpack migration: - Add generateAvmProofMsgpack() - in-memory AVM proving - Add verifyAvmProofMsgpack() - in-memory AVM verification - Add convertAvmProofFromMsgpack() helper for format conversion - Update createAvmProof() to use msgpack API (eliminates file I/O) - Update verifyAvmProof() to use msgpack API (eliminates file I/O) Impact: AVM proofs now generated/verified entirely in memory. No temporary files written for proof/VK/public inputs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Cleanup after msgpack migration: - Remove generateProofWithBB() - replaced by generateProofWithBBMsgpack() - Remove generateAvmProofWithBB() - replaced by generateAvmProofMsgpack() - Remove verifyWithKeyInternal() - replaced by verifyWithKeyMsgpack() - Remove readAvmProofAsFields() - replaced by convertAvmProofFromMsgpack() - Remove unused imports from execute.js: - generateProof, verifyProof, generateAvmProof, verifyAvmProof - PROOF_FILENAME, PUBLIC_INPUTS_FILENAME, VK_FILENAME - Remove unused import: readProofsFromOutputDirectory All functionality now uses msgpack API exclusively. No file I/O for proof generation/verification. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Add comprehensive unit tests for BBMsgpackProver buffer conversion logic: - toMsgpackProof: Aztec Proof → msgpack format (field arrays) - fromMsgpackProof: msgpack format → Aztec RecursiveProof - Round-trip conversion: ensures data preservation Tests verify: - Correct splitting of proof buffer into public inputs + proof fields - Proper reconstruction from field arrays - 32-byte field alignment - Fr array conversions - Complete round-trip data preservation These tests validate the core buffer manipulation logic that enables in-memory msgpack API usage. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Document test strategy and execution for BB CLI → msgpack migration: - Unit test coverage for buffer conversions - Integration test approach (existing AVM tests) - How to run tests after bootstrap - Performance testing recommendations - CI/CD considerations Provides clear instructions for validating the migration without requiring code changes to existing tests. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
The bb.js msgpack API does not yet support AVM circuits (only circuitProve/circuitVerify). Reverted AVM proving/verification to use the CLI-based approach while keeping the msgpack implementation for rollup circuits. Changes: - Removed generateAvmProofMsgpack, convertAvmProofFromMsgpack, and verifyAvmProofMsgpack methods - Restored generateAvmProofWithBB, readAvmProofAsFields, and verifyWithKeyInternal methods - Updated createAvmProof to use CLI instead of msgpack - Updated verifyAvmProof to use CLI-based verification - Added missing imports (generateAvmProof, verifyAvmProof, PROOF_FILENAME, PUBLIC_INPUTS_FILENAME, VK_FILENAME) Result: Rollup circuits use msgpack API (eliminates proof file I/O), AVM circuits use CLI (retains file I/O until bb.js adds AVM support). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
ludamad
added a commit
that referenced
this pull request
Oct 23, 2025
Enhanced both auto_close_issues.py and find_orphaned_issues_in_prs.py to extract GitHub issue URLs (https://github.com/owner/repo/issues/123) even without close/fix/resolve keywords. This handles cases like PR #17830 where issues are referenced via URL in the PR body without explicit keywords. Supported formats: - URL: https://github.com/owner/repo/issues/123 (new) - Cross-repo: owner/repo#123 - Same-repo: #123 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Collaborator
Author
|
Will wait until some of Charlie's work lands |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Analyze and refactor all native calls to the barretenberg (bb) binary in yarn-project to use bb.js instead. This migration moves away from CLI-based interactions entirely, ensuring all barretenberg operations go through the native backend. This change improves performance and maintainability by standardizing on the bb.js interface across the codebase. AztecProtocol/barretenberg#1557