feat: WIP integrate batched blobs into l1 contracts + ts by MirandaWood · Pull Request #14329 · AztecProtocol/aztec-packages

MirandaWood · 2025-05-15T10:43:49Z

Finalises integration of batched blobs

mw/blob-batching-integration adds batching to the rollup .nr circuits only (=> will not run in the repo). This PR brings those changes downstream to the typescript and L1 contracts. Main changes:

L1 Contracts:
- No longer calls the point evaluation precompile on propose, instead injects the blob commitments, check they correspond to the broadcast blobs, and stores them in the blobCommitmentsHash
- Does not store any blob public inputs apart from the blobCommitmentsHash (no longer required)
- Calls the point evaluation precompile once on submitEpochRootProof for ALL blobs in the epoch
- Uses the same precompile inputs as pubic inputs to the root proof verification alonge with the blobCommitmentsHash to link the circuit batched blob, real L1 blobs, and the batched blob verified on L1
Refactors mock blob oracle
Injects the final blob challenges used on each blob into all block building methods in orchestrator
Accumulates blobs in ts when building blocks and uses as inputs to each rollup circuit
Returns the blob inputs required for submitEpochRootProof on finaliseEpoch()
Updates nr structs in ts plus fixtures and tests

TODOs/Current issues

When using real proofs (e.g. yarn-project/prover-client/src/test/bb_prover_full_rollup.test.ts), the root rollup proof is generated correctly but fails verification checks in bb due to incorrect number of public inputs. Changing the number correctly updates vks and all constants elsewhere, but bb does not change. EDIT: solved - must include the is_inf point member for now (see below TODO)
The Prover.toml for block-root is not executing. The error manifests in the same way as that in fix: fix assert_split_transformed_value_arrays conditional access index underflow #12540 (but may be different). EDIT: temporarily fixed - details in this repro (Bigcurve/Bignum nargo execute failure repro #14381) and noir issue (println statement changes execution result of program noir-lang/noir#8563).
BLS points in noir take up 9 fields (4 for each coordinate as a limbed bignum, 1 for the is_inf flag) but can be compressed to only 2. For recursive verification in block root and above, would it be worth the gates to compress these? It depends whether the gate cost of compression is more/less than gate cost of recursively verifying 7 more public inputs.

PR Stack

mw/blob-batching <- main feature
^ mw/blob-batching-bls-utils <- BLS12-381 bigcurve and bignum utils (noir) (feat: blob batching methods #13583)
^ mw/blob-batching-bls-utils-ts <- BLS12-381 bigcurve and bignum utils (ts) (feat: blob batching methods (ts) #13606)
^ mw/blob-batching-integration <- Integrate batching into noir protocol circuits (feat: WIP batch blobs and validate in rollup #13817)
^ mw/blob-batching-integration-ts-sol <- Integrate batching into ts and solidity (feat: WIP integrate batched blobs into l1 contracts + ts #14329)

This reverts commit 68be71e.

…atching-bls-utils

… mw/blob-batching-bls-utils-ts

…atching-bls-utils

… mw/blob-batching-bls-utils-ts

…atching-bls-utils

… mw/blob-batching-bls-utils-ts

…nto mw/blob-batching-integration

… mw/blob-batching-bls-utils-ts

…nto mw/blob-batching-integration

…to mw/blob-batching-integration-ts-sol

iAmMichaelConnor

🚀

@iAmMichaelConnor

Ts only blob batching methods plus tests. Points to the parent methods PR: #13583. TODOs (Marked in files as `TODO(MW)`): - [ ] Remove the large trusted setup file? Not sure if it's required, but it is currently the only way I show in tests that our BLS12 methods match those in c-kzg. - [x] Add nr fixture where we can use `updateInlineTestData` for point compression. Other TODOs must wait until we actually integrate batching, otherwise I will break the repo. NB: The files `bls12_fields.ts` and `bls12_point.ts` and their tests are essentially copies of `./fields.ts` and `./point.ts`. When reviewing please keep that in mind and double check the original file if you see an issue before commenting (@iAmMichaelConnor ;) ). --- ## PR Stack - [ ] `mw/blob-batching` <- main feature - [ ] ^ `mw/blob-batching-bls-utils` <- BLS12-381 bigcurve and bignum utils (noir) (#13583) - [x] ^ `mw/blob-batching-bls-utils-ts` <- BLS12-381 bigcurve and bignum utils (ts) (#13606) - [ ] ^ `mw/blob-batching-integration` <- Integrate batching into noir protocol circuits (#13817) - [ ] ^ `mw/blob-batching-integration-ts-sol` <- Integrate batching into ts and solidity (#14329)

…atching-integration

…to mw/blob-batching-integration-ts-sol

…atching-integration

…to mw/blob-batching-integration-ts-sol

WIP TODOs - [ ] Compress BLS12 fq and fr values to fewer native fields to reduce number of public inputs (somewhat blocked by #13608 since that dictates how large bls12fr value gamma is) - [ ] Delete old `blob.nr` files and remove `pub`s w/o batching (will do this later so it's easier to review) - [x] Rework `RootRollupPublicInputs` so it doesn't contain unnecessary values not needed for L1 verification --- ## PR Stack - [ ] `mw/blob-batching` <- main feature - [ ] ^ `mw/blob-batching-bls-utils` <- BLS12-381 bigcurve and bignum utils (noir) (#13583) - [ ] ^ `mw/blob-batching-bls-utils-ts` <- BLS12-381 bigcurve and bignum utils (ts) (#13606) - [x] ^ `mw/blob-batching-integration` <- Integrate batching into noir protocol circuits (#13817) - [ ] ^ `mw/blob-batching-integration-ts-sol` <- Integrate batching into ts and solidity (#14329) --------- Co-authored-by: Tom French <15848336+TomAFrench@users.noreply.github.com>

…atching-integration-ts-sol

LHerskind · 2025-06-04T12:13:18Z

l1-contracts/src/core/libraries/rollup/BlobLib.sol

+   * input[1:] - blob commitments (48 bytes * num blobs in block)
+   * @param _blobsInput - The above bytes to verify our input blob commitments match real blobs
+   * @param _checkBlob - Whether to skip blob related checks. Hardcoded to true (See RollupCore.sol -> checkBlob), exists only to be overriden in tests.
   */


Please add @return for the different values here to make it a bit simpler to follow.

Also probably good to add an assumption about there not being any non-aztec blobs in the tx.

Would it be worth adding a loop from numBlobs to whatever the max is and ensuring the blobHash is 0 for those?

Added here: b90b979

LHerskind · 2025-06-04T12:18:46Z

l1-contracts/src/core/interfaces/IRollup.sol

  // to one another.
  mapping(address prover => mapping(Epoch epoch => bool claimed)) proverClaimed;
  RollupConfig config;
+  // TODO(#14646): We only ever need to store AZTEC_MAX_EPOCH_DURATION values below => make fixed length and overwrite once we start a new epoch


I'm guess the AZTEC_MAX_EPOCH_DURATION is something that is here to keep some flexibility for the circuits to not necessarily follow the actual epoch duration.

However, it is not sufficient to just store for one epoch, as prune can remove more than that, I think that the value you are probably interested in is the submission window.

Regardless, something that could be potentially of interest (not now I would expect) is keeping a number of running hashes in there instead. Essentially allowing you are the time of proof submission to just read one storage variable instead of one for each block.

Separately, why is this value in here, and not part of the BlockLog it seems more convenient if we keep them close (BlockLog) as it is much clearer when doing optimisations what can be grouped. For example storing a hash of the blocklog content and storing that reducing to a single store etc, and just simpler to figure out then.

as prune can remove more than that, I think that the value you are probably interested in is the submission window.

Ah ok. I didn't realise this. This was the reason why the value is here and not BlockLog - I wanted it to be separate so it would be possible to change later to a fixed length thing and just re use the slots once an epoch is proven. But that was based on a wrong assumption, so I'll add the value to BlockLog for now.

Regardless, something that could be potentially of interest (not now I would expect) is keeping a number of running hashes in there instead. Essentially allowing you are the time of proof submission to just read one storage variable instead of one for each block.

I had assumed this wouldn't work because of pruning, but thinking about it now this would work. We only need the 'current' value of the epoch, then iterate and update that value, unless it's the first block of an epoch where we initialise the value. I'll add an issue for this, thanks!

Fixed here: 32bf652
(Didn't have time to test yet, may fail)

LHerskind · 2025-06-04T12:29:36Z

l1-contracts/src/core/libraries/rollup/BlobLib.sol

+      blobInputStart += Constants.BLS12_POINT_COMPRESSED_BYTES;
+
+      // TODO(#14646): Use kzg_to_versioned_hash & VERSIONED_HASH_VERSION_KZG
+      // Using bytes32 array to force bytes into memory


Why you want to force it into memory vs on stack here?

Personally, I think it is slightly simpler to just read as some binary operations on the stack, e.g., as seen below. When seeing anything with arrays and memory, my mind will first be thinking "oh is it updating the length of the array" as that is what would have happened if it was not a fixed size.

bytes32 blobHashCheck = bytes32( ( uint256(sha256(blobCommitments[i])) & 0x00FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF ) | 0x0100000000000000000000000000000000000000000000000000000000000000 );

Purely because I thought it was a little clearer to see the 'overwriting' happen with mstore but you raise a good point. I will use your suggestion, thanks

Fixed here: b90b979

@iAmMichaelConnor

## The blobs are back in town. This PR reworks blobs so that instead of calling the point evaluation precompile for each blob (currently up to 3 per block => up to 96 (?) calls per epoch), we call it once per epoch by batching blobs to a single kzg commitment, opening, challenge, and proof. How we can be sure that this one pairing check is equivalent to a check per blob is covered in the maths by @iAmMichaelConnor [here](https://hackmd.io/WUtNusQxS5KAw-af3gxycA?view) 🎉 ## Overview Instead of pushing to a long array of `BlobPublicInputs`, which are then individually checked on L1, we batch each blob together to a single set of `BlobAccumulatorPublicInputs`. The `start` accumulator state is fed into each block root circuit, where the block's blobs are accumulated and the `end` state is set. Each block merge circuit checks that the state follows on correctly and, finally, the root circuit checks that the very `start` state was empty and finalises the last `end` state. This last `end` state makes up the set of inputs for the point evaluation precompile. If the pairing check in that precompile passes, we know that all blobs for all blocks in the epoch are valid and contain only the tx effects validated by the rollup. ### Circuits Key changes: - Integrate BLS12-381 curve operations with `bignum` and `bigcurve` libraries, plus tests. - Rework the `blob` package to batch blobs and store in reworked structs, plus tests. - Rework the rollup circuits from `block_root` above to handle blob accumulation state rather than a list of individual blob inputs, plus (you guessed it) tests. ### Contracts The contracts: - No longer call the point evaluation precompile on `propose`, instead inject the blob commitments, check they correspond to the broadcast blobs, and stores them in the `blobCommitmentsHash`. - Do not store any blob public inputs apart from the `blobCommitmentsHash`. - Call the point evaluation precompile once on `submitEpochRootProof` for ALL blobs in the epoch. - Use the same precompile inputs as pubic inputs to the root proof verification along with the `blobCommitmentsHash` to link the circuit batched blob, real L1 blobs, and the batched blob verified on L1. ### TypeScript Key changes: - Edit all the structs and methods reliant on the circuits/contracts to match the above changes. - Inject the final blob challenges used on each blob into all block building methods in `orchestrator`. - Accumulate blobs in ts when building blocks and use as inputs to each rollup circuit, plus tests. - Return the blob inputs required for `submitEpochRootProof` on `finaliseEpoch()`. ### TODOs/Related Issues - Choose field for hashing challenge: #13608 - Instead of exponentiating `gamma` (expensive!), hash it for each iteration: #13740 - Number of public inputs: BLS points in noir take up 9 fields (4 for each coordinate as a limbed bignum, 1 for the is_inf flag) but can be compressed to only 2. For recursive verification in block root and above, would it be worth the gates to compress these? It depends whether the gate cost of compression is more/less than gate cost of recursively verifying 7 more public inputs. - Remove the large trusted setup file from `yarn-project/blob-lib/src/trusted_setup_bit_reversed.json`? Used in testing, but may not be worth keeping (see code comments). - Cleanup old, unused blob stuff in #14637. ## PR Stack - [x] `mw/blob-batching` <- main feature - [x] ^ `mw/blob-batching-bls-utils` <- BLS12-381 bigcurve and bignum utils (noir) (#13583) - [x] ^ `mw/blob-batching-bls-utils-ts` <- BLS12-381 bigcurve and bignum utils (ts) (#13606) - [x] ^ `mw/blob-batching-integration` <- Integrate batching into noir protocol circuits (#13817) - [x] ^ `mw/blob-batching-integration-ts-sol` <- Integrate batching into ts and solidity (#14329) - [ ] ^ `mw/blob-batching-cleanup` <- Remove old blob code --------- Co-authored-by: Tom French <15848336+TomAFrench@users.noreply.github.com>

@iAmMichaelConnor

## The blobs are back in town. This PR reworks blobs so that instead of calling the point evaluation precompile for each blob (currently up to 3 per block => up to 96 (?) calls per epoch), we call it once per epoch by batching blobs to a single kzg commitment, opening, challenge, and proof. How we can be sure that this one pairing check is equivalent to a check per blob is covered in the maths by @iAmMichaelConnor [here](https://hackmd.io/WUtNusQxS5KAw-af3gxycA?view) 🎉 ## Overview Instead of pushing to a long array of `BlobPublicInputs`, which are then individually checked on L1, we batch each blob together to a single set of `BlobAccumulatorPublicInputs`. The `start` accumulator state is fed into each block root circuit, where the block's blobs are accumulated and the `end` state is set. Each block merge circuit checks that the state follows on correctly and, finally, the root circuit checks that the very `start` state was empty and finalises the last `end` state. This last `end` state makes up the set of inputs for the point evaluation precompile. If the pairing check in that precompile passes, we know that all blobs for all blocks in the epoch are valid and contain only the tx effects validated by the rollup. ### Circuits Key changes: - Integrate BLS12-381 curve operations with `bignum` and `bigcurve` libraries, plus tests. - Rework the `blob` package to batch blobs and store in reworked structs, plus tests. - Rework the rollup circuits from `block_root` above to handle blob accumulation state rather than a list of individual blob inputs, plus (you guessed it) tests. ### Contracts The contracts: - No longer call the point evaluation precompile on `propose`, instead inject the blob commitments, check they correspond to the broadcast blobs, and stores them in the `blobCommitmentsHash`. - Do not store any blob public inputs apart from the `blobCommitmentsHash`. - Call the point evaluation precompile once on `submitEpochRootProof` for ALL blobs in the epoch. - Use the same precompile inputs as pubic inputs to the root proof verification along with the `blobCommitmentsHash` to link the circuit batched blob, real L1 blobs, and the batched blob verified on L1. ### TypeScript Key changes: - Edit all the structs and methods reliant on the circuits/contracts to match the above changes. - Inject the final blob challenges used on each blob into all block building methods in `orchestrator`. - Accumulate blobs in ts when building blocks and use as inputs to each rollup circuit, plus tests. - Return the blob inputs required for `submitEpochRootProof` on `finaliseEpoch()`. ### TODOs/Related Issues - Choose field for hashing challenge: AztecProtocol#13608 - Instead of exponentiating `gamma` (expensive!), hash it for each iteration: AztecProtocol#13740 - Number of public inputs: BLS points in noir take up 9 fields (4 for each coordinate as a limbed bignum, 1 for the is_inf flag) but can be compressed to only 2. For recursive verification in block root and above, would it be worth the gates to compress these? It depends whether the gate cost of compression is more/less than gate cost of recursively verifying 7 more public inputs. - Remove the large trusted setup file from `yarn-project/blob-lib/src/trusted_setup_bit_reversed.json`? Used in testing, but may not be worth keeping (see code comments). - Cleanup old, unused blob stuff in AztecProtocol#14637. ## PR Stack - [x] `mw/blob-batching` <- main feature - [x] ^ `mw/blob-batching-bls-utils` <- BLS12-381 bigcurve and bignum utils (noir) (AztecProtocol#13583) - [x] ^ `mw/blob-batching-bls-utils-ts` <- BLS12-381 bigcurve and bignum utils (ts) (AztecProtocol#13606) - [x] ^ `mw/blob-batching-integration` <- Integrate batching into noir protocol circuits (AztecProtocol#13817) - [x] ^ `mw/blob-batching-integration-ts-sol` <- Integrate batching into ts and solidity (AztecProtocol#14329) - [ ] ^ `mw/blob-batching-cleanup` <- Remove old blob code --------- Co-authored-by: Tom French <15848336+TomAFrench@users.noreply.github.com>

MirandaWood and others added 30 commits April 15, 2025 19:27

feat: blob batching methods - nr only

97e2a01

chore: fmt, more tests, rearranging

52801e6

feat: BLS12 field, curve methods, blob batching methods, ts only

35e8be2

chore: lint, cleanup

db03339

chore: remove trusted setup file + test using it (size issues)

68be71e

Revert "chore: remove trusted setup file + test using it (size issues)"

1fa5d49

This reverts commit 68be71e.

chore: cleanup packages + increase playground size

33d62a7

feat: address some comments, cleanup

46b8866

chore: update some comments

346ca9a

Merge remote-tracking branch 'origin/mw/blob-batching' into mw/blob-b…

f655bf5

…atching-bls-utils

Merge remote-tracking branch 'origin/mw/blob-batching-bls-utils' into…

8d57216

… mw/blob-batching-bls-utils-ts

chore: renaming, cleanup

9490422

chore: renaming, cleanup

c300c77

chore: add issue nums (hopefully force ci cache reset)

63ffe96

feat: as isNegative to F, rename proof -> Q

606a942

Merge remote-tracking branch 'origin/mw/blob-batching' into mw/blob-b…

75d6d35

…atching-bls-utils

Merge remote-tracking branch 'origin/mw/blob-batching-bls-utils' into…

40ffd8b

… mw/blob-batching-bls-utils-ts

chore: bumped vite kb limit 1700 -> 1720

951d329

chore: bumped vite kb limit 1700 -> 1750

307cb09

feat: adding helpers, constants, docs, etc. for integration

5251cd1

Merge remote-tracking branch 'origin/mw/blob-batching-bls-utils' into…

de8ec1e

… mw/blob-batching-bls-utils-ts

feat: WIP batch blobs and validate in rollup

5f2fe64

feat: add more tests, cleanup, remove some warnings

070cbd2

chore: update to fixed noir version

f9e5364

feat: add final accumulator pub inputs for root

201711e

Merge remote-tracking branch 'origin/mw/blob-batching' into mw/blob-b…

ca0da9d

…atching-bls-utils

Merge remote-tracking branch 'origin/mw/blob-batching-bls-utils' into…

d5f11d4

… mw/blob-batching-bls-utils-ts

Merge remote-tracking branch 'origin/mw/blob-batching-bls-utils-ts' i…

34e9e1a

…nto mw/blob-batching-integration

chore: rename v -> blob_commitments_hash, move noir ref further up stack

d3ac058

Merge remote-tracking branch 'origin/mw/blob-batching-bls-utils' into…

6750476

… mw/blob-batching-bls-utils-ts

MirandaWood added 9 commits June 2, 2025 10:53

chore: renaming, bring down changes from integration branch, cleanup

92f6e5f

Merge remote-tracking branch 'origin/mw/blob-batching-bls-utils' into…

88b4b28

… mw/blob-batching-bls-utils-ts

chore: cleanup, bring down changes from other PRs

0ee11fd

Merge remote-tracking branch 'origin/mw/blob-batching-bls-utils-ts' i…

7429aac

…nto mw/blob-batching-integration

Merge remote-tracking branch 'origin/mw/blob-batching-integration' in…

bbaa96f

…to mw/blob-batching-integration-ts-sol

chore: renaming from methods branch

ac78bf3

Merge remote-tracking branch 'origin/mw/blob-batching-integration' in…

0716329

…to mw/blob-batching-integration-ts-sol

feat: address comments, docs, renaming

0d4ebdd

chore: update fixtures

3ae3a8b

iAmMichaelConnor approved these changes Jun 3, 2025

View reviewed changes

MirandaWood added 8 commits June 3, 2025 10:31

Merge remote-tracking branch 'origin/mw/blob-batching' into mw/blob-b…

4a6ee02

…atching-integration

chore: bump bignum post merge

4c0f1a9

Merge remote-tracking branch 'origin/mw/blob-batching-integration' in…

3c81612

…to mw/blob-batching-integration-ts-sol

chore: revert fix to epochs test (no longer req)

1a16d7a

Merge remote-tracking branch 'origin/mw/blob-batching' into mw/blob-b…

42230da

…atching-integration

Merge remote-tracking branch 'origin/mw/blob-batching-integration' in…

1f912b7

…to mw/blob-batching-integration-ts-sol

chore: bring down next fix from #14722

4f2320f

chore: bring down next fix again

5783929

Base automatically changed from mw/blob-batching-integration to mw/blob-batching June 4, 2025 08:13

MirandaWood added 3 commits June 4, 2025 08:17

Merge remote-tracking branch 'origin/mw/blob-batching' into mw/blob-b…

295c59d

…atching-integration-ts-sol

chore: clarify docs, add issue numbers to todos

39ff55d

fix: bad merge

83e7d3a

MirandaWood merged commit 3f75d58 into mw/blob-batching Jun 4, 2025
4 checks passed

MirandaWood deleted the mw/blob-batching-integration-ts-sol branch June 4, 2025 15:22

LHerskind reviewed Jun 4, 2025

View reviewed changes

MirandaWood mentioned this pull request Jun 4, 2025

Deferred Blob issues #14646

Open

17 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: WIP integrate batched blobs into l1 contracts + ts#14329

feat: WIP integrate batched blobs into l1 contracts + ts#14329
MirandaWood merged 141 commits intomw/blob-batchingfrom
mw/blob-batching-integration-ts-sol

MirandaWood commented May 15, 2025 •

edited

Loading

Uh oh!

iAmMichaelConnor left a comment

Uh oh!

Uh oh!

LHerskind Jun 4, 2025

Uh oh!

LHerskind Jun 4, 2025

Uh oh!

MirandaWood Jun 4, 2025

Uh oh!

MirandaWood Jun 4, 2025

Uh oh!

LHerskind Jun 4, 2025

Uh oh!

MirandaWood Jun 4, 2025

Uh oh!

MirandaWood Jun 4, 2025

Uh oh!

LHerskind Jun 4, 2025

Uh oh!

MirandaWood Jun 4, 2025

Uh oh!

MirandaWood Jun 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

MirandaWood commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Finalises integration of batched blobs

TODOs/Current issues

PR Stack

Uh oh!

iAmMichaelConnor left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

MirandaWood commented May 15, 2025 •

edited

Loading