feat: blob batching methods (ts) by MirandaWood · Pull Request #13606 · AztecProtocol/aztec-packages

MirandaWood · 2025-04-16T13:49:35Z

Ts only blob batching methods plus tests. Points to the parent methods PR: #13583.

TODOs (Marked in files as TODO(MW)):

Remove the large trusted setup file? Not sure if it's required, but it is currently the only way I show in tests that our BLS12 methods match those in c-kzg.
Add nr fixture where we can use updateInlineTestData for point compression.

Other TODOs must wait until we actually integrate batching, otherwise I will break the repo.

NB: The files bls12_fields.ts and bls12_point.ts and their tests are essentially copies of ./fields.ts and ./point.ts. When reviewing please keep that in mind and double check the original file if you see an issue before commenting (@iAmMichaelConnor ;) ).

PR Stack

mw/blob-batching <- main feature
^ mw/blob-batching-bls-utils <- BLS12-381 bigcurve and bignum utils (noir) (feat: blob batching methods #13583)
^ mw/blob-batching-bls-utils-ts <- BLS12-381 bigcurve and bignum utils (ts) (feat: blob batching methods (ts) #13606)
^ mw/blob-batching-integration <- Integrate batching into noir protocol circuits (feat: WIP batch blobs and validate in rollup #13817)
^ mw/blob-batching-integration-ts-sol <- Integrate batching into ts and solidity (feat: WIP integrate batched blobs into l1 contracts + ts #14329)

This reverts commit 68be71e.

MirandaWood · 2025-04-17T13:55:53Z

playground/vite.config.ts

      // }),
      bundlesize({
-        limits: [{ name: 'assets/index-*', limit: '1600kB' }],
+        limits: [{ name: 'assets/index-*', limit: '1650kB' }],


@Thunkar Was having problems just hitting the size limit here after adding a package to foundation. Not sure of the best way to solve it (since the package is needed) so bumped this - any advice appreciated!

It's been bumped to 1680kb due to bb.js changes, would that work?

It's ok as long as it's justified (and this is!), I just want to make sure it doesn't get completely out of control again!

Shouldn't be an issue! When I merge master if I need to add a lot more to get it working I will get in touch, thank you!

Had to bump from 1700 -> 1750* just now. Running CI to see if it works, but happy to take a look and find an alternative if you feel 1750* is pushing the limit!

iAmMichaelConnor

Amazing! Thanks! Only some really minor comments from me.

yarn-project/blob-lib/src/blob.ts

yarn-project/blob-lib/src/blob_batching.ts

yarn-project/foundation/src/fields/bls12_point.ts

yarn-project/foundation/src/fields/bls12_fields.ts

yarn-project/foundation/src/fields/bls12_point.ts

yarn-project/blob-lib/src/blob_batching.ts

…atching-bls-utils

… mw/blob-batching-bls-utils-ts

yarn-project/blob-lib/src/blob_batching.ts

yarn-project/foundation/src/fields/bls12_fields.ts

yarn-project/foundation/src/fields/bls12_point.ts

iAmMichaelConnor · 2025-04-22T13:59:21Z

(Oh, I haven't reviewed the nr files that got merged into this PR. Would you like me to? Or should I wait?)

…atching-bls-utils

… mw/blob-batching-bls-utils-ts

…atching-bls-utils

… mw/blob-batching-bls-utils-ts

yarn-project/blob-lib/src/blob_batching.test.ts

yarn-project/foundation/src/fields/bls12_fields.test.ts

LeilaWang · 2025-05-28T11:43:06Z

yarn-project/foundation/src/fields/bls12_fields.test.ts

+      const anotherString = '1000a000';
+      const anotherStringPrepended = '0x1000a000';
+
+      const expectedValueOfAnotherHexString = 268476416n;


Maybe define it (and others) as BigInt(0x1000a000) so that we know this is not taken from the value to be compared below and we end up just comparing two wrong values.

This was just copied over from fields.test.ts, I can change both?

yarn-project/foundation/src/fields/bls12_fields.ts

yarn-project/foundation/src/fields/bls12_point.test.ts

yarn-project/foundation/src/fields/bls12_point.ts

MirandaWood · 2025-05-29T15:37:58Z

yarn-project/foundation/src/fields/bls12_fields.ts

+ * Conversions from Buffer to BigInt and vice-versa are not cheap.
+ * We allow construction with either form and lazily convert to other as needed.
+ * We only check we are within the field modulus when initializing with bigint.


This was copied over from fields.ts - though in testing I found no discernable slowdown when initiating a field as assigning its buffer when inputting a bigint and vice versa. So I have since changed the constructor to assign both the bigint and buffer forms.

LeilaWang

🎉

… mw/blob-batching-bls-utils-ts

iAmMichaelConnor

🚀

…atching-bls-utils-ts

WIP TODOs - [ ] Compress BLS12 fq and fr values to fewer native fields to reduce number of public inputs (somewhat blocked by #13608 since that dictates how large bls12fr value gamma is) - [ ] Delete old `blob.nr` files and remove `pub`s w/o batching (will do this later so it's easier to review) - [x] Rework `RootRollupPublicInputs` so it doesn't contain unnecessary values not needed for L1 verification --- ## PR Stack - [ ] `mw/blob-batching` <- main feature - [ ] ^ `mw/blob-batching-bls-utils` <- BLS12-381 bigcurve and bignum utils (noir) (#13583) - [ ] ^ `mw/blob-batching-bls-utils-ts` <- BLS12-381 bigcurve and bignum utils (ts) (#13606) - [x] ^ `mw/blob-batching-integration` <- Integrate batching into noir protocol circuits (#13817) - [ ] ^ `mw/blob-batching-integration-ts-sol` <- Integrate batching into ts and solidity (#14329) --------- Co-authored-by: Tom French <15848336+TomAFrench@users.noreply.github.com>

## Finalises integration of batched blobs `mw/blob-batching-integration` adds batching to the rollup .nr circuits only (=> will not run in the repo). This PR brings those changes downstream to the typescript and L1 contracts. Main changes: - L1 Contracts: - No longer calls the point evaluation precompile on `propose`, instead injects the blob commitments, check they correspond to the broadcast blobs, and stores them in the `blobCommitmentsHash` - Does not store any blob public inputs apart from the `blobCommitmentsHash` (no longer required) - Calls the point evaluation precompile once on `submitEpochRootProof` for ALL blobs in the epoch - Uses the same precompile inputs as pubic inputs to the root proof verification alonge with the `blobCommitmentsHash` to link the circuit batched blob, real L1 blobs, and the batched blob verified on L1 - Refactors mock blob oracle - Injects the final blob challenges used on each blob into all block building methods in `orchestrator` - Accumulates blobs in ts when building blocks and uses as inputs to each rollup circuit - Returns the blob inputs required for `submitEpochRootProof` on `finaliseEpoch()` - Updates nr structs in ts plus fixtures and tests ## TODOs/Current issues - ~When using real proofs (e.g. `yarn-project/prover-client/src/test/bb_prover_full_rollup.test.ts`), the root rollup proof is generated correctly but fails verification checks in `bb` due to incorrect number of public inputs. Changing the number correctly updates vks and all constants elsewhere, but `bb` does not change.~ EDIT: solved - must include the `is_inf` point member for now (see below TODO) - ~The `Prover.toml` for block-root is not executing. The error manifests in the same way as that in #12540 (but may be different).~ EDIT: temporarily fixed - details in this repro (#14381) and noir issue (noir-lang/noir#8563). - BLS points in noir take up 9 fields (4 for each coordinate as a limbed bignum, 1 for the `is_inf` flag) but can be compressed to only 2. For recursive verification in block root and above, would it be worth the gates to compress these? It depends whether the gate cost of compression is more/less than gate cost of recursively verifying 7 more public inputs. ## PR Stack - [ ] `mw/blob-batching` <- main feature - [ ] ^ `mw/blob-batching-bls-utils` <- BLS12-381 bigcurve and bignum utils (noir) (#13583) - [ ] ^ `mw/blob-batching-bls-utils-ts` <- BLS12-381 bigcurve and bignum utils (ts) (#13606) - [ ] ^ `mw/blob-batching-integration` <- Integrate batching into noir protocol circuits (#13817) - [x] ^ `mw/blob-batching-integration-ts-sol` <- Integrate batching into ts and solidity (#14329) --------- Co-authored-by: Tom French <15848336+TomAFrench@users.noreply.github.com>

@iAmMichaelConnor

## The blobs are back in town. This PR reworks blobs so that instead of calling the point evaluation precompile for each blob (currently up to 3 per block => up to 96 (?) calls per epoch), we call it once per epoch by batching blobs to a single kzg commitment, opening, challenge, and proof. How we can be sure that this one pairing check is equivalent to a check per blob is covered in the maths by @iAmMichaelConnor [here](https://hackmd.io/WUtNusQxS5KAw-af3gxycA?view) 🎉 ## Overview Instead of pushing to a long array of `BlobPublicInputs`, which are then individually checked on L1, we batch each blob together to a single set of `BlobAccumulatorPublicInputs`. The `start` accumulator state is fed into each block root circuit, where the block's blobs are accumulated and the `end` state is set. Each block merge circuit checks that the state follows on correctly and, finally, the root circuit checks that the very `start` state was empty and finalises the last `end` state. This last `end` state makes up the set of inputs for the point evaluation precompile. If the pairing check in that precompile passes, we know that all blobs for all blocks in the epoch are valid and contain only the tx effects validated by the rollup. ### Circuits Key changes: - Integrate BLS12-381 curve operations with `bignum` and `bigcurve` libraries, plus tests. - Rework the `blob` package to batch blobs and store in reworked structs, plus tests. - Rework the rollup circuits from `block_root` above to handle blob accumulation state rather than a list of individual blob inputs, plus (you guessed it) tests. ### Contracts The contracts: - No longer call the point evaluation precompile on `propose`, instead inject the blob commitments, check they correspond to the broadcast blobs, and stores them in the `blobCommitmentsHash`. - Do not store any blob public inputs apart from the `blobCommitmentsHash`. - Call the point evaluation precompile once on `submitEpochRootProof` for ALL blobs in the epoch. - Use the same precompile inputs as pubic inputs to the root proof verification along with the `blobCommitmentsHash` to link the circuit batched blob, real L1 blobs, and the batched blob verified on L1. ### TypeScript Key changes: - Edit all the structs and methods reliant on the circuits/contracts to match the above changes. - Inject the final blob challenges used on each blob into all block building methods in `orchestrator`. - Accumulate blobs in ts when building blocks and use as inputs to each rollup circuit, plus tests. - Return the blob inputs required for `submitEpochRootProof` on `finaliseEpoch()`. ### TODOs/Related Issues - Choose field for hashing challenge: #13608 - Instead of exponentiating `gamma` (expensive!), hash it for each iteration: #13740 - Number of public inputs: BLS points in noir take up 9 fields (4 for each coordinate as a limbed bignum, 1 for the is_inf flag) but can be compressed to only 2. For recursive verification in block root and above, would it be worth the gates to compress these? It depends whether the gate cost of compression is more/less than gate cost of recursively verifying 7 more public inputs. - Remove the large trusted setup file from `yarn-project/blob-lib/src/trusted_setup_bit_reversed.json`? Used in testing, but may not be worth keeping (see code comments). - Cleanup old, unused blob stuff in #14637. ## PR Stack - [x] `mw/blob-batching` <- main feature - [x] ^ `mw/blob-batching-bls-utils` <- BLS12-381 bigcurve and bignum utils (noir) (#13583) - [x] ^ `mw/blob-batching-bls-utils-ts` <- BLS12-381 bigcurve and bignum utils (ts) (#13606) - [x] ^ `mw/blob-batching-integration` <- Integrate batching into noir protocol circuits (#13817) - [x] ^ `mw/blob-batching-integration-ts-sol` <- Integrate batching into ts and solidity (#14329) - [ ] ^ `mw/blob-batching-cleanup` <- Remove old blob code --------- Co-authored-by: Tom French <15848336+TomAFrench@users.noreply.github.com>

@iAmMichaelConnor

## The blobs are back in town. This PR reworks blobs so that instead of calling the point evaluation precompile for each blob (currently up to 3 per block => up to 96 (?) calls per epoch), we call it once per epoch by batching blobs to a single kzg commitment, opening, challenge, and proof. How we can be sure that this one pairing check is equivalent to a check per blob is covered in the maths by @iAmMichaelConnor [here](https://hackmd.io/WUtNusQxS5KAw-af3gxycA?view) 🎉 ## Overview Instead of pushing to a long array of `BlobPublicInputs`, which are then individually checked on L1, we batch each blob together to a single set of `BlobAccumulatorPublicInputs`. The `start` accumulator state is fed into each block root circuit, where the block's blobs are accumulated and the `end` state is set. Each block merge circuit checks that the state follows on correctly and, finally, the root circuit checks that the very `start` state was empty and finalises the last `end` state. This last `end` state makes up the set of inputs for the point evaluation precompile. If the pairing check in that precompile passes, we know that all blobs for all blocks in the epoch are valid and contain only the tx effects validated by the rollup. ### Circuits Key changes: - Integrate BLS12-381 curve operations with `bignum` and `bigcurve` libraries, plus tests. - Rework the `blob` package to batch blobs and store in reworked structs, plus tests. - Rework the rollup circuits from `block_root` above to handle blob accumulation state rather than a list of individual blob inputs, plus (you guessed it) tests. ### Contracts The contracts: - No longer call the point evaluation precompile on `propose`, instead inject the blob commitments, check they correspond to the broadcast blobs, and stores them in the `blobCommitmentsHash`. - Do not store any blob public inputs apart from the `blobCommitmentsHash`. - Call the point evaluation precompile once on `submitEpochRootProof` for ALL blobs in the epoch. - Use the same precompile inputs as pubic inputs to the root proof verification along with the `blobCommitmentsHash` to link the circuit batched blob, real L1 blobs, and the batched blob verified on L1. ### TypeScript Key changes: - Edit all the structs and methods reliant on the circuits/contracts to match the above changes. - Inject the final blob challenges used on each blob into all block building methods in `orchestrator`. - Accumulate blobs in ts when building blocks and use as inputs to each rollup circuit, plus tests. - Return the blob inputs required for `submitEpochRootProof` on `finaliseEpoch()`. ### TODOs/Related Issues - Choose field for hashing challenge: AztecProtocol#13608 - Instead of exponentiating `gamma` (expensive!), hash it for each iteration: AztecProtocol#13740 - Number of public inputs: BLS points in noir take up 9 fields (4 for each coordinate as a limbed bignum, 1 for the is_inf flag) but can be compressed to only 2. For recursive verification in block root and above, would it be worth the gates to compress these? It depends whether the gate cost of compression is more/less than gate cost of recursively verifying 7 more public inputs. - Remove the large trusted setup file from `yarn-project/blob-lib/src/trusted_setup_bit_reversed.json`? Used in testing, but may not be worth keeping (see code comments). - Cleanup old, unused blob stuff in AztecProtocol#14637. ## PR Stack - [x] `mw/blob-batching` <- main feature - [x] ^ `mw/blob-batching-bls-utils` <- BLS12-381 bigcurve and bignum utils (noir) (AztecProtocol#13583) - [x] ^ `mw/blob-batching-bls-utils-ts` <- BLS12-381 bigcurve and bignum utils (ts) (AztecProtocol#13606) - [x] ^ `mw/blob-batching-integration` <- Integrate batching into noir protocol circuits (AztecProtocol#13817) - [x] ^ `mw/blob-batching-integration-ts-sol` <- Integrate batching into ts and solidity (AztecProtocol#14329) - [ ] ^ `mw/blob-batching-cleanup` <- Remove old blob code --------- Co-authored-by: Tom French <15848336+TomAFrench@users.noreply.github.com>

MirandaWood added 4 commits April 15, 2025 19:27

feat: blob batching methods - nr only

97e2a01

chore: fmt, more tests, rearranging

52801e6

feat: BLS12 field, curve methods, blob batching methods, ts only

35e8be2

chore: lint, cleanup

db03339

MirandaWood requested a review from iAmMichaelConnor April 16, 2025 13:54

MirandaWood marked this pull request as ready for review April 16, 2025 13:54

MirandaWood mentioned this pull request Apr 16, 2025

feat(blobs): Add BLS12 field/curve methods for blob batching #13607

Closed

MirandaWood added 3 commits April 16, 2025 14:28

chore: remove trusted setup file + test using it (size issues)

68be71e

Revert "chore: remove trusted setup file + test using it (size issues)"

1fa5d49

This reverts commit 68be71e.

chore: cleanup packages + increase playground size

33d62a7

MirandaWood requested a review from Thunkar as a code owner April 17, 2025 13:53

MirandaWood commented Apr 17, 2025

View reviewed changes

iAmMichaelConnor approved these changes Apr 17, 2025

View reviewed changes

MirandaWood added 6 commits April 17, 2025 15:39

feat: address some comments, cleanup

46b8866

chore: update some comments

346ca9a

Merge remote-tracking branch 'origin/mw/blob-batching' into mw/blob-b…

f655bf5

…atching-bls-utils

Merge remote-tracking branch 'origin/mw/blob-batching-bls-utils' into…

8d57216

… mw/blob-batching-bls-utils-ts

chore: renaming, cleanup

9490422

chore: renaming, cleanup

c300c77

iAmMichaelConnor approved these changes Apr 22, 2025

View reviewed changes

MirandaWood added 4 commits April 22, 2025 15:32

chore: add issue nums (hopefully force ci cache reset)

63ffe96

feat: as isNegative to F, rename proof -> Q

606a942

Merge remote-tracking branch 'origin/mw/blob-batching' into mw/blob-b…

75d6d35

…atching-bls-utils

Merge remote-tracking branch 'origin/mw/blob-batching-bls-utils' into…

40ffd8b

… mw/blob-batching-bls-utils-ts

MirandaWood mentioned this pull request Apr 22, 2025

refactor(blobs): Extract y > (p-1)/2 into its own function #13730

Closed

MirandaWood added 4 commits April 22, 2025 16:10

chore: bumped vite kb limit 1700 -> 1720

951d329

chore: bumped vite kb limit 1700 -> 1750

307cb09

feat: adding helpers, constants, docs, etc. for integration

5251cd1

Merge remote-tracking branch 'origin/mw/blob-batching-bls-utils' into…

de8ec1e

… mw/blob-batching-bls-utils-ts

MirandaWood added 4 commits May 22, 2025 10:17

Merge remote-tracking branch 'origin/mw/blob-batching-bls-utils' into…

1b7fbf0

… mw/blob-batching-bls-utils-ts

Merge remote-tracking branch 'origin/mw/blob-batching' into mw/blob-b…

8667c5c

…atching-bls-utils

chore: bump bignum

ff52662

Merge remote-tracking branch 'origin/mw/blob-batching-bls-utils' into…

0c91085

… mw/blob-batching-bls-utils-ts

LeilaWang reviewed May 28, 2025

View reviewed changes

MirandaWood added 3 commits May 28, 2025 15:03

feat: address some comments

3064028

chore: test using toEqual in jest

b358b3e

feat: init bigint and buffer, remove static compress

fb8e45a

MirandaWood commented May 29, 2025

View reviewed changes

MirandaWood mentioned this pull request May 30, 2025

chore: post blob batching cleanup #14637

Merged

feat: replace empty blob assumption

c75232b

LeilaWang approved these changes May 30, 2025

View reviewed changes

MirandaWood mentioned this pull request May 30, 2025

feat!: blob batching #14648

Merged

6 tasks

iAmMichaelConnor approved these changes May 30, 2025

View reviewed changes

MirandaWood added 7 commits May 30, 2025 17:54

feat: address some comments

9e80ff2

Merge remote-tracking branch 'origin/mw/blob-batching-bls-utils' into…

2d1f35e

… mw/blob-batching-bls-utils-ts

chore: add extra check before blob acc init

ea19acd

Merge remote-tracking branch 'origin/mw/blob-batching-bls-utils' into…

e89fd4a

… mw/blob-batching-bls-utils-ts

chore: renaming, bring down changes from integration branch, cleanup

92f6e5f

Merge remote-tracking branch 'origin/mw/blob-batching-bls-utils' into…

88b4b28

… mw/blob-batching-bls-utils-ts

chore: cleanup, bring down changes from other PRs

0ee11fd

iAmMichaelConnor approved these changes Jun 3, 2025

View reviewed changes

Base automatically changed from mw/blob-batching-bls-utils to mw/blob-batching June 3, 2025 08:54

Merge remote-tracking branch 'origin/mw/blob-batching' into mw/blob-b…

f8ea4d2

…atching-bls-utils-ts

MirandaWood merged commit 2c45397 into mw/blob-batching Jun 3, 2025
3 of 4 checks passed

MirandaWood deleted the mw/blob-batching-bls-utils-ts branch June 3, 2025 09:19

Conversation

MirandaWood commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Stack

Uh oh!

MirandaWood Apr 17, 2025

Choose a reason for hiding this comment

Uh oh!

Thunkar Apr 21, 2025

Choose a reason for hiding this comment

Uh oh!

MirandaWood Apr 21, 2025

Choose a reason for hiding this comment

Uh oh!

MirandaWood Apr 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iAmMichaelConnor left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iAmMichaelConnor commented Apr 22, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LeilaWang May 28, 2025

Choose a reason for hiding this comment

Uh oh!

MirandaWood May 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MirandaWood May 29, 2025

Choose a reason for hiding this comment

Uh oh!

LeilaWang left a comment

Choose a reason for hiding this comment

Uh oh!

iAmMichaelConnor left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MirandaWood commented Apr 16, 2025 •

edited

Loading

MirandaWood Apr 22, 2025 •

edited

Loading