Skip to content

feat!: batch note validation check#14595

Merged
nventuro merged 9 commits intonextfrom
nv/process-pending-notes
May 29, 2025
Merged

feat!: batch note validation check#14595
nventuro merged 9 commits intonextfrom
nv/process-pending-notes

Conversation

@nventuro
Copy link
Contributor

@nventuro nventuro commented May 28, 2025

This massively improves performance of the note discovery process by enabling batching of note validation requests (i.e. checking that notes indeed exist in the note hash tree). We achieve this by migrating from sequential steps to a staged approach.

This continues the trend set by #13107 where we began to split the message discovery process into multiple steps (now modeled by aztec::message::processing from #14567), each of which starts and ends with a capsule array. This lets us consume and produce arbitrary amounts of values in unconstrained Noir functions, without having to deal with the limitations of fixed sized arrays. As of this PR, the process looks as follows:

  1. (pxe): contract simulation starts
  2. (nr) message discovery calls fetchTaggedLogs oracle
  3. (pxe) log discovery is performed by comunicating with the node, logs are placed in a capsule arrray
  4. (nr) each log is read from the capsule array, decrypted, decoded and processed. note validation requests are placed in a capsule array. pending partial notes are placed in a capsule array.
  5. (nr) pending partial notes are processed one by one (slow)
  6. (pxe) note validation requests are completed by comunicating with the node (in parallel!)

This pattern sidesteps having to deal with fixed-size oracles (which would require e.g. pagination, an even more stateful PXE) by leveraging the capsule array as the main building tool. It should be simple to later extend this to e.g. produce a capsule array of tags to be searched for, which would result in a second capsule array being populated with the results, so that partial note completion log queries need not be completed serially.

The obvious downside of this approach is that it introduces relatively strange oracle semantics, and results in comunication from the contract and pxe to occur not just through the oracle interfaces, but also the values in the capsule arrays (including their serialization).

@nventuro nventuro marked this pull request as ready for review May 28, 2025 20:10
@Thunkar Thunkar self-requested a review May 29, 2025 14:38
@nventuro nventuro enabled auto-merge May 29, 2025 18:31
@nventuro nventuro added this pull request to the merge queue May 29, 2025
github-merge-queue bot pushed a commit that referenced this pull request May 29, 2025
Discovered through
#14595

High probability of it being the source of boxes flakyness
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to a conflict with the base branch May 29, 2025
@nventuro nventuro enabled auto-merge May 29, 2025 19:26
@nventuro nventuro added this pull request to the merge queue May 29, 2025
Merged via the queue into next with commit 3777a06 May 29, 2025
4 checks passed
@nventuro nventuro deleted the nv/process-pending-notes branch May 29, 2025 20:15
github-merge-queue bot pushed a commit that referenced this pull request Jun 2, 2025
Combined with #14595, this will mean regular private notes will be
processed in two roundtrips: one for events, and one for note/nullifier
indexes (for all events and notes concurrently, not accounting for the
advancing of the tagging indexes).
github-merge-queue bot pushed a commit that referenced this pull request Jun 2, 2025
Combined with #14595, this will mean regular private notes will be
processed in two roundtrips: one for events, and one for note/nullifier
indexes (for all events and notes concurrently, not accounting for the
advancing of the tagging indexes).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants