feat(avm): merkle db hints (part 3) by fcarreiro · Pull Request #13199 · AztecProtocol/aztec-packages

fcarreiro · 2025-04-01T11:56:42Z

Sequential inserts for PUBLIC_DATA_TREE and NULLIFIER_TREE.

Bulk test yields

Initializing HintedRawMerkleDB with...
 * get_sibling_path hints: 39
 * get_previous_value_index hints: 37
 * get_leaf_preimage hints_public_data_tree: 28
 * get_leaf_preimage hints_nullifier_tree: 9
 * get_leaf_value_hints: 0
 * sequential_insert_hints_public_data_tree: 7
 * sequential_insert_hints_nullifier_tree: 2

fcarreiro · 2025-04-01T11:56:58Z

feat(avm): tx hint init #13218
feat(avm): merkle db hints (part 3) #13199 👈 (View in Graphite)
refactor: align TS and C++ indexed leaf types #13185
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

fcarreiro · 2025-04-01T11:58:47Z

barretenberg/cpp/src/barretenberg/vm2/simulation/lib/db_interfaces.hpp

        crypto::merkle_tree::index_t leaf_index) const = 0;
+
+    // Inserts a leaf into the public data tree sequentially, getting witnesses at every step.
+    // Note: This method doesn't support inserting empty leaves.


Comment added by cursor 🤷

barretenberg/cpp/src/barretenberg/vm2/simulation/lib/raw_data_dbs.cpp

fcarreiro · 2025-04-01T12:11:41Z

yarn-project/stdlib/src/avm/avm.ts

+// NOTE: I need this factory because in order to get hold of the schema, I need an actual instance of the class,
+// having the type doesn't suffice since TS does type erasure in the end.
+function AvmSequentialInsertHintFactory(klass: IndexedTreeLeafPreimagesClasses) {
+  return class AvmSequentialInsertHint {


@sirasistant this is everything that we'd get in C++ for an insertion (no other side-merkle op is triggered). Looks ok?

I think it looks good!

dbanks12

LGTM! Probably good to get @sirasistant's review too.

dbanks12 · 2025-04-01T15:15:05Z

barretenberg/cpp/src/barretenberg/vm2/common/avm_inputs.hpp

+template <typename Leaf> struct SequentialInsertHint {
+    AppendOnlyTreeSnapshot hintKey;
+    AppendOnlyTreeSnapshot stateAfter;


Meaning the "hintKey" is the start state? Maybe just add a comment since this member has two meanings.

Maybe I'll just move the stateAfter to after the //return section. All other merkle hints share this hintKey field.

dbanks12 · 2025-04-01T15:18:24Z

barretenberg/cpp/src/barretenberg/vm2/simulation/lib/raw_data_dbs.cpp

+world_state::SequentialInsertionResult<crypto::merkle_tree::PublicDataLeafValue> HintedRawMerkleDB::
+    insert_indexed_leaves_public_data_tree(const crypto::merkle_tree::PublicDataLeafValue& leaf_value)
+{
+    auto tree_info = get_tree_info(world_state::MerkleTreeId::PUBLIC_DATA_TREE);
+    SequentialInsertHintPublicDataTreeKey key = { tree_info, world_state::MerkleTreeId::PUBLIC_DATA_TREE, leaf_value };
+    auto it = sequential_insert_hints_public_data_tree.find(key);
+    if (it == sequential_insert_hints_public_data_tree.end()) {
+        throw std::runtime_error(format("Sequential insert hint (PUBLIC_DATA_TREE) not found for key (root: ",
+                                        tree_info.root,
+                                        ", size: ",
+                                        tree_info.nextAvailableLeafIndex,
+                                        ", leaf_value: ",
+                                        leaf_value,
+                                        ")"));
+    }
+    const auto& hint = it->second;
+
+    world_state::SequentialInsertionResult<crypto::merkle_tree::PublicDataLeafValue> result;
+
+    // Convert low leaves witness data
+    result.low_leaf_witness_data.emplace_back(
+        hint.lowLeavesWitnessData.leaf, hint.lowLeavesWitnessData.index, hint.lowLeavesWitnessData.path);
+
+    // Convert insertion witness data
+    result.insertion_witness_data.emplace_back(
+        hint.insertionWitnessData.leaf, hint.insertionWitnessData.index, hint.insertionWitnessData.path);
+
+    // Evolve state.
+    tree_roots.publicDataTree.root = hint.stateAfter.root;
+    tree_roots.publicDataTree.nextAvailableLeafIndex = hint.stateAfter.nextAvailableLeafIndex;
+
+    debug("Evolved state of PUBLIC_DATA_TREE: ",
+          tree_roots.publicDataTree.root,
+          " (size: ",
+          tree_roots.publicDataTree.nextAvailableLeafIndex,
+          ")");
+
+    return result;
+}


Will this function eventually make a call to the public data write gadget?

Nope! The other way around.

sirasistant · 2025-04-02T13:41:34Z

barretenberg/cpp/src/barretenberg/vm2/common/avm_inputs.hpp

+    Leaf leaf;
+    // return
+    crypto::merkle_tree::LeafUpdateWitnessData<Leaf> lowLeavesWitnessData;
+    crypto::merkle_tree::LeafUpdateWitnessData<Leaf> insertionWitnessData;


Empty if it's update?

Looks like so yup

sirasistant

fantastic

sirasistant · 2025-04-02T14:17:39Z

yarn-project/stdlib/src/avm/avm.ts

+// NOTE: I need this factory because in order to get hold of the schema, I need an actual instance of the class,
+// having the type doesn't suffice since TS does type erasure in the end.
+function AvmSequentialInsertHintFactory(klass: IndexedTreeLeafPreimagesClasses) {
+  return class AvmSequentialInsertHint {


I think it looks good!

🤖 I have created a new Aztec Packages release --- ## [0.83.0](v0.82.3...v0.83.0) (2025-04-03) ### ⚠ BREAKING CHANGES * operation mouthwash ([#13171](#13171)) * change version and protocol version to rollupVersion ([#13145](#13145)) * processing events in Aztec.nr ([#12957](#12957)) ### Features * accept multiple consensus client endpoints ([#13022](#13022)) ([1d0185d](1d0185d)) * add ecdsa non ssh account to cli wallet ([#13085](#13085)) ([c5f9984](c5f9984)) * Add parent log link to top of CI logs. ([#13170](#13170)) ([a851087](a851087)) * Add public data read gadget ([#13138](#13138)) ([6bb76db](6bb76db)) * **avm:** merkle db hints (part 3) ([#13199](#13199)) ([7f96676](7f96676)) * **Barretenberg:** static analyzer's routine ([#13207](#13207)) ([84890c2](84890c2)) * derived pending notes capsules slot ([#13102](#13102)) ([6307ba0](6307ba0)) * get mana limit from rollup by default. ([#13029](#13029)) ([a406c54](a406c54)) * Increase CIVC depth with no rollup cost ([#13106](#13106)) ([3a555ef](3a555ef)) * making SyncDataProvider throw before sync ([#13151](#13151)) ([9840241](9840241)) * more benchmarks ([#13103](#13103)) ([7a2c9b7](7a2c9b7)) * **noir:** Allow missing optional fields in msgpack ([#13141](#13141)) ([493dede](493dede)) * processing events in Aztec.nr ([#12957](#12957)) ([88c0e04](88c0e04)) * Prover id defaults to publisher address ([#13206](#13206)) ([f459c3e](f459c3e)) * purge of log decryption in TS ([#12992](#12992)) ([800ab8d](800ab8d)) * register private-only contracts in cli-wallet and misc improvements ([#13245](#13245)) ([ad0530f](ad0530f)) * remove unary trick in decomposition and constraints polishing ([#13080](#13080)) ([0e60255](0e60255)) * split public inputs from proof ([#12816](#12816)) ([ca7151e](ca7151e)) * use magic address for fee juice portal in msgs ([#13241](#13241)) ([6fbc378](6fbc378)) * util for computing proposer/forwarder address ([#13169](#13169)) ([87809b2](87809b2)) * Zw/goblin avm tests ([#12904](#12904)) ([baea4b3](baea4b3)) ### Bug Fixes * add check for rollup version in tx validator ([#13197](#13197)) ([c8220c9](c8220c9)), closes [#13192](#13192) * add flake ([13bdfa7](13bdfa7)) * add flakes ([547d50a](547d50a)) * **avm:** alu interface ([#13115](#13115)) ([101ff78](101ff78)) * await transaction to make proposal ([#13132](#13132)) ([180db9e](180db9e)) * bb mac publish on tag push ([#13174](#13174)) ([8d46c8c](8d46c8c)) * bb workdir permission issue ([#13095](#13095)) ([3685a80](3685a80)) * **bb:** dont publish .zst ([#13173](#13173)) ([4dba4e9](4dba4e9)) * boolean config helper for cli args works now ([#13110](#13110)) ([a93ce6e](a93ce6e)) * **docs:** Register FPC docs, contract deployment, events ([#13222](#13222)) ([49aabfb](49aabfb)) * don't log sepolia acc mnemonic on github action ([#13178](#13178)) ([809b44d](809b44d)) * eq instead of !== ([#13161](#13161)) ([921e347](921e347)) * fetch the correct vk in getSolidityVerifier ([#13157](#13157)) ([71b6719](71b6719)) * force anvil/blob networking to ipv4 on localhost. attempt to fix port flakes ([#13099](#13099)) ([970dae5](970dae5)) * fuzzing build issues ([#13114](#13114)) ([bacae3d](bacae3d)) * Handle proven chain events referring to unseen blocks ([#13144](#13144)) ([229515f](229515f)), closes [#13142](#13142) * handling multiple identical logs in a tx ([#13184](#13184)) ([c100499](c100499)) * incorrect blocknumber in syncTaggedLogs ([e58ea95](e58ea95)) * indexeddb multimap dupes ([#13254](#13254)) ([1e470f0](1e470f0)) * load two more points ([#13119](#13119)) ([2a2904a](2a2904a)) * Masternet to run with the nightly tag ([#13239](#13239)) ([f16b70e](f16b70e)) * newline weirdness in package.json ([#13111](#13111)) ([244ea99](244ea99)) * nightly deploy quotes ([#13253](#13253)) ([775eecf](775eecf)) * prover config read from L1 ([#13237](#13237)) ([13426fe](13426fe)) * Race condition while unwinding blocks ([#13148](#13148)) ([1c2291a](1c2291a)) * read and pass rollup version ([#13232](#13232)) ([4ee1e4a](4ee1e4a)) * recursive sumcheck bugs ([#12885](#12885)) ([a784802](a784802)) * reenable test cache ([4dfca94](4dfca94)) * remove [skip ci] edge condition ([#13228](#13228)) ([4e699ea](4e699ea)) * separator in pending partial notes capsule array slot ([#13153](#13153)) ([0d6ec63](0d6ec63)) * sysdig oops ([5cf6ad1](5cf6ad1)) * tagging bug ([#13061](#13061)) ([280e1a6](280e1a6)) * test tracking failures in merge queue ([#13235](#13235)) ([650fdc1](650fdc1)) * title and external check need to run in merge queue ([e326ed6](e326ed6)) * Transpile cmov ([#13194](#13194)) ([6b94555](6b94555)) * trying to fix EADDRINUSE ([#13176](#13176)) ([260a057](260a057)) * Uninstall gossipsub event handler on service stop ([#13190](#13190)) ([081f30d](081f30d)) * use version from registry for rollup instead of config ([#12938](#12938)) ([7dac390](7dac390)) * validate private double spends in txs with public funcs ([#13088](#13088)) ([4555871](4555871)) * yolo flake and alert fix ([f8de52b](f8de52b)) ### Miscellaneous * add container id to netlog ([9ecdf15](9ecdf15)) * Add e2e test to bootstrap test_cmds ([#13146](#13146)) ([09e4722](09e4722)) * add rollup version as universal cli option ([#13205](#13205)) ([#13213](#13213)) ([568d9e9](568d9e9)) * Add ultra versions of fuzzers in stdlib ([#13139](#13139)) ([aea210b](aea210b)) * align TS and C++ indexed leaf types ([#13185](#13185)) ([d2574cc](d2574cc)) * Assign bb test flake ([#13127](#13127)) ([69fdb04](69fdb04)) * **avm:** remove check_interaction from tests ([#13136](#13136)) ([7d875a6](7d875a6)) * change version and protocol version to rollupVersion ([#13145](#13145)) ([24d7f8b](24d7f8b)) * Cleanup config vars for alpha-testnet ([#13204](#13204)) ([153a606](153a606)) * convenient way to run app ivc from bb ([#13158](#13158)) ([cb1a857](cb1a857)) * Cron snapshot upload in spartan ([#13108](#13108)) ([7c520a8](7c520a8)) * Do not close store before stopping p2p client in tests ([#13223](#13223)) ([63a843e](63a843e)) * docker flake diagnostics. ([48da272](48da272)) * **docs:** alpha-testnet versioning ([#13016](#13016)) ([9ca5d3c](9ca5d3c)) * **docs:** Update CLI faucet command ([#13104](#13104)) ([6eb71de](6eb71de)) * **docs:** Update versions-updating.md ([#13090](#13090)) ([0310d4e](0310d4e)) * Enable debug logging for annoying unit tests ([#13191](#13191)) ([5556524](5556524)) * enable sentinel in sentinel test ([#13219](#13219)) ([f2cfe3f](f2cfe3f)) * Exclude nightly versions from stable fn ([#13196](#13196)) ([a832e90](a832e90)) * Fix flake in e2e fees failures ([#13229](#13229)) ([5defe47](5defe47)) * fix kind logs ([#13023](#13023)) ([c6c2727](c6c2727)), closes [#13053](#13053) * fuzzing build in ci ([#13105](#13105)) ([1c08d38](1c08d38)) * Improve callstacks for public dispatch fns ([#13120](#13120)) ([f67375d](f67375d)) * log out the slash factory when a new rollup is deployed ([#13131](#13131)) ([4643a31](4643a31)) * Merge alpha back to master ([#13128](#13128)) ([504c338](504c338)) * metric attributes ([#13126](#13126)) ([f87d5e3](f87d5e3)), closes [#13063](#13063) * minor tagging API improvement ([#13092](#13092)) ([f5bcece](f5bcece)) * more benchmarking ([#13211](#13211)) ([4b0a4ad](4b0a4ad)) * move unbound impl generics to their functions ([#13147](#13147)) ([62f497f](62f497f)) * operation mouthwash ([#13171](#13171)) ([384f4e5](384f4e5)) * remove catch-all branch from opcode match in transpiler ([#13210](#13210)) ([2bc9ca2](2bc9ca2)) * remove templating by flavor in merge protocol ([#13098](#13098)) ([cf5e217](cf5e217)) * rename journal dir and file to state manager & mv to up to public/ ([#13159](#13159)) ([f13be09](f13be09)) * replace relative paths to noir-protocol-circuits ([f18336d](f18336d)) * replace relative paths to noir-protocol-circuits ([d620c12](d620c12)) * replace relative paths to noir-protocol-circuits ([be3e74e](be3e74e)) * replace relative paths to noir-protocol-circuits ([5bfac63](5bfac63)) * replace relative paths to noir-protocol-circuits ([f41ab2c](f41ab2c)) * replace relative paths to noir-protocol-circuits ([806e560](806e560)) * streamlined log processing ([#13107](#13107)) ([b184865](b184865)) * use testnet optimized trace ([#13135](#13135)) ([5a4f2ac](5a4f2ac)) * uses a new larger redis cache. flake file tweaks. ([#13167](#13167)) ([0951fb6](0951fb6)) ### Documentation * Add intermediate migration note versions ([#13183](#13183)) ([f5de7b8](f5de7b8)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

fcarreiro mentioned this pull request Apr 1, 2025

refactor: align TS and C++ indexed leaf types #13185

Merged

fcarreiro requested review from dbanks12 and sirasistant April 1, 2025 11:57

fcarreiro commented Apr 1, 2025

View reviewed changes

barretenberg/cpp/src/barretenberg/vm2/simulation/lib/raw_data_dbs.cpp Show resolved Hide resolved

fcarreiro force-pushed the fc/avm-sequentialinsert-hints branch from 8b22161 to dfdc7dc Compare April 1, 2025 12:06

fcarreiro commented Apr 1, 2025

View reviewed changes

fcarreiro changed the base branch from fc/avm-hints-prework to graphite-base/13199 April 1, 2025 14:00

fcarreiro force-pushed the fc/avm-sequentialinsert-hints branch from dfdc7dc to 21a0c5b Compare April 1, 2025 14:03

fcarreiro changed the base branch from graphite-base/13199 to fc/avm-hints-prework April 1, 2025 14:03

fcarreiro force-pushed the fc/avm-hints-prework branch from 1f74223 to 198412a Compare April 1, 2025 14:18

fcarreiro force-pushed the fc/avm-sequentialinsert-hints branch from 21a0c5b to b8dead2 Compare April 1, 2025 14:18

fcarreiro force-pushed the fc/avm-hints-prework branch from 198412a to 93af52e Compare April 1, 2025 14:40

fcarreiro force-pushed the fc/avm-sequentialinsert-hints branch 2 times, most recently from 3149d01 to 94a8201 Compare April 1, 2025 15:07

fcarreiro force-pushed the fc/avm-hints-prework branch from 93af52e to 5a085ed Compare April 1, 2025 15:07

dbanks12 approved these changes Apr 1, 2025

View reviewed changes

fcarreiro force-pushed the fc/avm-sequentialinsert-hints branch from 94a8201 to b0de1c2 Compare April 1, 2025 15:47

fcarreiro force-pushed the fc/avm-hints-prework branch from 9e239c1 to c730ad7 Compare April 1, 2025 15:59

fcarreiro force-pushed the fc/avm-sequentialinsert-hints branch 2 times, most recently from 5972d5a to a0c5866 Compare April 1, 2025 16:47

fcarreiro mentioned this pull request Apr 1, 2025

feat(avm): tx hint init #13218

Merged

fcarreiro force-pushed the fc/avm-sequentialinsert-hints branch 2 times, most recently from 2965864 to 6592cea Compare April 2, 2025 10:26

fcarreiro force-pushed the fc/avm-hints-prework branch 2 times, most recently from 0d8ffaf to d1df374 Compare April 2, 2025 11:54

fcarreiro force-pushed the fc/avm-sequentialinsert-hints branch from 6592cea to 1aa0ebd Compare April 2, 2025 11:54

fcarreiro changed the base branch from fc/avm-hints-prework to graphite-base/13199 April 2, 2025 12:30

sirasistant reviewed Apr 2, 2025

View reviewed changes

sirasistant approved these changes Apr 2, 2025

View reviewed changes

fcarreiro force-pushed the graphite-base/13199 branch from d1df374 to 1d0185d Compare April 2, 2025 14:24

fcarreiro force-pushed the fc/avm-sequentialinsert-hints branch from 1aa0ebd to 748972a Compare April 2, 2025 14:24

fcarreiro changed the base branch from graphite-base/13199 to master April 2, 2025 14:24

fcarreiro marked this pull request as ready for review April 2, 2025 14:25

fcarreiro requested review from IlyasRidhuan and jeanmon as code owners April 2, 2025 14:25

fcarreiro enabled auto-merge April 2, 2025 14:25

fcarreiro force-pushed the fc/avm-sequentialinsert-hints branch from 748972a to a5282e1 Compare April 2, 2025 14:51

feat(avm): merkle db hints (part 3)

b354130

fcarreiro force-pushed the fc/avm-sequentialinsert-hints branch from a5282e1 to b354130 Compare April 2, 2025 15:45

mac fixes

2877ca9

fcarreiro added this pull request to the merge queue Apr 2, 2025

Merged via the queue into master with commit 7f96676 Apr 2, 2025
8 checks passed

fcarreiro deleted the fc/avm-sequentialinsert-hints branch April 2, 2025 17:53

AztecBot mentioned this pull request Apr 2, 2025

chore(master): release 0.83.0 #13112

Merged

Conversation

fcarreiro commented Apr 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fcarreiro commented Apr 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dbanks12 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sirasistant left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fcarreiro commented Apr 1, 2025 •

edited

Loading

fcarreiro commented Apr 1, 2025 •

edited

Loading