Merged
Conversation
spalladino
changed the title
spalladino
force-pushed
the
palla/snapshots-stdlib
branch
from
e415c21 to
50b4d8a
Compare
Maddiaa0
reviewed
Mar 25, 2025
| } | ||
|
|
||
| /** | ||
| * Forks the current DB into a new DB by backing it up to a temporary location and opening a new lmdb db. |
Maddiaa0
reviewed
Mar 25, 2025
|
|
||
| const INPUTS_PATH = 'inputs'; | ||
|
|
||
| // REFACTOR: Use the stdlib/file-store instead of referencing google-cloud-storage directly. |
Maddiaa0
reviewed
Mar 25, 2025
|
|
||
| const basePath = getBasePath(metadata); | ||
| const [archiverDataUrl, worldStateDataUrl] = await Promise.all([ | ||
| store.upload(`${basePath}/archiver-${date}-${metadata.l2BlockHash}.db`, archiverPath, { compress: true }), |
Member
There was a problem hiding this comment.
Is there a strong deason to have the date in the name, at a first glance it would make downloading more difficult, i suppose this is counteracted by the metadata, but im not convinced the metadata is required if we can just request based on the archive
Member
There was a problem hiding this comment.
Ah i see there is one index.json that points to the most recent snapshot
spalladino
force-pushed
the
palla/snapshots-stdlib
branch
from
50b4d8a to
6a22ba7
Compare
Contributor
Author
|
@Maddiaa0 I added commit "Wire snapshot upload" that completes the loop, feel free to review just that one. |
Contributor
Author
|
This is what an $ cat /tmp/aztec/aztec-31337-1-0x26db1dbde404abe656c354e6331217ea7c9cc21e/index.json {
"l1ChainId": 31337,
"l2Version": 1,
"rollupAddress": "0x26db1dbde404abe656c354e6331217ea7c9cc21e",
"snapshots": [
{
"l1BlockNumber": 131,
"l2BlockHash": "0x1ddc5c6b6464ac19f27e1d6ed86dab06d7b834a20d5a785825522c8d82d521c3",
"l2BlockNumber": 31,
"schemaVersions": {
"archiver": 1,
"worldState": 1
},
"timestamp": 1742940387189,
"dataUrls": {
"l1-to-l2-message-tree": "file:///tmp/aztec/aztec-31337-1-0x26db1dbde404abe656c354e6331217ea7c9cc21e/l1-to-l2-message-tree-20250325220627-0x1ddc5c6b6464ac19f27e1d6ed86dab06d7b834a20d5a785825522c8d82d521c3.db",
"archive-tree": "file:///tmp/aztec/aztec-31337-1-0x26db1dbde404abe656c354e6331217ea7c9cc21e/archive-tree-20250325220627-0x1ddc5c6b6464ac19f27e1d6ed86dab06d7b834a20d5a785825522c8d82d521c3.db",
"public-data-tree": "file:///tmp/aztec/aztec-31337-1-0x26db1dbde404abe656c354e6331217ea7c9cc21e/public-data-tree-20250325220627-0x1ddc5c6b6464ac19f27e1d6ed86dab06d7b834a20d5a785825522c8d82d521c3.db",
"note-hash-tree": "file:///tmp/aztec/aztec-31337-1-0x26db1dbde404abe656c354e6331217ea7c9cc21e/note-hash-tree-20250325220627-0x1ddc5c6b6464ac19f27e1d6ed86dab06d7b834a20d5a785825522c8d82d521c3.db",
"nullifier-tree": "file:///tmp/aztec/aztec-31337-1-0x26db1dbde404abe656c354e6331217ea7c9cc21e/nullifier-tree-20250325220627-0x1ddc5c6b6464ac19f27e1d6ed86dab06d7b834a20d5a785825522c8d82d521c3.db",
"archiver": "file:///tmp/aztec/aztec-31337-1-0x26db1dbde404abe656c354e6331217ea7c9cc21e/archiver-20250325220627-0x1ddc5c6b6464ac19f27e1d6ed86dab06d7b834a20d5a785825522c8d82d521c3.db"
}
},
{
"l2BlockNumber": 24,
"l2BlockHash": "0x06e938f0668bb121838607e573d3aa21b0a0ed3a247655de5849887ea7bc5425",
"l1BlockNumber": 117,
"timestamp": 1742940331978,
"schemaVersions": {
"archiver": 1,
"worldState": 1
},
"dataUrls": {
"l1-to-l2-message-tree": "file:///tmp/aztec/aztec-31337-1-0x26db1dbde404abe656c354e6331217ea7c9cc21e/l1-to-l2-message-tree-20250325220531-0x06e938f0668bb121838607e573d3aa21b0a0ed3a247655de5849887ea7bc5425.db",
"archive-tree": "file:///tmp/aztec/aztec-31337-1-0x26db1dbde404abe656c354e6331217ea7c9cc21e/archive-tree-20250325220531-0x06e938f0668bb121838607e573d3aa21b0a0ed3a247655de5849887ea7bc5425.db",
"public-data-tree": "file:///tmp/aztec/aztec-31337-1-0x26db1dbde404abe656c354e6331217ea7c9cc21e/public-data-tree-20250325220531-0x06e938f0668bb121838607e573d3aa21b0a0ed3a247655de5849887ea7bc5425.db",
"note-hash-tree": "file:///tmp/aztec/aztec-31337-1-0x26db1dbde404abe656c354e6331217ea7c9cc21e/note-hash-tree-20250325220531-0x06e938f0668bb121838607e573d3aa21b0a0ed3a247655de5849887ea7bc5425.db",
"nullifier-tree": "file:///tmp/aztec/aztec-31337-1-0x26db1dbde404abe656c354e6331217ea7c9cc21e/nullifier-tree-20250325220531-0x06e938f0668bb121838607e573d3aa21b0a0ed3a247655de5849887ea7bc5425.db",
"archiver": "file:///tmp/aztec/aztec-31337-1-0x26db1dbde404abe656c354e6331217ea7c9cc21e/archiver-20250325220531-0x06e938f0668bb121838607e573d3aa21b0a0ed3a247655de5849887ea7bc5425.db"
}
}
]
} |
Utility functions for managing snapshot indices and uploading and downloading from gcs.
spalladino
force-pushed
the
palla/snapshots-stdlib
branch
from
29b0750 to
104c922
Compare
alexghr
reviewed
Mar 26, 2025
Contributor
alexghr
left a comment
alexghr
left a comment
There was a problem hiding this comment.
Code looks good to me. Just a couple of questions around tightening up access to the snapshot mechanism
Comment on lines
+56
to
+60
| public async backupTo(path: string, compress = true): Promise<string> { | ||
| await this.db.backupTo(path, compress); | ||
| return join(path, 'data.mdb'); | ||
| } | ||
|
|
Contributor
There was a problem hiding this comment.
Not something for this PR but we should consider bundling the rollup address/schema version into the backup to avoid restoring incompatible versions?
Contributor
There was a problem hiding this comment.
Forgot to go back and update this comment: this is handled already. This info is part of the snapshot index
Contributor
Author
There was a problem hiding this comment.
I'm adding it to the snapshot metadata. We could write those as entries of the db itself though, and completely do away with the db_version file we use today. WDYT?
Comment on lines
+51
to
+54
| public async download(pathOrUrlStr: string, destPath: string): Promise<void> { | ||
| const file = await this.getFileObject(pathOrUrlStr); | ||
| await file.download({ destination: destPath }); | ||
| } |
Contributor
There was a problem hiding this comment.
Same note as above, bur for downloading
Comment on lines
+39
to
+45
| private getFullPath(pathOrUrl: string): string { | ||
| if (URL.canParse(pathOrUrl)) { | ||
| return new URL(pathOrUrl).pathname; | ||
| } else { | ||
| return resolve(this.basePath, pathOrUrl); | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
Should basePath always be enforced on reads and writes? Otherwise we can read/write arbitrary paths through this class
| return this.validatorsSentinel?.computeStats() ?? Promise.resolve({ stats: {}, slotWindow: 0 }); | ||
| } | ||
|
|
||
| public async startSnapshotUpload(location: string): Promise<void> { |
Contributor
There was a problem hiding this comment.
Hm, is this part of the admin API on the node? I'm just thinking someone could repeatedly call this function and fill up our disk
Contributor
Author
There was a problem hiding this comment.
Yep, it's admin API, and requires permissions on the remote file store. So we should be the only ones allowed to call it for our nodes and our bucket.
alexghr
reviewed
Mar 26, 2025
| }; | ||
| snapshotsIndex.snapshots.unshift(newSnapshotMetadata); | ||
|
|
||
| await store.save(getSnapshotIndexPath(metadata), Buffer.from(jsonStringify(snapshotsIndex, true))); |
Contributor
There was a problem hiding this comment.
A bit worried if this fails (pod dies or network issues) then we'd left with an invalid index file.
Contributor
Author
There was a problem hiding this comment.
I understand gcs ensures we don't end up with broken objects if an upload is aborted halfway. And the local store is just used for testing.
spalladino
requested review from
dbanks12,
fcarreiro and
sirasistant
as code owners
Contributor
Author
|
@alexghr I've addressed all comments or added issues, and renamed fast sync to snapshot (to avoid confusion with geth fast sync which is different). Other items remaining are:
|
Contributor
Author
|
Sample run. We start a node and then ask it to trigger a snapshot: The node pauses syncing, back ups databases, and resumes syncing while it uploads snapshot data to gcs: We run it again for the sake of it. The index at {
"l1ChainId": 31337,
"l2Version": 1,
"rollupAddress": "0x706f4477c3a1b1f123ba06b688df5c84f935b747",
"snapshots": [
{
"l1BlockNumber": 153,
"l2BlockHash": "0x2fcd18f31f17a50f726ccd73beb00204cea0e7e4d7fde5a680d9bac81d0f63ee",
"l2BlockNumber": 73,
"schemaVersions": {
"archiver": 1,
"worldState": 1
},
"timestamp": 1743085415918,
"dataUrls": {
"l1-to-l2-message-tree": "https://storage.googleapis.com/aztec-testnet/snapshots/aztec-31337-1-0x706f4477c3a1b1f123ba06b688df5c84f935b747/l1-to-l2-message-tree-20250327142335-0x2fcd18f31f17a50f726ccd73beb00204cea0e7e4d7fde5a680d9bac81d0f63ee.db",
"archive-tree": "https://storage.googleapis.com/aztec-testnet/snapshots/aztec-31337-1-0x706f4477c3a1b1f123ba06b688df5c84f935b747/archive-tree-20250327142335-0x2fcd18f31f17a50f726ccd73beb00204cea0e7e4d7fde5a680d9bac81d0f63ee.db",
"public-data-tree": "https://storage.googleapis.com/aztec-testnet/snapshots/aztec-31337-1-0x706f4477c3a1b1f123ba06b688df5c84f935b747/public-data-tree-20250327142335-0x2fcd18f31f17a50f726ccd73beb00204cea0e7e4d7fde5a680d9bac81d0f63ee.db",
"note-hash-tree": "https://storage.googleapis.com/aztec-testnet/snapshots/aztec-31337-1-0x706f4477c3a1b1f123ba06b688df5c84f935b747/note-hash-tree-20250327142335-0x2fcd18f31f17a50f726ccd73beb00204cea0e7e4d7fde5a680d9bac81d0f63ee.db",
"nullifier-tree": "https://storage.googleapis.com/aztec-testnet/snapshots/aztec-31337-1-0x706f4477c3a1b1f123ba06b688df5c84f935b747/nullifier-tree-20250327142335-0x2fcd18f31f17a50f726ccd73beb00204cea0e7e4d7fde5a680d9bac81d0f63ee.db",
"archiver": "https://storage.googleapis.com/aztec-testnet/snapshots/aztec-31337-1-0x706f4477c3a1b1f123ba06b688df5c84f935b747/archiver-20250327142335-0x2fcd18f31f17a50f726ccd73beb00204cea0e7e4d7fde5a680d9bac81d0f63ee.db"
}
},
{
"l2BlockNumber": 20,
"l2BlockHash": "0x1ab52595ee41c919647b5e5b580abe07d1e37e2df2e571a4afffb8884294857c",
"l1BlockNumber": 48,
"timestamp": 1743084997346,
"schemaVersions": {
"archiver": 1,
"worldState": 1
},
"dataUrls": {
"l1-to-l2-message-tree": "https://storage.googleapis.com/aztec-testnet/snapshots/aztec-31337-1-0x706f4477c3a1b1f123ba06b688df5c84f935b747/l1-to-l2-message-tree-20250327141637-0x1ab52595ee41c919647b5e5b580abe07d1e37e2df2e571a4afffb8884294857c.db",
"archive-tree": "https://storage.googleapis.com/aztec-testnet/snapshots/aztec-31337-1-0x706f4477c3a1b1f123ba06b688df5c84f935b747/archive-tree-20250327141637-0x1ab52595ee41c919647b5e5b580abe07d1e37e2df2e571a4afffb8884294857c.db",
"public-data-tree": "https://storage.googleapis.com/aztec-testnet/snapshots/aztec-31337-1-0x706f4477c3a1b1f123ba06b688df5c84f935b747/public-data-tree-20250327141637-0x1ab52595ee41c919647b5e5b580abe07d1e37e2df2e571a4afffb8884294857c.db",
"note-hash-tree": "https://storage.googleapis.com/aztec-testnet/snapshots/aztec-31337-1-0x706f4477c3a1b1f123ba06b688df5c84f935b747/note-hash-tree-20250327141637-0x1ab52595ee41c919647b5e5b580abe07d1e37e2df2e571a4afffb8884294857c.db",
"nullifier-tree": "https://storage.googleapis.com/aztec-testnet/snapshots/aztec-31337-1-0x706f4477c3a1b1f123ba06b688df5c84f935b747/nullifier-tree-20250327141637-0x1ab52595ee41c919647b5e5b580abe07d1e37e2df2e571a4afffb8884294857c.db",
"archiver": "https://storage.googleapis.com/aztec-testnet/snapshots/aztec-31337-1-0x706f4477c3a1b1f123ba06b688df5c84f935b747/archiver-20250327141637-0x1ab52595ee41c919647b5e5b580abe07d1e37e2df2e571a4afffb8884294857c.db"
}
}
]We then start another node with |
alexghr
approved these changes
Mar 27, 2025
spalladino
added a commit
that referenced
this pull request
Mar 27, 2025
PhilWindle
pushed a commit
that referenced
this pull request
Mar 27, 2025
🤖 I have created a new Aztec Packages release --- ## [0.82.3](v0.82.2...v0.82.3) (2025-03-27) ### Features * `msgpack` encoding for `Program` and `WitnessStack` ([#12841](#12841)) ([1e58eb1](1e58eb1)) * 64 bit log type id, 64 bit log metadata ([#12956](#12956)) ([20d734a](20d734a)) * AVM parsing tag validation ([#12936](#12936)) ([56b1f0d](56b1f0d)) * **avm:** add calldata & returndata to context ([#13008](#13008)) ([f03b2e5](f03b2e5)) * **avm:** merkle db hints (part 1) ([#12922](#12922)) ([34ec9e8](34ec9e8)) * **avm:** merkle hints (part 2) ([#13077](#13077)) ([fbbc6c7](fbbc6c7)) * **avm:** vm2 initial context ([#12972](#12972)) ([e2b1361](e2b1361)) * benchmark avm simulator ([#12985](#12985)) ([00fae1b](00fae1b)) * client flows benchmarks ([#13007](#13007)) ([9bf7568](9bf7568)) * gas benchmark for "normal usage" ([#13073](#13073)) ([4eb1156](4eb1156)) * Implement merkle writes in the merkle check gadget ([#13050](#13050)) ([c94fe50](c94fe50)) * LogEncryption trait ([#12942](#12942)) ([0b7e564](0b7e564)) * Node snapshot sync ([#12927](#12927)) ([afde851](afde851)), closes [#12926](#12926) * **p2p:** capture all gossipsub metrics ([#12930](#12930)) ([cc940cb](cc940cb)) * Prover node snapshot sync ([#13097](#13097)) ([1e77efb](1e77efb)) * staking asset handler ([#12968](#12968)) ([af48184](af48184)), closes [#12932](#12932) * stream crs data to disk ([#12996](#12996)) ([d016e4d](d016e4d)), closes [#12948](#12948) * track failed tests. add flake. ([f4936d7](f4936d7)) * Track test history. ([#13037](#13037)) ([036bb32](036bb32)) * track total tx fee ([#12601](#12601)) ([9612a4e](9612a4e)) * Validators sentinel ([#12818](#12818)) ([770695c](770695c)) ### Bug Fixes * added #[derive(Eq)] to EcdsaPublicKeyNote ([#12966](#12966)) ([0c21c74](0c21c74)) * Allow use of local blob sink client ([#13025](#13025)) ([ba8d654](ba8d654)) * **avm:** semicolons are hard ([#12999](#12999)) ([8871c83](8871c83)) * bootstrap network and sponsored fpc devnet ([#13044](#13044)) ([8a47d8b](8a47d8b)) * Bump tsc target ([#13052](#13052)) ([985e83b](985e83b)) * cycle_group fuzzer ([#12921](#12921)) ([69f426e](69f426e)) * **docs:** Fix import errors in aztec.js tutorial ([#12969](#12969)) ([856208a](856208a)) * **docs:** Load token artifact from the compiled source in the sample dapp tutorial ([#12802](#12802)) ([0838084](0838084)), closes [#12810](#12810) * **docs:** Update sponsored fpc docs to use 82.2 syntax ([#13054](#13054)) ([e5d425b](e5d425b)) * **e2e:** p2p ([#13002](#13002)) ([1ece539](1ece539)) * extend e2e 2 pxes timeout. strip color codes for error_regex. ([73820e4](73820e4)) * flake ([6cc9e81](6cc9e81)) * fuzzer on staking asset handler constructor test ([#13101](#13101)) ([d936285](d936285)) * invalid getCommittee function ([#13072](#13072)) ([327341f](327341f)) * mac publish should use clang 18 like x-compiler, and use it ([#12983](#12983)) ([7b83c45](7b83c45)) * make circuit parsing deterministic ([#11772](#11772)) ([76ef873](76ef873)) * parse away trailing slash from consensus host ([#12577](#12577)) ([6701806](6701806)) * prerelease versions should be pushed to install.aztec.network ([#13086](#13086)) ([c4e6039](c4e6039)) * smoke ([#13060](#13060)) ([7756b15](7756b15)) * some flake additions ([58638f1](58638f1)) * sponsored fpc arg parsed correctly ([#12976](#12976)) ([#12977](#12977)) ([a85f530](a85f530)) * starting the sandbox with no pxe should still deploy initial test accounts ([#13047](#13047)) ([d92d895](d92d895)) * Syntax error when running tests via jest after tsc build ([#13051](#13051)) ([f972db9](f972db9)) * Use the correct image in aztec start ([#13058](#13058)) ([06285cd](06285cd)) * yolo fix ([91e2f4b](91e2f4b)) * yolo fix nightly ([b3b3259](b3b3259)) * yolo fix obvious thing to track fails. ([2fee630](2fee630)) * yolo flakes ([e3b030a](e3b030a)) * yolo set -x ([bfd3205](bfd3205)) * yolo we suspect the halt is making tests fail that would have passed ([04e3fa2](04e3fa2)) ### Miscellaneous * `getIndexedTaggingSecretAsSender` oracle cleanup ([#13015](#13015)) ([8e71e55](8e71e55)) * Add a script to generate cpp files for AVM2 ([#13091](#13091)) ([7bb43a9](7bb43a9)) * add default native proving for cli wallet ([#12855](#12855)) ([c0f773c](c0f773c)) * add default native proving for cli wallet retry ([#13028](#13028)) ([b2f4785](b2f4785)) * Alpha testnet into master ([#13033](#13033)) ([d98fdbd](d98fdbd)) * AVM TS - move tag validation outside of instruction constructors ([#13038](#13038)) ([45548ab](45548ab)), closes [#12934](#12934) * **avm:** final codegen nuking ([#13089](#13089)) ([9c82f3f](9c82f3f)) * **avm:** remove codegen (all but flavor) ([#13079](#13079)) ([e1f2bdd](e1f2bdd)) * **bb:** minor acir buf C++ improvements ([#13042](#13042)) ([1ebd044](1ebd044)) * boxes dep cleanup ([#12979](#12979)) ([6540b7c](6540b7c)) * **ci:** less catch all e2e_p2p flakes ([#12737](#12737)) ([2134634](2134634)) * comprehensive cleanup of translator flavor and use inheritance properly in flavors ([#13041](#13041)) ([dc5f78f](dc5f78f)) * compress storage footprint ([#12871](#12871)) ([58c110f](58c110f)) * display warning when installing bb versions < 0.82.0 ([#13027](#13027)) ([7247fe7](7247fe7)) * **docs:** Update docs on fees and various other updates ([#12929](#12929)) ([1dec907](1dec907)) * dump dmesg/net/cpu/mem usage at end of ci run ([#12967](#12967)) ([8877792](8877792)) * fix governance util issue ([#13043](#13043)) ([d768d26](d768d26)) * redundant if in affine from projective constructor ([#13045](#13045)) ([3a7ba2d](3a7ba2d)) * remove addition of dummy ops in mock circuit producer ([#13003](#13003)) ([a64d1dc](a64d1dc)) * remove dummy ops in decider pk ([#13049](#13049)) ([da6d021](da6d021)) * replace relative paths to noir-protocol-circuits ([e1b88f6](e1b88f6)) * replace relative paths to noir-protocol-circuits ([849b4b0](849b4b0)) * replace relative paths to noir-protocol-circuits ([18a02d6](18a02d6)) * Revert "chore: add default native proving for cli wallet ([#12855](#12855))" ([#13013](#13013)) ([98e2576](98e2576)) * Speed up and deflake sentinel test ([#13078](#13078)) ([27f1eca](27f1eca)) * **testnet:** making consensus host mandatory input ([#12716](#12716)) ([d47c74a](d47c74a)) * towards no more mock op_queues ([#12984](#12984)) ([fefffa7](fefffa7)) * update bb version for noir 1.0.0-beta.0+ ([#13026](#13026)) ([dd68074](dd68074)) * update CODEOWNERS to reflect new sync method ([#12998](#12998)) ([a3d1915](a3d1915)) ### Documentation * Add fees to cli reference ([#12884](#12884)) ([4a0fd58](4a0fd58)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
mdb_copyfrom native world state and lmdb storebackupTomethods to archiver and world stateactionsto the aztec node for uploading and downloading snapshotsFixes #12926