Skip to content

feat: Native world state now supports checkpointing #11739

Merged
PhilWindle merged 87 commits intomasterfrom
pw/checkpointing
Feb 12, 2025
Merged

feat: Native world state now supports checkpointing #11739
PhilWindle merged 87 commits intomasterfrom
pw/checkpointing

Conversation

@PhilWindle
Copy link
Collaborator

@PhilWindle PhilWindle commented Feb 5, 2025

This PR introduces checkpointing to the native world state.

Checkpointing allows for state updates to be reverted to a previous state. This can be done to an arbitrary depth.

PhilWindle and others added 30 commits December 20, 2024 19:25
@PhilWindle
WIP
7753283
@PhilWindle
WIP
2f1b2af
@PhilWindle
WIP
77df462
@PhilWindle
Merge branch 'master' into pw/tree-concurrency
e0012b2
@PhilWindle
WIP
fe1ee9b
@PhilWindle
WIP
c4cc155
@PhilWindle
WIP
44f02f2
@PhilWindle
Moving files
dce4701
@PhilWindle
Moving files around
be77033
@PhilWindle
Renamed transactions
d02755f
@PhilWindle
Initial LMDBStore class
124682b
fix: remove circular includes
321543c
chore: move files
18c3945
refactor: extract module init outside of world_state
2b5a514
refactor!: rename world_state_napi -> nodejs_module
67de053
@PhilWindle
WIP
1878207
@PhilWindle
Merge branch 'feat/lmdb-wrapper' of github.com:AztecProtocol/aztec-pa…
b41e3e3 
…ckages into feat/lmdb-wrapper
@PhilWindle
Merge remote-tracking branch 'origin/master' into feat/lmdb-wrapper
df48bf5
feat: first pass
2b4e4ae
refactor: extract msg processing
9ccfd36
feat: add @aztec/native
6f063a6
@PhilWindle
WIP
5ab8325
@PhilWindle
WIP
0ee01c1
@PhilWindle
Merge branch 'feat/lmdb-wrapper' of github.com:AztecProtocol/aztec-pa…
d6996f3 
…ckages into feat/lmdb-wrapper
@PhilWindle
More work on cursors and duplicates
58359d3
@PhilWindle
Work on duplicates
bd0702f
@PhilWindle
Refactoring of tree stores
35133ed
@PhilWindle
Test refactoring
77d8591
@PhilWindle
Batch put operations
8c5e594
feat: nodejs lmdb wrapper (dummy backend)
66e3ec7
This was linked to issues Feb 10, 2025
Refactor uncommitted cache #9524
Closed
Create journalled caching implementation #9525
Closed
Plumb branch commit/rollback functionality through to TS #9526
Closed
alexghr
alexghr approved these changes Feb 11, 2025
View reviewed changes
Copy link
Contributor

@alexghr alexghr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an elegant solution to the problem!

@PhilWindle PhilWindle enabled auto-merge (squash) February 12, 2025 19:04
@PhilWindle PhilWindle merged commit 6464059 into master Feb 12, 2025
82 checks passed
@PhilWindle PhilWindle deleted the pw/checkpointing branch February 12, 2025 19:37
@AztecBot AztecBot mentioned this pull request Feb 12, 2025
Merged
sklppy88 pushed a commit that referenced this pull request Feb 13, 2025
@AztecBot
chore(master): Release 0.76.3 (#11924)
47fe362 
🤖 I have created a release *beep* *boop*
---


<details><summary>aztec-package: 0.76.3</summary>

##
[0.76.3](aztec-package-v0.76.2...aztec-package-v0.76.3)
(2025-02-12)


### Features

* Add undici
([#11818](#11818))
([8503c7a](8503c7a))
* Initial multi-proof test
([#11779](#11779))
([f54db75](f54db75))
</details>

<details><summary>barretenberg.js: 0.76.3</summary>

##
[0.76.3](barretenberg.js-v0.76.2...barretenberg.js-v0.76.3)
(2025-02-12)


### Features

* **perf:** Speed up construction of bbjs Frs & cache zero hashes in
ephemeral trees (redo)
([#11894](#11894))
([e093acf](e093acf))


### Bug Fixes

* Memory fragmentation fixes to cut UltraHonk memory usage by 26%
([#11895](#11895))
([b4e2264](b4e2264))
</details>

<details><summary>aztec-packages: 0.76.3</summary>

##
[0.76.3](aztec-packages-v0.76.2...aztec-packages-v0.76.3)
(2025-02-12)


### Features

* Add undici
([#11818](#11818))
([8503c7a](8503c7a))
* **avm:** Sequential lookup resolution
([#11769](#11769))
([3980f6c](3980f6c))
* Enable ws for reth on devnet
([#11922](#11922))
([7124664](7124664)),
closes
[#11921](#11921)
* Initial multi-proof test
([#11779](#11779))
([f54db75](f54db75))
* Native world state now supports checkpointing
([#11739](#11739))
([6464059](6464059))
* **perf:** Speed up construction of bbjs Frs & cache zero hashes in
ephemeral trees (redo)
([#11894](#11894))
([e093acf](e093acf))


### Bug Fixes

* Correctly configure batch queue
([#11934](#11934))
([4908df8](4908df8))
* De-dup pubkey conversion of cli-wallet param
([#11948](#11948))
([5529871](5529871))
* **docs:** Update the token bridge tutorial
([#11578](#11578))
([aaf42a7](aaf42a7))
* Don't restart kind control plane automatically
([#11923](#11923))
([c23c0f9](c23c0f9))
* Empty blocks can now be unwound
([#11920](#11920))
([fdc2042](fdc2042))
* Gcloud logs
([#11944](#11944))
([c53b1c5](c53b1c5)),
closes
[#11887](#11887)
* Lmdb cmake race condition
([#11959](#11959))
([031200d](031200d))
* Memory fragmentation fixes to cut UltraHonk memory usage by 26%
([#11895](#11895))
([b4e2264](b4e2264))
* Npm packages noir resolution
([#11946](#11946))
([d3e3f20](d3e3f20))
* Set resource limits on Loki pods
([#11940](#11940))
([6999982](6999982))


### Miscellaneous

* Only take FF (and not Flavor) in compute_logderivative_inverse
([#11938](#11938))
([bbbded3](bbbded3))
* Op queue cleanup
([#11925](#11925))
([082ed66](082ed66))
* Renaming `pxe_db.nr` as `capsules.nr`
([#11905](#11905))
([14d873c](14d873c))
* Replace relative paths to noir-protocol-circuits
([9ce401a](9ce401a))
* Use realistic size Client IVC proofs during simulations
([#11692](#11692))
([90b9fbf](90b9fbf))
* Use RelationChecker in relation correctness tests and add Translator
interleaving test
([#11878](#11878))
([ed215e8](ed215e8))
</details>

<details><summary>barretenberg: 0.76.3</summary>

##
[0.76.3](barretenberg-v0.76.2...barretenberg-v0.76.3)
(2025-02-12)


### Features

* **avm:** Sequential lookup resolution
([#11769](#11769))
([3980f6c](3980f6c))
* Native world state now supports checkpointing
([#11739](#11739))
([6464059](6464059))


### Bug Fixes

* Empty blocks can now be unwound
([#11920](#11920))
([fdc2042](fdc2042))
* Lmdb cmake race condition
([#11959](#11959))
([031200d](031200d))
* Memory fragmentation fixes to cut UltraHonk memory usage by 26%
([#11895](#11895))
([b4e2264](b4e2264))


### Miscellaneous

* Only take FF (and not Flavor) in compute_logderivative_inverse
([#11938](#11938))
([bbbded3](bbbded3))
* Op queue cleanup
([#11925](#11925))
([082ed66](082ed66))
* Use RelationChecker in relation correctness tests and add Translator
interleaving test
([#11878](#11878))
([ed215e8](ed215e8))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
AztecBot added a commit to AztecProtocol/barretenberg that referenced this pull request Feb 14, 2025
@AztecBot
chore(master): Release 0.76.3 (#11924)
ad2e48f 
🤖 I have created a release *beep* *boop*
---


<details><summary>aztec-package: 0.76.3</summary>

##
[0.76.3](AztecProtocol/aztec-packages@aztec-package-v0.76.2...aztec-package-v0.76.3)
(2025-02-12)


### Features

* Add undici
([#11818](AztecProtocol/aztec-packages#11818))
([8503c7a](AztecProtocol/aztec-packages@8503c7a))
* Initial multi-proof test
([#11779](AztecProtocol/aztec-packages#11779))
([f54db75](AztecProtocol/aztec-packages@f54db75))
</details>

<details><summary>barretenberg.js: 0.76.3</summary>

##
[0.76.3](AztecProtocol/aztec-packages@barretenberg.js-v0.76.2...barretenberg.js-v0.76.3)
(2025-02-12)


### Features

* **perf:** Speed up construction of bbjs Frs & cache zero hashes in
ephemeral trees (redo)
([#11894](AztecProtocol/aztec-packages#11894))
([e093acf](AztecProtocol/aztec-packages@e093acf))


### Bug Fixes

* Memory fragmentation fixes to cut UltraHonk memory usage by 26%
([#11895](AztecProtocol/aztec-packages#11895))
([b4e2264](AztecProtocol/aztec-packages@b4e2264))
</details>

<details><summary>aztec-packages: 0.76.3</summary>

##
[0.76.3](AztecProtocol/aztec-packages@aztec-packages-v0.76.2...aztec-packages-v0.76.3)
(2025-02-12)


### Features

* Add undici
([#11818](AztecProtocol/aztec-packages#11818))
([8503c7a](AztecProtocol/aztec-packages@8503c7a))
* **avm:** Sequential lookup resolution
([#11769](AztecProtocol/aztec-packages#11769))
([3980f6c](AztecProtocol/aztec-packages@3980f6c))
* Enable ws for reth on devnet
([#11922](AztecProtocol/aztec-packages#11922))
([7124664](AztecProtocol/aztec-packages@7124664)),
closes
[#11921](AztecProtocol/aztec-packages#11921)
* Initial multi-proof test
([#11779](AztecProtocol/aztec-packages#11779))
([f54db75](AztecProtocol/aztec-packages@f54db75))
* Native world state now supports checkpointing
([#11739](AztecProtocol/aztec-packages#11739))
([6464059](AztecProtocol/aztec-packages@6464059))
* **perf:** Speed up construction of bbjs Frs & cache zero hashes in
ephemeral trees (redo)
([#11894](AztecProtocol/aztec-packages#11894))
([e093acf](AztecProtocol/aztec-packages@e093acf))


### Bug Fixes

* Correctly configure batch queue
([#11934](AztecProtocol/aztec-packages#11934))
([4908df8](AztecProtocol/aztec-packages@4908df8))
* De-dup pubkey conversion of cli-wallet param
([#11948](AztecProtocol/aztec-packages#11948))
([5529871](AztecProtocol/aztec-packages@5529871))
* **docs:** Update the token bridge tutorial
([#11578](AztecProtocol/aztec-packages#11578))
([aaf42a7](AztecProtocol/aztec-packages@aaf42a7))
* Don't restart kind control plane automatically
([#11923](AztecProtocol/aztec-packages#11923))
([c23c0f9](AztecProtocol/aztec-packages@c23c0f9))
* Empty blocks can now be unwound
([#11920](AztecProtocol/aztec-packages#11920))
([fdc2042](AztecProtocol/aztec-packages@fdc2042))
* Gcloud logs
([#11944](AztecProtocol/aztec-packages#11944))
([c53b1c5](AztecProtocol/aztec-packages@c53b1c5)),
closes
[#11887](AztecProtocol/aztec-packages#11887)
* Lmdb cmake race condition
([#11959](AztecProtocol/aztec-packages#11959))
([031200d](AztecProtocol/aztec-packages@031200d))
* Memory fragmentation fixes to cut UltraHonk memory usage by 26%
([#11895](AztecProtocol/aztec-packages#11895))
([b4e2264](AztecProtocol/aztec-packages@b4e2264))
* Npm packages noir resolution
([#11946](AztecProtocol/aztec-packages#11946))
([d3e3f20](AztecProtocol/aztec-packages@d3e3f20))
* Set resource limits on Loki pods
([#11940](AztecProtocol/aztec-packages#11940))
([6999982](AztecProtocol/aztec-packages@6999982))


### Miscellaneous

* Only take FF (and not Flavor) in compute_logderivative_inverse
([#11938](AztecProtocol/aztec-packages#11938))
([bbbded3](AztecProtocol/aztec-packages@bbbded3))
* Op queue cleanup
([#11925](AztecProtocol/aztec-packages#11925))
([082ed66](AztecProtocol/aztec-packages@082ed66))
* Renaming `pxe_db.nr` as `capsules.nr`
([#11905](AztecProtocol/aztec-packages#11905))
([14d873c](AztecProtocol/aztec-packages@14d873c))
* Replace relative paths to noir-protocol-circuits
([9ce401a](AztecProtocol/aztec-packages@9ce401a))
* Use realistic size Client IVC proofs during simulations
([#11692](AztecProtocol/aztec-packages#11692))
([90b9fbf](AztecProtocol/aztec-packages@90b9fbf))
* Use RelationChecker in relation correctness tests and add Translator
interleaving test
([#11878](AztecProtocol/aztec-packages#11878))
([ed215e8](AztecProtocol/aztec-packages@ed215e8))
</details>

<details><summary>barretenberg: 0.76.3</summary>

##
[0.76.3](AztecProtocol/aztec-packages@barretenberg-v0.76.2...barretenberg-v0.76.3)
(2025-02-12)


### Features

* **avm:** Sequential lookup resolution
([#11769](AztecProtocol/aztec-packages#11769))
([3980f6c](AztecProtocol/aztec-packages@3980f6c))
* Native world state now supports checkpointing
([#11739](AztecProtocol/aztec-packages#11739))
([6464059](AztecProtocol/aztec-packages@6464059))


### Bug Fixes

* Empty blocks can now be unwound
([#11920](AztecProtocol/aztec-packages#11920))
([fdc2042](AztecProtocol/aztec-packages@fdc2042))
* Lmdb cmake race condition
([#11959](AztecProtocol/aztec-packages#11959))
([031200d](AztecProtocol/aztec-packages@031200d))
* Memory fragmentation fixes to cut UltraHonk memory usage by 26%
([#11895](AztecProtocol/aztec-packages#11895))
([b4e2264](AztecProtocol/aztec-packages@b4e2264))


### Miscellaneous

* Only take FF (and not Flavor) in compute_logderivative_inverse
([#11938](AztecProtocol/aztec-packages#11938))
([bbbded3](AztecProtocol/aztec-packages@bbbded3))
* Op queue cleanup
([#11925](AztecProtocol/aztec-packages#11925))
([082ed66](AztecProtocol/aztec-packages@082ed66))
* Use RelationChecker in relation correctness tests and add Translator
interleaving test
([#11878](AztecProtocol/aztec-packages#11878))
([ed215e8](AztecProtocol/aztec-packages@ed215e8))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alexghr alexghr alexghr approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Plumb branch commit/rollback functionality through to TS Create journalled caching implementation Refactor uncommitted cache

2 participants

@PhilWindle @alexghr