feat: new proving broker by alexghr · Pull Request #10174 · AztecProtocol/aztec-packages

alexghr · 2024-11-25T09:51:57Z

This PR integrates the new proving broker implementation added in #8495 into the orchestrator and adds a new caching layer between the orchestrator and the job broker able to resolve jobs that have already been finished (in order to provide crash recovery for the prover-node).

spalladino

Quite a big one! Looks great, there's just one blocker at the very beginning caused by one of the most fearsome concepts that a developer must deal with: the boolean.

yarn-project/aztec/src/cli/cmds/start_prover_agent.ts

spalladino · 2024-11-26T13:26:53Z

yarn-project/circuit-types/src/interfaces/proving-job.ts

Love seeing all that cleanup

yarn-project/end-to-end/webpack.config.js

spalladino · 2024-11-26T13:28:44Z

yarn-project/prover-client/src/prover-agent/memory-proving-queue.ts

+
+    if (!this.queue.put(item as ProvingJobWithResolvers<any>)) {


Hey, that's cheating, you just wrapped the any in another type! :-P

spalladino · 2024-11-26T13:30:57Z

yarn-project/prover-client/src/proving_broker/caching_broker_facade.ts

+  getAvmProof(
+    inputs: AvmCircuitInputs,
+    signal?: AbortSignal,
+    _blockNumber?: number,
+  ): Promise<ProofAndVerificationKey<typeof AVM_PROOF_LENGTH_IN_FIELDS>> {
+    return this.enqueueAndWaitForJob(
+      this.generateId(ProvingRequestType.PUBLIC_VM, inputs),
+      ProvingRequestType.PUBLIC_VM,
+      inputs,
+      signal,
+    );
+  }


At some point I'd like to delete all these methods from the Prover interface, and just have a getProof with a type discriminator. It would remove so much boilerplate.

yarn-project/prover-client/src/proving_broker/proving_broker_database/persisted.ts

yarn-project/prover-client/src/proving_broker/rpc.ts

yarn-project/prover-client/src/tx-prover/tx-prover.ts

yarn-project/prover-node/src/prover-cache/cache_manager.ts

yarn-project/prover-node/src/prover-node.ts

Co-authored-by: Santiago Palladino <santiago@aztecprotocol.com>

alexghr · 2024-11-27T16:38:20Z

yarn-project/prover-client/src/proving_broker/proof_store.ts

+    const encoded = encodeURIComponent(jsonStringify(obj));
+    return (PREFIX + SEPARATOR + encoded) as ProofUri;


@spalladino proofs are now string-encoded into the data-uri to avoid double b64 encoding

🤖 I have created a release *beep* *boop* --- <details><summary>aztec-package: 0.65.2</summary> ## [0.65.2](aztec-package-v0.65.1...aztec-package-v0.65.2) (2024-11-28) ### Features * New proving broker ([#10174](#10174)) ([6fd5fc1](6fd5fc1)) </details> <details><summary>barretenberg.js: 0.65.2</summary> ## [0.65.2](barretenberg.js-v0.65.1...barretenberg.js-v0.65.2) (2024-11-28) ### Miscellaneous * **barretenberg.js:** Synchronize aztec-packages versions </details> <details><summary>aztec-packages: 0.65.2</summary> ## [0.65.2](aztec-packages-v0.65.1...aztec-packages-v0.65.2) (2024-11-28) ### Features * Fee foresight support ([#10262](#10262)) ([9e19244](9e19244)) * New proving broker ([#10174](#10174)) ([6fd5fc1](6fd5fc1)) * Sequential insertion in indexed trees ([#10111](#10111)) ([bfd9fa6](bfd9fa6)) * Swap polys to facilitate dynamic trace overflow ([#9976](#9976)) ([b7b282c](b7b282c)) ### Bug Fixes * Don't store indices of zero leaves. ([#10270](#10270)) ([c22be8b](c22be8b)) * Expect proper duplicate nullifier error patterns in e2e tests ([#10256](#10256)) ([4ee8344](4ee8344)) ### Miscellaneous * Check artifact consistency ([#10271](#10271)) ([6a49405](6a49405)) * Dont import things that themselves import jest in imported functions ([#10260](#10260)) ([9440c1c](9440c1c)) * Fix bad merge in integration l1 publisher ([#10272](#10272)) ([b5a6aa4](b5a6aa4)) * Fixing sol warnings ([#10276](#10276)) ([3d113b2](3d113b2)) * Pull out sync changes ([#10274](#10274)) ([391a6b7](391a6b7)) * Pull value merger code from sync ([#10080](#10080)) ([3392629](3392629)) * Remove default gas settings ([#10163](#10163)) ([c9a4d88](c9a4d88)) * Replace relative paths to noir-protocol-circuits ([654d801](654d801)) * Teardown context in prover coordination test ([#10257](#10257)) ([7ea3888](7ea3888)) </details> <details><summary>barretenberg: 0.65.2</summary> ## [0.65.2](barretenberg-v0.65.1...barretenberg-v0.65.2) (2024-11-28) ### Features * Sequential insertion in indexed trees ([#10111](#10111)) ([bfd9fa6](bfd9fa6)) * Swap polys to facilitate dynamic trace overflow ([#9976](#9976)) ([b7b282c](b7b282c)) ### Bug Fixes * Don't store indices of zero leaves. ([#10270](#10270)) ([c22be8b](c22be8b)) ### Miscellaneous * Pull value merger code from sync ([#10080](#10080)) ([3392629](3392629)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- <details><summary>aztec-package: 0.65.2</summary> ## [0.65.2](AztecProtocol/aztec-packages@aztec-package-v0.65.1...aztec-package-v0.65.2) (2024-11-28) ### Features * New proving broker ([#10174](AztecProtocol/aztec-packages#10174)) ([6fd5fc1](AztecProtocol/aztec-packages@6fd5fc1)) </details> <details><summary>barretenberg.js: 0.65.2</summary> ## [0.65.2](AztecProtocol/aztec-packages@barretenberg.js-v0.65.1...barretenberg.js-v0.65.2) (2024-11-28) ### Miscellaneous * **barretenberg.js:** Synchronize aztec-packages versions </details> <details><summary>aztec-packages: 0.65.2</summary> ## [0.65.2](AztecProtocol/aztec-packages@aztec-packages-v0.65.1...aztec-packages-v0.65.2) (2024-11-28) ### Features * Fee foresight support ([#10262](AztecProtocol/aztec-packages#10262)) ([9e19244](AztecProtocol/aztec-packages@9e19244)) * New proving broker ([#10174](AztecProtocol/aztec-packages#10174)) ([6fd5fc1](AztecProtocol/aztec-packages@6fd5fc1)) * Sequential insertion in indexed trees ([#10111](AztecProtocol/aztec-packages#10111)) ([bfd9fa6](AztecProtocol/aztec-packages@bfd9fa6)) * Swap polys to facilitate dynamic trace overflow ([#9976](AztecProtocol/aztec-packages#9976)) ([b7b282c](AztecProtocol/aztec-packages@b7b282c)) ### Bug Fixes * Don't store indices of zero leaves. ([#10270](AztecProtocol/aztec-packages#10270)) ([c22be8b](AztecProtocol/aztec-packages@c22be8b)) * Expect proper duplicate nullifier error patterns in e2e tests ([#10256](AztecProtocol/aztec-packages#10256)) ([4ee8344](AztecProtocol/aztec-packages@4ee8344)) ### Miscellaneous * Check artifact consistency ([#10271](AztecProtocol/aztec-packages#10271)) ([6a49405](AztecProtocol/aztec-packages@6a49405)) * Dont import things that themselves import jest in imported functions ([#10260](AztecProtocol/aztec-packages#10260)) ([9440c1c](AztecProtocol/aztec-packages@9440c1c)) * Fix bad merge in integration l1 publisher ([#10272](AztecProtocol/aztec-packages#10272)) ([b5a6aa4](AztecProtocol/aztec-packages@b5a6aa4)) * Fixing sol warnings ([#10276](AztecProtocol/aztec-packages#10276)) ([3d113b2](AztecProtocol/aztec-packages@3d113b2)) * Pull out sync changes ([#10274](AztecProtocol/aztec-packages#10274)) ([391a6b7](AztecProtocol/aztec-packages@391a6b7)) * Pull value merger code from sync ([#10080](AztecProtocol/aztec-packages#10080)) ([3392629](AztecProtocol/aztec-packages@3392629)) * Remove default gas settings ([#10163](AztecProtocol/aztec-packages#10163)) ([c9a4d88](AztecProtocol/aztec-packages@c9a4d88)) * Replace relative paths to noir-protocol-circuits ([654d801](AztecProtocol/aztec-packages@654d801)) * Teardown context in prover coordination test ([#10257](AztecProtocol/aztec-packages#10257)) ([7ea3888](AztecProtocol/aztec-packages@7ea3888)) </details> <details><summary>barretenberg: 0.65.2</summary> ## [0.65.2](AztecProtocol/aztec-packages@barretenberg-v0.65.1...barretenberg-v0.65.2) (2024-11-28) ### Features * Sequential insertion in indexed trees ([#10111](AztecProtocol/aztec-packages#10111)) ([bfd9fa6](AztecProtocol/aztec-packages@bfd9fa6)) * Swap polys to facilitate dynamic trace overflow ([#9976](AztecProtocol/aztec-packages#9976)) ([b7b282c](AztecProtocol/aztec-packages@b7b282c)) ### Bug Fixes * Don't store indices of zero leaves. ([#10270](AztecProtocol/aztec-packages#10270)) ([c22be8b](AztecProtocol/aztec-packages@c22be8b)) ### Miscellaneous * Pull value merger code from sync ([#10080](AztecProtocol/aztec-packages#10080)) ([3392629](AztecProtocol/aztec-packages@3392629)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

Instead of processing blocks in sequencer during epoch proving and trigger proving jobs as each block is processed, we fetch the state prior to each block by forking off a world state following the pending (not proven) chain and process each block (ie execute public functions) in parallel. This means tx execution is less of a bottleneck for proving. Main change is that the epoch orchestrator now requires not a db, but something that can return forks at given block numbers of the db. It also means that the orchestrator accepts out-of-order operations for block building, so multiple blocks can be started, and their txs added in any order (though following the order within each block). Builds on #10174 Fixes #10265 Pending: - Ensuring the state after processing each block matches what we had used from world-state

alexghr changed the title ~~feat: new proving broker~~ feat: new proving broker [WIP] Nov 25, 2024

alexghr added the e2e-all label Nov 25, 2024

alexghr force-pushed the ag/proving-changes branch from 8e2fa60 to 1cd15c5 Compare November 25, 2024 20:07

This was linked to issues Nov 26, 2024

[Prover] Orchestrator persists partial proving state as proofs are generated #9528

Closed

[Prover] add k8s files for ProvingBroker and small/big ProvingAgent #10108

Closed

alexghr force-pushed the ag/proving-changes branch from 112c837 to 30de3bc Compare November 26, 2024 12:42

alexghr changed the title ~~feat: new proving broker [WIP]~~ feat: new proving broker Nov 26, 2024

spalladino reviewed Nov 26, 2024

View reviewed changes

alexghr force-pushed the ag/proving-changes branch from 606e6d9 to 2a834ba Compare November 26, 2024 16:21

Alex Gherghisan and others added 7 commits November 27, 2024 13:49

refactor: add a separate db for proving input/output

844c35b

feat: integrate new proving broker

2582174

fix: k8s

7b31cdc

fix: native network

98ac426

fix: provernet docker compose

76bcbd7

refactor: prover broker host

ef8d742

Apply suggestions from code review

072316f

Co-authored-by: Santiago Palladino <santiago@aztecprotocol.com>

alexghr force-pushed the ag/proving-changes branch from b5bb9fb to 072316f Compare November 27, 2024 13:53

Alex Gherghisan added 2 commits November 27, 2024 14:42

fix: code review

38cd368

fix: serialisation

bb5ff3c

alexghr commented Nov 27, 2024

View reviewed changes

spalladino mentioned this pull request Nov 27, 2024

feat: Process blocks in parallel during epoch proving #10263

Merged

spalladino approved these changes Nov 28, 2024

View reviewed changes

PhilWindle merged commit 6fd5fc1 into master Nov 28, 2024

PhilWindle deleted the ag/proving-changes branch November 28, 2024 13:19

AztecBot mentioned this pull request Nov 28, 2024

chore(master): Release 0.65.2 #10258

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: new proving broker#10174

feat: new proving broker#10174
PhilWindle merged 9 commits intomasterfrom
ag/proving-changes

alexghr commented Nov 25, 2024 •

edited

Loading

Uh oh!

spalladino left a comment

Uh oh!

Uh oh!

spalladino Nov 26, 2024

Uh oh!

Uh oh!

spalladino Nov 26, 2024

Uh oh!

spalladino Nov 26, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexghr Nov 27, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		if (!this.queue.put(item as ProvingJobWithResolvers<any>)) {

		const encoded = encodeURIComponent(jsonStringify(obj));
		return (PREFIX + SEPARATOR + encoded) as ProofUri;

Conversation

alexghr commented Nov 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

spalladino left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

spalladino Nov 26, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

spalladino Nov 26, 2024

Choose a reason for hiding this comment

Uh oh!

spalladino Nov 26, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexghr Nov 27, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alexghr commented Nov 25, 2024 •

edited

Loading