From 7a89774d8b07f59aea95fbedded635bace6e0b40 Mon Sep 17 00:00:00 2001 From: Anatoly Yakovenko Date: Wed, 29 Mar 2023 16:42:34 -0600 Subject: [PATCH 01/13] design --- .../0023-asynchronous-program-execution.md | 143 ++++++++++++++++++ 1 file changed, 143 insertions(+) create mode 100644 proposals/0023-asynchronous-program-execution.md diff --git a/proposals/0023-asynchronous-program-execution.md b/proposals/0023-asynchronous-program-execution.md new file mode 100644 index 000000000..4839ede04 --- /dev/null +++ b/proposals/0023-asynchronous-program-execution.md @@ -0,0 +1,143 @@ +--- +simd: '0023' +title: Asynchronous Program Execution and Broadcast +authors: + - Anatoly Yakovenko +category: Standard/Meta +type: Core +status: Draft +created: 2024-04-29 +feature: (fill in with feature tracking issues once accepted) +--- + +## Summary + +This feature changes how the ledger is broadcast and executed. It +separates proposing blocks full of user transactions from blocks +with votes. It allows for N concurrent proposers of user transaction +blocks, and it allows for asynchronous execution of user blocks. + +## Motivation + +1. Single leader for user transactions is a bottleneck. Leaders +are in a single spot in the world, while clients are all over the +world. Latency is therefore based on how close a client is to a +leader. Leaders also have to spend a ton of resources on prioritization +of transactions. + +2. Executing programs before voting is a bottleneck. Fork choice +doesn't depend on program execution, and delaying voting creates +forks and also delays the next block producer from starting. + +## Alternatives Considered + +APEX design that was proposed in https://github.com/solana-labs/solana/pull/24127 + +## New Terminology + +* UserBlock - a block full of none vote transactions. + +* UserBlockEntry - a hash of the block that was proposed + +* UserBlockSlot - a slot measuring half a normal slot in ticks. + +* Leader - the current leader for the slot that will propose a PoH +ledger full of Votes and UserBlockEntry + +* Proposer - a node that is scheduled to propose a block with non +vote transactions + +## Detailed Design + +### Overview +Leaders are scheduled to propose blocks as they are currently by +the LeaderSchedule. + +Proposer's are scheduled along side leaders to propose blocks. N +number of proposers can be scheudled per slot. Proposers are +scheudled at 2x the rate of leaders. So for every leader slot, +there are N proposers producing 2 UserBlocks. + +While a leader is scheduled, they receive and encode votes as normal. +Any well formed UserBlocks that were received from the **previous** +UserBlockSlot are added to the leaders PoH ledger as UserBlockEntry. + +The N concurrent Proposers have 200ms slots to create blocks out +of user transactions. These are transmitted to the network. The +leader receives and decodes them and generates a UserBlockEntry, +and adds it to PoH as soon as the leaders PoH has passed the +UserBlockSlot. + +### Fork Choice + +If a validator doesn't have the proposer's UserBlock, the validator +doesn't vote on the proposed block and tries to repair the UserBlock. +That fork is ignored for fork choice. + +Otherwise the validator evaluates the forks as usual. + +### UserBlock execution + +Each UserBlock is assumed to have been created simultaneously +during its scheduled slot. For each UserBlock, the transactions +are ordered by priority fee before execution. If two transactions +from two different blocks have the same priority, the earliest +transaction by UserBlock PoH is executed first, otherwise by which +UserBlock appears first in the leaders PoH. + +### BankHash + +Votes must include the most recent computed BankHash along with the +slot for the BankHash computation. This slot is different +from the tower vote slot. + +Each validator that is computing a BankHash will be able to verify +the BankHash values created by block producers for previous slots +because each BankHash has a link to the previous one. + +During verification, if a validator sees that > 1/3 of the network +has incorrectly computed a state transition, they should halt +immediately. + +### Client Confirmations + +Client can receive a confirmation as soon as the UserBlockEntry has +been optimistically confirmed. Client can also wait for a status +code weighted by stake weight, as each validator's execution catches +up to the slot and includes a BankHash confirming the state of the +execution. + +It may be sufficient to wait for 1/3+ validators executing the state +transition instead of the full 2/3+, because if 1/3+ are incorrect +the network will halt anyways. + +## Impact + +Multiple nodes can operate as Proposers on the network concurrently. +So clients can pick the nearest one, and the bandwidth to schedule +and prioritize transactions is doubled. There needs to be a design +for BundleTransactions that allow the bundler to prioritize a batch +of transactions for execution by paying a priority fee for all of +them, and executing the whole batch together. + +Priority fees now also imply execution priority. + +Fork choice and voting is not blocked by user block propagation or +execution of user transactions. + +## Security Considerations + +* Validators may include incorrect BankHash values which may not +be detected instantly. Validators may skip evaluating their own +BankHash and copy results from other validators. + +* Network halts if it cannot compute a epoch snapshot hash once an +epoch because it is overloaded with user transactions. + +## Drawbacks + +[TBD] + +## Backwards Compatibility + +[TBD] From 0f5393fc86e0445b8a46a8f7717359035b54d150 Mon Sep 17 00:00:00 2001 From: Anatoly Yakovenko Date: Wed, 29 Mar 2023 16:42:52 -0600 Subject: [PATCH 02/13] rename --- ...on.md => 0023-asynchronous-program-execution-and-broadcast.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename proposals/{0023-asynchronous-program-execution.md => 0023-asynchronous-program-execution-and-broadcast.md} (100%) diff --git a/proposals/0023-asynchronous-program-execution.md b/proposals/0023-asynchronous-program-execution-and-broadcast.md similarity index 100% rename from proposals/0023-asynchronous-program-execution.md rename to proposals/0023-asynchronous-program-execution-and-broadcast.md From dbf0775fa55d6b40b7ebe01d12090418aba57e0f Mon Sep 17 00:00:00 2001 From: Anatoly Yakovenko Date: Wed, 29 Mar 2023 17:18:43 -0600 Subject: [PATCH 03/13] update --- .../0023-asynchronous-program-execution-and-broadcast.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/proposals/0023-asynchronous-program-execution-and-broadcast.md b/proposals/0023-asynchronous-program-execution-and-broadcast.md index 4839ede04..2c50da038 100644 --- a/proposals/0023-asynchronous-program-execution-and-broadcast.md +++ b/proposals/0023-asynchronous-program-execution-and-broadcast.md @@ -125,6 +125,11 @@ Priority fees now also imply execution priority. Fork choice and voting is not blocked by user block propagation or execution of user transactions. +Need a mechanism for splitting the reward between the leader and +proposer. + +Proposers are going to capture all the MEV. + ## Security Considerations * Validators may include incorrect BankHash values which may not @@ -134,6 +139,9 @@ BankHash and copy results from other validators. * Network halts if it cannot compute a epoch snapshot hash once an epoch because it is overloaded with user transactions. +* Leader could censor the proposer. But then they would miss out +on the total fees proposed in the block. + ## Drawbacks [TBD] From 25d9375e0ba49b8349a70d196fbd55e9681c178a Mon Sep 17 00:00:00 2001 From: Anatoly Yakovenko Date: Wed, 29 Mar 2023 17:47:43 -0600 Subject: [PATCH 04/13] rename proposer to builder --- ...hronous-program-execution-and-broadcast.md | 27 +++++++++---------- 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/proposals/0023-asynchronous-program-execution-and-broadcast.md b/proposals/0023-asynchronous-program-execution-and-broadcast.md index 2c50da038..599490889 100644 --- a/proposals/0023-asynchronous-program-execution-and-broadcast.md +++ b/proposals/0023-asynchronous-program-execution-and-broadcast.md @@ -14,7 +14,7 @@ feature: (fill in with feature tracking issues once accepted) This feature changes how the ledger is broadcast and executed. It separates proposing blocks full of user transactions from blocks -with votes. It allows for N concurrent proposers of user transaction +with votes. It allows for N concurrent builders of user transaction blocks, and it allows for asynchronous execution of user blocks. ## Motivation @@ -44,7 +44,7 @@ APEX design that was proposed in https://github.com/solana-labs/solana/pull/2412 * Leader - the current leader for the slot that will propose a PoH ledger full of Votes and UserBlockEntry -* Proposer - a node that is scheduled to propose a block with non +* Builder - a node that is scheduled to propose a block with non vote transactions ## Detailed Design @@ -53,16 +53,16 @@ vote transactions Leaders are scheduled to propose blocks as they are currently by the LeaderSchedule. -Proposer's are scheduled along side leaders to propose blocks. N -number of proposers can be scheudled per slot. Proposers are +Builder's are scheduled along side leaders to propose blocks. N +number of builders can be scheudled per slot. Builder are scheudled at 2x the rate of leaders. So for every leader slot, -there are N proposers producing 2 UserBlocks. +there are N builders producing 2 UserBlocks. While a leader is scheduled, they receive and encode votes as normal. -Any well formed UserBlocks that were received from the **previous** +Any well formed UserBlocks that were received from the previous UserBlockSlot are added to the leaders PoH ledger as UserBlockEntry. -The N concurrent Proposers have 200ms slots to create blocks out +The N concurrent Builder have 200ms slots to create blocks out of user transactions. These are transmitted to the network. The leader receives and decodes them and generates a UserBlockEntry, and adds it to PoH as soon as the leaders PoH has passed the @@ -70,7 +70,7 @@ UserBlockSlot. ### Fork Choice -If a validator doesn't have the proposer's UserBlock, the validator +If a validator doesn't have the builder's UserBlock, the validator doesn't vote on the proposed block and tries to repair the UserBlock. That fork is ignored for fork choice. @@ -113,7 +113,7 @@ the network will halt anyways. ## Impact -Multiple nodes can operate as Proposers on the network concurrently. +Multiple nodes can operate as Builder on the network concurrently. So clients can pick the nearest one, and the bandwidth to schedule and prioritize transactions is doubled. There needs to be a design for BundleTransactions that allow the bundler to prioritize a batch @@ -125,10 +125,7 @@ Priority fees now also imply execution priority. Fork choice and voting is not blocked by user block propagation or execution of user transactions. -Need a mechanism for splitting the reward between the leader and -proposer. - -Proposers are going to capture all the MEV. +Builders are going to capture all the MEV. ## Security Considerations @@ -139,8 +136,8 @@ BankHash and copy results from other validators. * Network halts if it cannot compute a epoch snapshot hash once an epoch because it is overloaded with user transactions. -* Leader could censor the proposer. But then they would miss out -on the total fees proposed in the block. +* Leader could censor the builder. But then they would miss out +on all the fees in the block. ## Drawbacks From 3636f73298654b22d678da97dc17456e296f61db Mon Sep 17 00:00:00 2001 From: Anatoly Yakovenko Date: Thu, 30 Mar 2023 10:03:49 -0600 Subject: [PATCH 05/13] more details --- ...hronous-program-execution-and-broadcast.md | 100 ++++++++++++++---- 1 file changed, 82 insertions(+), 18 deletions(-) diff --git a/proposals/0023-asynchronous-program-execution-and-broadcast.md b/proposals/0023-asynchronous-program-execution-and-broadcast.md index 599490889..0fe2351e9 100644 --- a/proposals/0023-asynchronous-program-execution-and-broadcast.md +++ b/proposals/0023-asynchronous-program-execution-and-broadcast.md @@ -47,6 +47,13 @@ ledger full of Votes and UserBlockEntry * Builder - a node that is scheduled to propose a block with non vote transactions +* VoteHash - like BankHash, but for vote only transactions. + +* BundleTransactions - a transaction signed by the builder that can +list transactions in the buildres proposed block to be executed in +one ordered batch. This transaction can also add a priority fee +such that the entire bundle is prioritized for execution as a batch. + ## Detailed Design ### Overview @@ -59,13 +66,14 @@ scheudled at 2x the rate of leaders. So for every leader slot, there are N builders producing 2 UserBlocks. While a leader is scheduled, they receive and encode votes as normal. -Any well formed UserBlocks that were received from the previous -UserBlockSlot are added to the leaders PoH ledger as UserBlockEntry. +Any well formed UserBlocks that were received from the previous or +current UserBlockSlot are added to the leaders PoH ledger as +UserBlockEntry. The N concurrent Builder have 200ms slots to create blocks out of user transactions. These are transmitted to the network. The leader receives and decodes them and generates a UserBlockEntry, -and adds it to PoH as soon as the leaders PoH has passed the +and adds it to PoH as soon as the leaders PoH has started the UserBlockSlot. ### Fork Choice @@ -78,12 +86,46 @@ Otherwise the validator evaluates the forks as usual. ### UserBlock execution -Each UserBlock is assumed to have been created simultaneously -during its scheduled slot. For each UserBlock, the transactions -are ordered by priority fee before execution. If two transactions -from two different blocks have the same priority, the earliest -transaction by UserBlock PoH is executed first, otherwise by which -UserBlock appears first in the leaders PoH. +Each UserBlock is assumed to have been created simultaneously during +the UserBlockSlot that it was encoded by the leader. For each +UserBlock, the transactions are ordered by priority fee before +execution. If two transactions from two different blocks have the +same priority, the earliest transaction by UserBlock is executed +first, otherwise by which UserBlock appears first in the leaders +PoH. + +If the leader's proposed block is attached to the heaviest fork, +the validator can start execution of the UserBlocks optimistically. +Otherwise the validator should only execute UserBlocks on the +heaviest fork. + +### UserBlock format + +UserBlock follows the standard solana ledger format, except it +doesn't contain any PoH ticks. + +### UserBlock broadcast + +Builders can form blocks at any time. Each scheduled builder can +transmit shreds over a pre-allocated shred range for the slot. + +For example, if the network is configured with 5k shred blocks, and +2 builders: + +Leader: [0-1000] +UserBlockSlot 0, Builder 0: [1000-2000] +UserBlockSlot 0, Builder 1: [2000-3000] +UserBlockSlot 1, Builder 0: [3000-4000] +UserBlockSlot 1, Builder 1: [4000-5000] + +Leader must fit their entire block in shreds 0 to 1000, including +PoH tick entries, votes and UserBlockEntries. + +Builder 0 transmits shreds 1000-2000 and 3000-4000 +Builder 1 transmits shreds 2000-3000 and 4000-5000 + +UserBlockEntries for UserBlockSlot 1 must be included by the leader +after the halfway point in the leaders block. ### BankHash @@ -99,13 +141,24 @@ During verification, if a validator sees that > 1/3 of the network has incorrectly computed a state transition, they should halt immediately. +### VoteHash + +All validators will execute all the vote transactions on every fork, +as they do now. After computing all the vote state transitions the +validator will compute the VoteHash, identical to the BankHash in +structure, but covers only the vote state transitions. The VoteHash +is included in the vote for the slot that the validator is voting +for. + ### Client Confirmations -Client can receive a confirmation as soon as the UserBlockEntry has -been optimistically confirmed. Client can also wait for a status -code weighted by stake weight, as each validator's execution catches -up to the slot and includes a BankHash confirming the state of the -execution. +Clients connected to a trusted RPC can confirm the transaction as +soon as the RPC has executed teh UserBlock and the block has been +optimistically confirmed. + +Client can also wait for a status code weighted by stake weight, +as each validator's execution catches up to the slot and includes +a BankHash confirming the state of the execution. It may be sufficient to wait for 1/3+ validators executing the state transition instead of the full 2/3+, because if 1/3+ are incorrect @@ -127,17 +180,28 @@ execution of user transactions. Builders are going to capture all the MEV. +Votes contain additional data, instead of BankHash, they contain +(VoteHash, (BankHash, slot)) + ## Security Considerations * Validators may include incorrect BankHash values which may not -be detected instantly. Validators may skip evaluating their own -BankHash and copy results from other validators. +be detected instantly. + +* Validators may skip evaluating their own BankHash and copy results +from other validators. * Network halts if it cannot compute a epoch snapshot hash once an epoch because it is overloaded with user transactions. -* Leader could censor the builder. But then they would miss out -on all the fees in the block. +* Leader could censor or delay the builder. + +* Under heavy forking, validators will skip executing all the +UserBlocks on minor forks. + +## Economic Considerations + +* Builders should be shuffled and scheduled according to stake weight. ## Drawbacks From c27b6758d4ba5add6a17334607578ade6c171be5 Mon Sep 17 00:00:00 2001 From: anatoly yakovenko Date: Thu, 30 Mar 2023 13:04:39 -0700 Subject: [PATCH 06/13] Update proposals/0023-asynchronous-program-execution-and-broadcast.md Co-authored-by: segfaultdoctor <17258903+segfaultdoc@users.noreply.github.com> --- proposals/0023-asynchronous-program-execution-and-broadcast.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0023-asynchronous-program-execution-and-broadcast.md b/proposals/0023-asynchronous-program-execution-and-broadcast.md index 0fe2351e9..73439a334 100644 --- a/proposals/0023-asynchronous-program-execution-and-broadcast.md +++ b/proposals/0023-asynchronous-program-execution-and-broadcast.md @@ -60,7 +60,7 @@ such that the entire bundle is prioritized for execution as a batch. Leaders are scheduled to propose blocks as they are currently by the LeaderSchedule. -Builder's are scheduled along side leaders to propose blocks. N +Builder's are scheduled along side leaders to build blocks. N number of builders can be scheudled per slot. Builder are scheudled at 2x the rate of leaders. So for every leader slot, there are N builders producing 2 UserBlocks. From 8472fab00919aef9805cd1a184fd28a38702b0a4 Mon Sep 17 00:00:00 2001 From: Anatoly Yakovenko Date: Thu, 30 Mar 2023 14:13:47 -0700 Subject: [PATCH 07/13] update --- ...hronous-program-execution-and-broadcast.md | 117 ++++++++++++++---- 1 file changed, 90 insertions(+), 27 deletions(-) diff --git a/proposals/0023-asynchronous-program-execution-and-broadcast.md b/proposals/0023-asynchronous-program-execution-and-broadcast.md index 73439a334..e0c922433 100644 --- a/proposals/0023-asynchronous-program-execution-and-broadcast.md +++ b/proposals/0023-asynchronous-program-execution-and-broadcast.md @@ -35,24 +35,30 @@ APEX design that was proposed in https://github.com/solana-labs/solana/pull/2412 ## New Terminology -* UserBlock - a block full of none vote transactions. +* UserBlock - a block full of non vote transactions. -* UserBlockEntry - a hash of the block that was proposed +* UserBlockEntry - this is the entry that leader creates in PoH for +the UserBlock, it contains a hash of the built UserBlock. -* UserBlockSlot - a slot measuring half a normal slot in ticks. +* UserBlockSlot - A slot for builders to transmit the UserBlock to +the cluster over turbine. Cluster could be configured with more +then one UserBlockSlots per slot with a default setting of 2. * Leader - the current leader for the slot that will propose a PoH -ledger full of Votes and UserBlockEntry +ledger full of Votes and UserBlockEntries. -* Builder - a node that is scheduled to propose a block with non +* Builder - a node that is scheduled to build a block with non vote transactions * VoteHash - like BankHash, but for vote only transactions. * BundleTransactions - a transaction signed by the builder that can -list transactions in the buildres proposed block to be executed in -one ordered batch. This transaction can also add a priority fee +list transactions in the builders UserBlock to be executed in +one ordered batch. This transaction can also add a priority fee such that the entire bundle is prioritized for execution as a batch. +This transaction can only appear in the builders UserBlock and can +only reference transactions prior to it in the UserBlock. TBD on +format. ## Detailed Design @@ -60,21 +66,22 @@ such that the entire bundle is prioritized for execution as a batch. Leaders are scheduled to propose blocks as they are currently by the LeaderSchedule. -Builder's are scheduled along side leaders to build blocks. N -number of builders can be scheudled per slot. Builder are -scheudled at 2x the rate of leaders. So for every leader slot, -there are N builders producing 2 UserBlocks. +Builder's are scheduled along side leaders to build UserBlocks - +blocks of non vote transactions. N number of builders can be scheudled +concurrently to build blocks. While a leader is scheduled, they receive and encode votes as normal. Any well formed UserBlocks that were received from the previous or current UserBlockSlot are added to the leaders PoH ledger as UserBlockEntry. -The N concurrent Builder have 200ms slots to create blocks out -of user transactions. These are transmitted to the network. The -leader receives and decodes them and generates a UserBlockEntry, -and adds it to PoH as soon as the leaders PoH has started the -UserBlockSlot. +The N concurrent Builder create blocks out of user transactions. +These are transmitted to the cluster via turbine. The leader +receives and decodes them and generates a UserBlockEntry, and adds +it to PoH as soon as the leaders PoH has started the UserBlockSlot. + +Validators can vote on leader blocks by executing the votes, but +before completing the execution of UserBlock transactions. ### Fork Choice @@ -84,6 +91,23 @@ That fork is ignored for fork choice. Otherwise the validator evaluates the forks as usual. +### UserBlock schedule + +There are 2 UserBlockSlots per slot. Builders produce 1 UserBlock +per UserBlockSlot. So if the cluster is configured with 3 Builders, +each leader slot would see a total of 6 UserBlocks per slot. 3 in +the first half of the leaders slot, and 3 in the second half. + +The current leader schedule rotates each leader every 4 slots, but +builders could be rotated at a different schedule then the leaders +and rotation could be staggered. + +### UserBlock Compute Limits + +If the overall compute capacity for user transactions per leader +block is 48m CU, and cluster is configured with 2 builders, then +each UserBlock can use no more then 48m/4 or 12m CU. + ### UserBlock execution Each UserBlock is assumed to have been created simultaneously during @@ -99,17 +123,37 @@ the validator can start execution of the UserBlocks optimistically. Otherwise the validator should only execute UserBlocks on the heaviest fork. +Duplicate transactions in the UserBlock are skipped over without +any state changes. They must still be retained in the ledger because +they have been hashed into the UserBlockEntry. + ### UserBlock format UserBlock follows the standard solana ledger format, except it -doesn't contain any PoH ticks. +MUST not contain any PoH ticks or votes. + +Invalid transactions are skipped, including duplicates, or those +with invalid signatures, or completely malformed, and the rest of +the block is processed. + +The first entry must start with the hash of the builders public +key, the last entry must contain a signature from the builder of +the previous entry hash. This will ensure that leader cannot +manipulate the UserBlock, and that the entire block must be included +when computing the UserBlockEntry. + +### UserBlockEntry format + +Contains the hash of all the data in the UserBlock. TBD if the +hash should contain a vector commitment for data availability +sampling. ### UserBlock broadcast Builders can form blocks at any time. Each scheduled builder can transmit shreds over a pre-allocated shred range for the slot. -For example, if the network is configured with 5k shred blocks, and +For example, if the cluster is configured with 5k shred blocks, and 2 builders: Leader: [0-1000] @@ -127,6 +171,14 @@ Builder 1 transmits shreds 2000-3000 and 4000-5000 UserBlockEntries for UserBlockSlot 1 must be included by the leader after the halfway point in the leaders block. +### Leader Blocks + +Leader produces blocks as usual, but blocks are valid if they +only contain Vote transactions and UserBlockEntries and PoH ticks. + +UserBlockEntries in the leaders block must be recent, must not be +in the future, and must not be duplicates. + ### BankHash Votes must include the most recent computed BankHash along with the @@ -137,7 +189,7 @@ Each validator that is computing a BankHash will be able to verify the BankHash values created by block producers for previous slots because each BankHash has a link to the previous one. -During verification, if a validator sees that > 1/3 of the network +During verification, if a validator sees that > 1/3 of the cluster has incorrectly computed a state transition, they should halt immediately. @@ -162,11 +214,19 @@ a BankHash confirming the state of the execution. It may be sufficient to wait for 1/3+ validators executing the state transition instead of the full 2/3+, because if 1/3+ are incorrect -the network will halt anyways. +the cluster will halt anyways. + +### State synchronization + +Nodes must be able to compute a full snapshot at least once an +epoch. So overall CU limits for the block must be set such that +synchronous execution is possible. But because asynchronous execution +is possible, replay can take advantage of much larger batches of +execution. ## Impact -Multiple nodes can operate as Builder on the network concurrently. +Multiple nodes can operate as Builder on the cluster concurrently. So clients can pick the nearest one, and the bandwidth to schedule and prioritize transactions is doubled. There needs to be a design for BundleTransactions that allow the bundler to prioritize a batch @@ -185,23 +245,26 @@ Votes contain additional data, instead of BankHash, they contain ## Security Considerations -* Validators may include incorrect BankHash values which may not +Validators may include incorrect BankHash values which may not be detected instantly. -* Validators may skip evaluating their own BankHash and copy results +Validators may skip evaluating their own BankHash and copy results from other validators. -* Network halts if it cannot compute a epoch snapshot hash once an +Network halts if it cannot compute a epoch snapshot hash once an epoch because it is overloaded with user transactions. -* Leader could censor or delay the builder. +Leader could censor or delay the builder. -* Under heavy forking, validators will skip executing all the +Under heavy forking, validators will skip executing all the UserBlocks on minor forks. ## Economic Considerations -* Builders should be shuffled and scheduled according to stake weight. +Builders should be shuffled and scheduled according to stake weight. + +TBD, deciding how should builders and leaders split the fees from +user transactions. ## Drawbacks From 92f953cb1f5a7ec8da3c945912f75f7f1bebfe94 Mon Sep 17 00:00:00 2001 From: Anatoly Yakovenko Date: Thu, 30 Mar 2023 14:45:35 -0700 Subject: [PATCH 08/13] update --- .../0023-asynchronous-program-execution-and-broadcast.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/proposals/0023-asynchronous-program-execution-and-broadcast.md b/proposals/0023-asynchronous-program-execution-and-broadcast.md index e0c922433..e966d2052 100644 --- a/proposals/0023-asynchronous-program-execution-and-broadcast.md +++ b/proposals/0023-asynchronous-program-execution-and-broadcast.md @@ -136,11 +136,10 @@ Invalid transactions are skipped, including duplicates, or those with invalid signatures, or completely malformed, and the rest of the block is processed. -The first entry must start with the hash of the builders public -key, the last entry must contain a signature from the builder of -the previous entry hash. This will ensure that leader cannot -manipulate the UserBlock, and that the entire block must be included -when computing the UserBlockEntry. +The first entry must start with the value 0, the last entry must +contain a signature from the builder of the previous entry hash. +This will ensure that leader cannot manipulate the UserBlock, and +that the entire block must be included when computing the UserBlockEntry. ### UserBlockEntry format From 247b79b6d9bfcd676a218016d3ce9e484320da6d Mon Sep 17 00:00:00 2001 From: Anatoly Yakovenko Date: Thu, 30 Mar 2023 15:56:52 -0700 Subject: [PATCH 09/13] update --- ...hronous-program-execution-and-broadcast.md | 22 ++++++++++++------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/proposals/0023-asynchronous-program-execution-and-broadcast.md b/proposals/0023-asynchronous-program-execution-and-broadcast.md index e966d2052..70dfdc4a0 100644 --- a/proposals/0023-asynchronous-program-execution-and-broadcast.md +++ b/proposals/0023-asynchronous-program-execution-and-broadcast.md @@ -44,7 +44,7 @@ the UserBlock, it contains a hash of the built UserBlock. the cluster over turbine. Cluster could be configured with more then one UserBlockSlots per slot with a default setting of 2. -* Leader - the current leader for the slot that will propose a PoH +* Leader - the current leader for the slot that will build a PoH ledger full of Votes and UserBlockEntries. * Builder - a node that is scheduled to build a block with non @@ -63,7 +63,7 @@ format. ## Detailed Design ### Overview -Leaders are scheduled to propose blocks as they are currently by +Leaders are scheduled to build blocks as they are currently by the LeaderSchedule. Builder's are scheduled along side leaders to build UserBlocks - @@ -110,15 +110,21 @@ each UserBlock can use no more then 48m/4 or 12m CU. ### UserBlock execution +Execution of UserBlocks can happen asynchronously to voting. When +voting validators transmit their most recent BankHash, but it may +be for an older parent slot then the slot they are voting on. + Each UserBlock is assumed to have been created simultaneously during -the UserBlockSlot that it was encoded by the leader. For each -UserBlock, the transactions are ordered by priority fee before -execution. If two transactions from two different blocks have the -same priority, the earliest transaction by UserBlock is executed -first, otherwise by which UserBlock appears first in the leaders +the UserBlockSlot that it was encoded by the leader. + +For each UserBlock in the PoH section spanning the UesrBlockSlot, +the transactions are ordered by priority fee before execution. + +If two transactions from two different blocks have the same priority, +they are ordered by which UserBlock appears first in the leaders PoH. -If the leader's proposed block is attached to the heaviest fork, +If the leader's block is attached to the heaviest fork, the validator can start execution of the UserBlocks optimistically. Otherwise the validator should only execute UserBlocks on the heaviest fork. From a16ecc1531d6e710995b7766615f681a8c207194 Mon Sep 17 00:00:00 2001 From: Anatoly Yakovenko Date: Fri, 31 Mar 2023 15:22:55 -0700 Subject: [PATCH 10/13] fix date --- proposals/0023-asynchronous-program-execution-and-broadcast.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0023-asynchronous-program-execution-and-broadcast.md b/proposals/0023-asynchronous-program-execution-and-broadcast.md index 70dfdc4a0..122823b68 100644 --- a/proposals/0023-asynchronous-program-execution-and-broadcast.md +++ b/proposals/0023-asynchronous-program-execution-and-broadcast.md @@ -6,7 +6,7 @@ authors: category: Standard/Meta type: Core status: Draft -created: 2024-04-29 +created: 2024-03-29 feature: (fill in with feature tracking issues once accepted) --- From 5d0a76f389a49cbb83ed5633b545c56a2a5670a9 Mon Sep 17 00:00:00 2001 From: Anatoly Yakovenko Date: Wed, 14 Jun 2023 11:11:28 -0700 Subject: [PATCH 11/13] Add design for persistent userblock builders --- ...hronous-program-execution-and-broadcast.md | 85 ++++++++++++++----- 1 file changed, 66 insertions(+), 19 deletions(-) diff --git a/proposals/0023-asynchronous-program-execution-and-broadcast.md b/proposals/0023-asynchronous-program-execution-and-broadcast.md index 122823b68..adb3e223f 100644 --- a/proposals/0023-asynchronous-program-execution-and-broadcast.md +++ b/proposals/0023-asynchronous-program-execution-and-broadcast.md @@ -67,7 +67,7 @@ Leaders are scheduled to build blocks as they are currently by the LeaderSchedule. Builder's are scheduled along side leaders to build UserBlocks - -blocks of non vote transactions. N number of builders can be scheudled +blocks of non vote transactions. N number of builders can be scheduled concurrently to build blocks. While a leader is scheduled, they receive and encode votes as normal. @@ -75,10 +75,11 @@ Any well formed UserBlocks that were received from the previous or current UserBlockSlot are added to the leaders PoH ledger as UserBlockEntry. -The N concurrent Builder create blocks out of user transactions. -These are transmitted to the cluster via turbine. The leader -receives and decodes them and generates a UserBlockEntry, and adds -it to PoH as soon as the leaders PoH has started the UserBlockSlot. +The N concurrent builders create blocks out of user transactions. +These are transmitted to the cluster via turbine concurrently with +the scheduled leader and other builders. The leader receives and +decodes them and generates a UserBlockEntry, and adds it to PoH as +soon as the leaders PoH has started the UserBlockSlot. Validators can vote on leader blocks by executing the votes, but before completing the execution of UserBlock transactions. @@ -89,26 +90,63 @@ If a validator doesn't have the builder's UserBlock, the validator doesn't vote on the proposed block and tries to repair the UserBlock. That fork is ignored for fork choice. -Otherwise the validator evaluates the forks as usual. +Otherwise the validator evaluates the forks as usual. -### UserBlock schedule +### UserBlock builder schedule -There are 2 UserBlockSlots per slot. Builders produce 1 UserBlock -per UserBlockSlot. So if the cluster is configured with 3 Builders, -each leader slot would see a total of 6 UserBlocks per slot. 3 in -the first half of the leaders slot, and 3 in the second half. +Each block is sloted to support N user blocks. Some of these builders +are randomly scheduled, some of these are persistent. For example, +there are 10 UserBlock builders per network block, that means 10 +builders are each assigned 10% of the shreds and 10% of the compute +available to the block. -The current leader schedule rotates each leader every 4 slots, but -builders could be rotated at a different schedule then the leaders -and rotation could be staggered. +#### Randomized UserBlock builders -### UserBlock Compute Limits +Randomized UserBlock builders are assigned at the same time as the +leader schedule is created based on stake weighted leader distribution. + +The benefit to more than 1 random UserBlock builder per leader +block is that they are likely geographically distributed and users +will be able to pick the closest one. + +The downside is that resource management becomes more complex. Each +builder has 1/N compute and shred capacity and users have no idea +which one is saturated when sending their transaction. It's likely +that priority fee floor will be different at each UserBlock builder. + +#### Persistent UserBlock slots + +Persistent UserBlock slots for the epoch are auctioned of to the top +N bidders who burn the most lamports. + +In the first half of the epoch each builder deposits the lamports +they are planning on burning, in the second half the builders may +withdraw excess lamports. The top N builders are assigned the slots +in a dutch auction according to their remaining bids. If there are +no bidders the capacity is relinquished to the randomized UserBlock +builders. + +The benefit to persistent UserBlock builders is predictability of +scheduling. Applications can request the percentage of block +bandwidth they need for operation and create dedicated sequencers +that guarantee eventual settlement into the chain. The drawback +is censorship, but because some space is available for +randomized UserBlock builders there is no way for the persistent +block builders to prevent transactions from eventually landing into +the chain or to be outbid in the next epoch. + +Future work may include assigning program state to a specific +persistent builder so only those builders can schedule transactions +that call those programs. + +#### UserBlock Compute Limits If the overall compute capacity for user transactions per leader -block is 48m CU, and cluster is configured with 2 builders, then -each UserBlock can use no more then 48m/4 or 12m CU. +block is 48m CU, and cluster is configured with 2 builders and +UserBlockSlots are configured to 200ms, then each UserBlock can +use no more then 48m/4 or 12m CU. -### UserBlock execution +### Priority ordering for UserBlock transaction execution Execution of UserBlocks can happen asynchronously to voting. When voting validators transmit their most recent BankHash, but it may @@ -273,7 +311,16 @@ user transactions. ## Drawbacks -[TBD] +The major drawback is figuring out how to manage resource allocation +between UserBlocks. If each UserBlock has 1/N capacity, each one +is much more likely to saturate and have a higher priority fee floor +then if the capacity was aggregated into 1 builder. + +The design should consider rolling some unused compute capacity +forward, because of asynchronous execution the network is able to +deal with bursts of greater than expected demand on compute as long +as the average demand allows all the nodes to create a snapshot +hash at least once an epoch. ## Backwards Compatibility From 6ca0683dcf3bfd290f16e5e08eb6c9a52520776f Mon Sep 17 00:00:00 2001 From: Anatoly Yakovenko Date: Wed, 14 Jun 2023 11:15:34 -0700 Subject: [PATCH 12/13] fixup! Add design for persistent userblock builders --- proposals/0023-asynchronous-program-execution-and-broadcast.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0023-asynchronous-program-execution-and-broadcast.md b/proposals/0023-asynchronous-program-execution-and-broadcast.md index adb3e223f..08810f714 100644 --- a/proposals/0023-asynchronous-program-execution-and-broadcast.md +++ b/proposals/0023-asynchronous-program-execution-and-broadcast.md @@ -114,7 +114,7 @@ builder has 1/N compute and shred capacity and users have no idea which one is saturated when sending their transaction. It's likely that priority fee floor will be different at each UserBlock builder. -#### Persistent UserBlock slots +#### Persistent UserBlock builders Persistent UserBlock slots for the epoch are auctioned of to the top N bidders who burn the most lamports. From 43e1f3b2decb6323c9bf4640ca8893b163a99c54 Mon Sep 17 00:00:00 2001 From: Anatoly Yakovenko Date: Fri, 23 Jun 2023 09:12:23 -0700 Subject: [PATCH 13/13] update --- ...hronous-program-execution-and-broadcast.md | 64 ++++++++++++++++--- 1 file changed, 54 insertions(+), 10 deletions(-) diff --git a/proposals/0023-asynchronous-program-execution-and-broadcast.md b/proposals/0023-asynchronous-program-execution-and-broadcast.md index 08810f714..91772b744 100644 --- a/proposals/0023-asynchronous-program-execution-and-broadcast.md +++ b/proposals/0023-asynchronous-program-execution-and-broadcast.md @@ -63,6 +63,7 @@ format. ## Detailed Design ### Overview + Leaders are scheduled to build blocks as they are currently by the LeaderSchedule. @@ -302,6 +303,16 @@ Leader could censor or delay the builder. Under heavy forking, validators will skip executing all the UserBlocks on minor forks. +## Slashing UserBlock Builders + +There shouldn't be a reason to slash UserBlock builders. Duplicate +UserBlocks will be resolved by the leader, and it is up to the +leader to pick one. If turbine failed and the rest of the network +has a different UserBlock, the leaders proposed block is dropped. + +If slashing is required, then stakeweighted or a bond based auction +would be necessary. + ## Economic Considerations Builders should be shuffled and scheduled according to stake weight. @@ -309,19 +320,52 @@ Builders should be shuffled and scheduled according to stake weight. TBD, deciding how should builders and leaders split the fees from user transactions. -## Drawbacks +## Capacity Management + +Because capacity is split between UserBlocks, max UserBlock size +as well as max account limit per UserBlock must be 1/N CUs, where +N is the number of concurrent builders. + +To prevent under-utilization, unused resources from blocks that are +partially filled can be used by future blocks up at most N * (block +or account limit). This would allow the network to handle bursts +of demand and on average maintain the desired load. + +## Multi leader spam + +Spammers would be motivated to send the same message to every +leader. + +1. builders censor fee payers by most significant bit of the fee +payer public key -The major drawback is figuring out how to manage resource allocation -between UserBlocks. If each UserBlock has 1/N capacity, each one -is much more likely to saturate and have a higher priority fee floor -then if the capacity was aggregated into 1 builder. +2. users generate N fee payers for N concurrent builders and use +the fee payer that is geographically closest. -The design should consider rolling some unused compute capacity -forward, because of asynchronous execution the network is able to -deal with bursts of greater than expected demand on compute as long -as the average demand allows all the nodes to create a snapshot -hash at least once an epoch. +If spammers want to send messages to multiple nodes, they would +need to use multiple fee payers and pay per node. + +## Drawbacks + +[TBD] ## Backwards Compatibility [TBD] + +## Implementation Roadmap + +1. Bankless leaders - separate out leader functionality such that +it doesn't depend on state. This would require relaxing that all +fee payers must be valid in a block. + +2. Fixed sized voting committees - configure network to run with +400 nodes voting per epoch, this requires consensus changes. Consensus +then has a fixed resource cost on the network. + +3. UserBlocks - separate votes from non votes, and transmit non-votes +in user blocks. Execution is still synchronous. + +4. Asynchronous execution - nodes skip user block execution before voting. + +5. Multiple builders - requires capacity management