From 5100ff72477ccb76757c21e2f79e361343929677 Mon Sep 17 00:00:00 2001 From: jherrera-jump Date: Tue, 3 Jun 2025 00:11:47 +0000 Subject: [PATCH 1/8] SIMD-0307: Add Block Header --- proposals/0307-add-block-header.md | 318 +++++++++++++++++++++++++++++ 1 file changed, 318 insertions(+) create mode 100644 proposals/0307-add-block-header.md diff --git a/proposals/0307-add-block-header.md b/proposals/0307-add-block-header.md new file mode 100644 index 000000000..d84d0f21f --- /dev/null +++ b/proposals/0307-add-block-header.md @@ -0,0 +1,318 @@ +--- +simd: '0307' +title: Add Block Header +authors: + - jherrera-jump (Firedancer) +category: Standard +type: Core +status: Review +created: 2025-06-17 +feature: +development: + - Anza - TBD +--- + +## Summary + +Add a block header to solana blocks and expose header fields in the +`getBlock` rpc endpoint. + +## Motivation + +For the purposes of historical monitoring, development, and auditing, it is +important to know exactly who produced a block and when it was produced. +Currently, this information can be partially inferred from Gossip and from vote +timestamps. Unfortunately there are some problems with the current approach: + +- The information from gossip is ephemeral. Currently a peer needs to record + and persist it. This may cause synchronization issues when matching client + updates in gossip with the correct slot. +- Gossip lacks important information that may useful for monitoring (e.g. + scheduler used, mods, configuration settings, etc). +- Vote timestamps have a granularity of 1-second, so they cannot be used to + estimate block duration. +- Vote timestamps will be removed with Alpenglow. + +This SIMD solves these issues by including relevant information in a static +block header. + +## New Terminology + +No new terms, but the following definitions are given for clarity: + +- Client - The software run by leaders to interface with a solana cluster. + (e.g. `agave` or `frankendancer`) +- Block Producer - The client that produced a given block +- Scheduler - The system responsible for processing incoming transactions and + ordering them for block construction. +- Forward Error Correction set (FEC set) - A collection of shreds. At a high + level, this is a construct that leverages Reed-Solomon encoding to overcome + the problem of data loss from packet drops. +- Shreds - A fixed chunk of encoded raw block data. +- Entry Batch - An array of entries. +- Entry - An array of transactions. + +## Detailed Design + +### Data Layout + +Solana blocks are organized in abstraction layers not entirely unlike the +arrangement of a typical network packet (e.g. MAC -> IP -> TCP -> HTTP). At the +highest layer a block consists of some number (~100+) FEC sets. A single FEC +set contains a handful of shreds (~32). Once sufficient shreds are available +the raw block data is reconstructed and reinterpreted as an array of entry +batches. Entry batches do not cross shred boundaries. + +This SIMD add the following header at the beginning of the raw block data. This +puts it on the same abstraction layer as serialized entry batch data. Put +differently, the serialized header will be prepended to the first serialized +entry batch in the block. + +``` +< -- 64 bits --> ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| block_header_flag | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| version | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| header_length | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| block_producer_time_nanos | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| block_user_agent | +| | +⋮ +30 ⋮ ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| ... future fields ... | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +``` + +- `block_header_flag: u64` will always be zero. The first 8 bytes of an entry +batch are always a positive number (the number of entries in the batch), so +this flag allows parsers to differentiate between a normal entry batch and one +with a header prepended. Though not strictly necessary, this may facilitate +parsing block data, and allows us to make the header optional if we ever need +to. + +- `version: u64` is a positive integer which changes anytime a change is made to +the header. The initial version will be 1. + +- `header_length: u64` is the length of the rest of the header in bytes (i.e. +not including the `block_header_flag`, `version`, and `header_length` fields). + +- `block_producer_time_nanos: u64` is a nanosecond UNIX timestamp representing +the time when the block producer became leader and started constructing the +block. + +- `block_user_agent: [u8; 256]` is a string that provides identifying +information about the block producer. + +- `future fields` any other fields that are deemed necessary in the future may +be added with a corresponding change to `version` / `header_length`. For +example, SIMD +[0298](https://github.com/solana-foundation/solana-improvement-documents/pull/298) +proposes a field header, which could be added as a subsequent SIMD (or folded +into this one). + +### Header Field Specification + +Header fields will be unilaterally populated by their respective block producer +without any enforced constraint on their contents. This SIMD includes the +following fields in the header + +- `block_producer_time_nanos`: u64 +- `block_user_agent`: [u8; 256] + +Because it is desirable to maintain cluster-wide diagnostics this SIMD provides +a suggested format for the `block_user_agent` string which includes basic +information about the block producer. This should be an UTF-8 encoded, null +terminated string. The null character should terminate valid UTF-8 data. Any +data following the null character is ignored by parsers and may contain +arbitrary information. It is expected that all producers use this format, +though this will not be enforced. Clients that choose to opt out of the +suggested format should set the first byte of the field to 0 (i.e. the null +character). The format is loosely based on HTTP `user-agent` header format +specification: + +``` +/ +``` + +The first entry will always be the software client. + +``` +client/client_version +``` + +Options for `client` currently include: + +- `agave` +- `frankendancer` +- `firedancer` + +`client_version` should be consistent with the information stored on-chain (in +`ConfigProgram`). Software forks (e.g. `jito-agave`) should put one of +the 3 base clients and can specify details about the fork in the comment. + +The comment should be in parentheses and contain a semicolon separated +list of flags. A flag has an unrestricted format, but should represent a +feature that is contained and enabled in the client it describes. + +e.g. + +``` +agave/v2.2.15 (jito; doublezero; some-mod/v1.2.3) +``` + +Sometimes there may be software that coexists or runs alongside a validator +client. For example, current client development aims to make the transaction +scheduler modular, which would allow the transaction scheduler to be developed +independently from the client codebase. Validator clients that use +complementary software like this should add additional +`/ ` entries in the user agent string. + +For example: + +``` +agave/v3.0.0 (doublezero) greedy-scheduler/v3 (mode:perf; another-flag) +``` + +### RPC Protocol Changes + +The `getBlock` RPC response will be extended to, optionally, include all header +fields. The request will be extended with the `header` parameter, which lets +the client signal that they want the header fields in the response. By default, +header fields will be included in the response. + +Sample Request Payload + +```json +{ + "jsonrpc": "2.0", + "id": 1, + "method": "getBlock", + "params": [ + 378967388, + { + "encoding": "json", + "maxSupportedTransactionVersion": 0, + "transactionDetails": "full", + "rewards": false, + "header": true + } + ] +} +``` + +Sample Response Payload + + +```json +{ + "jsonrpc": "2.0", + "result": { + "blockHeight": 428, + "blockTime": null, + "blockhash": "3Eq21vXNB5s86c62bVuUfTeaMif1N2kUqRPBmGRJhyTA", + "parentSlot": 429, + "previousBlockhash": "mfcyqEXB3DnHXki6KjjmZck6YjmZLvpAByy2fj4nh6B", + "header": { + "blockProducerTimeNanos": 1750176982899968023, + "blockUserAgent": "agave/v3.0.0 (doublezero) greedy-scheduler/v3 (mode:perf; another-flag)", + }, + "transactions": [ + { + "meta": { + "err": null, + "fee": 5000, + "innerInstructions": [], + "logMessages": [], + "postBalances": [499998932500, 26858640, 1, 1, 1], + "postTokenBalances": [], + "preBalances": [499998937500, 26858640, 1, 1, 1], + "preTokenBalances": [], + "rewards": null, + "status": { + "Ok": null + } + }, + "transaction": { + "message": { + "accountKeys": [ + "3UVYmECPPMZSCqWKfENfuoTv51fTDTWicX9xmBD2euKe", + "AjozzgE83A3x1sHNUR64hfH7zaEBWeMaFuAN9kQgujrc", + "SysvarS1otHashes111111111111111111111111111", + "SysvarC1ock11111111111111111111111111111111", + "Vote111111111111111111111111111111111111111" + ], + "header": { + "numReadonlySignedAccounts": 0, + "numReadonlyUnsignedAccounts": 3, + "numRequiredSignatures": 1 + }, + "instructions": [ + { + "accounts": [1, 2, 3, 0], + "data": "37u9WtQpcm6ULa3WRQHmj49EPs4if7o9f1jSRVZpm2dvihR9C8jY4NqEwXUbLwx15HBSNcP1", + "programIdIndex": 4 + } + ], + "recentBlockhash": "mfcyqEXB3DnHXki6KjjmZck6YjmZLvpAByy2fj4nh6B" + }, + "signatures": [ + "2nBhEBYYvfaAe16UMNqRHre4YNSskvuYgx3M6E4JP1oDYvZEJHvoPzyUidNgNX5r9sTyN1J9UxtbCXy2rqYcuyuv" + ] + } + } + ] + }, + "id": 1 +} +``` + + +## Alternatives Considered + +- Do nothing + - We can't estimate block time / duration with sufficient granularity. We + won't be able to estimate at all when votes are changed in alpenglow. + - We will continue to have an incomplete, ephemeral record of who produced + blocks. +- derive timestamp header field from consensus and enforce user agent format + - This can and probably should be implemented as a future SIMD. Meanwhile, + these fields are still useful since + 1. most of the cluster is expected to + be honest, so monitoring tools may still use them for cluster-wide + analytics and + 2. block producers still use these fields to self-monitor + their performance. +- Send block producer information via gossip instead + - The information is short-lived and depends on physical network availability +- Update this information in an on-chain account instead (e.g. ConfigProgram) + - Same issue as above, the information is short-lived. + +## Impact + +This change will enable more reliable monitoring and benchmarking for operators +and for the community. Clients and indexers will need to extend both in-memory +and long-term block storage to be aware of the new columns added to the block +header. The client rpc engine will need to change to support the new fields. + +## Security Considerations + +- The header fields are untrusted and purely informational. Tools that expose + these fields to external users should clearly communicate their untrusted + nature. + +## Drawbacks + +- No expected drawbacks beyond minimal resource overhead. + +## Backwards Compatibility + +- RPC requests for old slots should properly document and return a suitable + default value (e.g. None). +- Clients that don't implement this SIMD will reject new blocks because they +will fail to parse the new header. +- Because this header is mandatory, leaders that produce blocks without a +header will skip, since the header is required. \ No newline at end of file From 154d60a95ca9c3f59647d9fefb9fa281ce5e8840 Mon Sep 17 00:00:00 2001 From: jherrera-jump Date: Wed, 18 Jun 2025 22:21:51 +0000 Subject: [PATCH 2/8] change header_length type, change block_producer_time_nanos description --- proposals/0307-add-block-header.md | 71 +++++++++++++++--------------- 1 file changed, 36 insertions(+), 35 deletions(-) diff --git a/proposals/0307-add-block-header.md b/proposals/0307-add-block-header.md index d84d0f21f..4f4bb1cec 100644 --- a/proposals/0307-add-block-header.md +++ b/proposals/0307-add-block-header.md @@ -69,22 +69,23 @@ differently, the serialized header will be prepended to the first serialized entry batch in the block. ``` -< -- 64 bits --> -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| block_header_flag | -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| version | -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| header_length | -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| block_producer_time_nanos | -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| block_user_agent | -| | -⋮ +30 ⋮ -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| ... future fields ... | -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + Block Header Layout ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| block_header_flag (64 bits) | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| version (64 bits) | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| header_length (16 bits) | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| block_producer_time_nanos (64 bits) | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| block_user_agent_len (8 bits) | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| block_user_agent (0-255 bytes) | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + +Note that header fields are packed together without any alignment requirements +or padding. ``` - `block_header_flag: u64` will always be zero. The first 8 bytes of an entry @@ -97,22 +98,23 @@ to. - `version: u64` is a positive integer which changes anytime a change is made to the header. The initial version will be 1. -- `header_length: u64` is the length of the rest of the header in bytes (i.e. +- `header_length: u16` is the length of the rest of the header in bytes (i.e. not including the `block_header_flag`, `version`, and `header_length` fields). - `block_producer_time_nanos: u64` is a nanosecond UNIX timestamp representing -the time when the block producer became leader and started constructing the -block. +the time when the block producer started constructing the block. -- `block_user_agent: [u8; 256]` is a string that provides identifying -information about the block producer. +- `block_user_agent_len: u8` the length of the `block_user_agent` string in +bytes. -- `future fields` any other fields that are deemed necessary in the future may -be added with a corresponding change to `version` / `header_length`. For -example, SIMD +- `block_user_agent: String` is a variable length utf-8 encoded string that +provides identifying information about the block producer. + +Any other fields that are deemed necessary in the future may be added with a +corresponding change to `version` / `header_length`. For example, SIMD [0298](https://github.com/solana-foundation/solana-improvement-documents/pull/298) -proposes a field header, which could be added as a subsequent SIMD (or folded -into this one). +proposes a field header, which could be added as a subsequent SIMD (or even +folded into this one). ### Header Field Specification @@ -121,18 +123,17 @@ without any enforced constraint on their contents. This SIMD includes the following fields in the header - `block_producer_time_nanos`: u64 -- `block_user_agent`: [u8; 256] +- `block_user_agent_len`: u8 +- `block_user_agent`: String Because it is desirable to maintain cluster-wide diagnostics this SIMD provides a suggested format for the `block_user_agent` string which includes basic -information about the block producer. This should be an UTF-8 encoded, null -terminated string. The null character should terminate valid UTF-8 data. Any -data following the null character is ignored by parsers and may contain -arbitrary information. It is expected that all producers use this format, -though this will not be enforced. Clients that choose to opt out of the -suggested format should set the first byte of the field to 0 (i.e. the null -character). The format is loosely based on HTTP `user-agent` header format -specification: +information about the block producer. This should be a UTF-8 encoded variable +length string (up to 255 bytes long). It is not necessarily null-terminated. It +is expected that all producers use the format specified here, though this will +not be enforced. Clients are encouraged to at the very least use a valid utf-8 +string and include extraneous data in a way that coheres with the specification. +The format is loosely based on HTTP `user-agent` header format specification: ``` / From 66a0d14cb1fce4eca2ca21b1a43485eb6bc68135 Mon Sep 17 00:00:00 2001 From: jherrera-jump Date: Wed, 2 Jul 2025 19:39:41 +0000 Subject: [PATCH 3/8] use footer instead of header --- ...ock-header.md => 0307-add-block-footer.md} | 81 +++++++++++-------- 1 file changed, 46 insertions(+), 35 deletions(-) rename proposals/{0307-add-block-header.md => 0307-add-block-footer.md} (80%) diff --git a/proposals/0307-add-block-header.md b/proposals/0307-add-block-footer.md similarity index 80% rename from proposals/0307-add-block-header.md rename to proposals/0307-add-block-footer.md index 4f4bb1cec..884254f4a 100644 --- a/proposals/0307-add-block-header.md +++ b/proposals/0307-add-block-footer.md @@ -1,6 +1,6 @@ --- simd: '0307' -title: Add Block Header +title: Add Block Footer authors: - jherrera-jump (Firedancer) category: Standard @@ -14,7 +14,7 @@ development: ## Summary -Add a block header to solana blocks and expose header fields in the +Add a block footer to Solana blocks and expose Footer fields in the `getBlock` rpc endpoint. ## Motivation @@ -34,7 +34,7 @@ timestamps. Unfortunately there are some problems with the current approach: - Vote timestamps will be removed with Alpenglow. This SIMD solves these issues by including relevant information in a static -block header. +block footer. ## New Terminology @@ -63,19 +63,19 @@ set contains a handful of shreds (~32). Once sufficient shreds are available the raw block data is reconstructed and reinterpreted as an array of entry batches. Entry batches do not cross shred boundaries. -This SIMD add the following header at the beginning of the raw block data. This +This SIMD add the following footer at the end of the raw block data. This puts it on the same abstraction layer as serialized entry batch data. Put -differently, the serialized header will be prepended to the first serialized +differently, the serialized footer will be appended after the last serialized entry batch in the block. ``` - Block Header Layout + Block Footer Layout +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| block_header_flag (64 bits) | +| block_footer_flag (64 bits) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | version (64 bits) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| header_length (16 bits) | +| footer_length (16 bits) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | block_producer_time_nanos (64 bits) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ @@ -84,25 +84,29 @@ entry batch in the block. | block_user_agent (0-255 bytes) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -Note that header fields are packed together without any alignment requirements +Note that footer fields are packed together without any alignment requirements or padding. ``` -- `block_header_flag: u64` will always be zero. The first 8 bytes of an entry +- `block_footer_flag: u64` will always be zero. The first 8 bytes of an entry batch are always a positive number (the number of entries in the batch), so -this flag allows parsers to differentiate between a normal entry batch and one -with a header prepended. Though not strictly necessary, this may facilitate -parsing block data, and allows us to make the header optional if we ever need -to. +this flag allows parsers to differentiate the footer from a normal entry batch. +This facilitates parsing block data and would also allow us to make the footer +optional if that's ever needed. - `version: u64` is a positive integer which changes anytime a change is made to -the header. The initial version will be 1. +the footer. The initial version will be 1. -- `header_length: u16` is the length of the rest of the header in bytes (i.e. -not including the `block_header_flag`, `version`, and `header_length` fields). +- `footer_length: u16` is the length of the rest of the footer in bytes (i.e. +not including the `block_footer_flag`, `version`, and `footer_length` fields). - `block_producer_time_nanos: u64` is a nanosecond UNIX timestamp representing the time when the block producer started constructing the block. +"started constructing" is the point at which, from the perspective of the +leader, all of the consensus checks required for assuming leadership have +"just passed". For example, in Agave's pre-alpenglow implementations, this would +be in replay/maybe_start_leader. In a post-Alpenglow implementation, this would +be after receiving the proper vote/skip certificate for the previous slot. - `block_user_agent_len: u8` the length of the `block_user_agent` string in bytes. @@ -111,16 +115,16 @@ bytes. provides identifying information about the block producer. Any other fields that are deemed necessary in the future may be added with a -corresponding change to `version` / `header_length`. For example, SIMD +corresponding change to `version` / `footer_length`. For example, SIMD [0298](https://github.com/solana-foundation/solana-improvement-documents/pull/298) -proposes a field header, which could be added as a subsequent SIMD (or even +proposes a header field, which could be added as a subsequent SIMD (or even folded into this one). -### Header Field Specification +### Footer Field Specification -Header fields will be unilaterally populated by their respective block producer +Footer fields will be unilaterally populated by their respective block producer without any enforced constraint on their contents. This SIMD includes the -following fields in the header +following fields in the footer - `block_producer_time_nanos`: u64 - `block_user_agent_len`: u8 @@ -180,10 +184,10 @@ agave/v3.0.0 (doublezero) greedy-scheduler/v3 (mode:perf; another-flag) ### RPC Protocol Changes -The `getBlock` RPC response will be extended to, optionally, include all header -fields. The request will be extended with the `header` parameter, which lets -the client signal that they want the header fields in the response. By default, -header fields will be included in the response. +The `getBlock` RPC response will be extended to, optionally, include all footer +fields. The request will be extended with the `footer` parameter, which lets +the client signal that they want the footer fields in the response. By default, +footer fields will be included in the response. Sample Request Payload @@ -199,7 +203,7 @@ Sample Request Payload "maxSupportedTransactionVersion": 0, "transactionDetails": "full", "rewards": false, - "header": true + "footer": true } ] } @@ -217,7 +221,7 @@ Sample Response Payload "blockhash": "3Eq21vXNB5s86c62bVuUfTeaMif1N2kUqRPBmGRJhyTA", "parentSlot": 429, "previousBlockhash": "mfcyqEXB3DnHXki6KjjmZck6YjmZLvpAByy2fj4nh6B", - "header": { + "footer": { "blockProducerTimeNanos": 1750176982899968023, "blockUserAgent": "agave/v3.0.0 (doublezero) greedy-scheduler/v3 (mode:perf; another-flag)", }, @@ -246,7 +250,7 @@ Sample Response Payload "SysvarC1ock11111111111111111111111111111111", "Vote111111111111111111111111111111111111111" ], - "header": { + "footer": { "numReadonlySignedAccounts": 0, "numReadonlyUnsignedAccounts": 3, "numRequiredSignatures": 1 @@ -272,6 +276,13 @@ Sample Response Payload ``` +### Mandating the block footer + +While it is possible to make the block footer optional thanks to the +`block_footer_flag` field, this proposal makes it mandatory. Blocks that don't +include a valid footer in the block payload will be flagged as dead blocks and +skipped by the other nodes in the cluster. + ## Alternatives Considered - Do nothing @@ -279,7 +290,7 @@ Sample Response Payload won't be able to estimate at all when votes are changed in alpenglow. - We will continue to have an incomplete, ephemeral record of who produced blocks. -- derive timestamp header field from consensus and enforce user agent format +- derive timestamp footer field from consensus and enforce user agent format - This can and probably should be implemented as a future SIMD. Meanwhile, these fields are still useful since 1. most of the cluster is expected to @@ -297,11 +308,11 @@ Sample Response Payload This change will enable more reliable monitoring and benchmarking for operators and for the community. Clients and indexers will need to extend both in-memory and long-term block storage to be aware of the new columns added to the block -header. The client rpc engine will need to change to support the new fields. +footer. The client rpc engine will need to change to support the new fields. ## Security Considerations -- The header fields are untrusted and purely informational. Tools that expose +- The footer fields are untrusted and purely informational. Tools that expose these fields to external users should clearly communicate their untrusted nature. @@ -314,6 +325,6 @@ header. The client rpc engine will need to change to support the new fields. - RPC requests for old slots should properly document and return a suitable default value (e.g. None). - Clients that don't implement this SIMD will reject new blocks because they -will fail to parse the new header. -- Because this header is mandatory, leaders that produce blocks without a -header will skip, since the header is required. \ No newline at end of file +will fail to parse the new footer. +- Because this footer is mandatory, leaders that produce blocks without a +footer will skip. \ No newline at end of file From 6428df04247980c2e93d63a63f3781dbb6af521b Mon Sep 17 00:00:00 2001 From: jherrera-jump Date: Tue, 8 Jul 2025 18:00:04 +0000 Subject: [PATCH 4/8] clarify wording in data layout + mandate sections --- proposals/0307-add-block-footer.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/proposals/0307-add-block-footer.md b/proposals/0307-add-block-footer.md index 884254f4a..705a5808a 100644 --- a/proposals/0307-add-block-footer.md +++ b/proposals/0307-add-block-footer.md @@ -61,12 +61,22 @@ arrangement of a typical network packet (e.g. MAC -> IP -> TCP -> HTTP). At the highest layer a block consists of some number (~100+) FEC sets. A single FEC set contains a handful of shreds (~32). Once sufficient shreds are available the raw block data is reconstructed and reinterpreted as an array of entry -batches. Entry batches do not cross shred boundaries. +batches. Entry batches are aligned with shred boundaries (i.e. they will +start/stop at a shred boundary). This SIMD add the following footer at the end of the raw block data. This puts it on the same abstraction layer as serialized entry batch data. Put differently, the serialized footer will be appended after the last serialized -entry batch in the block. +entry batch in the block as its own pseudo-entry-batch. Parsers should, +however, support the footer actually being placed anywhere in the block, +between any other entry batches. The footer should be parsed as its own +pseudo-entry-batch and will be differentiated from other entry batches using +the `block_footer_flag` field. Allowing the footer anywhere in the block +gives us the flexibility to fix it as a header or a footer in a future SIMD. +Currently, we call it a "footer" and encourage block producers to add it to +the end of the block since we think future SIMD's may require the footer to +be computed after constructing the block (e.g. a new timing metric, async +execution). ``` Block Footer Layout @@ -280,7 +290,7 @@ Sample Response Payload While it is possible to make the block footer optional thanks to the `block_footer_flag` field, this proposal makes it mandatory. Blocks that don't -include a valid footer in the block payload will be flagged as dead blocks and +include a valid footer in the block payload must be flagged as dead blocks and skipped by the other nodes in the cluster. ## Alternatives Considered From 531dbac8439402577430f8d41797e7e7cc184c27 Mon Sep 17 00:00:00 2001 From: jherrera-jump Date: Thu, 7 Aug 2025 20:38:09 +0000 Subject: [PATCH 5/8] generalize the footer as a block meta variant --- proposals/0307-add-block-footer.md | 147 ++++++++++++++++++----------- 1 file changed, 91 insertions(+), 56 deletions(-) diff --git a/proposals/0307-add-block-footer.md b/proposals/0307-add-block-footer.md index 705a5808a..e1986365c 100644 --- a/proposals/0307-add-block-footer.md +++ b/proposals/0307-add-block-footer.md @@ -14,7 +14,7 @@ development: ## Summary -Add a block footer to Solana blocks and expose Footer fields in the +Add a block footer to Solana blocks and expose footer fields in the `getBlock` rpc endpoint. ## Motivation @@ -51,6 +51,9 @@ No new terms, but the following definitions are given for clarity: - Shreds - A fixed chunk of encoded raw block data. - Entry Batch - An array of entries. - Entry - An array of transactions. +- Block Meta Chunk - A chunk of structured non-transaction data, typically + metadata fields, that can be placed before, after, or in-between entry + batches in a block. ## Detailed Design @@ -58,34 +61,75 @@ No new terms, but the following definitions are given for clarity: Solana blocks are organized in abstraction layers not entirely unlike the arrangement of a typical network packet (e.g. MAC -> IP -> TCP -> HTTP). At the -highest layer a block consists of some number (~100+) FEC sets. A single FEC -set contains a handful of shreds (~32). Once sufficient shreds are available -the raw block data is reconstructed and reinterpreted as an array of entry -batches. Entry batches are aligned with shred boundaries (i.e. they will -start/stop at a shred boundary). - -This SIMD add the following footer at the end of the raw block data. This -puts it on the same abstraction layer as serialized entry batch data. Put -differently, the serialized footer will be appended after the last serialized -entry batch in the block as its own pseudo-entry-batch. Parsers should, -however, support the footer actually being placed anywhere in the block, -between any other entry batches. The footer should be parsed as its own -pseudo-entry-batch and will be differentiated from other entry batches using -the `block_footer_flag` field. Allowing the footer anywhere in the block -gives us the flexibility to fix it as a header or a footer in a future SIMD. -Currently, we call it a "footer" and encourage block producers to add it to -the end of the block since we think future SIMD's may require the footer to -be computed after constructing the block (e.g. a new timing metric, async -execution). +highest layer a block consists of some number (~100+) FEC sets. A single FEC set +contains a handful of shreds (~32). Once sufficient shreds are available the raw +block data is reconstructed and reinterpreted as an array of entry batches. +Entry batches are aligned with shred boundaries (i.e. they will start/stop at a +shred boundary). + +This SIMD introduces the idea of a block metadata chunk (block meta). This is a +piece of data that would take the place of an entry batch in an incoming shred +stream. Entry batch data starts with an 8 byte value that represents the number +of entries in the batch. This number cannot be zero. By including 8 zero bytes +at the beginning of the block meta header, a replay parser can differentiate it +from a regular entry batch. A block meta chunk has the following versioned +header: ``` - Block Footer Layout + + Block Meta Header Layout ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| block_meta_flag (64 bits of 0) | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| version=1 (16 bits) | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| variant (8 bits) | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| length (16 bits) | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| --payload-- | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + +Note that all data fields below are packed together without any alignment or +padding constraints. + +``` + +`block_meta_flag: u64`: will always be zero. The first 8 bytes of an entry batch +are always a positive number (the number of entries in the batch), so this flag +allows parsers to differentiate the block meta from a normal entry batch. + +- `version: u16` is a positive integer which changes anytime a change is made to +the block meta header. The initial version will be 1. + +- `variant: u8` is a positive integer which identifies the structure of the +block meta payload. For example, the block footer will be identified by +`variant=1`. New metadata may be added without changing the footer by adding a +new variant which corresponds to a differently specified payload. + +- `length: u16` is the length of the block meta payload in bytes (i.e. not +including the `block_meta_flag`, `version`, `variant`, and `length` fields). +Though not necessary, this will make it easier for block parsers to ignore +certain variants. + +This SIMD also proposes the following block meta variant with an additional +constraint: it must occur once after the last entry batch in a block. The block +footer is meant to contain general block and producer metadata, along with any +metrics that must be computed after the block has been produced. + +``` + + Block Footer +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| block_footer_flag (64 bits) | +| block_meta_flag (64 bits of 0) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| version (64 bits) | +| version=1 (16 bits) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| footer_length (16 bits) | +| variant=1 (8 bits) | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| length (16 bits) | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| footer_version=1 (16 bits) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | block_producer_time_nanos (64 bits) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ @@ -94,29 +138,21 @@ execution). | block_user_agent (0-255 bytes) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -Note that footer fields are packed together without any alignment requirements -or padding. -``` - -- `block_footer_flag: u64` will always be zero. The first 8 bytes of an entry -batch are always a positive number (the number of entries in the batch), so -this flag allows parsers to differentiate the footer from a normal entry batch. -This facilitates parsing block data and would also allow us to make the footer -optional if that's ever needed. +Note that all data fields below are packed together without any alignment or +padding constraints. -- `version: u64` is a positive integer which changes anytime a change is made to -the footer. The initial version will be 1. +``` -- `footer_length: u16` is the length of the rest of the footer in bytes (i.e. -not including the `block_footer_flag`, `version`, and `footer_length` fields). +- `footer_version: u16` is a positive integer which changes anytime a change is +made to the footer. The initial version will be 1. - `block_producer_time_nanos: u64` is a nanosecond UNIX timestamp representing -the time when the block producer started constructing the block. -"started constructing" is the point at which, from the perspective of the -leader, all of the consensus checks required for assuming leadership have -"just passed". For example, in Agave's pre-alpenglow implementations, this would -be in replay/maybe_start_leader. In a post-Alpenglow implementation, this would -be after receiving the proper vote/skip certificate for the previous slot. +the time when the block producer started constructing the block. "started +constructing" is the point at which, from the perspective of the leader, all of +the consensus checks required for assuming leadership have "just passed". For +example, in Agave's pre-alpenglow implementations, this would be in +replay/maybe_start_leader. In a post-Alpenglow implementation, this would be +after receiving the proper vote/skip certificate for the previous slot. - `block_user_agent_len: u8` the length of the `block_user_agent` string in bytes. @@ -124,11 +160,11 @@ bytes. - `block_user_agent: String` is a variable length utf-8 encoded string that provides identifying information about the block producer. -Any other fields that are deemed necessary in the future may be added with a -corresponding change to `version` / `footer_length`. For example, SIMD -[0298](https://github.com/solana-foundation/solana-improvement-documents/pull/298) -proposes a header field, which could be added as a subsequent SIMD (or even -folded into this one). +Any other fields that are deemed necessary in the future may be added in one of +two ways. + +- amend the `footer_version` and add the field the footer +- create a new block meta variant and add the field to its payload ### Footer Field Specification @@ -176,7 +212,7 @@ feature that is contained and enabled in the client it describes. e.g. ``` -agave/v2.2.15 (jito; doublezero; some-mod/v1.2.3) +agave/v2.2.15 (jito; double0; some-mod/v1.2.3) ``` Sometimes there may be software that coexists or runs alongside a validator @@ -189,7 +225,7 @@ complementary software like this should add additional For example: ``` -agave/v3.0.0 (doublezero) greedy-scheduler/v3 (mode:perf; another-flag) +agave/v3.0.0 (paladin; double0) greedy-scheduler/v3 (mode:perf; another-flag) ``` ### RPC Protocol Changes @@ -233,7 +269,7 @@ Sample Response Payload "previousBlockhash": "mfcyqEXB3DnHXki6KjjmZck6YjmZLvpAByy2fj4nh6B", "footer": { "blockProducerTimeNanos": 1750176982899968023, - "blockUserAgent": "agave/v3.0.0 (doublezero) greedy-scheduler/v3 (mode:perf; another-flag)", + "blockUserAgent": "agave/v3.0.0 (double0) greedy-scheduler/v3 (mode:perf; another-flag)", }, "transactions": [ { @@ -286,12 +322,11 @@ Sample Response Payload ``` -### Mandating the block footer +### Mandating the footer -While it is possible to make the block footer optional thanks to the -`block_footer_flag` field, this proposal makes it mandatory. Blocks that don't -include a valid footer in the block payload must be flagged as dead blocks and -skipped by the other nodes in the cluster. +This proposal makes the block footer. Blocks that don't include a valid footer +in the block payload must be flagged as dead blocks and skipped by the other +nodes in the cluster. ## Alternatives Considered From 3feaaec36083cefd818a0757a536b9acc58d6e8e Mon Sep 17 00:00:00 2001 From: jherrera-jump Date: Fri, 8 Aug 2025 20:59:19 +0000 Subject: [PATCH 6/8] change block meta to block marker --- proposals/0307-add-block-footer.md | 44 ++++++++++++++---------------- 1 file changed, 21 insertions(+), 23 deletions(-) diff --git a/proposals/0307-add-block-footer.md b/proposals/0307-add-block-footer.md index e1986365c..7a691254f 100644 --- a/proposals/0307-add-block-footer.md +++ b/proposals/0307-add-block-footer.md @@ -51,9 +51,8 @@ No new terms, but the following definitions are given for clarity: - Shreds - A fixed chunk of encoded raw block data. - Entry Batch - An array of entries. - Entry - An array of transactions. -- Block Meta Chunk - A chunk of structured non-transaction data, typically - metadata fields, that can be placed before, after, or in-between entry - batches in a block. +- Block Marker - A chunk of structured non-transaction data that can be placed + before, after, or in-between entry batches in a block. ## Detailed Design @@ -67,19 +66,18 @@ block data is reconstructed and reinterpreted as an array of entry batches. Entry batches are aligned with shred boundaries (i.e. they will start/stop at a shred boundary). -This SIMD introduces the idea of a block metadata chunk (block meta). This is a -piece of data that would take the place of an entry batch in an incoming shred -stream. Entry batch data starts with an 8 byte value that represents the number -of entries in the batch. This number cannot be zero. By including 8 zero bytes -at the beginning of the block meta header, a replay parser can differentiate it -from a regular entry batch. A block meta chunk has the following versioned -header: +This SIMD introduces the idea of a block marker. This is a piece of data that +would take the place of an entry batch in an incoming shred stream. Entry batch +data starts with an 8 byte value that represents the number of entries in the +batch. This number cannot be zero. By including 8 zero bytes at the beginning of +the block marker header, a replay parser can differentiate it from a regular +entry batch. A block marker chunk has the following versioned header: ``` - Block Meta Header Layout + Block Marker Header Layout +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| block_meta_flag (64 bits of 0) | +| block_marker_flag (64 bits of 0) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | version=1 (16 bits) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ @@ -95,33 +93,33 @@ padding constraints. ``` -`block_meta_flag: u64`: will always be zero. The first 8 bytes of an entry batch -are always a positive number (the number of entries in the batch), so this flag -allows parsers to differentiate the block meta from a normal entry batch. +`block_marker_flag: u64`: will always be zero. The first 8 bytes of an entry +batch are always a positive number (the number of entries in the batch), so this +flag allows parsers to differentiate the block marker from a normal entry batch. - `version: u16` is a positive integer which changes anytime a change is made to -the block meta header. The initial version will be 1. +the block marker header. The initial version will be 1. - `variant: u8` is a positive integer which identifies the structure of the -block meta payload. For example, the block footer will be identified by +block marker payload. For example, the block footer will be identified by `variant=1`. New metadata may be added without changing the footer by adding a new variant which corresponds to a differently specified payload. -- `length: u16` is the length of the block meta payload in bytes (i.e. not -including the `block_meta_flag`, `version`, `variant`, and `length` fields). +- `length: u16` is the length of the block marker payload in bytes (i.e. not +including the `block_marker_flag`, `version`, `variant`, and `length` fields). Though not necessary, this will make it easier for block parsers to ignore certain variants. -This SIMD also proposes the following block meta variant with an additional +This SIMD also proposes the following block marker variant with an additional constraint: it must occur once after the last entry batch in a block. The block footer is meant to contain general block and producer metadata, along with any metrics that must be computed after the block has been produced. ``` - Block Footer + Block Marker Variant -- Footer +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| block_meta_flag (64 bits of 0) | +| block_marker_flag (64 bits of 0) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | version=1 (16 bits) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ @@ -164,7 +162,7 @@ Any other fields that are deemed necessary in the future may be added in one of two ways. - amend the `footer_version` and add the field the footer -- create a new block meta variant and add the field to its payload +- create a new block marker variant and add the field to its payload ### Footer Field Specification From 9af16174645d3ec30975e45e97a14e2fe06a6fb6 Mon Sep 17 00:00:00 2001 From: jherrera-jump Date: Mon, 11 Aug 2025 15:40:28 +0000 Subject: [PATCH 7/8] change variant to start at variant=0 --- proposals/0307-add-block-footer.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/proposals/0307-add-block-footer.md b/proposals/0307-add-block-footer.md index 7a691254f..6548f7461 100644 --- a/proposals/0307-add-block-footer.md +++ b/proposals/0307-add-block-footer.md @@ -100,10 +100,10 @@ flag allows parsers to differentiate the block marker from a normal entry batch. - `version: u16` is a positive integer which changes anytime a change is made to the block marker header. The initial version will be 1. -- `variant: u8` is a positive integer which identifies the structure of the -block marker payload. For example, the block footer will be identified by -`variant=1`. New metadata may be added without changing the footer by adding a -new variant which corresponds to a differently specified payload. +- `variant: u8` is an integer which identifies the structure of the block marker +payload. For example, the block footer will be identified by `variant=0`. New +metadata may be added without changing the footer by adding a new variant which +corresponds to a differently specified payload. - `length: u16` is the length of the block marker payload in bytes (i.e. not including the `block_marker_flag`, `version`, `variant`, and `length` fields). @@ -123,7 +123,7 @@ metrics that must be computed after the block has been produced. +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | version=1 (16 bits) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| variant=1 (8 bits) | +| variant=0 (8 bits) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | length (16 bits) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ From 814b50dfafafeced64a7b1028eb9b936aed7a76f Mon Sep 17 00:00:00 2001 From: jherrera-jump Date: Wed, 13 Aug 2025 15:00:32 +0000 Subject: [PATCH 8/8] block marker header fields explanation --- proposals/0307-add-block-footer.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/proposals/0307-add-block-footer.md b/proposals/0307-add-block-footer.md index 6548f7461..bf44908b3 100644 --- a/proposals/0307-add-block-footer.md +++ b/proposals/0307-add-block-footer.md @@ -110,6 +110,14 @@ including the `block_marker_flag`, `version`, `variant`, and `length` fields). Though not necessary, this will make it easier for block parsers to ignore certain variants. +This header would precede any block marker variant inserted into the block data. +Although the block marker data length can be inferred from the version/variant, +an explicit length field is included for non-client parsers which discard +non-transactional data from the block. This ensures that we don't create a +dependency on these parsers when pushing out new/updated block markers. It also +greatly simplifies the implementation of hardware parsers, especially +considering that some variants contain variable length fields. + This SIMD also proposes the following block marker variant with an additional constraint: it must occur once after the last entry batch in a block. The block footer is meant to contain general block and producer metadata, along with any