-
Notifications
You must be signed in to change notification settings - Fork 5.7k
Add address lookup tables to minimized snapshot #30158
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -39,7 +39,10 @@ use { | |
| poh_timing_point::{send_poh_timing_point, PohTimingSender, SlotPohTimingInfo}, | ||
| }, | ||
| solana_rayon_threadlimit::get_max_thread_count, | ||
| solana_runtime::hardened_unpack::{unpack_genesis_archive, MAX_GENESIS_ARCHIVE_UNPACKED_SIZE}, | ||
| solana_runtime::{ | ||
| bank::Bank, | ||
| hardened_unpack::{unpack_genesis_archive, MAX_GENESIS_ARCHIVE_UNPACKED_SIZE}, | ||
| }, | ||
| solana_sdk::{ | ||
| clock::{Slot, UnixTimestamp, DEFAULT_TICKS_PER_SECOND, MS_PER_TICK}, | ||
| genesis_config::{GenesisConfig, DEFAULT_GENESIS_ARCHIVE, DEFAULT_GENESIS_FILE}, | ||
|
|
@@ -2835,6 +2838,7 @@ impl Blockstore { | |
| /// Used by ledger-tool to create a minimized snapshot | ||
| pub fn get_accounts_used_in_range( | ||
| &self, | ||
| bank: &Bank, | ||
| starting_slot: Slot, | ||
| ending_slot: Slot, | ||
| ) -> DashSet<Pubkey> { | ||
|
|
@@ -2844,11 +2848,27 @@ impl Blockstore { | |
| .into_par_iter() | ||
| .for_each(|slot| { | ||
| if let Ok(entries) = self.get_slot_entries(slot, 0) { | ||
| entries.par_iter().for_each(|entry| { | ||
| entry.transactions.iter().for_each(|tx| { | ||
| tx.message.static_account_keys().iter().for_each(|pubkey| { | ||
| result.insert(*pubkey); | ||
| }); | ||
| entries.into_par_iter().for_each(|entry| { | ||
| entry.transactions.into_iter().for_each(|tx| { | ||
| if let Some(lookups) = tx.message.address_table_lookups() { | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @t-nelson here |
||
| lookups.iter().for_each(|lookup| { | ||
| result.insert(lookup.account_key); | ||
| }); | ||
| } | ||
| // howdy, anybody who reached here from the panic messsage! | ||
| // the .unwrap() below could indicate there was an odd error or there | ||
| // could simply be a tx with a new ALT, which is just created/updated | ||
| // in this range. too bad... this edge case isn't currently supported. | ||
| // see: https://github.com/solana-labs/solana/issues/30165 | ||
| // for casual use, please choose different slot range. | ||
| let sanitized_tx = bank.fully_verify_transaction(tx).unwrap(); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Definitely better than it was, as when I wrote this originally I didn't even know lookup tables were a thing - but I'm not 100% convinced this is correct either. If a lookup table is extended in the slot range, then a tx after that could fail to verify this check (bank is looking at beginning of range) and we wouldn't add the associated keys for the transaction.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @apfitzge hehe, i know there's buried compromise here: how about this helpful, cheerful, encouraging comment?: db26aed92b7a101a3fb010f86b4f669860d9b8ba
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. patches are welcome but it'll take some good deal of efforts to support this fully....
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ^ agreed. I'm not sure how many people actually use this subcommand - but not sure its worth distracting us over scheduler. I'd update the comment to include this issue I just made: #30165 😄
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Yeah, it might be that we just have to replay the slot range and add accounts as we see them that way which is a bit of effort. With @xiangzhu70's change for keeping and loading from snapshot files will allow us to do this in a timely manner!
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
thanks for creating issue; just done: a2b8565
Oh sounds promising. :) |
||
| sanitized_tx | ||
| .message() | ||
| .account_keys() | ||
| .iter() | ||
| .for_each(|&pubkey| { | ||
| result.insert(pubkey); | ||
| }); | ||
| }); | ||
| }); | ||
| } | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1912,6 +1912,7 @@ impl Bank { | |
| additional_builtins, | ||
| debug_do_not_add_builtins, | ||
| ); | ||
| bank.fill_missing_sysvar_cache_entries(); | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. so, this single new line will be most objectionable change in this pr without in-source comments... Arguably, this is touching an important code path. after some research, in short, i think this is a simple overlook at both https://github.com/solana-labs/solana/pull/21108/files#r1098224198 #21108 and https://github.com/solana-labs/solana/pull/22455/files#r1098222965 #22455 So, how this unintentional omission relates to this pr?: bank fails to create sanitized tx from valid As for justification of touching this code path, I considered to relax However, I concluded the risk is small overall (use of sysvar cache on snapshot bank is VERY rare as this isn't noticed for months) then, that this correct (yet broad) fix should instead be preferred to prevent potential future subtle bugs and another dev suffering from this unhappy rabbit hole.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I think normal validator operations would never hit this case since we replay and create child banks. If there were a network outage (plz no) and we needed to restart from a snapshot, would we hit this case and be unable to restart w/o this patch?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
fortunately, it's not. the pesky hard network outage (hard-fork) case is rigorously tested by local-cluster tests. as far as i checked, However, we need populated sysvar_cache for minimized snapshot as a kind of unusual use case here. So, the other component which is affected would be rpc's tx preflight, with alt tx and with air-gapped node. (this is little known trick, but you can inspect account state at single point of time with rich set of rpc functionality if you make machine stand-alone with empty ledger with single snapshot). On the other hand, starting to populate sysvar_cache shouldn't cause any bad effects as well. After all, it's rather harmless operation.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Got it; it does seem like it was an oversight from the previous PRs on just not including this code on the initialization from fields.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes you are right. Can't think of any reason why it shouldn't be called here as well. Seems like I simply missed it and it did not break any tests. |
||
|
|
||
| // Sanity assertions between bank snapshot and genesis config | ||
| // Consider removing from serializable bank state | ||
|
|
@@ -6920,6 +6921,13 @@ impl Bank { | |
| Ok(sanitized_tx) | ||
| } | ||
|
|
||
| pub fn fully_verify_transaction( | ||
| &self, | ||
| tx: VersionedTransaction, | ||
| ) -> Result<SanitizedTransaction> { | ||
| self.verify_transaction(tx, TransactionVerificationMode::FullVerification) | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. i love type safety. but it's too verbose to impose this fn signature at call-site, imo.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. also, these small wrapper fns avoid new |
||
| } | ||
|
|
||
| /// only called from ledger-tool or tests | ||
| fn calculate_capitalization(&self, debug_verify: bool) -> u64 { | ||
| let is_startup = true; | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it looks like this speeds up a bit? Feature parity with other
solana-ledger-toolsubcommands.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah probably a good option to allow - nice.