(WIP) Prototyping the new Accounts DB storage

[yhchiang-sol]

WIP.

Based on #28550

[yhchiang-sol]

Hello, @jeffwashington. If you want to test out the PR, this can be done by compiling the ledger-tool and run the following command:

solana-ledger-tool new_ads_file --append-vec <append_vec_path> --ads-file-name <new_file_path> --ledger <ledger>

Here's my script for running it for example:

 /ds-file-nameUsers/keelar/solana/yhchiang-sol/target/debug/solana-ledger-tool new_ads_file --append-vec /Users/keelar/solana/account-db-data/$1 --ads-file-name /Users/keelar/solana/account-db-data/ads-$1 --ledger /Users/keelar/solana/disks/ledger/

The above command will create the new accounts-data-storage file from the specified append_vec. It will also verify whether the two files have the same account meta and data.

For each account, the command will dump its data from both append_vec and the new format and verify whether they are the same:

[2022-12-02T20:28:01.556305000Z INFO  solana_ledger_tool] verifing account ByMuSmKJUYTqFpUZWDDCgqPHwCzBeGNWkN2SZmyM24Rs
[2022-12-02T20:28:01.556333000Z INFO  solana_ledger_tool]     lamport: append_vec(2039280) vs ads_file (2039280)
entry = AccountMetaStorageEntry { lamports: 2039280, block_offset: 1697855, owner_local_id: 0, uncompressed_length: 165, local_offset: 1041 }
[2022-12-02T20:28:01.556393000Z INFO  solana_ledger_tool]  StoredMeta: append_vec(StoredMeta { write_version: 457658454976, pubkey: ByMuSmKJUYTqFpUZWDDCgqPHwCzBeGNWkN2SZmyM24Rs, data_len: 165 })
[2022-12-02T20:28:01.556425000Z INFO  solana_ledger_tool]              ads_file  (StoredMeta { write_version: 457658454976, pubkey: ByMuSmKJUYTqFpUZWDDCgqPHwCzBeGNWkN2SZmyM24Rs, data_len: 165 })
[2022-12-02T20:28:01.556458000Z INFO  solana_ledger_tool]  AccountSharedData: append_vec(Account { lamports: 2039280, data.len: 165, owner: TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA, executable: false, rent_epoch: 358, data: 0b3338a0ab2cc841d5b014bc6a3cf756291874b319c9517d9bbfa9e4e9661ef96a9b111acda5f839cc36127979c2d0f29a3089fe40509696e77b210d04595108 })
[2022-12-02T20:28:01.556512000Z INFO  solana_ledger_tool]                     ads_file  (Account { lamports: 2039280, data.len: 165, owner: TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA, executable: false, rent_epoch: 358, data: 0b3338a0ab2cc841d5b014bc6a3cf756291874b319c9517d9bbfa9e4e9661ef96a9b111acda5f839cc36127979c2d0f29a3089fe40509696e77b210d04595108 })

In my experiment, the new format could save up to 75% of the storage size:

-rw-r--r--  1 keelar  staff  21524480 Dec  2 00:05 154931095.1633108
-rw-r--r--  1 keelar  staff   5545380 Dec  2 12:23 ads-154931095.1633108

[yhchiang-sol]

@jeffwashington: FYI: this is the command that creates the new accounts-data-storage from from an append_vec file and verify its correctness.