Accounts DB proposal: Tiered-storage design and cold-storage format

[yhchiang-sol]

Summary of Changes

This PR includes the proposal for the tiered-storage design and
cold-storage format for accounts DB.

It is an updated design proposal based on the previous proposal (#28550),
sketch implementation (#28790), and online/offline discussions with
jwash, brooks, and many Solana engineers.

[brooksprumo]

While reviewing the reference impl for SIMD 16, one thing that would be good to think though is if we can make this format flexible to changes. Specifically adding/removing fields within AccountMeta.

Here's the specific part I'm referring to, where they would like to add an "application fees" field to the account, but that's hard with the current design: #30137 (comment)

[yhchiang-sol]

if we can make this format flexible to changes. Specifically adding/removing fields within AccountMeta.

Yes, the format handles optional fields by storing flags in the account meta where each bit represents whether this account has an optional field. And the optional field is stored in the data block.

[lurais]

@yhchiang-sol hello , is there a brief implementation of the proposal ? And is there a brief test about the implementation of the proposal? As far as I can see, there is only a format of the cold storage proposed now, is that right?

[CantelopePeel]

It's not entirely clear what happens to appendvec under this proposal. Does it get replaced?

[yhchiang-sol]

@yhchiang-sol hello , is there a brief implementation of the proposal ? And is there a brief test about the implementation of the proposal?

Hello, @lurais. The brief sketch is in #28790. It is still WIP as the current get_account API is still tied to hot storage as it requires &'a for its return value.

As far as I can see, there is only a format of the cold storage proposed now, is that right?

I am currently extending the format to allow it to support hot storage as well. The key idea is to use aligned & uncompressed data block for hot-storage, and have a footer field to describe the data block format (in the current doc, the compression_algorithm field will be changed to data_block_format.

I will share more details and update the doc once the idea is more polished.

It's not entirely clear what happens to appendvec under this proposal. Does it get replaced?

@CantelopePeel : If the new format handles hot-storage well (will update the proposal), appendvec will be replaced eventually.
During the transition period, appendvec will be served as the hot-storage.

[yhchiang-sol]

Revised and published a newer version of the proposal under SIMD: solana-foundation/solana-improvement-documents#37

The new version includes both hot- and cold- storage formats.

[yhchiang-sol]

Moved the proposal back to solana/docs/src/proposal. Here's the V3 which includes both cold and hot storage: #30315