Hierarchical state diffs in hot DB

[dapplion]

Proposed Changes

This PR implements #5978 (tree-states) but on the hot DB. It allows Lighthouse to massively reduce its disk footprint during non-finality and overall I/O in all cases.

Closes #6580

Conga into #6744

TODOs

• Fix OOM in CI Fix test OOM issues on tree-states-hot #7176
• optimise store_hot_state to avoid storing a duplicate state if the summary already exists (should be safe from races now that pruning is cleaner)
• mispelled: get_ancenstor_state_root
• get_ancestor_state_root should use state summaries
• Prevent split from changing during ancestor calc
• Use same hierarchy for hot and cold

TODO Good optimization for future PRs

• On the migration, if the latest hot snapshot is aligned with the cold snapshot migrate the diffs instead of the full states.

align slot  time
10485760    Nov-26-2024
12582912    Sep-14-2025
14680064    Jul-02-2026

TODO Maybe things good to have

• Rename anchor_slot https://github.com/sigp/lighthouse/compare/tree-states-hot-rebase-oom...dapplion:lighthouse:tree-states-hot-anchor-slot-rename?expand=1
• Make anchor fields not public such that they must be mutated through a method. To prevent un-wanted changes of the anchor_slot

NOTTODO

• Use fork-choice and a new method descendants_of_checkpoint to filter only the state summaries that descend of finalized checkpoint]

Additional Info

Original WIP PR with lots of context and discussion

• HDiff in the hot DB dapplion/lighthouse#39

Some resources to get up to speed with HDiff concepts:

• https://github.com/dapplion/tree-states-review-guide/blob/main/persisted_hdiff.md
• https://hackmd.io/G82DNSdvR5Osw2kg565lBA

[chong-he]

I have done some testing on this on the Holesky testnet as mentioned in #6775. Regular checkpoint sync works, so does syncing from a checkpoint >24 hours ago. Some findings:

1. Seeing the following logs when syncing an archive node:

Jan 26 08:04:01.957 ERRO Database write failed, action: reverting blob DB changes, error: MissingHotStateSummary(0x554bd524427dfc0a0efcabbdc65f087dcd053385fd378cc83aa78a5d9b9db167), service: freezer_db, module: store::hot_cold_store:1390
Jan 26 08:04:01.957 ERRO Database write failed!, error: MissingHotStateSummary(0x554bd524427dfc0a0efcabbdc65f087dcd053385fd378cc83aa78a5d9b9db167), msg: Restoring fork choice from disk, service: beacon, module: beacon_chain::beacon_chain:3957

(The first line of this is until slot 3498015, I truncated it)
Shortly after this, the following appears:

Jan 26 08:04:03.946 CRIT Beacon block processing error, error: DBError(MissingHotStateSummary(0x554bd524427dfc0a0efcabbdc65f087dcd053385fd378cc83aa78a5d9b9db167)), service: beacon, module: beacon_chain::beacon_chain:3377
Jan 26 08:05:49.160 CRIT Beacon block processing error, error: MissingBeaconBlock(0x831d25d70b5bdfe7544c0cac196cb781fcea3e9c7c5d6e902ad82059d27fc8ed), service: beacon, module: beacon_chain::beacon_chain:3377
Jan 26 08:12:17.256 WARN BlockProcessingFailure, outcome: MissingBeaconBlock(0x831d25d70b5bdfe7544c0cac196cb781fcea3e9c7c5d6e902ad82059d27fc8ed), msg: unexpected condition in processing block., module: network::network_beacon_processor::sync_methods:747

The beacon node wouldn't be able to start upon a restart:

Jan 27 05:57:58.731 CRIT Failed to start beacon node, reason: Failed to build beacon chain: Head block not found in store, module: lighthouse:721

Another point worth noting is that the logs seem to be in a loop. Similar logs appear repeatedly in the log file, to the extent that it would only take a few seconds/minutes to fill up a 200MB log file. These are the repeated logs:

Jan 26 08:02:47.632 DEBG Created block lookup, id: 1753067, awaiting_parent: none, block_root: 0x523bff6d855c77b4f14ed5d7639a3f4975cef5115e0f0631f2f730665229fbc6, peer_ids: [PeerId("16Uiu2HAm4qmfW7E7E9rkvNVAWeZNWMhZ4VefKJHhjtFDUKaJAGAb"), PeerId("16Uiu2HAkvWFRpCLuiRyxvse9JzXP5HjgTMJfwPcHUCCa5NGwaike"),  PeerId("16Uiu2HAm9pjmagyJFvMkn3bVDPWTVwgyLJ3QdsejndLJnRhz2wfZ")], service: lookup_sync, service: sync, module: network::sync::block_lookups:401
Jan 26 08:02:47.632 DEBG Dropping completed lookup, id: 1753067, block: 0x523bff6d855c77b4f14ed5d7639a3f4975cef5115e0f0631f2f730665229fbc6, service: lookup_sync, service: sync, module: network::sync::block_lookups:781
Jan 26 08:02:47.632 DEBG Continuing child lookup, block_root: 0xf8eb5af88a4ea962e91a6fa77c69e1efa17005c346e7921d4218d0e84ff5a149, id: 113701, parent_root: 0x523bff6d855c77b4f14ed5d7639a3f4975cef5115e0f0631f2f730665229fbc6, service: lookup_sync, service: sync, module: network::sync::block_lookups:733
Jan 26 08:02:47.632 DEBG Sending blobs for processing, id: 113701, block: 0xf8eb5af88a4ea962e91a6fa77c69e1efa17005c346e7921d4218d0e84ff5a149, service: sync, module: network::sync::network_context:1086
Jan 26 08:02:47.632 DEBG RPC blobs received, commitments: [0xb8b1…3151, 0xb0a9…8f7d, 0xb29d…876a], slot: 3497985, block_root: 0xf8eb5af88a4ea962e91a6fa77c69e1efa17005c346e7921d4218d0e84ff5a149, indices: [0, 1, 2], module: network::network_beacon_processor::sync_methods:267
Jan 26 08:02:47.632 WARN Error when importing rpc blobs, slot: 3497985, block_hash: 0xf8eb5af88a4ea962e91a6fa77c69e1efa17005c346e7921d4218d0e84ff5a149, error: ParentUnknown { parent_root: 0x523bff6d855c77b4f14ed5d7639a3f4975cef5115e0f0631f2f730665229fbc6 }, module: network::network_beacon_processor::sync_methods:318
Jan 26 08:02:47.632 DEBG Received lookup processing result, result: Err(ParentUnknown { parent_root: 0x523bff6d855c77b4f14ed5d7639a3f4975cef5115e0f0631f2f730665229fbc6 }), id: 113701, block_root: 0xf8eb5af88a4ea962e91a6fa77c69e1efa17005c346e7921d4218d0e84ff5a149, component: Blob, service: lookup_sync, service: sync, module: network::sync::block_lookups:552
Jan 26 08:02:47.632 DEBG Marking lookup as awaiting parent, parent_root: 0x523bff6d855c77b4f14ed5d7639a3f4975cef5115e0f0631f2f730665229fbc6, block_root: 0xf8eb5af88a4ea962e91a6fa77c69e1efa17005c346e7921d4218d0e84ff5a149, id: 113701,

(The first line of the above log is truncated)

2. On another try of syncing an archive node:

Jan 22 01:27:34.393 ERRO Database write failed, action: reverting blob DB changes, error: MissingHotStateSummary(0x21262323cc023403605249a57849c267debad4612d7c340523fab29ded71bdb5), service: freezer_db, module: store::hot_cold_store:1390
Jan 22 01:27:34.393 ERRO Database write failed!, error: MissingHotStateSummary(0x21262323cc023403605249a57849c267debad4612d7c340523fab29ded71bdb5), msg: Restoring fork choice from disk, service: beacon, module: beacon_chain::beacon_chain:3957
Jan 22 01:28:51.854 WARN Hot DB pruning failed, error: PruningError(MissingSummaryForFinalizedCheckpoint(0xd395c3f4e6d3eafd733a5554b3191e20f670c444916d868d68157aa07f0887fb)), service: beacon, module: beacon_chain::migrate:381

The beacon node wouldn't be able to start upon a restart:

Jan 22 03:55:27.130 INFO Blob DB initialized                     oldest_data_column_slot: None, oldest_blob_slot: Some(Slot(3336224)), path: "/var/lib/lighthouse_test1/beacon/blobs_db", service: freezer_db
Jan 22 03:55:28.186 CRIT Failed to start beacon node             reason: Unable to open database: HotColdDBError(MissingSplitState(0x855420d7ff98dca1dbe028ef624b18a81a027ab0cb1f95e4c048ad9b3c4510c4, Slot(3467232)))
Jan 22 03:55:28.186 INFO Internal shutdown received              reason: Failed to start beacon node

3. I have not successfully reconstructed all states (i.e., finished syncing an archived node) on this branch (due to the above errors). It is either stuck during historical blocks download or during state reconstruction. A similar error to no. 2 occurs regularly (during state reconstruction):

Jan 28 07:44:58.442 ERRO Database write failed                   action: reverting blob DB changes, error: Hdiff(LessThanStart(Slot(3504672), Slot(3504704))), service: freezer_db
Jan 28 07:44:58.443 WARN Hot DB pruning failed                   error: DBError(HotColdDBError(Rollback)), service: beacon

(slot 3504704 is the anchor slot from the database info)

4. Upgrading from v6.0.1:
   Checkpoint sync using v6.0.1, stop Lighthouse once it has checkpoint synced, then change the binary file to this PR branch. Sometimes it can upgrade smoothly, but sometimes it has error:

Jan 28 08:49:27.224 INFO Blob DB initialized                     oldest_data_column_slot: None, oldest_blob_slot: Some(Slot(3497888)), path: "/home/ck/.lighthouse/holesky/beacon/blobs_db", service: freezer_db
Jan 28 08:49:27.556 CRIT Failed to start beacon node             reason: Unable to open database: MigrationError("Migrating from SchemaVersion(22) to SchemaVersion(23): MigrationError(\"error computing states summaries dag MissingStateSummary(0x9d6efd378136fd8a08574b9a7f11b3f04ff5c5a4c15c0eaa64eac0c82f2aadee)\")")
Jan 28 08:49:27.557 INFO Internal shutdown received              reason: Failed to start beacon node

If this error occurs and I change back to v6.0.1 and let it syncs for a few minutes, and then change back again to this binary PR, it could work (upgrade is successful) and the database schema shows V23:

Jan 28 08:54:08.619 INFO Blob DB initialized                     oldest_data_column_slot: None, oldest_blob_slot: Some(Slot(3497504)), path: "/home/ck/.lighthouse/holesky/beacon/blobs_db", service: freezer_db
Jan 28 08:54:14.692 INFO Hot states migration in progress        summaries_written: 33, diff_written: 2
Jan 28 08:54:17.910 INFO Hot states migration complete           summaries_written: 99, diff_written: 3
Jan 28 08:54:20.345 INFO Refusing to checkpoint sync             msg: database already exists, use --purge-db to force checkpoint sync, service: beacon

However, I notice the anchor_slot is changed after upgrading to V23 (e.g., advancing by 1 epoch). Is this expected?

[mergify]

This PR is amazing @dapplion, therefore you don't have to fix the conflicts yet. Please take your time and enjoy the day 💮 ❤️

[michaelsproul]

Let's merge 😁