fix(rocksdb): proper shard logic for history indices

[yongkangc]

Summary
Adds proper shard management for RocksDB history indices.

Changes

• Treat empty RocksDB tables as first-run scenario in invariants
• Handle sentinel-only entries in consistency checks
• Use proper shard logic for history indices in write_blocks_data

Testing
Invariant checks pass.

---

PR Stack

main
  ↑
joshie/par-save-blocks-rocksdb (base)
  ↑
#21063 (rocksdb-either-reader)
  ↑
#21064 (rocksdb-shard-logic) ◀
  ↑
#21065 (rocksdb-stage-utils)
  ↑
#21066 (rocksdb-index-history-stages)
  ↑
#21068 (rocksdb-init-txlookup)
  ↑
#21069 (rocksdb-cli-testutils)
  ↑
#21070 (rocksdb-cli)
  ↑
#21071 (rocksdb-docs)

• feat(storage): add EitherReader for routing history queries to MDBX or RocksDB #21063
• fix(rocksdb): proper shard logic for history indices #21064 ◀
• feat(stages): add RocksDB helper functions for stage operations #21065
• feat(stages): implement RocksDB unwind for index history stages #21066
• feat(storage): RocksDB support for genesis init and tx lookup stage #21068
• fix: update CLI commands and test utilities for RocksDB support #21069
• feat(cli): add --storage.rocksdb flag for RocksDB history indices #21070
• docs: add CLI documentation for --storage.rocksdb flag #21071

[gakonst]

Code Review: PR #21064

CI Status

• ❌ fmt failure - needs cargo fmt
• ❌ lint success failure (due to fmt)
• ⚠️ Merge conflicts detected

Correctness Concerns

🔴 Major: Write-path shard append race condition

append_account_history_shard() and append_storage_history_shard() read existing shards from the committed DB but writes are deferred into pending_batches. If save_blocks() can run multiple times before commit():

1. Call #1 appends shard → produces batch A (pending)
2. Call #2 appends shard → reads old committed shard (not batch A)
3. Call #2 overwrites the shard key(s) in batch B
4. Indices from batch A are lost

Recommendation: Either:

• Commit RocksDB history writes immediately in write_blocks_data (simplest)
• Use a transactional write context that supports read-your-writes
• Maintain an in-memory per-tx accumulator for open shard values

🟡 Medium: Missing "open shard" edge case

Both append functions assume the "open shard" key (highest_block_number == u64::MAX) exists or history is empty. If only closed shards exist (partial state), get(last_key) returns None and the append starts a fresh open shard, potentially creating overlaps.

Recommendation: Consider a prefix scan for the last shard if u64::MAX lookup fails, or add an invariant assertion.

🟡 Medium: Rechunk never deletes obsolete shards

Rechunking writes new closed shard keys plus a new u64::MAX shard, but never deletes pre-existing shard keys. This is probably fine if only the u64::MAX shard can grow and closed shards are immutable, but worth confirming this invariant.

Nits

1. Thread naming: RocksDB thread panic uses format!("rocksdb write thread {i} panicked"). Consider naming by component (tx_hash_numbers, account_history, storage_history) like StaticFile does for debuggability.

2. Import cleanup: There are some duplicated/unused imports in the diff. Worth a cleanup pass.

3. RocksDBBatch docs vs behavior: Docs say "does NOT support read-your-writes," but append_*_history_shard is fundamentally a read-modify-write operation. Add docs clarifying this works only if called once per commit cycle.

Summary

The read-path logic (compute_history_rank, prev-shard check) looks correct. The parallel write infrastructure for static files and RocksDB is well-designed. The main blocker is the write-path shard append correctness issue - please address before merge.

---

Review by @tempo_ai

[yongkangc]

Merged into #21063 to simplify the PR stack. The 3 shard logic commits have been cherry-picked into the EitherReader PR.