(--rocksdb.block-cache-size, env ETHREX_ROCKSDB_BLOCK_CACHE_SIZE) with a default
of 20 GiB. Because the previous commit moved index and bloom-filter blocks into
the bounded block cache, the cache size now governs total RocksDB resident memory
and significantly influences block-import throughput. Measured on a synced mainnet
node: at a 4 GiB cache, filter blocks monopolize the cache and block exec is ~76%
slower than the unbounded baseline; at 20 GiB the cache comfortably holds the
filter + index working set plus the EVM's hot data and exec is at parity. The
help text spells the trade-off out explicitly and only recommends lowering it on
resource-constrained hosts.
Plumbed through a new StoreConfig struct (exposed from ethrex-storage) and
Store::new_with_config / new_from_genesis_with_config /
{init,load,open}_store_with_config variants. The existing zero-config
constructors continue to use the default and remain unchanged for tests and
tools, so callers that don't need to override the cache size are unaffected.
Motivation
ethrex's RocksDB backend ties resident memory to database size with no upper bound: as the on-disk state grows, so does the in-heap footprint, with no ceiling. On any long-running node this presents as resident memory that climbs indefinitely — operationally indistinguishable from a memory leak — and on a large enough database it will eventually exhaust the host. The mechanism behind this (RocksDB keeping all SST files' index and filter blocks pinned in heap, outside its bounded LRU) is detailed below.
Description
This PR ships in two commits.
1. Store index and filter blocks in the shared block cache (
67f3492f).Enables
cache_index_and_filter_blocks(true)+pin_l0_filter_and_index_blocks_in_cache(true)on every column family. With this change, RocksDB stops pinning every open SST's index and bloom-filter blocks in heap and instead routes them through its shared LRU cache. Total RocksDB resident memory now tracks the block cache size, not the database size.2. Expose the block cache size as a CLI option (
32ffd479).Adds
--rocksdb.block-cache-size <BYTES>(envETHREX_ROCKSDB_BLOCK_CACHE_SIZE), default 20 GiB. Plumbed through a newStoreConfigstruct and*_with_configconstructor variants onStoreand theinit_store/load_store/open_storehelpers; the existing zero-config constructors keep working with the default and are unchanged for tests, tools, and L2 callers.The cache size now governs the memory vs. block-import-throughput trade-off: filter and index blocks share the cache with data blocks, so a cache that is too small to hold the filter + index working set plus a useful amount of hot data will stall execution. The CLI help text states this explicitly and warns against lowering the value below the default.
Validation (live on mainnet, 60-block window of head-following, same chain segment)
A jemalloc heap profile of the unfixed baseline attributed ~92% of resident memory to RocksDB, dominated by ~8 GB of index and bloom-filter blocks (~6 GB of which are bloom filters). With the fix applied, the corresponding
PrefetchIndexAndFilterBlocksallocations drop from ~8 GB to under 1 GB — the rest is now demand-loaded into the bounded cache viaGetOrReadFilterBlock.At the 20 GiB default, block-import is at parity with the unfixed baseline and resident memory is bounded forever regardless of database growth.
Trade-off worth noting
At today's ~500 GB mainnet database the default 20 GiB cache uses more memory than the unfixed baseline (~27 GB vs ~16 GB). The value of the fix is bounded memory forever — the unfixed baseline keeps climbing as the database grows (state DBs only grow); the crossover lands around a ~1 TB database. Operators who need a lower ceiling at the cost of throughput can lower the cache size; the help text documents this.