ethdb: enlarge the number of memory tables

[flywukong]

Description

In the current BSC validator and miner workflow, write stalls frequently occur during the commit or block synchronization processes. Specifically, operations like writeStateID or writing block data often experience write delays of several hundred milliseconds, which negatively impacts commit performance and causes miners to exceed the 750ms block production time.

Pebble allows configuring multiple memtables for buffering unflushed data，if too many memtables are blocked and are waiting to be flushed, subsequent writes will stall, especially when Pebble performs a background flush while compaction is also ongoing. By default, Pebble is configured with only 2 memtables, each potentially exceeding 1GB in size. This behavior can be mitigated by increasing the number of memtables, which allows background flushes to be performed more smoothly and efficiently in a queue-like manner, thereby reducing the likelihood of write stalls under heavy load.
After shortening the BSC block interval, more frequent writes and compactions significantly increase the likelihood of write stalls. Adjusting the number of Pebble memtables is necessary to mitigate this issue.

Rationale

Improve the performance of commit db and reduce write stall. Testing shows that after tuning the Pebble parameters, write stalls were reduced by approximately 75%, and commit performance at the p99 percentile improved by around 30%.

Example

add an example CLI or API response...

Changes

Notable changes:

• add each change in a bullet point here
• ...

[Copilot]

Pull Request Overview

This PR increases Pebble's in-memory table count from 2 to 6 to smooth out flush operations and reduce write stalls under heavy load.

• Updated default memTableLimit from 2 to 6
• Adjusted comments to reflect the new memtable configuration

Comments suppressed due to low confidence (2)

ethdb/pebble/pebble.go:174

• [nitpick] Consider exposing memTableLimit as a configuration parameter (e.g., via CLI flag or config file) instead of hardcoding it, so users can tune this value without modifying source code.

memTableLimit := 6

ethdb/pebble/pebble.go:174

• Add or extend unit tests to cover scenarios with different memTableLimit values to ensure buffer sizing and flush logic remains correct and performant.

memTableLimit := 6