perf: add fast MVCC snapshots for MemDb in FlatDb tests

[asdacap]

Summary

Replaces slow O(n) full-copy snapshots with fast O(1) MVCC-based snapshots in FlatDb test infrastructure.

Changes

New Implementation

• SnapshotableMemDb: MVCC-based in-memory database with O(1) snapshot creation

  • Uses SortedDictionary<byte[], VersionedEntry> with version tracking
  • Implements IKeyValueStoreWithSnapshot and ISortedKeyValueStore
  • Automatic version garbage collection when snapshots are disposed
  • Thread-safe with lock-based concurrency control

• SnapshotableMemColumnsDb: Column database wrapper providing atomic snapshots across all columns

  • Uses SnapshotableMemDb per column
  • Snapshots capture consistent view across all columns

Test Infrastructure Migration

• Updated PseudoNethermindModule to register SnapshotableMemColumnsDb for FlatDb tests
• Fixed FlatTrieVerifierTests helper methods to use IDb interface

Performance Impact

Before: FlatDb test snapshots performed full database copy (O(n))
After: Snapshots are O(1) version captures

This significantly improves test performance when FlatDb creates frequent snapshots during test execution.

Testing

• ✅ 21 SnapshotableMemDb unit tests (isolation, concurrency, sorted iteration, GC)
• ✅ 11 SnapshotableMemColumnsDb unit tests (atomic snapshots, write batches)
• ✅ 22 FlatTrieVerifierTests (all passing with new implementation)

Type of change

• Performance improvement
• Refactoring (test infrastructure improvement)
• New tests

Documentation/Changelog

• Readme/Documentation was updated
• Changelog was updated

[Copilot]

Pull request overview

This PR introduces MVCC-style in-memory DB implementations to speed up FlatDb-related tests by making snapshot creation O(1), then wires the new DB into FlatDb test infrastructure.

Changes:

• Add SnapshotableMemDb (MVCC snapshot-capable in-memory DB) and SnapshotableMemColumnsDb (multi-column wrapper).
• Migrate FlatDb test DI and FlatTrieVerifierTests to use the new snapshot-capable column DB.
• Add unit tests for the new DBs and update spelling dictionary for new terms.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
src/Nethermind/Nethermind.State.Flat.Test/FlatTrieVerifierTests.cs	Switch FlatDb test setup from TestMemColumnsDb to SnapshotableMemColumnsDb and remove concrete DB casts.
src/Nethermind/Nethermind.Db/SnapshotableMemDb.cs	New MVCC-based in-memory DB with snapshot + sorted view support and version pruning.
src/Nethermind/Nethermind.Db/SnapshotableMemColumnsDb.cs	New in-memory columns DB that creates per-column snapshots.
src/Nethermind/Nethermind.Db.Test/SnapshotableMemDbTests.cs	New unit tests for MVCC DB behavior (including a currently ignored pruning regression).
src/Nethermind/Nethermind.Db.Test/SnapshotableMemColumnsDbTests.cs	New unit tests for columns DB snapshot behavior and write batches.
src/Nethermind/Nethermind.Core.Test/Modules/PseudoNethermindModule.cs	Register SnapshotableMemColumnsDb for FlatDb tests (with pruning disabled).
cspell.json	Add “MVCC” and “Snapshotable” to spelling allowlist.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

This test is newly introduced but marked [Ignore] due to a known pruning bug. Shipping a new ignored test usually means the PR knowingly introduces incomplete/incorrect behavior; it also leaves the broken pruning path untested. Prefer fixing the pruning logic in SnapshotableMemDb and enabling this test, or remove the pruning feature until it’s correct.

[Copilot]

FlatTrieVerifierTests constructs SnapshotableMemColumnsDb with default settings (pruning enabled), but SnapshotableMemDb currently has a known pruning bug (see ignored test in SnapshotableMemDbTests). If these persistence paths rely on snapshots, this risks intermittent wrong reads depending on snapshot disposal order. Either fix pruning in SnapshotableMemDb, or construct the test DB with neverPrune: true until pruning is correct.

Suggested change

	_columnsDb = new SnapshotableMemColumnsDb<FlatDbColumns>();
	_columnsDb = new SnapshotableMemColumnsDb<FlatDbColumns>(neverPrune: true);

[Copilot]

Registering SnapshotableMemColumnsDb with neverPrune: true disables version GC entirely. Given FlatDb creates many snapshots, this can cause unbounded memory growth during longer test runs. Once pruning is fixed, prefer enabling pruning (or at least adding a bounded GC strategy) so tests don’t risk OOMs.

Suggested change

	.AddSingleton<IColumnsDb<FlatDbColumns>>((_) => new SnapshotableMemColumnsDb<FlatDbColumns>(neverPrune: true))
	.AddSingleton<IColumnsDb<FlatDbColumns>>((_) => new SnapshotableMemColumnsDb<FlatDbColumns>(neverPrune: false))

[Copilot]

GetValueAtVersion enumerates _db.Reverse() which is LINQ Enumerable.Reverse() over a SortedDictionary. That buffers the entire dictionary on every read and then scans linearly, making reads O(n) with per-call allocations (and will dominate performance in FlatDb tests). Consider a data structure that supports floor queries without full scans (e.g., per-key version list + binary search, or a composite key approach with a lookup to the last <= version using custom range search).

Suggested change

	// Find the latest entry for this key with version <= requested version
	// Since entries are sorted by (key, version), we can iterate backwards from the upper bound
	// Get all entries that could match: same key with version <= requested version
	// We need to find the entry with the highest version <= requested version
	foreach (var kvp in _db.Reverse())
	{
	(byte[] entryKey, int entryVersion) = kvp.Key;
	// If we've gone past this key entirely, stop
	if (entryKey.AsSpan().SequenceCompareTo(key) < 0)
	{
	break;
	}
	// If this entry is for our key and version is <= requested
	if (entryKey.AsSpan().SequenceCompareTo(key) == 0 && entryVersion <= version)
	{
	return kvp.Value; // This could be null (tombstone)
	}
	}
	return null;
	// Find the latest entry for this key with version <= requested version.
	// Entries are sorted by (key, version) in ascending order.
	byte[]? result = null;
	int bestVersion = int.MinValue;
	foreach (var kvp in _db)
	{
	(byte[] entryKey, int entryVersion) = kvp.Key;
	int cmp = entryKey.AsSpan().SequenceCompareTo(key);
	// Skip entries for keys that are less than the requested key.
	if (cmp < 0)
	{
	continue;
	}
	// Once we pass the requested key, no further entries can match.
	if (cmp > 0)
	{
	break;
	}
	// Same key: track the highest version <= requested version.
	if (entryVersion <= version && entryVersion > bestVersion)
	{
	bestVersion = entryVersion;
	result = kvp.Value; // This could be null (tombstone)
	}
	}
	return result;

[Copilot]

SnapshotableMemColumnsDb claims/aims to provide consistent snapshots across columns, but CreateSnapshot takes independent per-column snapshots without any shared version/lock. With concurrent writes across columns, the returned IColumnDbSnapshot can observe different points-in-time per column (unlike RocksDb ColumnsDb which uses a single underlying snapshot). If atomicity is required, consider sharing a global version/lock across all column stores, or redesign so writes go through a wrapper that can coordinate snapshot boundaries.

[Copilot]

_columnDbs is a plain Dictionary accessed without synchronization in GetColumnDb/ColumnKeys/CreateSnapshot/Dispose/Flush. Since this type is used in tests marked parallelizable and the PR description mentions thread-safety, this can race if columns are accessed from multiple threads. Consider using a lock (or ConcurrentDictionary) around dictionary mutations/iteration and ensure CreateSnapshot enumerates a stable snapshot of the columns.

[asdacap]

Oit! Reverse is slow.

[asdacap]

WTF

[asdacap]

WTF, you are suppose to iterate the sorted dictionary!

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

With neverPrune=false, versions are only garbage-collected from _db when a snapshot is disposed (via OnSnapshotDisposed). If the store is used without ever creating snapshots, every Set/Remove keeps accumulating historical versions indefinitely, causing unbounded memory growth.

Consider pruning on write when _activeSnapshotVersions.Count == 0 (e.g., remove previous versions for the same key immediately), so the non-snapshot case behaves like a normal MemDb and doesn't leak memory.

Suggested change

	_currentVersion++;
	_currentVersion++;
	if (!_neverPrune && _activeSnapshotVersions.Count == 0)
	{
	// No active snapshots: we can safely drop all previous versions for this key
	List<(byte[] Key, int Version)> versionsToRemove = new();
	foreach ((byte[] existingKey, int existingVersion) in _db.Keys)
	{
	if (existingVersion < _currentVersion && existingKey.AsSpan().SequenceEqual(keyArray))
	{
	versionsToRemove.Add((existingKey, existingVersion));
	}
	}
	foreach ((byte[] removeKey, int removeVersion) in versionsToRemove)
	{
	_db.Remove((removeKey, removeVersion));
	}
	}

[Copilot]

GetValueAtVersion linearly scans _db from the beginning for every Get/KeyExists, making reads O(total entries up to the key) rather than O(log n). With MVCC this can become very expensive as versions accumulate.

Consider changing the data structure to index by key first (e.g., SortedDictionary<byte[], VersionChain> where VersionChain supports binary search for the latest version <= requested) so point lookups are logarithmic and don't require full scans.

[Copilot]

MemDbSortedView.MoveNext() (and StartBefore) re-scan the entire _db._db from the beginning on every call. This makes iterating a range O(n²) for n keys and can negate the intended performance improvements when FlatDb performs long co-iterations.

Consider implementing ISortedView with a persistent enumerator / iterator state (or pre-materialize the range once) so each MoveNext() is amortized O(1) within the chosen range.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

GetAll() holds _versionLock while yielding results. Because this is an iterator block, the lock remains held for the entire enumeration, which can block writers/readers and can deadlock if the caller iterates and calls back into the DB. Prefer materializing the results (or keys) under the lock and returning an already-built collection/array so the lock is not held across consumer code (same pattern applies to GetAllKeys/GetAllValues/GetAllAtVersion).

Suggested change

	lock (_versionLock)
	{
	foreach (byte[] key in GetAllUniqueKeys())
	{
	byte[]? value = GetValueAtVersion(key, _currentVersion);
	if (value is not null)
	{
	yield return new KeyValuePair<byte[], byte[]?>(key, value);
	}
	}
	}
	}
	public IEnumerable<byte[]> GetAllKeys(bool ordered = false)
	{
	lock (_versionLock)
	{
	foreach (byte[] key in GetAllUniqueKeys())
	{
	if (GetValueAtVersion(key, _currentVersion) is not null)
	{
	yield return key;
	}
	}
	}
	List<KeyValuePair<byte[], byte[]?>> result;
	lock (_versionLock)
	{
	result = new List<KeyValuePair<byte[], byte[]?>>();
	foreach (byte[] key in GetAllUniqueKeys())
	{
	byte[]? value = GetValueAtVersion(key, _currentVersion);
	if (value is not null)
	{
	result.Add(new KeyValuePair<byte[], byte[]?>(key, value));
	}
	}
	}
	foreach (KeyValuePair<byte[], byte[]?> item in result)
	{
	yield return item;
	}
	}
	public IEnumerable<byte[]> GetAllKeys(bool ordered = false)
	{
	List<byte[]> result;
	lock (_versionLock)
	{
	result = new List<byte[]>();
	foreach (byte[] key in GetAllUniqueKeys())
	{
	if (GetValueAtVersion(key, _currentVersion) is not null)
	{
	result.Add(key);
	}
	}
	}
	foreach (byte[] key in result)
	{
	yield return key;
	}

[Copilot]

MemDbSortedView.StartBefore() returns false when the computed range is empty (e.g., when key is before _firstKey). In existing ISortedView implementations in this repo (e.g., TestMemDb.FakeSortedView), StartBefore typically returns false only when the view itself is empty, and callers may interpret false as “no data”. Consider handling key < _firstKey by positioning to “before start” (so the next MoveNext yields the first element) and returning true when the view has any elements.

Suggested change

	_currentKey = null;
	_currentValue = null;
	return false;
	// key is before the first key in this view: position "before start"
	_currentKey = null;
	_currentValue = null;
	// Determine if there is any visible element in the view at this version.
	var fullLower = (_firstKey, 0, (byte[]?)null);
	var fullUpper = (_lastKey, 0, (byte[]?)null);
	var fullView = _db._db.GetViewBetween(fullLower, fullUpper);
	byte[]? bestKeyInRange = null;
	byte[]? bestValueInRange = null;
	byte[]? candidateKeyInRange = null;
	byte[]? candidateValueInRange = null;
	foreach (var entry in fullView)
	{
	if (candidateKeyInRange is not null && candidateKeyInRange.AsSpan().SequenceCompareTo(entry.Key) != 0)
	{
	if (candidateValueInRange is not null)
	{
	bestKeyInRange = candidateKeyInRange;
	bestValueInRange = candidateValueInRange;
	}
	candidateKeyInRange = null;
	candidateValueInRange = null;
	}
	if (entry.Version <= _version)
	{
	candidateKeyInRange = entry.Key;
	candidateValueInRange = entry.Value;
	}
	}
	if (candidateValueInRange is not null)
	{
	bestKeyInRange = candidateKeyInRange;
	bestValueInRange = candidateValueInRange;
	}
	// Return true if there is at least one visible element in the view.
	return bestKeyInRange is not null;

[benaadams]

Does it need locking? Also why IDictionary when is privately assigned Dictionary

[asdacap]

No it doesn't need locking, columns are initialized in the constructor and only read after. Changed IDictionary to Dictionary. Fixed in d39d3c3.

-- Claude (on behalf of asdacap)

[benaadams]

Suggested change

	private readonly object _versionLock = new();
	private readonly Lock _versionLock = new();

[asdacap]

Done, changed to Lock. Fixed in d39d3c3.

-- Claude (on behalf of asdacap)