HDDS-13847. Introduce Snapshot Content Lock to lock table contents

[swamirishi]

What changes were proposed in this pull request?

Introduce Snapshot Content Lock to prevent writes on a snapshot during snapshot defrag. Even though GC lock would be acquired during defrag on leader the contents in the snapshot is only guaranteed to be same for leader however defrag on follower can happen at any time, thus a lock should be acquired while the defrag to prevent the contents from changing in the snapshot which would mainly happen in 3 transactions during OM double buffer flush on followers i.e. 
OMSnapshotMoveTableKeysResponse, OMKeyPurgeResponse, OMDirectoriesPurgeResponseWithFSO.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13847

How was this patch tested?

Additional unit tests added.

[swamirishi]

@SaketaChalamchala can you review this patch?

[smengcl]

do we only use read lock on SNAPSHOT_DB_CONTENT_LOCK?

[swamirishi]

We would acquire a write lock when we want to do a defrag

[swamirishi]

Acquire write GC_LOCK
Snapshot is flushed

   take a checkpoint on tmp path
   compact if first snapshot(kforce Compaction)
   update keyTable, fileTable, directory()
   Acquire write SNAPSHOT_DB_CONTENT_LOCK
    truncate and ingest other tables(dumpToSst and ingestSstFile)
    delete if new prodPath if exists and mv tmp to production version snapshotDB path
    bump version snapshotLocalDataYaml -> new versionPath
    SNapshot cache lock on snapshotId(previosu rocksdb instance)
        delete of old version of snapshot path
    release snapshot cache lock on snapshotId
   releaseWrite content Lock
Release gc lock

[Copilot]

Pull Request Overview

This PR introduces a new lock type SNAPSHOT_DB_CONTENT_LOCK to protect snapshot RocksDB contents during operations that read or modify snapshot data. The changes add proper locking around snapshot database access during key purge and snapshot move operations.

Key changes:

• Added a new SNAPSHOT_DB_CONTENT_LOCK flat resource for locking snapshot database contents
• Enhanced MultiSnapshotLocks to accept an initial capacity parameter for better memory efficiency
• Added lock acquisition/release around snapshot database operations in purge and move responses
• Added test coverage to verify lock acquisition behavior

Reviewed Changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
FlatResource.java	Adds new SNAPSHOT_DB_CONTENT_LOCK enum value
MultiSnapshotLocks.java	Adds constructor parameter for initial list capacity optimization
OMSnapshotMoveTableKeysResponse.java	Acquires and releases snapshot DB content locks around snapshot operations
OMKeyPurgeResponse.java	Acquires and releases snapshot DB content locks during key purge
OMDirectoriesPurgeResponseWithFSO.java	Acquires and releases snapshot DB content locks during directory purge
ReclaimableFilter.java	Passes capacity hint to MultiSnapshotLocks constructor
SnapshotDeletingService.java	Passes capacity hint to MultiSnapshotLocks constructor
SnapshotDefragService.java	Passes capacity hint to MultiSnapshotLocks constructor
TestSnapshotRequestAndResponse.java	Wraps metadata manager with spy, adds spy import, removes trailing blank line
TestOMSnapshotMoveTableKeysResponse.java	Adds test coverage for lock acquisition verification
TestOMKeyRequest.java	Removes trailing blank line
TestOMKeyPurgeRequestAndResponse.java	Adds test coverage for lock acquisition verification
TestOMDirectoriesPurgeRequestAndResponse.java	Adds test coverage for lock acquisition verification

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[swamirishi]

thank you @smengcl for reviewing the patch