HDDS-13003. [POC] Snapshot Defragmentation to reduce storage footprint#8954

Closed

smengcl wants to merge 33 commits intoapache:masterfrom

smengcl:HDDS-13003-Defrag-POC-rebased-1

Commits on Aug 19, 2025

HDDS-13003. [POC] Snapshot Defragmentation to reduce storage footprint
smengcl
committed
[split] Add new OM layout feature SNAPSHOT_DEFRAGMENTATION (doesn't do anything else for now. need to handle upgrade later. e.g. block defrag service until finalization is performed)
smengcl
committed
Make `RDBSstFileWriter` public to be used in defrag service.
smengcl
committed
Implement `delete(key)` in `RDBSstFileWriter`.
smengcl
committed
Add config `ozone.snapshot.defrag.service.timeout`, `ozone.snapshot.defrag.limit.per.task`, `ozone.snapshot.defrag.service.interval`.
smengcl
committed
Rename term: compact -> defrag
smengcl
committed
[split] HDDS-13009. Implement SnapshotDefragService POC.
smengcl
committed
[split] Implement manual trigger of snapshot defrag. e.g. `ozone admin om triggerSnapshotDefrag -host=om`
smengcl
committed
Add tests (for debugging)
smengcl
committed
[split] yaml: TODO: Add OM_SLD_DB_CHECKPOINT_DIR field
smengcl
committed
[drop][dev] Disable WAL archival
smengcl
committed
SDS: Write diff to defragged DB directly instead of writing to SST file with RDBSstFileWriter then ingest.
smengcl
committed
[dev] Disable WAL in write options for testing
smengcl
committed
[dev] Update native library handling in Ozone build and shell
smengcl
committed
[dev] Add debug log for failed System.loadLibrary call
smengcl
committed
Update default values in OmSnapshotLocalData constructor
smengcl
committed
[dev] testUpdateYaml to set fields in .yaml before triggering defrag
smengcl
committed
Remove TestSnapshotDefragService test class
smengcl
committed
Reorder constant definition in OMConfigKeys
smengcl
committed
[dev] Update compose/ozone volumes/profiles and config defaults for local dev testing
smengcl
committed
om: Use triggerSnapshotDefragOnce for snapshot defrag manual triggering
smengcl
committed
SDS: Do not fall back to full defrag on inc defrag error
smengcl
committed
SDS: Clean up, add TODOs
smengcl
committed
SDS: Fix extractKeyFromPath for FSO (hack).
smengcl
committed
Rename CHECKPOINT_STATE_DEFRAGED_DIR -> CHECKPOINT_STATE_DEFRAGGED_DIR
smengcl
committed
Update TestSnapshotDefragService2 for FSO buckets and config tuning
smengcl
committed
[split] freon RandomKeyGenerator to handle existing volumes and buckets
smengcl
committed
[dev] Update key comparison logic in SnapshotDiffManager to count modificationTime change as MODIFY instead of RENAME for zero-byte keys
smengcl
committed
Make RocksCheckpoint and related methods public
smengcl
committed
[split] Handle STAND_ALONE replication type in QuotaUtil (was printing a lot of warning in the new integration test)
smengcl
committed
[split] TODO that we should disable db.snapshots creation for scm and others
smengcl
committed
Remove TestOmSnapshotDefrag integration test
smengcl
committed

Commits on Aug 22, 2025

Update docker-compose.yaml
smengcl
authored