HDDS-13003. [Design Doc] Snapshot Defragmentation to reduce storage footprint #8514
Conversation
Change-Id: Ieb6a1145c732ffbbbc6811565734a78bd12e30ef
jojochuang
added
documentation
There was a problem hiding this comment.
Pull Request Overview
This PR introduces a design document explaining the snapshot compaction strategy in Apache Ozone to reduce the storage footprint of snapshots.
- Introduces new metadata fields (e.g., needsCompaction, lastCompactionTime, sstFiles) for managing snapshot compaction.
- Details the proposed workflow for compaction including directory restructuring and diff computation.
Comments suppressed due to low confidence (1)
hadoop-hdds/docs/content/feature/SnapshotCompaction.md:20
- The type 'List' appears to be inconsistent; if the intention is to store a list of Long values, consider using 'List' for clarity and consistency.
A new list of Map<String, List<Longs>> (`sstFiles`) also needs to be added to snapshot info; this would be storing the original list of sst files in the uncompacted copy of the snapshot corresponding to keyTable/fileTable/DirectoryTable.
jojochuang
left a comment
jojochuang
left a comment
There was a problem hiding this comment.
The Markdown is rendered from the design doc attached to the jira: https://docs.google.com/document/d/1Xw1AtKAlDm97UiLXd8egjeLIaYq4rpClv1xD7x5Xvww/edit?usp=sharing
jojochuang
left a comment
jojochuang
left a comment
There was a problem hiding this comment.
Please check if the following workflow chart look correct
Change-Id: I29d688c22cdf29aa55cf2bbce6b750720887f6a0
Change-Id: I9761006b9b9697f8392aab68c01b400793996d06
|
|
||
| 3. ### Directory Structure Changes | ||
|
|
||
| Snapshots currently reside in the `db.checkpoints` directory. The proposal introduces a `db.checkpoints.compacted` directory for compacted snapshots. The directory format should be as follows: |
There was a problem hiding this comment.
clarification
| Snapshots currently reside in the `db.checkpoints` directory. The proposal introduces a `db.checkpoints.compacted` directory for compacted snapshots. The directory format should be as follows: | |
| Snapshots currently reside in the `db.checkpoints` directory. The proposal introduces a `db.checkpoints.compacted` directory for compacted snapshots. And the OM DB checkpoint directory name format inside the `db.checkpoints.compacted` directory should be as follows: |
There was a problem hiding this comment.
Actually, snapshots are under db.snapshots. e.g.:
/var/lib/hadoop-ozone/om/data/db.snapshots
/var/lib/hadoop-ozone/om/data/db.snapshots/diffState
/var/lib/hadoop-ozone/om/data/db.snapshots/diffState/compaction-sst-backup
/var/lib/hadoop-ozone/om/data/db.snapshots/diffState/snapDiff
/var/lib/hadoop-ozone/om/data/db.snapshots/diffState/compaction-log
/var/lib/hadoop-ozone/om/data/db.snapshots/checkpointState
Newly created FS snapshots would be under ./db.snapshots/checkpointState
|
|
||
| 5. ### Snapshot Compaction Workflow | ||
|
|
||
| Snapshot compaction should only occur once the snapshot has undergone SST filtering. The following steps outline the process: |
There was a problem hiding this comment.
Thanks @swamirishi for working on this. Does this mean the SstFilteringService would run it?
|
I suggest adding a section that summarises how each table is handled during compaction. The information is scattered now, grouping it together might make the doc easier to read. |
| 2. **Flush changed objects** into separate SST files using the SST file writer, categorizing them by table type. | ||
| 3. **Ingest these SST files** into the RocksDB checkpoint using the `ingestFile` API. | ||
| 3. Truncate `deletedTable,deletedDirectoryTable,snapshotRenamedTable etc. (All tables excepting keyTable/fileTable/directoryTable)` in checkpointed rocksdb and ingest the entire table from deletedTable and deletedDirectoryTable from the current snapshot rocksdb. | ||
| 4. **Acquire the snapshot cache lock** to prevent snapshot access during directory updates. |
thanks @chungen0126 for reviewing the doc. I will update the doc shortly capturing this info |
jojochuang
left a comment
jojochuang
left a comment
There was a problem hiding this comment.
What conditions trigger a Snapshot Compaction? For sure not when there's no snapshots. But does it get triggered when there's one snpahsot? Or a predefined threshold of snapshots / diffs between snapshots?
|
One potential race condition would be when the snapshot compaction process is ongoing, a new OM node bootstraps and tries to download the DBs. In such case I believe we had existing lock in-place to prevent that but we need to double check. |
smengcl
changed the title
Co-authored-by: Wei-Chiu Chuang <[email protected]>
|
I am going to push an update this design doc. Mostly about the change of terms. |
|
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days. |
|
Thank you for your contribution. This PR is being closed due to inactivity. If needed, feel free to reopen it. |
4 participants
What changes were proposed in this pull request?
In Apache Ozone, snapshots currently take a checkpoint of the Active Object Store (AOS) RocksDB each time a snapshot is created and track the compaction of SST files over time. This model works efficiently when snapshots are short-lived, as they merely serve as hard links to the AOS RocksDB. However, over time, if an older snapshot persists while significant churn occurs in the AOS RocksDB (due to compactions and writes), the snapshot RocksDB may diverge significantly from both the AOS RocksDB and other snapshot RocksDB instances. This divergence increases storage requirements linearly with the number of snapshots.
The primary inefficiency in the current snapshotting mechanism stems from constant RocksDB compactions in AOS, which can cause a key, file, or directory entry to appear in multiple SST files. Ideally, each unique key, file, or directory entry should reside in only one SST file, eliminating redundant storage and mitigating the multiplier effect caused by snapshots. If implemented correctly, the total RocksDB size would be proportional to the total number of unique keys in the system rather than the number of snapshots.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13003
How was this patch tested?
No tests required