HDDS-12712. Design Document of "Aggressive DB Compaction with Minimal Degradation"

[peterxcli]

What changes were proposed in this pull request?

Short Introduction

Use the numEntries and numDeletion in TableProperties which stores statistics for each SST as "guidance" to determine how to split tables into finer ranges for compaction.

Motivation

Our current approach of compacting entire column families directly would significantly impact online performance through excessive write amplification. After researching TiKV and RocksDB compaction mechanisms, it's clear we need a more sophisticated solution that better balances maintenance operations with user workloads.

TiKV runs background tasks for compaction and logically splits key ranges into table regions (with default size limits of 256MB per region), allowing gradual scanning and compaction of known ranges. While we can use the built-in TableProperties in SST files to check metrics like num_entries and num_deletion, these only represent operation counts without deduplicating keys. TiKV addresses this with a custom MVCTablePropertiesCollector for more accurate results, but unfortunately, the Java API doesn't currently support custom collectors, forcing us to rely on built-in statistics.

For the Ozone Manager implementation, we face a different challenge since OM lacks the concept of size-based key range splits. The most logical division we can use is the bucket prefix (file table). For FSO buckets, we can further divide key ranges based on directory parent_id, enabling more granular and targeted compaction that minimizes disruption to ongoing operations.

By implementing bucket-level compaction with proper paging mechanisms like next_bucket and potentially next_parent_id for directory-related tables, we can achieve more efficient storage utilization while maintaining performance. The Java APIs currently provide enough support to implement these ideas, making this approach viable for Ozone Manager.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-12712

How was this patch tested?

no, only doc added.

[Copilot]

Pull Request Overview

This PR adds a design document outlining an "Aggressive DB Compaction with Minimal Degradation" approach. It details new configuration parameters, describes two types of compactors for different table layouts (OBS and FSO), and provides pseudocode examples to illustrate the proposed compaction strategies.

[adoroszlai]

Thanks @peterxcli for the doc. Design docs should be added in hadoop-hdds/docs/content/design/ (please check existing docs there for metadata to add in header).

[jojochuang]

Thanks! I had glanced over the design and on a high level makes sense to me.
I'm not that familiar with RocksDB and didn't realize RocksDB provides the APIs to gather info at this granularity. That's great!

[guohao-rosicky]

hi, perter @peterxcli thanks work on this, Refer to this code for numDeletions of the sstable.

    RocksDatabase rocksDB = ((RDBStore)getStore()).getDb();
    List<LiveFileMetaData> liveFileMetaDataList =
        rocksDB.getLiveFilesMetaData();
    LiveFileMetaData mostDeletionSST = null;
    for (LiveFileMetaData metadata : liveFileMetaDataList) {
      if (mostDeletionSST == null) {
        mostDeletionSST = metadata;
      } else {
        if (mostDeletionSST.numDeletions() < metadata.numDeletions()) {
          mostDeletionSST = metadata;
        }
      }
    }
    
    if (mostDeletionSST != null) {
      ColumnFamily columnFamily = null;
      for (ColumnFamily cf : rocksDB.getExtraColumnFamilies()) {
        if (cf.getName().equals(mostDeletionSST.columnFamilyName())) {
          columnFamily = cf;
          break;
        }
      }
      if (columnFamily != null) {
        ManagedCompactRangeOptions options =
            new ManagedCompactRangeOptions();
        options.setBottommostLevelCompaction(
            ManagedCompactRangeOptions.BottommostLevelCompaction.kForce);
        rocksDB.compactRange(columnFamily,
            mostDeletionSST.smallestKey(),
            mostDeletionSST.largestKey(),
            options);
      }
    }

[peterxcli]

Thanks @guohao-rosicky！

[ChenSammi]

@peterxcli , thanks for the proposal. Looking forward to see the benchmark data.

[Tejaskriya]

Thanks for working on this proposal @peterxcli, the overall design looks good. Just wondering how the performance of creating and compacting these ranges would be. Benchmarking data would be helpful!

[Tejaskriya]

We can also include the multipartInfoTable here, for anyone using multipart uploads, this table also could cross the numEntries and numDeletes thresholds.

[peterxcli]

Just wondering how the performance of creating and compacting these ranges would be. Benchmarking data would be helpful!

Thanks for the review, I will start to work on this ASAP!

[aryangupta1998]

Thanks for the patch @peterxcli, the design looks good to me!
One suggestion, can we consider adding a config (e.g., max_parallel_compactions) to control the number of parallel compactions (across multiple tables)? Alternatively, we could handle multiple compaction requests through a CommandQueue mechanism to avoid overloading RocksDB.

[peterxcli]

Thanks for the patch @peterxcli, the design looks good to me! One suggestion, can we consider adding a config (e.g., max_parallel_compactions) to control the number of parallel compactions (across multiple tables)? Alternatively, we could handle multiple compaction requests through a CommandQueue mechanism to avoid overloading RocksDB.

Yeah, we can also use priority queue to prioritise the compaction range that has highest tombstone percentage.

[peterxcli]

I have just completed the benchmark on the OBS table: https://github.com/peterxcli/ozone-helper/blob/main/benchmark/range-compaction/README.md#experimental-results

I plan to update the design document in the next few days based on these results.

For reference, my POC branch for the benchmark is here: peterxcli#2
Note that it uses my RocksDB fork build, which addresses the JNI return type issue with the getPropertiesOfTablesInRange function. If there's consensus on this proposal, I plan to adapt the existing rocks native module with a patch to achieve the same fix(Thanks for @swamirishi's opinion).

Let me know if you have any feedback or suggestions!

[github-actions]

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

[github-actions]

Thank you for your contribution. This PR is being closed due to inactivity. If needed, feel free to reopen it.