HDDS-13114. Make MaxSubCompactions configureable for manual compaction #8572
Conversation
peterxcli
changed the title
swamirishi
left a comment
swamirishi
left a comment
There was a problem hiding this comment.
@peterxcli Thanks for working on the patch. I have left some comments inline.
| } | ||
|
|
||
| return new RDBStore(dbFile, rocksDBOption, statistics, writeOptions, tableConfigs, | ||
| ManagedCompactRangeOptions rangeCompactionOptions = new ManagedCompactRangeOptions(); |
There was a problem hiding this comment.
This shouldn't be done on the automatic background compactions. We need these compaction options only for the manual compactions done by the BackgroundCompactionService & SnapshotCache. These default options on the rocksdb options should be coming from an ini file we don't want individual ozone configuration corresponding to each and every rocksdb configuration this might get very tough to handle later.
https://issues.apache.org/jira/browse/HDDS-12484
cc: @Galsza
There was a problem hiding this comment.
…pactionbytes-and-maxsubcompactions-configurable
peterxcli
force-pushed
the
codex/make-maxcompactionbytes-and-maxsubcompactions-configurable
branch
from
91bc8b3 to
6385e7e
Compare
| defaultValue = "1", | ||
| tags = {OM}, | ||
| description = "Maximum number of sub compactions to use when manual compacting. Default 1 means no limit.") | ||
| private int manualCompactionMaxSubCompactions = 1; |
There was a problem hiding this comment.
https://github.com/facebook/rocksdb/wiki/Subcompaction#options
The option max_subcompactions limits the max number of subcompactions for each compaction. By default, max_subcompactions = 1, which means it is disabled.
peterxcli
changed the title
peterxcli
changed the title
|
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days. |
|
Thank you for your contribution. This PR is being closed due to inactivity. If needed, feel free to reopen it. |
2 participants
What changes were proposed in this pull request?
#8141 (comment)
In #8141
RDBStore#compactTablewas introduced.We should set
options.setMaxCompactionBytes()when we open the snapshot rocksdb this could be useful so that we ensure one sub compaction doesn't take up a lot of memory. Also look into making ManagedCompactRangeOptions.setMaxSubCompactions() configurable so that we don't use a lot of CPU for this operation. It is ok if the compactions take time.This can be achieved by adding
ManagedRangeCompactionOptionintoAbstractRDBStore,RDBStore,RDBStoreBuilder. Also the corresponding rocksdb config keys.What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13114
How was this patch tested?
CI:
https://github.com/peterxcli/ozone/actions/runs/15484301525