HDDS-11513. All deletion configurations should be configurable without restart

[sarvekshayr]

What changes were proposed in this pull request?

#8391 adds support for invoking a callback once reconfiguration completes, allowing background services to be updated dynamically. Few more deletion related configurations are now dynamically reconfigurable without requiring a restart.

OM:

• ozone.thread.number.dir.deletion

DATANODE:

• ozone.block.deleting.service.interval
• ozone.block.deleting.service.timeout

What is the link to the Apache JIRA

HDDS-11513

How was this patch tested?

OM:

bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 status
OM: Reconfiguring status for node [om:9862]: started at Tue Jun 10 07:19:01 UTC 2025 and finished at Tue Jun 10 07:19:01 UTC 2025.
SUCCESS: Changed property ozone.thread.number.dir.deletion
        From: "15"
        To: "19"
SUCCESS: Changed property ozone.directory.deleting.service.interval
        From: "2m"
        To: "3m"

Logs

2025-06-10 07:19:01,307 [Reconfiguration Task] INFO conf.ReconfigurableBase: Starting reconfiguration task.
2025-06-10 07:19:01,314 [Reconfiguration Task] INFO conf.ReconfigurableBase: Change property: ozone.directory.deleting.service.interval from "2m" to "3m".
2025-06-10 07:19:01,315 [Reconfiguration Task] INFO conf.ReconfigurableBase: Change property: ozone.thread.number.dir.deletion from "15" to "19".
2025-06-10 07:19:01,315 [Reconfiguration Task] INFO service.DirectoryDeletingService: Updating and restarting DirectoryDeletingService with interval 180 seconds and core pool size 19
2025-06-10 07:19:01,315 [Reconfiguration Task] INFO utils.BackgroundService: Shutting down service DirectoryDeletingService
2025-06-10 07:19:01,317 [Reconfiguration Task] INFO utils.BackgroundService: Starting service DirectoryDeletingService with interval 180 seconds
2025-06-10 07:19:01,317 [Reconfiguration Task] INFO conf.ReconfigurationHandler: Reconfiguration completed with 2 updated properties.

DN:

bash-5.1$ ozone admin reconfig --service=DATANODE --address=datanode:19864 status
DN: Reconfiguring status for node [datanode:19864]: started at Tue Jun 10 07:19:44 UTC 2025 and finished at Tue Jun 10 07:19:44 UTC 2025.
SUCCESS: Changed property ozone.block.deleting.service.timeout
        From: "300000ms"
        To: "2m"
SUCCESS: Changed property ozone.block.deleting.service.interval
        From: "1m"
        To: "3m"
SUCCESS: Changed property ozone.block.deleting.service.workers
        From: "10"
        To: "30"

Logs

2025-06-10 07:19:44,896 [Reconfiguration Task] INFO conf.ReconfigurableBase: Starting reconfiguration task.
2025-06-10 07:19:44,907 [Reconfiguration Task] INFO conf.ReconfigurableBase: Change property: ozone.block.deleting.service.timeout from "300000ms" to "2m".
2025-06-10 07:19:44,908 [Reconfiguration Task] INFO conf.ReconfigurableBase: Change property: ozone.block.deleting.service.workers from "10" to "30".
2025-06-10 07:19:44,908 [Reconfiguration Task] INFO conf.ReconfigurableBase: Change property: ozone.block.deleting.service.interval from "1m" to "3m".
2025-06-10 07:19:44,908 [Reconfiguration Task] INFO impl.BlockDeletingService: Updating and restarting BlockDeletingService with interval 180 seconds, core pool size 30 and timeout 120000000000 nanoseconds
2025-06-10 07:19:44,908 [Reconfiguration Task] INFO utils.BackgroundService: Shutting down service BlockDeletingService
2025-06-10 07:19:44,908 [Reconfiguration Task] INFO utils.BackgroundService: BlockDeletingService timeout is 120000000000 nanoseconds
2025-06-10 07:19:44,909 [Reconfiguration Task] INFO utils.BackgroundService: Starting service BlockDeletingService with interval 180 seconds
2025-06-10 07:19:44,909 [Reconfiguration Task] INFO conf.ReconfigurationHandler: Reconfiguration completed with 3 updated properties.

[adoroszlai]

Thanks @sarvekshayr for the patch.

Making properties reconfigurable is a good first step, but the services do not pick up the updated configurations. dirDeletingServiceInterval etc. are only used by the tests.

[aryangupta1998]

@adoroszlai, the dirDeletingServiceInterval(ozone.directory.deleting.service.interval) and blockDeletingServiceInterval(ozone.block.deleting.service.interval) are used by the background services to determine the intervals between their execution. These intervals, apart from being used in tests, are configured for the background services that delete directories and blocks. The background service uses scheduleWithFixedDelay to schedule the tasks with the specified intervals.
public void start() { exec.scheduleWithFixedDelay(service, 0, interval, unit); }

[adoroszlai]

BlockDeletingService is created with interval and timeout taken from config:

ozone/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java

Lines 245 to 260 in 10945c8

	Duration blockDeletingSvcInterval = dnConf.getBlockDeletionInterval();
	long blockDeletingServiceTimeout = config
	.getTimeDuration(OZONE_BLOCK_DELETING_SERVICE_TIMEOUT,
	OZONE_BLOCK_DELETING_SERVICE_TIMEOUT_DEFAULT,
	TimeUnit.MILLISECONDS);
	int blockDeletingServiceWorkerSize = config
	.getInt(OZONE_BLOCK_DELETING_SERVICE_WORKERS,
	OZONE_BLOCK_DELETING_SERVICE_WORKERS_DEFAULT);
	blockDeletingService =
	new BlockDeletingService(this, blockDeletingSvcInterval.toMillis(),
	blockDeletingServiceTimeout, TimeUnit.MILLISECONDS,
	blockDeletingServiceWorkerSize, config,
	datanodeDetails.threadNamePrefix(),
	context.getParent().getReconfigurationHandler());

They are stored in the base BackgroundService class:

ozone/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundService.java

Lines 58 to 65 in 10945c8

	public BackgroundService(String serviceName, long interval,
	TimeUnit unit, int threadPoolSize, long serviceTimeout,
	String threadNamePrefix) {
	this.interval = interval;
	this.unit = unit;
	this.serviceName = serviceName;
	this.serviceTimeoutInNanos = TimeDuration.valueOf(serviceTimeout, unit)
	.toLong(TimeUnit.NANOSECONDS);

which schedules the task when started:

ozone/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundService.java

Lines 108 to 109 in 10945c8

	public void start() {
	exec.scheduleWithFixedDelay(service, 0, interval, unit);

Reconfiguring does not change values in BackgroundService (they are final), and even updating interval wouldn't change the task already scheduled with fixed delay (it needs to be rescheduled).

[aryangupta1998]

Thanks for the clarification, @adoroszlai. I checked the code, for these configs to be updated, we need to restart the OM. The "ozone om" command starts the key manager, which then initializes the block deleting and directory deleting service with their intervals. So, simply updating these properties without starting the OM again won't be effective here.

[adoroszlai]

simply updating these properties without starting the OM again won't be effective here

That's my point. OM can be tweaked to pick up the config without restart, but the current patch is not enough for that yet.

[sarvekshayr]

@sumitagrawl @aryangupta1998 @Tejaskriya Please review this PR.

[sumitagrawl]

@sarvekshayr Thanks for working over this, few comments given

[Copilot]

Pull Request Overview

Adds dynamic reconfiguration support for deletion-related settings in OM and DataNode so they can be changed at runtime without restarting.

• OM now supports changing the number of directory-deletion threads (ozone.thread.number.dir.deletion).
• DataNode supports changing block-deletion interval and timeout (ozone.block.deleting.service.interval and ozone.block.deleting.service.timeout) at runtime.
• BackgroundService timeout is now mutable, and integration tests are updated to cover the new OM reconfig.

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java	Register and implement reconfOzoneThreadNumberDirDeletion callback
integration-test/src/test/java/org/apache/hadoop/ozone/reconfig/TestOmReconfiguration.java	Add test for thread-count reconfiguration in OM
integration-test/src/test/java/org/apache/hadoop/ozone/reconfig/TestDatanodeReconfiguration.java	Include block-deleting interval/timeout keys in expected reconfigurable set
container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/BlockDeletingService.java	Register complete callback and implement updateAndRestart on interval change
container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java	Register DN reconfig callbacks for interval and timeout and apply timeout
common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundService.java	Make serviceTimeoutInNanos mutable and add setter

Comments suppressed due to low confidence (3)

hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java:292

• The reconfigBlockDeletingServiceInterval callback only updates the configuration key but does not apply the new interval to the BlockDeletingService at runtime. Consider invoking a service update (e.g., calling updateAndRestart on BlockDeletingService) or setting the interval directly so the change takes effect immediately.

.register(OZONE_BLOCK_DELETING_SERVICE_INTERVAL, this::reconfigBlockDeletingServiceInterval)

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java:5164

• [nitpick] The method prefix reconf is inconsistent with the reconfig prefix used elsewhere (e.g., in HddsDatanodeService). Consider standardizing on a single naming convention for reconfiguration callbacks for clarity.

private String reconfOzoneThreadNumberDirDeletion(String newVal) {

hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/reconfig/TestDatanodeReconfiguration.java:55

• There are currently no tests verifying that runtime reconfiguration of block-deleting interval and timeout actually affects the service behavior on the DataNode. Consider adding integration or unit tests to assert that the BlockDeletingService applies new values without restart.

.add(OZONE_BLOCK_DELETING_SERVICE_TIMEOUT)

[aryangupta1998]

Thanks for updating the patch @sarvekshayr!

patch updated

[aryangupta1998]

Thanks for updating the patch @sarvekshayr, LGTM!

[sumitagrawl]

LGTM