HDDS-13045. Implement Immediate Triggering of Heartbeat when Volume Full

[siddhantsangwan]

What changes were proposed in this pull request?

Trigger datanode heartbeat immediately when the container being written to is (close) to full, or volume is full, or container is unhealthy. The immediate heartbeat will contain close container action.

• Changes made to HddsDispatcher.
• Per-container throttling. Heartbeat will be triggered immediately once for that particular container. After that, container close container action can be sent in regular heartbeat if needed. This acts as a retry mechanism in case the immediate heartbeat is not processed/lost or any other runtime failure reason.
• The volume full check is changed to reserved - available - min free <= 0. This fixes a bug.

See #8492 (comment) for full context and #8460 for a design discussion. The design discussion isn't fully updated yet, but both the above pull requests contain discussions about alternatives that we didn't decide to take.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13045

How was this patch tested?

Modified existing unit tests.

The PR is ready for review but marked as draft while CI is running in my fork.

I also did some manual testing using the docker compose ozone definition and some temporary changes to simulate a full volume.

Started with 150 MB available space and 100 MB min free space:

2025-06-09 07:27:33,660 [main] INFO volume.HddsVolume: HddsVolume: { id=DS-6f8612f1-b7b1-4914-854d-bdb1b25114ad dir=/data/hdds/hdds type=DISK capacity=2147268899 used=1990197248 available=157071651 minFree=104857600 committed=0 }

Wrote 150 MB of data, Datanode detected full volume/container and triggered a heartbeat:

 2025-06-09 07:28:07,530 [4cbef4c3-3e47-436e-809f-9282b2b1b1f9-EndpointStateMachineTaskThread-scm/172.18.0.6:9861-0 ] INFO endpoint.HeartbeatEndpointTask: Sending heartbeat message : containerActions {
   containerID: 1
   action: CLOSE
   reason: CONTAINER_FULL
 }
 
 2025-06-09 07:28:07,531 [4cbef4c3-3e47-436e-809f-9282b2b1b1f9-EndpointStateMachineTaskThread-recon/172.18.0.5:9891-0 ] INFO endpoint.HeartbeatEndpointTask: Sending heartbeat message : containerActions {
   containerID: 1
   action: CLOSE
   reason: CONTAINER_FULL
 }
 
 2025-06-09 07:28:07,882 [4cbef4c3-3e47-436e-809f-9282b2b1b1f9-ContainerOp-99fd4123-d677-4ed0-b872-d86353836135-4] WARN impl.HddsDispatcher: Volume [/data/hdds/hdds] is full. containerID: 1. Volume usage: [capacity=2147268899, used=2042626048, available=104642851].

SCM received the heartbeat at the same time:

2025-06-09 07:28:07,534 [IPC Server handler 59 on default port 9861] INFO server.SCMDatanodeHeartbeatDispatcher: Dispatching Container Actions. containerActions {
   containerID: 1
   action: CLOSE
   reason: CONTAINER_FULL
 }
 
 2025-06-09 07:28:07,535 [IPC Server handler 29 on default port 9861] INFO server.SCMDatanodeHeartbeatDispatcher: Dispatching Container Actions. containerActions {
   containerID: 1
   action: CLOSE
   reason: CONTAINER_FULL
 }
 
 2025-06-09 07:28:07,535 [EventQueue-CloseContainerForCloseContainerEventHandler] INFO container.CloseContainerEventHandler: Close container Event triggered for container : #1, current state: OPEN
 2025-06-09 07:28:07,535 [IPC Server handler 43 on default port 9861] INFO server.SCMDatanodeHeartbeatDispatcher: Dispatching Container Actions. containerActions {
   containerID: 1
   action: CLOSE
   reason: CONTAINER_FULL
 }

[sumitagrawl]

@siddhantsangwan have minor comments, other LGTM

[Copilot]

Pull Request Overview

This PR addresses the immediate triggering of a datanode heartbeat when a container or volume is full by modifying the close container action logic and updating tests to verify the per-container throttling behavior. Key changes include:

• Adding immediate heartbeat logic using an AtomicBoolean flag in ContainerData and HddsDispatcher.
• Updating the volume full check in HddsVolume.
• Enhancing unit tests in TestHddsDispatcher to verify heartbeat triggering and throttling.

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestHddsDispatcher.java	Updated test scenarios to verify immediate heartbeat triggering and throttling behavior.
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java	Updated volume full check logic.
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/HddsDispatcher.java	Introduced immediate heartbeat triggering logic and replaced volume full checking with an adjusted method.
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerData.java	Added an AtomicBoolean to track immediate close action triggering.

Comments suppressed due to low confidence (2)

hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestHddsDispatcher.java:312

• The increment value has been changed from 50 to 60 bytes; please update the accompanying comment to explain why the increment was adjusted so that the test scenario accurately reflects the volume full condition.

containerData.getVolume().getVolumeUsage().ifPresent(usage -> usage.incrementUsedSpace(60));

hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/HddsDispatcher.java:629

• [nitpick] The method name is quite verbose; consider using a more concise name (e.g., isVolumeFullAdjusted) that still conveys that committed space is excluded from the check.

private boolean isVolumeFullExcludingCommittedSpace(Container container) {

[siddhantsangwan]

In the latest commit I added verification in the unit tests that heartbeat throttling is done per container.

[sumitagrawl]

LGTM

[nandakumar131]

If we are closing the container because of full volume, we can set the reason to VOLUME_FULL.

[siddhantsangwan]

@nandakumar131 Thanks for the review! Since VOLUME_FULL isn't strictly required, in the interest of time let's merge this PR? I'd like to avoid making a proto change in this one. We can make the improvement in a future PR if needed.

[nandakumar131]

Yeah, we can do that as a followup.

[adoroszlai]

Thanks @siddhantsangwan for the patch.

Please consider using StorageLocationReport to capture all volume information in one step, and use it for both calculation and logging for consistency.

Example:

ozone/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/HddsDispatcher.java

Lines 617 to 621 in 6bf009c

	StorageLocationReport volumeReport = volume.getReport();
	boolean full = volumeReport.getUsableSpace() <= 0;
	if (full) {
	LOG.info("Container {} volume is full: {}", container.getContainerData().getContainerID(), volumeReport);
	}

[adoroszlai]

Thanks @siddhantsangwan for the patch, @nandakumar131, @sumitagrawl for the review.