HDDS-10749. Shutdown datanode when RatisServer is down #6587
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
A normal DN shutdown log, first XceiverServerRatis is stopped, "Stopping XceiverServerRatis 01effdc6-dad1-4bf3-916a-749d9aa7e5e5", then ContainerStateMachine is stopped, "Stopping ContainerStateMachine for group-5EA60976374E". |
Contributor
Author
|
A DN shutdown due to Ratis server is shutdown. First ContainerStateMachine is closed, "Container statemachine is closed by ratis, terminating HddsDatanodeService", then XceiverServerRatis is stopped, "Stopping XceiverServerRatis 01effdc6-dad1-4bf3-916a-749d9aa7e5e5". |
smengcl
reviewed
Apr 26, 2024
Comment on lines
1143
to
1145
| // wait a while for other pipeline's ContainerStateMachine.close() called. | ||
| try { | ||
| Thread.sleep(10000); |
Contributor
There was a problem hiding this comment.
Is there a more reliable way to wait for other pipeline closure other than sleep here?
And what happens if there are still unclosed pipeline after 10 seconds' wait?
Contributor
Author
There was a problem hiding this comment.
Here the ContainerStateMachine.java, which is called by each pipeline, it doesn't have the knowledge of other pipelines. 10s here is try to let other pipelines have time to close. And it's only memory operation in ContainerStateMachine.close() call. Missed a call to ContainerStateMachine.close() is not a big issue. So Shutdown immediately, wait 5s or 10s, has no big difference. Just think wait a while would be better, like when executor pool is shutdown.
adoroszlai
previously requested changes
Apr 29, 2024
Contributor
adoroszlai
left a comment
adoroszlai
left a comment
There was a problem hiding this comment.
Thanks @ChenSammi for working on this.
- Each Raft group, which corresponds to a pipeline membership in Datanode, has its own
ContainerStateMachine. - When SCM detects a dead datanode, it closes all pipelines associated with it, which triggers close of the state machine in the other two nodes.
So with this patch stopping a datanode or closing a pipeline kills other datanodes.
Note that due with multi-Raft, the effect can be cascading, since datanodes may be associated with different sets of other nodes for each Raft group.
Repro:
cd hadoop-ozone/dist/target/ozone-1.5.0-SNAPSHOT/compose/ozone
OZONE_DATANODES=6 ./run.sh -d
docker-compose exec scm ozone admin safemode wait -t 60
docker-compose ps
docker-compose up -d --no-recreate --scale datanode=5
docker-compose ps
sleep 120
docker-compose ps
Datanodes at the last step:
Name Command State Ports
-----------------------------------------------------------------------------------------------------------------------------------------------------
ozone_datanode_1 /usr/local/bin/dumb-init - ... Up 0.0.0.0:33008->19864/tcp,:::33008->19864/tcp, 0.0.0.0:33011->9882/tcp,:::33011->9882/tcp
ozone_datanode_2 /usr/local/bin/dumb-init - ... Exit 1
ozone_datanode_3 /usr/local/bin/dumb-init - ... Up 0.0.0.0:33014->19864/tcp,:::33014->19864/tcp, 0.0.0.0:33015->9882/tcp,:::33015->9882/tcp
ozone_datanode_4 /usr/local/bin/dumb-init - ... Up 0.0.0.0:33006->19864/tcp,:::33006->19864/tcp, 0.0.0.0:33007->9882/tcp,:::33007->9882/tcp
ozone_datanode_5 /usr/local/bin/dumb-init - ... Exit 1
Contributor
Author
|
@adoroszlai , I noticed the impact to the integration test too. It looks like terminate the DN in ContainerStateMachine is not a good idea for DN. Let me think if there is other solutions. |
Contributor
Author
|
Wait for RATIS release including https://issues.apache.org/jira/browse/RATIS-2066. |
Contributor
|
I was made aware that for OM if Ratis server experiences a long pause, Ratis state machine crashes itself and that shuts down OM: https://issues.apache.org/jira/browse/HDDS-6141 |
Contributor
Author
Both OM and SCM will shutdown itself after a long pause. |
Contributor
Author
|
Manual close the datanode, related datanode log Manual pause DN process and then resume the process |
Contributor
Author
|
All three failed misc acceptance runs are due to It cannot tell from the current logs why it failed. @adoroszlai , do you have any idea about this issue? |
Contributor
Author
|
Looks like the problem is |
Contributor
@ChenSammi please see #6893. This should be OK after merging from |
Contributor
|
Looks like all comments are addressed. HDDS-11092 is merged into HDDS-7593 so the previous error is no longer seen. |
Contributor
Author
|
Thanks @smengcl @jojochuang @adoroszlai for the review. |
adoroszlai
added a commit
that referenced
this pull request
Aug 13, 2024
ivandika3
pushed a commit
to ivandika3/ozone
that referenced
this pull request
Oct 12, 2025
(cherry picked from commit 8e701bf)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Currently, when RatisServer is down(mainly due to long GC which exceeds the ratis close threshold), Datanode is still running and in HEALTHY and IN_SERVICE state, which is confusing.
This tasks will shutdown the Datanode after RatisServer is down.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10749
How was this patch tested?
Manual test