HDFS-16320. Datanode retrieve slownode information from NameNode #3654
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
🎊 +1 overall
This message was automatically generated. |
Contributor
|
@symious Thanks. |
Contributor
|
@aajisaka @Hexiaoqiao @jojochuang Would you give any advices? |
Contributor
Contributor
Author
|
@ferhui Thank you for the review. Updated a document to better illustrate the idea. https://docs.google.com/document/d/10-qrEJ6n-wVCSKlJXsykvKzcS3UBOi6EkuKFm4Z-k40/edit?usp=sharing |
Contributor
|
@symious Thanks for your document, I think it's a good solution. |
Member
|
@symious Thanks for submitting the PR and sharing your document. I think it's a good solution too.
I want to clarify about "the clients already using the slownode". Does it mean the case that the DataNode is not a slownode when the client starts writing the file, but it becomes a slownode while the client is writing the file? Does it happen often? |
Contributor
Author
|
@tasanuma Thanks for the review.
Yes, normally it shouldn't happen quite often if we enable the config. But the consequence might be not acceptable. In our cluster, the client complains writing a file costs 1.5h, which is not acceptable especially in a streaming job. |
Member
|
@symious Thank you for your reply. That definitely should be fixed. And getting such information from the datanodes makes sense to me. |
Contributor
|
@symious You can fix the checkstyle issues. If no objection, you can continue to do the next work mentioned in your document. |
Contributor
Author
|
@ferhui Sure. Updated the patch. Please have a check. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
ferhui
approved these changes
Nov 20, 2021
Contributor
Contributor
Author
|
@ferhui @tasanuma @Hexiaoqiao Thank you for the review. The follow-up ticket is https://issues.apache.org/jira/browse/HDFS-16348. Please help to check. |
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
The current information of slownode is reported by reportingNode, and stored in NameNode.
This ticket is to let the slownode retrieve the information from NameNode, so that it can do other performance improvement actions based on this information.
Jira ticket: https://issues.apache.org/jira/browse/HDFS-16320
Document: https://docs.google.com/document/d/10-qrEJ6n-wVCSKlJXsykvKzcS3UBOi6EkuKFm4Z-k40/edit?usp=sharing
How was this patch tested?
unit test
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?