Skip to content

HDFS-16290.Make log more standardized when executing verifyAndSetNamespaceInfo().#3600

Merged
ayushtkn merged 1 commit intoapache:trunkfrom
jianghuazhu:HDFS-16290
Oct 31, 2021
Merged

HDFS-16290.Make log more standardized when executing verifyAndSetNamespaceInfo().#3600
ayushtkn merged 1 commit intoapache:trunkfrom
jianghuazhu:HDFS-16290

Conversation

@jianghuazhu
Copy link
Copy Markdown
Contributor

@jianghuazhu jianghuazhu commented Oct 29, 2021

Description of PR

When the DataNode starting, BPOfferService#verifyAndSetNamespaceInfo() will be executed. If the namenode being connected is in Active state, the following information will be recorded:
'
2021-10-27 18:08:36,242 [50867]-INFO [Thread-33:BPOfferService@376]-Acknowledging ACTIVE Namenode during handshakeBlock pool BP-597961518-xxx.xxx.xxx.xxx-1534758275943 (Datanode Uuid 9b2aedc9-f8b2 -4ee2-b99f-877bc6e42c87) service to xxxx.xxx.xxx.org/xxxx.xxx.xxx.xxx:8021
'
Here, the connection between'handshake' and'Block pool' is too tight, and the readability is not good.

Details: HDFS-16290

How was this patch tested?

This is related to the log, and there is less information to change. For testing, there is not much pressure.

@hadoop-yetus
Copy link
Copy Markdown

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 9s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 35m 22s trunk passed
+1 💚 compile 1m 37s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 1m 21s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 checkstyle 1m 3s trunk passed
+1 💚 mvnsite 1m 32s trunk passed
+1 💚 javadoc 1m 7s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 1m 29s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 3m 37s trunk passed
+1 💚 shadedclient 25m 55s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 20s the patch passed
+1 💚 compile 1m 29s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javac 1m 29s the patch passed
+1 💚 compile 1m 16s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 javac 1m 16s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 54s the patch passed
+1 💚 mvnsite 1m 23s the patch passed
+1 💚 javadoc 0m 53s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 1m 26s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 3m 43s the patch passed
+1 💚 shadedclient 25m 35s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 354m 48s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 42s The patch does not generate ASF License warnings.
464m 38s
Reason Tests
Failed junit tests hadoop.hdfs.TestHDFSFileSystemContract
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3600/1/artifact/out/Dockerfile
GITHUB PR #3600
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux f85bd3e36fc2 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 2ec74aa91dc58bedae20f5e1d39d2ebb9fdec17f
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3600/1/testReport/
Max. process+thread count 1983 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3600/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@jianghuazhu
Copy link
Copy Markdown
Contributor Author

jianghuazhu commented Oct 29, 2021

Some exceptions occurred here, during the operation of jenkins.
E.g:
TestHDFSFileSystemContract
TestDecommission
TestLeaseRecovery
TestHdfsCryptoStreams
TestSeveralNameNodes

It seems that these exceptions have little to do with the code I submitted.
@ayushtkn @tomscut, would you like to spend some time reviewing this pr.
thank you very much.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can use logger format:

      LOG.info("Acknowledging ACTIVE Namenode during handshake {}", actor);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @ayushtkn .

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your comments and reviews, @ayushtkn @tomscut.
I will update it later.

@hadoop-yetus
Copy link
Copy Markdown

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 19s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 36m 17s trunk passed
+1 💚 compile 1m 34s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 1m 23s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 checkstyle 1m 3s trunk passed
+1 💚 mvnsite 1m 34s trunk passed
+1 💚 javadoc 1m 7s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 1m 31s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 3m 51s trunk passed
+1 💚 shadedclient 26m 5s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 21s the patch passed
+1 💚 compile 1m 31s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javac 1m 31s the patch passed
+1 💚 compile 1m 19s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 javac 1m 19s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 56s the patch passed
+1 💚 mvnsite 1m 27s the patch passed
+1 💚 javadoc 0m 54s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 1m 26s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 3m 45s the patch passed
+1 💚 shadedclient 26m 4s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 353m 9s hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 41s The patch does not generate ASF License warnings.
464m 59s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3600/2/artifact/out/Dockerfile
GITHUB PR #3600
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux 0773cb26ebfa 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / bb326ae
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3600/2/testReport/
Max. process+thread count 1944 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3600/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@jianghuazhu
Copy link
Copy Markdown
Contributor Author

Would you like to spend some time reviewing this pr again, @ayushtkn @tomscut.
Thank you very much.

Copy link
Copy Markdown
Member

@ayushtkn ayushtkn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ayushtkn ayushtkn merged commit 0a68413 into apache:trunk Oct 31, 2021
HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022
…spaceInfo(). (apache#3600)

HDFS-16290.Make log more standardized when executing verifyAndSetNamespaceInfo(). (apache#3600). Contributed by JiangHua Zhu.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants