Skip to content

Conversation

@lzx404243
Copy link
Contributor

The test org.apache.hadoop.hdfs.nfs.nfs3.TestViewfsWithNfs3.testNfsRenameSingleNN is not idempotent and fails if run twice in the same JVM, because it pollutes state shared among tests. It may be good to clean this state pollution so that some other tests do not fail in the future due to the shared state polluted by this test.

Details

Running TestViewfsWithNfs3.testNfsRenameSingleNN twice would result in the second run failing with the a NullPointer exception:

[ERROR] Errors:
[ERROR] TestViewfsWithNfs3.testNfsRenameSingleNN:317 NullPointer

The reason for this is that the /user1/renameSingleNN file is created in setup(), but gets renamed intestNfsRenameSingleNN. When the second run of testNfsRenameSingleNN tries to get info of the file by its original name, it returns a NullPointer since the file no longer exists.

The fix for this is to rename the file back to the original when the test is done.

With the proposed fix, the test does not pollute the shared state (and passes when run twice in the same JVM).

Link to JIRA issue: https://issues.apache.org/jira/browse/HDFS-15862

@lzx404243
Copy link
Contributor Author

Hi @aajisaka , could you please help reviewing this change? It resolves similar issues as #2499 . Thanks!

Copy link
Member

@aajisaka aajisaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the 2nd rename failed, the test will be non-idempotent. I think renameSingleNN should be created in the test case instead of @BeforeClass. In addition, we need to remove renameSingleNNSucess if exists before running rename.

@lzx404243 lzx404243 requested a review from aajisaka August 31, 2021 05:01
@lzx404243
Copy link
Contributor Author

@aajisaka Thanks for the feedback! I've made the corresponding changes following your suggestion. Please let me know what you think.

Thanks!

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 13m 8s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 34m 27s trunk passed
+1 💚 compile 0m 30s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 0m 27s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 checkstyle 0m 24s trunk passed
+1 💚 mvnsite 0m 32s trunk passed
+1 💚 javadoc 0m 26s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 0m 26s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 0m 50s trunk passed
+1 💚 shadedclient 15m 35s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 26s the patch passed
+1 💚 compile 0m 24s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javac 0m 24s the patch passed
+1 💚 compile 0m 22s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 javac 0m 22s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 15s the patch passed
+1 💚 mvnsite 0m 21s the patch passed
+1 💚 javadoc 0m 21s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 0m 18s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 0m 51s the patch passed
+1 💚 shadedclient 16m 50s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 58s hadoop-hdfs-nfs in the patch passed.
+1 💚 asflicense 0m 33s The patch does not generate ASF License warnings.
93m 10s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2724/2/artifact/out/Dockerfile
GITHUB PR #2724
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux 3441aa40f3c3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 75d6bb6
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2724/2/testReport/
Max. process+thread count 636 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-nfs U: hadoop-hdfs-project/hadoop-hdfs-nfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2724/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@lzx404243 lzx404243 requested a review from aajisaka November 13, 2021 22:32
@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 40s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 32m 21s trunk passed
+1 💚 compile 0m 31s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 0m 29s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 checkstyle 0m 26s trunk passed
+1 💚 mvnsite 0m 34s trunk passed
+1 💚 javadoc 0m 29s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 0m 27s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 0m 47s trunk passed
+1 💚 shadedclient 19m 22s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 22s the patch passed
+1 💚 compile 0m 21s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javac 0m 21s the patch passed
+1 💚 compile 0m 19s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 javac 0m 19s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 14s the patch passed
+1 💚 mvnsite 0m 22s the patch passed
+1 💚 javadoc 0m 19s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 0m 17s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 0m 47s the patch passed
+1 💚 shadedclient 19m 16s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 42s hadoop-hdfs-nfs in the patch passed.
+1 💚 asflicense 0m 33s The patch does not generate ASF License warnings.
84m 5s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2724/3/artifact/out/Dockerfile
GITHUB PR #2724
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux 9a28fb974821 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 6c75449
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2724/3/testReport/
Max. process+thread count 549 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-nfs U: hadoop-hdfs-project/hadoop-hdfs-nfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2724/3/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@aajisaka aajisaka changed the title HDFS-15862. Rename file back to original in TestViewfsWithNfs3.testNfsRenameSingleNN() HDFS-15862. Make TestViewfsWithNfs3.testNfsRenameSingleNN() idempotent Dec 8, 2021
@aajisaka aajisaka merged commit 7ba91c1 into apache:trunk Dec 8, 2021
@aajisaka
Copy link
Member

aajisaka commented Dec 8, 2021

Merged. Thank you @lzx404243

aajisaka pushed a commit that referenced this pull request Dec 8, 2021
aajisaka pushed a commit that referenced this pull request Dec 8, 2021
sunchao pushed a commit that referenced this pull request Jan 4, 2022
HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants