Skip to content

HDFS-15457. TestFsDatasetImpl fails intermittently. #2407

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
goiri merged 1 commit into from
Oct 28, 2020
Merged

HDFS-15457. TestFsDatasetImpl fails intermittently. #2407

goiri merged 1 commit into from
Oct 28, 2020

Conversation

@amahussein
Copy link
Contributor

@amahussein amahussein commented Oct 23, 2020

NOTICE

Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

@goiri goiri changed the title hdfs-15457. fix TestFsDatasetImpl readLock unit tests. HDFS-15457. Fix TestFsDatasetImpl readLock unit tests. Oct 23, 2020
@hadoop-yetus
Copy link

hadoop-yetus commented Oct 23, 2020

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 31s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 0m 0s test4tests The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 30m 9s trunk passed
+1 💚 compile 1m 20s trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚 compile 1m 14s trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚 checkstyle 0m 51s trunk passed
+1 💚 mvnsite 1m 17s trunk passed
+1 💚 shadedclient 16m 50s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 55s trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚 javadoc 1m 29s trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+0 🆗 spotbugs 3m 4s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 3m 2s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 1m 14s the patch passed
+1 💚 compile 1m 13s the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚 javac 1m 13s the patch passed
+1 💚 compile 1m 7s the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚 javac 1m 7s the patch passed
+1 💚 checkstyle 0m 39s the patch passed
+1 💚 mvnsite 1m 9s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 14m 25s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 48s the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚 javadoc 1m 22s the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚 findbugs 3m 5s the patch passed
_ Other Tests _
-1 ❌ unit 95m 5s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 45s The patch does not generate ASF License warnings.
180m 35s
Reason Tests
Failed junit tests hadoop.hdfs.TestFileChecksumCompositeCrc
hadoop.hdfs.server.balancer.TestBalancer
hadoop.hdfs.TestFileChecksum
hadoop.hdfs.TestDFSUpgradeFromImage
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2407/1/artifact/out/Dockerfile
GITHUB PR #2407
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 5e1fbdff6356 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 7bdf165
Default Java Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2407/1/testReport/
Max. process+thread count 4109 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2407/1/console
versions git=2.17.1 maven=3.6.0 findbugs=4.1.3
Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@amahussein
Copy link
Contributor Author

amahussein commented Oct 28, 2020

Thank you @goiri for the review.
This fix is good to go. The failing units are among the ones we are working to fix. The couple others may be failing randomly.

@goiri goiri changed the title HDFS-15457. Fix TestFsDatasetImpl readLock unit tests. HDFS-15457. TestFsDatasetImpl fails intermittently. Oct 28, 2020
@goiri goiri merged commit 98097b8 into apache:trunk Oct 28, 2020
@goiri goiri assigned goiri and amahussein and unassigned goiri Oct 28, 2020
amahussein added a commit to amahussein/hadoop that referenced this pull request Sep 1, 2021
@amahussein
HDFS-15457. TestFsDatasetImpl fails intermittently (apache#2407)
dcaccc1 
(cherry picked from commit 98097b8)
Hexiaoqiao pushed a commit that referenced this pull request Sep 11, 2021
@amahussein @LeonGao91
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various…
318bc5f 
… FsDatasetImpl methods should use datanode readlock. (#3200)

* HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <[email protected]>
(cherry picked from commit d7c136b)

* HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <[email protected]>
(cherry picked from commit 2a67e2b)

* HDFS-15457. TestFsDatasetImpl fails intermittently (#2407)
(cherry picked from commit 98097b8)

* HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679)
(cherry picked from commit 9434c1e)

* HDFS-15160. amend to fix javac error supressing unchecked warning
Co-authored-by: Stephen O'Donnell <[email protected]>
Co-authored-by: LeonGao <[email protected]>
asfgit pushed a commit that referenced this pull request Sep 14, 2021
@amahussein @Hexiaoqiao
HDFS-15457. TestFsDatasetImpl fails intermittently (#2407)
33c60ba 
(cherry picked from commit 98097b8)
asfgit pushed a commit that referenced this pull request Sep 14, 2021
@amahussein @Hexiaoqiao
HDFS-15457. TestFsDatasetImpl fails intermittently (#2407)
0879f4e 
(cherry picked from commit 98097b8)
tasanuma pushed a commit that referenced this pull request Nov 16, 2021
@amahussein @tasanuma
HDFS-15457. TestFsDatasetImpl fails intermittently (#2407)
25bcc7e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@goiri goiri goiri approved these changes

Assignees

@amahussein amahussein

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@amahussein @hadoop-yetus @goiri