Skip to content

HDDS-2214. TestSCMContainerPlacementRackAware has an intermittent failure #1560

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
elek wants to merge 1 commit into from
Closed

HDDS-2214. TestSCMContainerPlacementRackAware has an intermittent failure #1560

elek wants to merge 1 commit into from

Conversation

@elek
Copy link
Member

@elek elek commented Oct 1, 2019 •
edited
Loading

For example from the nightly build:

  <testcase name="testNoFallback[8]" classname="org.apache.hadoop.hdds.scm.container.placement.algorithms.TestSCMContainerPlacementRackAware" time="0.014">
            <failure type="java.lang.AssertionError">java.lang.AssertionError
        	at org.junit.Assert.fail(Assert.java:86)
        	at org.junit.Assert.assertTrue(Assert.java:41)
        	at org.junit.Assert.assertTrue(Assert.java:52)
        	at org.apache.hadoop.hdds.scm.container.placement.algorithms.TestSCMContainerPlacementRackAware.testNoFallback(TestSCMContainerPlacementRackAware.java:276)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:498)
        	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)

The problem is in the testNoFallback:

Let's say we have 11 nodes (from parameter) and we would like to choose 5 nodes (hard coded in the test).

As the first two replicas are chosen from the same rack an all the other from different racks it's not possible, so we except a failure.

But we have an assertion that the success count is at least 3. But this is true only if the first two replicas are placed to the rack1 (5 nodes) or rack2 (5nodes). If the replica is placed to the rack3 (one node) it will fail immediately:

Lucky case when we have success count > 3

 rack1 -- node1 
 rack1 -- node2 -- FIRST replica
 rack1 -- node3 -- SECOND replica
 rack1 -- node4
 rack1 -- node5 
 rack2 -- node6
 rack2 -- node7 -- THIRD replica
 rack2 -- node8
 rack2 -- node9 
 rack2 -- node10
 rack3 -- node11 -- FOURTH replica{code}

 The specific case when we have success count == 1, as we can't choose the second replica on rack3 (This is when the test is failing)

 rack1 -- node1 
 rack1 -- node2
 rack1 -- node3
 rack1 -- node4
 rack1 -- node5 
 rack2 -- node6
 rack2 -- node7
 rack2 -- node8
 rack2 -- node9 
 rack2 -- node10
 rack3 -- node11 -- FIRST replica{code}
 ```

@elek elek added the ozone label Oct 1, 2019
@hadoop-yetus
Copy link

hadoop-yetus commented Oct 1, 2019

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 76 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
-1 mvninstall 29 hadoop-hdds in trunk failed.
-1 mvninstall 32 hadoop-ozone in trunk failed.
-1 compile 19 hadoop-hdds in trunk failed.
-1 compile 13 hadoop-ozone in trunk failed.
+1 checkstyle 47 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 922 branch has no errors when building and testing our client artifacts.
-1 javadoc 19 hadoop-hdds in trunk failed.
-1 javadoc 17 hadoop-ozone in trunk failed.
0 spotbugs 1009 Used deprecated FindBugs config; considering switching to SpotBugs.
-1 findbugs 29 hadoop-hdds in trunk failed.
-1 findbugs 17 hadoop-ozone in trunk failed.
_ Patch Compile Tests _
-1 mvninstall 32 hadoop-hdds in the patch failed.
-1 mvninstall 35 hadoop-ozone in the patch failed.
-1 compile 21 hadoop-hdds in the patch failed.
-1 compile 16 hadoop-ozone in the patch failed.
-1 javac 21 hadoop-hdds in the patch failed.
-1 javac 16 hadoop-ozone in the patch failed.
+1 checkstyle 51 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 800 patch has no errors when building and testing our client artifacts.
-1 javadoc 19 hadoop-hdds in the patch failed.
-1 javadoc 16 hadoop-ozone in the patch failed.
-1 findbugs 29 hadoop-hdds in the patch failed.
-1 findbugs 17 hadoop-ozone in the patch failed.
_ Other Tests _
-1 unit 25 hadoop-hdds in the patch failed.
-1 unit 22 hadoop-ozone in the patch failed.
+1 asflicense 29 The patch does not generate ASF License warnings.
2437
Subsystem Report/Notes
Docker Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/artifact/out/Dockerfile
GITHUB PR #1560
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux fcce16454f30 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 8efd25b
Default Java 1.8.0_222
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/artifact/out/branch-compile-hadoop-hdds.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/artifact/out/branch-compile-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/artifact/out/branch-findbugs-hadoop-hdds.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/artifact/out/patch-compile-hadoop-hdds.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/artifact/out/patch-compile-hadoop-ozone.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/artifact/out/patch-compile-hadoop-hdds.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/artifact/out/patch-compile-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/artifact/out/patch-findbugs-hadoop-hdds.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/artifact/out/patch-findbugs-hadoop-ozone.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/testReport/
Max. process+thread count 340 (vs. ulimit of 5500)
modules C: hadoop-hdds/server-scm U: hadoop-hdds/server-scm
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1560/1/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

adoroszlai
adoroszlai approved these changes Oct 2, 2019
View reviewed changes
Copy link
Contributor

@adoroszlai adoroszlai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @elek for digging into this intermittent issue.

@elek
Copy link
Member Author

elek commented Oct 13, 2019

Thank you very much to open this pull request.

During the weekend the Ozone source code has been moved out from apache/hadoop repository to apache/hadoop-ozone repository.

This git commits are rewritten, but the branch of this pull request is also transformed (state of Saturday morning), you can use the new, migrated branch to recreate this pull request.

Your pull request is important for us: Can you please re-create your pull request in the new repository?

1. Create a new fork of https://github.com/apache/hadoop-ozone

2. Clone it and have both your fork and the apache repo as remotes:

git clone git@github.com:elek/hadoop-ozone.git
cd hadoop-ozone
git remote add apache git@github.com:apache/hadoop-ozone.git
git fetch apache

3. Fetch your migrated branch and push it to your fork.

git checkout -b HDDS-2214 apache/HDDS-2214
git push origin HDDS-2214

4. And create the new pull request on the new repository.

https://github.com/apache/hadoop-ozone/compare/master...elek:HDDS-2214?expand=1

If you need more information, please check this wiki page or contact with me (my github user name + apache.org).

Thank you, and sorry for the inconvenience.

@elek
Copy link
Member Author

elek commented Oct 13, 2019

Moved to apache/ozone#6

@elek elek closed this Oct 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adoroszlai adoroszlai adoroszlai approved these changes

Assignees

No one assigned

Labels

ozone

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@elek @hadoop-yetus @adoroszlai