HBASE-24183 [flakey test] replication.TestAddToSerialReplicationPeer #1514

huaxiangsun · 2020-04-14T21:28:15Z

Will put up a root cause analysis later.

Apache-HBase · 2020-04-14T22:02:05Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 40s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 34s	master passed
+1 💚	checkstyle	1m 6s	master passed
+1 💚	spotbugs	2m 2s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 16s	the patch passed
+1 💚	checkstyle	1m 5s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	10m 42s	Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2.
+1 💚	spotbugs	2m 10s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		31m 49s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1514/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1514
Optional Tests	dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
uname	Linux d8f55374dded 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `4fc3527`
Max. process+thread count	94 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1514/1/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

huaxiangsun · 2020-04-15T01:07:15Z

There are two flakies.

testAddToSerialPeer failure is that it just needs to make sure the source RS's inmemory map contains only the new wal file. (not the RS which region moves to)

After that, there is still one failure which is common to testAddToSerialPeer and testChangeToSerial.

If the old wal file before rollover is still in the inmemory map of replicateSourceManager, during peer disable/enable/config update, it could be still be replicated over from the begin to peer cluster. If that happens, the old wal entries and the new wal entries will be written to the same wal file (which results in out of order seq numbers).

waitUntilReplicatedToTheCurrentWALFile() does not really guarantee that the inmemory map is forwarded to the new wal file, there is a small window that there is only one wal file in map which is the old wal file. Added a new check to make sure that inmemory map only has the new wal file.

This could happen in the production cluster as well, however I do not think this is the purpose of these two test cases. @Apache9, please provide your input, thanks.

Apache9

+1

Apache-HBase · 2020-04-15T01:51:23Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 39s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 45s	master passed
+1 💚	compile	1m 12s	master passed
+1 💚	shadedjars	5m 56s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 43s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 13s	the patch passed
+1 💚	compile	1m 9s	the patch passed
+1 💚	javac	1m 9s	the patch passed
+1 💚	shadedjars	6m 1s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 39s	the patch passed
		_ Other Tests _
+1 💚	unit	231m 57s	hbase-server in the patch passed.
		260m 39s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1514/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR	#1514
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 9778e249c96c 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `4fc3527`
Default Java	1.8.0_232
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1514/1/testReport/
Max. process+thread count	2572 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1514/1/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-04-15T02:06:34Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 40s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	5m 23s	master passed
+1 💚	compile	1m 23s	master passed
+1 💚	shadedjars	6m 50s	branch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 56s	hbase-server in master failed.
		_ Patch Compile Tests _
+1 💚	mvninstall	5m 11s	the patch passed
+1 💚	compile	1m 22s	the patch passed
+1 💚	javac	1m 22s	the patch passed
+1 💚	shadedjars	7m 18s	patch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	1m 0s	hbase-server in the patch failed.
		_ Other Tests _
-1 ❌	unit	242m 39s	hbase-server in the patch failed.
		275m 52s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1514/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#1514
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux c6538c99b2b6 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `4fc3527`
Default Java	2020-01-14
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1514/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1514/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
unit	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1514/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1514/1/testReport/
Max. process+thread count	2745 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1514/1/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

saintstack · 2020-04-15T04:33:21Z

Test failures related @huaxiangsun ? Let me rerun the tests to see.... Otherwise, +1 on patch. Lets try it. I like the root cause analysis. Stick that up on the issue tooo.

huaxiangsun · 2020-04-15T05:03:34Z

Thanks @Apache9 and @saintstack. The failed tests are not related cause the patch is in the specific test case. I checked the failed three cases, they are all due to malformat xml error so could related to the known native thread issue.

huaxiangsun · 2020-04-15T05:05:58Z

Forwarded the comments to the jira.

Apache-HBase · 2020-04-15T05:16:10Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 40s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 33s	master passed
+1 💚	checkstyle	1m 32s	master passed
+1 💚	spotbugs	2m 54s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 24s	the patch passed
+1 💚	checkstyle	1m 29s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	14m 19s	Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2.
+1 💚	spotbugs	2m 49s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		41m 44s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1514/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1514
Optional Tests	dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
uname	Linux 688a9101935e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `38954ed`
Max. process+thread count	94 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1514/2/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-04-15T06:53:40Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 33s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 30s	master passed
+1 💚	compile	0m 57s	master passed
+1 💚	shadedjars	4m 39s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 38s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 14s	the patch passed
+1 💚	compile	0m 54s	the patch passed
+1 💚	javac	0m 54s	the patch passed
+1 💚	shadedjars	4m 40s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 36s	the patch passed
		_ Other Tests _
+1 💚	unit	117m 34s	hbase-server in the patch passed.
		139m 23s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1514/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR	#1514
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux f23df99253c4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `38954ed`
Default Java	1.8.0_232
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1514/2/testReport/
Max. process+thread count	4538 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1514/2/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-04-15T07:18:13Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 38s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 23s	master passed
+1 💚	compile	1m 6s	master passed
+1 💚	shadedjars	5m 37s	branch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 42s	hbase-server in master failed.
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 4s	the patch passed
+1 💚	compile	1m 7s	the patch passed
+1 💚	javac	1m 7s	the patch passed
+1 💚	shadedjars	5m 40s	patch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 38s	hbase-server in the patch failed.
		_ Other Tests _
-1 ❌	unit	138m 5s	hbase-server in the patch failed.
		163m 55s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1514/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#1514
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 280136dfc0cd 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `38954ed`
Default Java	2020-01-14
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1514/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1514/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
unit	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1514/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1514/2/testReport/
Max. process+thread count	4011 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1514/2/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

huaxiangsun · 2020-04-15T16:57:11Z

Test failures are unrelated, merging, thanks for the review.

…pache#1514) Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: stack <[email protected]>

…1514) (#1525) Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: stack <[email protected]>

…1514) (#1526) Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: stack <[email protected]>

…pache#1514) Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: stack <[email protected]>

…pache#1514) (apache#1526) Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: stack <[email protected]>

HBASE-24183 [flakey test] replication.TestAddToSerialReplicationPeer

22414cd

huaxiangsun requested a review from Apache9 April 15, 2020 01:07

Apache9 approved these changes Apr 15, 2020

View reviewed changes

huaxiangsun merged commit 3421423 into apache:master Apr 15, 2020

huaxiangsun added a commit to huaxiangsun/hbase that referenced this pull request Apr 15, 2020

HBASE-24183 [flakey test] replication.TestAddToSerialReplicationPeer (a…

7ac2cca

…pache#1514) Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: stack <[email protected]>

huaxiangsun added a commit to huaxiangsun/hbase that referenced this pull request Apr 15, 2020

HBASE-24183 [flakey test] replication.TestAddToSerialReplicationPeer (a…

ad5c130

…pache#1514) Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: stack <[email protected]>

This was referenced Apr 15, 2020

Backport: HBASE-24183 [flakey test] replication.TestAddToSerialReplicationPeer … #1525

Merged

Backport: HBASE-24183 [flakey test] replication.TestAddToSerialReplicationPeer … #1526

Merged

huaxiangsun added a commit that referenced this pull request Apr 15, 2020

HBASE-24183 [flakey test] replication.TestAddToSerialReplicationPeer (#…

efb5ff0

…1514) (#1525) Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: stack <[email protected]>

huaxiangsun added a commit that referenced this pull request Apr 15, 2020

HBASE-24183 [flakey test] replication.TestAddToSerialReplicationPeer (#…

0edf6cf

…1514) (#1526) Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: stack <[email protected]>

thangTang pushed a commit to thangTang/hbase that referenced this pull request Apr 16, 2020

HBASE-24183 [flakey test] replication.TestAddToSerialReplicationPeer (a…

ef93789

…pache#1514) Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: stack <[email protected]>

thangTang pushed a commit to thangTang/hbase that referenced this pull request Apr 16, 2020

HBASE-24183 [flakey test] replication.TestAddToSerialReplicationPeer (a…

6958e02

…pache#1514) Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: stack <[email protected]>

thangTang pushed a commit to thangTang/hbase that referenced this pull request Apr 16, 2020

HBASE-24183 [flakey test] replication.TestAddToSerialReplicationPeer (a…

50350ef

…pache#1514) Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: stack <[email protected]>

thangTang pushed a commit to thangTang/hbase that referenced this pull request Apr 16, 2020

HBASE-24183 [flakey test] replication.TestAddToSerialReplicationPeer (a…

1ed9b9e

…pache#1514) Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: stack <[email protected]>

ddupg pushed a commit to ddupg/hbase that referenced this pull request Apr 20, 2020

HBASE-24183 [flakey test] replication.TestAddToSerialReplicationPeer (a…

266b6bb

…pache#1514) (apache#1526) Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: stack <[email protected]>

HBASE-24183 [flakey test] replication.TestAddToSerialReplicationPeer #1514

HBASE-24183 [flakey test] replication.TestAddToSerialReplicationPeer #1514

Uh oh!

Conversation

huaxiangsun commented Apr 14, 2020

Uh oh!

Apache-HBase commented Apr 14, 2020

Uh oh!

huaxiangsun commented Apr 15, 2020

Uh oh!

Apache9 left a comment

Choose a reason for hiding this comment

Uh oh!

Apache-HBase commented Apr 15, 2020

Uh oh!

Apache-HBase commented Apr 15, 2020

Uh oh!

saintstack commented Apr 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

huaxiangsun commented Apr 15, 2020

Uh oh!

huaxiangsun commented Apr 15, 2020

Uh oh!

Apache-HBase commented Apr 15, 2020

Uh oh!

Apache-HBase commented Apr 15, 2020

Uh oh!

Apache-HBase commented Apr 15, 2020

Uh oh!

huaxiangsun commented Apr 15, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

saintstack commented Apr 15, 2020 •

edited

Loading