HBASE-24795 : RegionMover to deal with unknown region while (un)loading #2172

virajjasani · 2020-07-29T18:47:09Z

RegionMover to ignore move failures for split/merged regions with ack mode
Refactor MoveWithAck and MoveWithoutAck as high level classes
UT for RegionMover gracefully handling split/merged regions while loading regions and throwing failure while loading offline regions.

tedyu · 2020-07-29T20:04:48Z

hbase-server/src/main/java/org/apache/hadoop/hbase/util/MoveWithAck.java

+        if (!sameServer) {
+          break;
+        }
+        Thread.sleep(100);


Maybe the interval can be a bit longer since maxWait is in seconds.

Agree, this was old code anyways, we can update this to 1 sec at least.

hbase-server/src/main/java/org/apache/hadoop/hbase/util/MoveWithAck.java

tedyu · 2020-07-29T20:12:43Z

hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java

-        LOG.error("Got Exception From Thread While moving region " + e.getMessage(), e);
-        throw e;
+        if (e.getCause() instanceof UnknownRegionException) {
+          LOG.debug("Ignore unknown region, it might have been split/merged.");


Should this be logged at more prominent level ?

sure, let me use info

tedyu · 2020-07-30T07:25:09Z

+1

busbey · 2020-07-30T16:52:03Z

is there a particular reason we're conflating formatting changes here?

are the qa failures related?

hbase-server/src/main/java/org/apache/hadoop/hbase/util/MoveWithoutAck.java

hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java

hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestRegionMover.java

virajjasani · 2020-07-30T17:04:36Z

is there a particular reason we're conflating formatting changes here?

are the qa failures related?

QA failures are not relevant, tried locally and both builds have one common test failure: TestReplicationEndpointWithMultipleAsyncWAL, works quite well locally with the patch.

I just thought of refactoring MoveWithAck and MoveWithoutAck as high level classes since RegionMover is quite huge.

busbey · 2020-07-30T17:05:36Z

I just thought of refactoring MoveWithAck and MoveWithoutAck as high level classes since RegionMover is quite huge.

that's a good idea; please make sure there is a note about that in the commit message body.

virajjasani

Incorporated review comments

hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java

hbase-server/src/main/java/org/apache/hadoop/hbase/util/MoveWithAck.java

busbey · 2020-07-31T00:22:54Z

hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java

      ExecutorService moveRegionsPool = Executors.newFixedThreadPool(this.maxthreads);
      List<Future<Boolean>> taskList = new ArrayList<>();
      int serverIndex = 0;
      while (counter < regionsToMove.size()) {


nit: this should be a for each loop on the regionsToMove list

hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestRegionMover2.java

virajjasani · 2020-07-31T11:43:33Z

Precommit run is not getting scheduled, waiting logs are present for long time:

Still waiting to schedule task
Waiting for next available executor on ‘Hadoop’

I tried manual trigger but the run is still not scheduled.

busbey · 2020-07-31T15:07:26Z

Precommit run is not getting scheduled, waiting logs are present for long time

it's backlog waiting for executors to open up on the new ci host:

https://ci-hadoop.apache.org/label/Hadoop/load-statistics

looks like the queue size started to climb yesterday around midnight CDT. peaked about 5 hours ago at ~300. is now sitting around 280.

hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestRegionMover1.java

hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestRegionMover2.java

Apache-HBase · 2020-08-01T01:19:12Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 31s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 25s	master passed
+1 💚	checkstyle	1m 5s	master passed
+1 💚	spotbugs	1m 57s	master passed
-0 ⚠️	patch	2m 6s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 23s	the patch passed
-0 ⚠️	checkstyle	1m 3s	hbase-server: The patch generated 5 new + 12 unchanged - 2 fixed = 17 total (was 14)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	11m 8s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1.
+1 💚	spotbugs	2m 7s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		32m 0s

Subsystem	Report/Notes
Docker	Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2172/5/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#2172
Optional Tests	dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
uname	Linux 2d279defa66d 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `7c4d66a`
checkstyle	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2172/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count	94 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2172/5/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-08-01T03:40:16Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 11s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 42s	master passed
+1 💚	compile	0m 55s	master passed
+1 💚	shadedjars	5m 30s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 37s	master passed
-0 ⚠️	patch	6m 20s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 29s	the patch passed
+1 💚	compile	0m 54s	the patch passed
+1 💚	javac	0m 54s	the patch passed
+1 💚	shadedjars	5m 38s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 37s	the patch passed
		_ Other Tests _
+1 💚	unit	147m 13s	hbase-server in the patch passed.
		172m 3s

Subsystem	Report/Notes
Docker	Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2172/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#2172
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux b062846db5cc 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `7c4d66a`
Default Java	1.8.0_232
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2172/5/testReport/
Max. process+thread count	4576 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2172/5/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-08-01T03:50:20Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 38s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	6m 5s	master passed
+1 💚	compile	1m 27s	master passed
+1 💚	shadedjars	8m 0s	branch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 52s	hbase-server in master failed.
-0 ⚠️	patch	9m 2s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	5m 2s	the patch passed
+1 💚	compile	1m 23s	the patch passed
+1 💚	javac	1m 23s	the patch passed
+1 💚	shadedjars	8m 2s	patch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 47s	hbase-server in the patch failed.
		_ Other Tests _
+1 💚	unit	147m 33s	hbase-server in the patch passed.
		182m 1s

Subsystem	Report/Notes
Docker	Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2172/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#2172
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux d89b95a5e414 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `7c4d66a`
Default Java	2020-01-14
javadoc	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2172/5/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
javadoc	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2172/5/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2172/5/testReport/
Max. process+thread count	4190 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2172/5/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-08-01T06:43:28Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 41s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 1s	master passed
+1 💚	checkstyle	1m 22s	master passed
+1 💚	spotbugs	2m 19s	master passed
-0 ⚠️	patch	2m 27s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 54s	the patch passed
-0 ⚠️	checkstyle	1m 19s	hbase-server: The patch generated 3 new + 14 unchanged - 0 fixed = 17 total (was 14)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	13m 17s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1.
+1 💚	spotbugs	2m 16s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		37m 44s

Subsystem	Report/Notes
Docker	Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2172/6/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#2172
Optional Tests	dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
uname	Linux daff207dec36 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `7c4d66a`
checkstyle	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2172/6/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count	94 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2172/6/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-08-01T09:00:47Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 29s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 57s	master passed
+1 💚	compile	0m 59s	master passed
+1 💚	shadedjars	6m 16s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 40s	master passed
-0 ⚠️	patch	7m 7s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 35s	the patch passed
+1 💚	compile	0m 54s	the patch passed
+1 💚	javac	0m 54s	the patch passed
+1 💚	shadedjars	5m 24s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 35s	the patch passed
		_ Other Tests _
+1 💚	unit	150m 19s	hbase-server in the patch passed.
		175m 7s

Subsystem	Report/Notes
Docker	Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2172/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#2172
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux b4daf8bcc314 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `7c4d66a`
Default Java	1.8.0_232
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2172/6/testReport/
Max. process+thread count	4063 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2172/6/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-08-01T09:06:56Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 42s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	6m 0s	master passed
+1 💚	compile	1m 21s	master passed
+1 💚	shadedjars	7m 0s	branch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 47s	hbase-server in master failed.
-0 ⚠️	patch	7m 58s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 50s	the patch passed
+1 💚	compile	1m 21s	the patch passed
+1 💚	javac	1m 21s	the patch passed
+1 💚	shadedjars	7m 23s	patch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 46s	hbase-server in the patch failed.
		_ Other Tests _
+1 💚	unit	148m 7s	hbase-server in the patch passed.
		180m 59s

Subsystem	Report/Notes
Docker	Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2172/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#2172
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux b054d8c5bb71 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `7c4d66a`
Default Java	2020-01-14
javadoc	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2172/6/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
javadoc	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2172/6/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2172/6/testReport/
Max. process+thread count	4485 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2172/6/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

* RegionMover to ignore move failures for split/merged regions with ack mode * Refactor MoveWithAck and MoveWithoutAck as high level classes * UT for RegionMover gracefully handling split/merged regions while loading regions and throwing failure while loading offline regions Closes #2172 Signed-off-by: Sean Busbey <[email protected]> Signed-off-by: Ted Yu <[email protected]>

* RegionMover to ignore move failures for split/merged regions with ack mode * Refactor MoveWithAck and MoveWithoutAck as high level classes * UT for RegionMover gracefully handling split/merged regions while loading regions and throwing failure while loading offline regions Closes apache#2172 Signed-off-by: Sean Busbey <[email protected]> Signed-off-by: Ted Yu <[email protected]>

HBASE-24795 : RegionMover to deal with unknown region while (un)loading

c83e789

This comment has been minimized.

Sign in to view

tedyu reviewed Jul 29, 2020

View reviewed changes

This comment has been minimized.

Sign in to view

update based on review

51a74d6

This comment has been minimized.

Sign in to view

busbey requested changes Jul 30, 2020

View reviewed changes

update based on review

bdec03c

virajjasani commented Jul 30, 2020

View reviewed changes

hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

busbey requested changes Jul 31, 2020

View reviewed changes

This comment has been minimized.

Sign in to view

addressing review

433cb91

busbey requested changes Jul 31, 2020

View reviewed changes

moved table creation in Before and deletion in After

feced89

virajjasani requested a review from busbey July 31, 2020 17:12

busbey approved these changes Jul 31, 2020

View reviewed changes

virajjasani closed this in ad7caf7 Aug 3, 2020

virajjasani deleted the HBASE-24795-trunk branch August 4, 2020 18:06

HBASE-24795 : RegionMover to deal with unknown region while (un)loading #2172

HBASE-24795 : RegionMover to deal with unknown region while (un)loading #2172

Uh oh!

Conversation

virajjasani commented Jul 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

tedyu Jul 29, 2020

Choose a reason for hiding this comment

Uh oh!

virajjasani Jul 30, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tedyu Jul 29, 2020

Choose a reason for hiding this comment

Uh oh!

virajjasani Jul 30, 2020

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

tedyu commented Jul 30, 2020

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

busbey commented Jul 30, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

virajjasani commented Jul 30, 2020

Uh oh!

busbey commented Jul 30, 2020

Uh oh!

virajjasani left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

busbey Jul 31, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

virajjasani commented Jul 31, 2020

Uh oh!

busbey commented Jul 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Apache-HBase commented Aug 1, 2020

Uh oh!

Apache-HBase commented Aug 1, 2020

Uh oh!

Apache-HBase commented Aug 1, 2020

Uh oh!

Apache-HBase commented Aug 1, 2020

Uh oh!

Apache-HBase commented Aug 1, 2020

Uh oh!

Apache-HBase commented Aug 1, 2020

Uh oh!

Reviewers

Assignees

virajjasani commented Jul 29, 2020 •

edited

Loading

busbey commented Jul 31, 2020 •

edited

Loading