HDDS-2164 : om.db.checkpoints is getting filling up fast.

[avijayanhwx]

What changes were proposed in this pull request?

• Fixed the underlying issue which was not causing the OM DB checkpoints to be cleaned up on OM host whenever recon or some other client requests DB checkpoint.
• Changed the 2 step process in the OM DB checkpoint servlet (Creating a tar file for OM DB + Writing to outputstream) to a single step process (Writing the compressed tar file as a stream directly to output stream).

How was this patch tested?

Manually tested on live cluster with a Ozone Manager instance and Recon. Verified that the OM DB checkpoints that are being generated in OM host are being cleaned up irrespective of whether the download to Recon succeeds or not.

Apache JIRA

https://issues.apache.org/jira/browse/HDDS-2164

[avijayanhwx]

/label ozone

[avijayanhwx]

@swagle / @vivekratnavel

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	0	Docker mode activated.
-1	patch	9	#1536 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem	Report/Notes
GITHUB PR	#1536
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/1/console
versions	git=2.17.1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[vivekratnavel]

/retest

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	77	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 4 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	70	Maven dependency ordering for branch
-1	mvninstall	37	hadoop-hdds in trunk failed.
-1	mvninstall	42	hadoop-ozone in trunk failed.
-1	compile	19	hadoop-hdds in trunk failed.
-1	compile	13	hadoop-ozone in trunk failed.
+1	checkstyle	60	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	946	branch has no errors when building and testing our client artifacts.
-1	javadoc	22	hadoop-hdds in trunk failed.
-1	javadoc	16	hadoop-ozone in trunk failed.
0	spotbugs	1037	Used deprecated FindBugs config; considering switching to SpotBugs.
-1	findbugs	31	hadoop-hdds in trunk failed.
-1	findbugs	17	hadoop-ozone in trunk failed.
-0	patch	1065	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
0	mvndep	26	Maven dependency ordering for patch
-1	mvninstall	31	hadoop-hdds in the patch failed.
-1	mvninstall	34	hadoop-ozone in the patch failed.
-1	compile	20	hadoop-hdds in the patch failed.
-1	compile	16	hadoop-ozone in the patch failed.
-1	javac	20	hadoop-hdds in the patch failed.
-1	javac	16	hadoop-ozone in the patch failed.
+1	checkstyle	24	The patch passed checkstyle in hadoop-hdds
+1	checkstyle	27	hadoop-ozone: The patch generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	793	patch has no errors when building and testing our client artifacts.
-1	javadoc	19	hadoop-hdds in the patch failed.
-1	javadoc	16	hadoop-ozone in the patch failed.
-1	findbugs	30	hadoop-hdds in the patch failed.
-1	findbugs	16	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	24	hadoop-hdds in the patch failed.
-1	unit	23	hadoop-ozone in the patch failed.
+1	asflicense	29	The patch does not generate ASF License warnings.
		2594

Subsystem	Report/Notes
Docker	Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/artifact/out/Dockerfile
GITHUB PR	#1536
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux fa8a6c6e8a26 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 13b427f
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/artifact/out/branch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/artifact/out/branch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/artifact/out/patch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/artifact/out/patch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/artifact/out/patch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/artifact/out/patch-findbugs-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/testReport/
Max. process+thread count	335 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/integration-test hadoop-ozone/ozone-manager hadoop-ozone/recon U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/2/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[avijayanhwx]

/retest

[anuengineer]

Can you please check if the rat failures are real? thx

[nandakumar131]

@anuengineer Rat failures are real, but they are not related to this PR.
They were introduced in HDDS-2193 and fixed in HDDS-1146.

[nandakumar131]

@avijayanhwx findbugs violations are related to this change, can you take a look at it?

[anuengineer]

/retest

[avijayanhwx]

@avijayanhwx findbugs violations are related to this change, can you take a look at it?

My latest patch should fix the findbugs issues.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	71	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 4 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	64	Maven dependency ordering for branch
-1	mvninstall	34	hadoop-hdds in trunk failed.
-1	mvninstall	33	hadoop-ozone in trunk failed.
-1	compile	19	hadoop-hdds in trunk failed.
-1	compile	13	hadoop-ozone in trunk failed.
+1	checkstyle	49	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	931	branch has no errors when building and testing our client artifacts.
-1	javadoc	19	hadoop-hdds in trunk failed.
-1	javadoc	16	hadoop-ozone in trunk failed.
0	spotbugs	1017	Used deprecated FindBugs config; considering switching to SpotBugs.
-1	findbugs	29	hadoop-hdds in trunk failed.
-1	findbugs	17	hadoop-ozone in trunk failed.
-0	patch	1045	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
0	mvndep	23	Maven dependency ordering for patch
-1	mvninstall	32	hadoop-hdds in the patch failed.
-1	mvninstall	34	hadoop-ozone in the patch failed.
-1	compile	21	hadoop-hdds in the patch failed.
-1	compile	16	hadoop-ozone in the patch failed.
-1	javac	21	hadoop-hdds in the patch failed.
-1	javac	16	hadoop-ozone in the patch failed.
+1	checkstyle	51	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	786	patch has no errors when building and testing our client artifacts.
-1	javadoc	19	hadoop-hdds in the patch failed.
-1	javadoc	16	hadoop-ozone in the patch failed.
-1	findbugs	29	hadoop-hdds in the patch failed.
-1	findbugs	16	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	24	hadoop-hdds in the patch failed.
-1	unit	23	hadoop-ozone in the patch failed.
+1	asflicense	29	The patch does not generate ASF License warnings.
		2538

Subsystem	Report/Notes
Docker	Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/artifact/out/Dockerfile
GITHUB PR	#1536
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 13dcca6818b6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / aa24add
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/artifact/out/branch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/artifact/out/branch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/artifact/out/patch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/artifact/out/patch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/artifact/out/patch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/artifact/out/patch-findbugs-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/testReport/
Max. process+thread count	307 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/integration-test hadoop-ozone/ozone-manager hadoop-ozone/recon U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1536/3/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.