HADOOP-17509. Parallelize building of dependencies

[GauthamBanasandra]

• Added -j option to parallelize the
  building of Protocol buffers and Intel
  ISA-L dependencies.

[hadoop-yetus]

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2669/1/console in case of problems.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	10m 18s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 32s		Maven dependency ordering for branch
+1 💚	shadedclient	23m 31s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 30s		Maven dependency ordering for patch
-1 ❌	hadolint	0m 4s	/diff-patch-hadolint.txt	The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
+1 💚	shellcheck	0m 0s		There were no new shellcheck issues.
+1 💚	shelldocs	0m 15s		There were no new shelldocs issues.
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	20m 6s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		72m 22s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2669/1/artifact/out/Dockerfile
GITHUB PR	#2669
Optional Tests	dupname asflicense hadolint shellcheck shelldocs
uname	Linux e334271442f7 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / b541346
Max. process+thread count	572 (vs. ulimit of 5500)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2669/1/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0 hadolint=1.11.1-0-g0e692dd
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2669/2/console in case of problems.

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	4m 48s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 16s		Maven dependency ordering for branch
+1 💚	shadedclient	13m 27s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 32s		Maven dependency ordering for patch
+1 💚	hadolint	0m 4s		There were no new hadolint issues.
+1 💚	shellcheck	0m 0s		There were no new shellcheck issues.
+1 💚	shelldocs	0m 20s		There were no new shelldocs issues.
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	12m 56s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		48m 57s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2669/2/artifact/out/Dockerfile
GITHUB PR	#2669
Optional Tests	dupname asflicense hadolint shellcheck shelldocs
uname	Linux abf3a8386921 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 9bf2ac0
Max. process+thread count	677 (vs. ulimit of 5500)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2669/2/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0 hadolint=1.11.1-0-g0e692dd
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[aajisaka]

+1. Great!

[aajisaka]

Thank you @GauthamBanasandra for your contribution!

[smengcl]

Hey @GauthamBanasandra sorry I'm late to this.

I can verify that on Ubuntu 20.04 I am able to docker build . with the change. But not on macOS 10.15.7 (should be fine for now).

[GauthamBanasandra]

Hi @smengcl ,
Are you saying that you encountered an error while doing a docker build on macOS? If so, could you please post the error here?
I don't own a mac. I'll try to see if I can spin one up on the cloud and verify.

[smengcl]

@GauthamBanasandra Yes. I just ran docker build again and it still fails, seemingly because of boost library compilation failure:

common.copy /usr/lib/libboost_wave.a
...failed updating 11 targets...
...skipped 24 targets...
...updated 16944 targets...
The command '/bin/bash -o pipefail -c mkdir -p /opt/boost-library     && curl -L https://sourceforge.net/projects/boost/files/boost/1.72.0/boost_1_72_0.tar.bz2/download > boost_1_72_0.tar.bz2     && mv boost_1_72_0.tar.bz2 /opt/boost-library     && cd /opt/boost-library     && tar --bzip2 -xf boost_1_72_0.tar.bz2     && cd /opt/boost-library/boost_1_72_0     && ./bootstrap.sh --prefix=/usr/     && ./b2 --without-python install     && cd /root     && rm -rf /opt/boost-library' returned a non-zero code: 1

This should have nothing to do with this PR. Don't worry. :)

I posted the full log here.

[GauthamBanasandra]

@smengcl I just had a look at the logs. I couldn't find the exact error but I found this - https://gist.github.com/smengcl/200668f706306031f72b8010d7c728a0#file-docker_build-log-L777-L778

I'm not sure if this is a kill signal that was sent externally. Would you happen to know the exact reason for compilation failure?

[smengcl]

@GauthamBanasandra Good find!

I wonder if the reason is Mac Docker's mem limit.
-- Yes. The build succeeded after I increased the Docker global memory limit from 3GB to 5GB (4GB doesn't work). Wonder why I haven't hit the mem limit before 🤔

Thanks for the help :D