TEZ-4381: Speed up TestSecureShuffle

[mark-bathori]

I've changed the recommended TEZ_AM_TASK_MAX_FAILED_ATTEMPTS configuration value and made some test run to see the run times. The default value for maximum failed attempts per task was 4 so I did 4 different run for each test where I reduced the config value by 1 in each run.

[sslInCluster:true, resultWithTezSSL:0, resultWithoutTezSSL:1, asyncHttp:false, taskMaxFailed:4] 1min 25sec
[sslInCluster:true, resultWithTezSSL:0, resultWithoutTezSSL:1, asyncHttp:false, taskMaxFailed:3] 1min 0sec
[sslInCluster:true, resultWithTezSSL:0, resultWithoutTezSSL:1, asyncHttp:false, taskMaxFailed:2] 48sec 655ms
[sslInCluster:true, resultWithTezSSL:0, resultWithoutTezSSL:1, asyncHttp:false, taskMaxFailed:1] 53sec 320ms

[sslInCluster:true, resultWithTezSSL:0, resultWithoutTezSSL:1, asyncHttp:true, taskMaxFailed:4] 7min 59sec
[sslInCluster:true, resultWithTezSSL:0, resultWithoutTezSSL:1, asyncHttp:true, taskMaxFailed:3] 5min 55sec
[sslInCluster:true, resultWithTezSSL:0, resultWithoutTezSSL:1, asyncHttp:true, taskMaxFailed:2] 3min 47sec
[sslInCluster:true, resultWithTezSSL:0, resultWithoutTezSSL:1, asyncHttp:true, taskMaxFailed:1] 1min 41sec

[sslInCluster:false, resultWithTezSSL:1, resultWithoutTezSSL:0, asyncHttp:true, taskMaxFailed:4] 51sec 613ms
[sslInCluster:false, resultWithTezSSL:1, resultWithoutTezSSL:0, asyncHttp:true, taskMaxFailed:3] 44sec 417ms
[sslInCluster:false, resultWithTezSSL:1, resultWithoutTezSSL:0, asyncHttp:true, taskMaxFailed:2] 41sec 378ms
[sslInCluster:false, resultWithTezSSL:1, resultWithoutTezSSL:0, asyncHttp:true, taskMaxFailed:1] 39sec 172ms

[sslInCluster:false, resultWithTezSSL:1, resultWithoutTezSSL:0, asyncHttp:false, taskMaxFailed:4] 51sec 87ms
[sslInCluster:false, resultWithTezSSL:1, resultWithoutTezSSL:0, asyncHttp:false, taskMaxFailed:3] 47sec 374ms
[sslInCluster:false, resultWithTezSSL:1, resultWithoutTezSSL:0, asyncHttp:false, taskMaxFailed:2] 55sec 452ms
[sslInCluster:false, resultWithTezSSL:1, resultWithoutTezSSL:0, asyncHttp:false, taskMaxFailed:1] 39sec 101ms

[tez-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 9s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ master Compile Tests _
+1 💚	mvninstall	13m 36s	master passed
+1 💚	compile	0m 29s	master passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	compile	0m 27s	master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 54s	master passed
+1 💚	javadoc	0m 35s	master passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 22s	master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+0 🆗	spotbugs	1m 7s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 4s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 19s	the patch passed
+1 💚	compile	0m 20s	the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javac	0m 20s	the patch passed
+1 💚	compile	0m 17s	the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 17s	the patch passed
+1 💚	checkstyle	0m 10s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 14s	the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 12s	the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	findbugs	0m 40s	the patch passed
		_ Other Tests _
+1 💚	unit	34m 41s	tez-tests in the patch passed.
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		56m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-180/1/artifact/out/Dockerfile
GITHUB PR	#180
JIRA Issue	TEZ-4381
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux 2687692243b1 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / 4e3eb95
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-180/1/testReport/
Max. process+thread count	1151 (vs. ulimit of 5500)
modules	C: tez-tests U: tez-tests
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-180/1/console
versions	git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[abstractdog]

double-check full unit test
before: [INFO] Total time: 10:53 min (Wall Clock)
after: [INFO] Total time: 03:47 min (Wall Clock)

merging this, thanks @mbathori-cloudera for the patch and testing