TEZ-4553. Default task scheduler to DagAwareTaskScheduler to avoid hang in TEZ-3535

[jteagles]

No description provided.

[tez-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	26m 18s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	6m 42s	Maven dependency ordering for branch
+1 💚	mvninstall	11m 43s	master passed
+1 💚	compile	3m 46s	master passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚	compile	3m 30s	master passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
+1 💚	checkstyle	2m 0s	master passed
+1 💚	javadoc	2m 21s	master passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚	javadoc	2m 0s	master passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
+0 🆗	spotbugs	6m 3s	Used deprecated FindBugs config; considering switching to SpotBugs.
+0 🆗	findbugs	0m 45s	branch/tez-ui no findbugs output file (findbugsXml.xml)
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 15s	Maven dependency ordering for patch
+1 💚	mvninstall	5m 35s	the patch passed
+1 💚	compile	3m 49s	the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚	javac	3m 49s	the patch passed
+1 💚	compile	3m 33s	the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
+1 💚	javac	3m 33s	the patch passed
+1 💚	checkstyle	1m 35s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
+1 💚	javadoc	2m 9s	the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚	javadoc	2m 4s	the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
+0 🆗	findbugs	0m 48s	tez-ui has no data from findbugs
		_ Other Tests _
+1 💚	unit	2m 30s	tez-api in the patch passed.
+1 💚	unit	1m 45s	tez-ui in the patch passed.
-1 ❌	unit	10m 2s	root in the patch failed.
+1 💚	asflicense	1m 12s	The patch does not generate ASF License warnings.
		112m 2s

Reason	Tests
Failed junit tests	tez.runtime.library.conf.TestUnorderedPartitionedKVEdgeConfig

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-347/1/artifact/out/Dockerfile
GITHUB PR	#347
Optional Tests	dupname asflicense javac javadoc unit xml compile spotbugs findbugs checkstyle
uname	Linux 95a339b00ea8 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / f080031
Default Java	Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
unit	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-347/1/artifact/out/patch-unit-root.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-347/1/testReport/
Max. process+thread count	2101 (vs. ulimit of 5500)
modules	C: tez-api tez-ui . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-347/1/console
versions	git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[ayushtkn]

Hi @jteagles

• Is the maven change intentional? related to this?
• UI changes?
• How come changing the default be a solution to a problem, like avoiding Hang.
• Can you describe the problem in the description, X -> Y -> Hence hang
• You mentioned: TEZ-3535, it has ticket linked which talks about tez job getting hang, by any chance you aren't missing that? TEZ-3491

[jteagles]

Hi @jteagles

• Is the maven change intentional? related to this?
• UI changes?
• How come changing the default be a solution to a problem, like avoiding Hang.
• Can you describe the problem in the description, X -> Y -> Hence hang
• You mentioned: TEZ-3535, it has ticket linked which talks about tez job getting hang, by any chance you aren't missing that? TEZ-3491

This jira is changing the default scheduler away from the one with the known hang in TEZ-3535 and to the one added by TEZ-3491. I've been running the TEZ-3491 scheduler DagAwareTaskScheduler for 8 years.
Fixed the extra commit changes

[tez-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	24m 13s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+1 💚	mvninstall	14m 36s	master passed
+1 💚	compile	0m 33s	master passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚	compile	0m 32s	master passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
+1 💚	checkstyle	1m 18s	master passed
+1 💚	javadoc	0m 48s	master passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚	javadoc	0m 37s	master passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
+0 🆗	spotbugs	1m 42s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 40s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 23s	the patch passed
+1 💚	compile	0m 25s	the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚	javac	0m 25s	the patch passed
+1 💚	compile	0m 22s	the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
+1 💚	javac	0m 22s	the patch passed
+1 💚	checkstyle	0m 13s	the patch passed
+1 💚	whitespace	0m 1s	The patch has no whitespace issues.
+1 💚	javadoc	0m 24s	the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚	javadoc	0m 27s	the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
+1 💚	findbugs	1m 4s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 19s	tez-api in the patch passed.
+1 💚	asflicense	0m 16s	The patch does not generate ASF License warnings.
		51m 10s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-347/2/artifact/out/Dockerfile
GITHUB PR	#347
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux e9a2ad51aff2 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / f080031
Default Java	Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-347/2/testReport/
Max. process+thread count	335 (vs. ulimit of 5500)
modules	C: tez-api U: tez-api
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-347/2/console
versions	git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[ayushtkn]

LGTM

[jteagles]

Thanks, @ayushtkn.