TEZ-4488: TaskSchedulerManager might not be initialized when the first DAG comes #280

abstractdog · 2023-04-06T06:26:01Z

No description provided.

rbalamohan · 2023-04-11T05:27:32Z

tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java

        taskSchedulerManager);
    addIfServiceDependency(taskSchedulerManager, clientRpcServer);

+    appMasterReadinessService = createAppMasterReadinessService();


Issue is happening as a corner case, as the taskscheduler could be taking longer to init. In this case, possibly "LlapTaskSchedulerService" is taking time to init due to ZK dependency.

Instead of adding one more dependency, why not throw RuntimeException in TaskSchedulerManager::getTaskSchedulerClassName (i.e until all taskSchedulers are initied w.r.t appContext) ?

abstractdog · 2023-04-11T12:43:59Z

@rbalamohan: I'm afraid we have no other choice than blocking the first DAG here, as throwing an exception from TaskSchedulerManager::getTaskSchedulerClassName would lead to a similar exception that I'm trying to avoid here
this solution simply blocks the very first DAG, and then lets the rest of them run after a simple AtomicBoolean check

…t DAG comes

rbalamohan · 2023-04-20T10:16:41Z

Lgtm. +1 on the recent patch with timeout.

abstractdog · 2023-04-20T10:27:16Z

Lgtm. +1 on the recent patch with timeout.

thanks @rbalamohan , waiting for green tests before merging

tez-yetus · 2023-04-20T10:46:33Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 36s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ master Compile Tests _
+0 🆗	mvndep	6m 0s	Maven dependency ordering for branch
+1 💚	mvninstall	10m 0s	master passed
+1 💚	compile	1m 18s	master passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu122.04
+1 💚	compile	1m 12s	master passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~22.04-b09
+1 💚	checkstyle	1m 13s	master passed
+1 💚	javadoc	1m 21s	master passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu122.04
+1 💚	javadoc	1m 12s	master passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~22.04-b09
+0 🆗	spotbugs	1m 16s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 38s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 9s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 46s	the patch passed
+1 💚	compile	0m 52s	the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu122.04
+1 💚	javac	0m 52s	the patch passed
+1 💚	compile	0m 43s	the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~22.04-b09
+1 💚	javac	0m 43s	the patch passed
+1 💚	checkstyle	0m 31s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 46s	the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu122.04
+1 💚	javadoc	0m 45s	the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~22.04-b09
+1 💚	findbugs	2m 5s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 14s	tez-api in the patch passed.
+1 💚	unit	5m 4s	tez-dag in the patch passed.
+1 💚	asflicense	0m 22s	The patch does not generate ASF License warnings.
		40m 59s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-280/8/artifact/out/Dockerfile
GITHUB PR	#280
JIRA Issue	TEZ-4488
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux 42a5134d6175 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `2fe3c46`
Default Java	Private Build-1.8.0_362-8u362-ga-0ubuntu1~22.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu122.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~22.04-b09
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-280/8/testReport/
Max. process+thread count	416 (vs. ulimit of 5500)
modules	C: tez-api tez-dag U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-280/8/console
versions	git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Change-Id: Ic81998359bd372b47b7e9699382aa2c80b814cae

tez-yetus · 2023-04-20T11:13:50Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 37s	Docker mode activated.
-1 ❌	patch	0m 5s	#280 does not apply to master. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/TEZ/How+to+Contribute+to+Tez for help.

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-280/9/artifact/out/Dockerfile
GITHUB PR	#280
JIRA Issue	TEZ-4488
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-280/9/console
versions	git=2.34.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

…t DAG comes (apache#280) (Laszlo Bodor reviewed by Rajesh Balamohan) (cherry picked from commit 249e017)

…t DAG comes (apache#280) (Laszlo Bodor reviewed by Rajesh Balamohan) (cherry picked from commit 249e017) (cherry picked from commit 41343dc)

…t DAG comes (apache#280) (Laszlo Bodor reviewed by Rajesh Balamohan) (cherry picked from commit 249e017) (cherry picked from commit 41343dc) (cherry picked from commit 6d81766)

…n the first DAG comes (apache#280) (Laszlo Bodor reviewed by Rajesh Balamohan) (#23) (cherry picked from commit 249e017) (cherry picked from commit 41343dc) (cherry picked from commit 6d81766) Co-authored-by: Bodor Laszlo <[email protected]>