YARN-10822. Containers going from New to Scheduled transition for kil… #3632

minni31 · 2021-11-08T18:04:16Z

…led container on recovery

Description of PR

How was this patch tested?

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

…led container on recovery

hadoop-yetus · 2021-11-08T20:06:11Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 58s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 7s		trunk passed
+1 💚	compile	1m 28s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 23s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	0m 32s		trunk passed
+1 💚	mvnsite	0m 44s		trunk passed
+1 💚	javadoc	0m 38s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 32s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	1m 31s		trunk passed
+1 💚	shadedclient	23m 4s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 38s		the patch passed
+1 💚	compile	1m 36s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 36s		the patch passed
+1 💚	compile	1m 40s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	1m 40s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 26s		the patch passed
+1 💚	mvnsite	0m 37s		the patch passed
+1 💚	javadoc	0m 33s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 31s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	1m 44s		the patch passed
+1 💚	shadedclient	23m 50s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	22m 39s		hadoop-yarn-server-nodemanager in the patch passed.
+1 💚	asflicense	0m 30s		The patch does not generate ASF License warnings.
		120m 37s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3632/1/artifact/out/Dockerfile
GITHUB PR	#3632
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux f66fcf996109 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ffb53fa`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3632/1/testReport/
Max. process+thread count	521 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3632/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

goiri · 2021-11-11T18:07:23Z

...java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java

-        return ContainerState.SCHEDULED;
-      } else if (container.recoveredAsKilled &&
-          container.recoveredStatus == RecoveredContainerStatus.REQUESTED) {
+      } else if (container.recoveredAsKilled && (


This condition is getting a little obfuscated.
Probably we should have a static method with this condition and explaining what's the sequence.

goiri · 2021-11-11T18:08:28Z

...org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManagerRecovery.java

+    Container c = containers.get(cid);
+    assertEquals(ContainerState.DONE, c.getContainerState());
+    app = context.getApplications().get(appId);
+    Thread.sleep(1000);


Can we wait for something specific instead of sleeping blindly?

Since ContainerState has been verified with the expected state from StateTransition, sleep is not required.

...org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManagerRecovery.java

.../test/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMMemoryStateStoreService.java

hadoop-yetus · 2021-11-23T08:39:19Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 40s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 43s		trunk passed
+1 💚	compile	1m 47s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 33s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	0m 40s		trunk passed
+1 💚	mvnsite	0m 55s		trunk passed
+1 💚	javadoc	0m 48s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 38s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	1m 46s		trunk passed
+1 💚	shadedclient	22m 43s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 43s		the patch passed
+1 💚	compile	1m 35s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 35s		the patch passed
+1 💚	compile	1m 25s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	1m 25s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 25s		the patch passed
+1 💚	mvnsite	0m 44s		the patch passed
+1 💚	javadoc	0m 35s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 32s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	1m 40s		the patch passed
+1 💚	shadedclient	22m 34s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	23m 46s		hadoop-yarn-server-nodemanager in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		121m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3632/2/artifact/out/Dockerfile
GITHUB PR	#3632
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 36ebdc4cc086 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `bd1b164`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3632/2/testReport/
Max. process+thread count	546 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3632/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-11-23T09:00:57Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 0s		trunk passed
+1 💚	compile	1m 37s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 31s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	0m 32s		trunk passed
+1 💚	mvnsite	0m 49s		trunk passed
+1 💚	javadoc	0m 48s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 35s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	1m 37s		trunk passed
+1 💚	shadedclient	24m 13s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 41s		the patch passed
+1 💚	compile	1m 33s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 33s		the patch passed
+1 💚	compile	1m 28s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	1m 28s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 26s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 1 new + 164 unchanged - 0 fixed = 165 total (was 164)
+1 💚	mvnsite	0m 43s		the patch passed
+1 💚	javadoc	0m 36s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 29s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	1m 38s		the patch passed
+1 💚	shadedclient	23m 49s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	23m 46s		hadoop-yarn-server-nodemanager in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		123m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3632/3/artifact/out/Dockerfile
GITHUB PR	#3632
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux d8f859b06fe6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `37074c8`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3632/3/testReport/
Max. process+thread count	543 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3632/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

goiri · 2021-11-23T17:16:22Z

...java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java

    }
+
+    static boolean isContainerRecoveredAsKilled(ContainerImpl container)
+    {


the { should go in the previous line

...java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java

hadoop-yetus · 2021-11-30T14:31:37Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 2s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	39m 37s		trunk passed
+1 💚	compile	1m 44s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 34s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	0m 42s		trunk passed
+1 💚	mvnsite	0m 55s		trunk passed
+1 💚	javadoc	0m 47s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 40s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	1m 51s		trunk passed
+1 💚	shadedclient	27m 32s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 45s		the patch passed
+1 💚	compile	1m 44s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 44s		the patch passed
+1 💚	compile	1m 31s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	1m 31s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 26s		the patch passed
+1 💚	mvnsite	0m 42s		the patch passed
+1 💚	javadoc	0m 34s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 30s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	1m 42s		the patch passed
+1 💚	shadedclient	27m 26s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	24m 20s		hadoop-yarn-server-nodemanager in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		136m 19s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3632/4/artifact/out/Dockerfile
GITHUB PR	#3632
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 90a18840e9c9 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `89f941f`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3632/4/testReport/
Max. process+thread count	582 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3632/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

apache#3632)

YARN-10822. Containers going from New to Scheduled transition for kil…

ffb53fa

…led container on recovery

goiri reviewed Nov 11, 2021

View reviewed changes

Minni Mittal added 2 commits November 23, 2021 12:05

Added assert statement in UT

bd1b164

move functionality to static method

37074c8

goiri reviewed Nov 23, 2021

View reviewed changes

Adding sequence in method def

89f941f

goiri approved these changes Dec 6, 2021

View reviewed changes

goiri merged commit 87abc43 into apache:trunk Feb 1, 2022

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

YARN-10822. Containers going from New to Scheduled transition for kil… (

49430da

apache#3632)

symious pushed a commit to symious/hadoop that referenced this pull request Nov 21, 2024

YARN-10822. Containers going from New to Scheduled transition for kil… (

1cf78a0

apache#3632)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

YARN-10822. Containers going from New to Scheduled transition for kil… #3632

YARN-10822. Containers going from New to Scheduled transition for kil… #3632

Uh oh!

minni31 commented Nov 8, 2021

Uh oh!

hadoop-yetus commented Nov 8, 2021

Uh oh!

goiri Nov 11, 2021

Uh oh!

goiri Nov 11, 2021

Uh oh!

minni31 Nov 23, 2021

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Nov 23, 2021

Uh oh!

hadoop-yetus commented Nov 23, 2021

Uh oh!

goiri Nov 23, 2021

Uh oh!

Uh oh!

hadoop-yetus commented Nov 30, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

YARN-10822. Containers going from New to Scheduled transition for kil… #3632

YARN-10822. Containers going from New to Scheduled transition for kil… #3632

Uh oh!

Conversation

minni31 commented Nov 8, 2021

Description of PR

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented Nov 8, 2021

Uh oh!

goiri Nov 11, 2021

Choose a reason for hiding this comment

Uh oh!

goiri Nov 11, 2021

Choose a reason for hiding this comment

Uh oh!

minni31 Nov 23, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Nov 23, 2021

Uh oh!

hadoop-yetus commented Nov 23, 2021

Uh oh!

goiri Nov 23, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hadoop-yetus commented Nov 30, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants