YARN-11081. TestYarnConfigurationFields consistently keeps failing #4027

virajjasani · 2022-02-24T07:36:19Z

Description of PR

TestYarnConfigurationFields consistently keeps failing:

Error 
Messageclass org.apache.hadoop.yarn.conf.YarnConfiguration has 1 variables missing in yarn-default.xml Entries:   yarn.scheduler.app-placement-allocator.class expected:<0> but was:<1>Stacktracejava.lang.AssertionError: class org.apache.hadoop.yarn.conf.YarnConfiguration has 1 variables missing in yarn-default.xml Entries:   yarn.scheduler.app-placement-allocator.class expected:<0> but was:<1>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.apache.hadoop.conf.TestConfigurationFieldsBase.testCompareConfigurationClassAgainstXml(TestConfigurationFieldsBase.java:493)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)

How was this patch tested?

Locally

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?

hadoop-yetus · 2022-02-24T09:11:13Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 46s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	33m 36s		trunk passed
+1 💚	compile	0m 46s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	compile	0m 38s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 20s		trunk passed
+1 💚	mvnsite	0m 46s		trunk passed
+1 💚	javadoc	0m 41s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 34s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	2m 10s		trunk passed
+1 💚	shadedclient	24m 25s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 41s		the patch passed
+1 💚	compile	0m 44s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javac	0m 44s		the patch passed
+1 💚	compile	0m 35s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 35s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 15s		the patch passed
+1 💚	mvnsite	0m 40s		the patch passed
+1 💚	javadoc	0m 32s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 29s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	2m 6s		the patch passed
+1 💚	shadedclient	22m 27s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 56s		hadoop-yarn-api in the patch passed.
+1 💚	asflicense	0m 33s		The patch does not generate ASF License warnings.
		93m 34s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4027/1/artifact/out/Dockerfile
GITHUB PR	#4027
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 224aaf6f6225 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `07e2694`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4027/1/testReport/
Max. process+thread count	692 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4027/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

aajisaka

I think we should document the parameter in yarn-default.xml.

virajjasani · 2022-03-03T15:10:13Z

Thanks @aajisaka. Based on the ApplicationPlacementAllocatorFactory, it looks like the code has sufficient null check to handle missing property value:

    try {
      if (appPlacementAllocatorName == null) {
        policyClass = ApplicationSchedulingConfig.DEFAULT_APPLICATION_PLACEMENT_TYPE_CLASS;
      } else {
        policyClass = Class.forName(appPlacementAllocatorName);
      }
    } catch (ClassNotFoundException e) {
      policyClass = ApplicationSchedulingConfig.DEFAULT_APPLICATION_PLACEMENT_TYPE_CLASS;
    }

Moreover, I just checked latest 5-6 configs that are added to configurationPrefixToSkipCompare in the UT and none are present in yarn-default.xml.

aajisaka · 2022-03-03T15:52:39Z

The unit test is to guarantee that all the YARN parameter is documented in yarn-default.xml. We should not add any parameter to configurationPrefixToSkipCompare without some specific reason (e.g., deprecated parameter).

virajjasani · 2022-03-04T12:07:08Z

We should not add any parameter to configurationPrefixToSkipCompare without some specific reason (e.g., deprecated parameter).

I see, the configs that I saw were deprecated ones. Thanks for the info. Addressed the comment in the latest commit.

hadoop-yetus · 2022-03-05T13:32:56Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 28s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 53s		Maven dependency ordering for branch
+1 💚	mvninstall	23m 0s		trunk passed
+1 💚	compile	10m 29s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	compile	8m 58s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 37s		trunk passed
+1 💚	mvnsite	3m 5s		trunk passed
+1 💚	javadoc	2m 41s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	2m 50s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 4s		trunk passed
+1 💚	shadedclient	21m 20s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 18s		the patch passed
+1 💚	compile	9m 54s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javac	9m 54s		the patch passed
+1 💚	compile	8m 57s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	8m 57s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 38s		the patch passed
+1 💚	mvnsite	2m 53s		the patch passed
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	javadoc	2m 33s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	2m 30s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 30s		the patch passed
+1 💚	shadedclient	21m 38s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 11s		hadoop-yarn-api in the patch passed.
+1 💚	unit	4m 59s		hadoop-yarn-common in the patch passed.
+1 💚	unit	98m 52s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 49s		The patch does not generate ASF License warnings.
		271m 19s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4027/3/artifact/out/Dockerfile
GITHUB PR	#4027
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux 190e9ae6b671 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `1dbff9d`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4027/3/testReport/
Max. process+thread count	975 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4027/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-03-05T13:33:29Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 27s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 54s		Maven dependency ordering for branch
+1 💚	mvninstall	23m 31s		trunk passed
+1 💚	compile	10m 33s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	compile	8m 57s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 35s		trunk passed
+1 💚	mvnsite	3m 4s		trunk passed
+1 💚	javadoc	2m 52s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	2m 51s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 58s		trunk passed
+1 💚	shadedclient	21m 28s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 26s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 21s		the patch passed
+1 💚	compile	9m 44s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javac	9m 44s		the patch passed
+1 💚	compile	8m 54s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	8m 54s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 30s		the patch passed
+1 💚	mvnsite	2m 54s		the patch passed
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	javadoc	2m 28s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	2m 33s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 28s		the patch passed
+1 💚	shadedclient	21m 25s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 7s		hadoop-yarn-api in the patch passed.
+1 💚	unit	4m 56s		hadoop-yarn-common in the patch passed.
+1 💚	unit	99m 8s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 53s		The patch does not generate ASF License warnings.
		271m 53s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4027/4/artifact/out/Dockerfile
GITHUB PR	#4027
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux 5f3f7040d857 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `1dbff9d`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4027/4/testReport/
Max. process+thread count	943 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4027/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

aajisaka

+1. Thank you @virajjasani

…pache#4027) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

YARN-11081. TestYarnConfigurationFields consistently keeps failing

07e2694

aajisaka requested changes Mar 3, 2022

View reviewed changes

addressing review

1dbff9d

virajjasani requested a review from aajisaka March 5, 2022 16:53

aajisaka approved these changes Mar 5, 2022

View reviewed changes

aajisaka merged commit 56d8071 into apache:trunk Mar 8, 2022

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

YARN-11081. TestYarnConfigurationFields consistently keeps failing (a…

5ae2e18

…pache#4027) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

YARN-11081. TestYarnConfigurationFields consistently keeps failing #4027

YARN-11081. TestYarnConfigurationFields consistently keeps failing #4027

Uh oh!

virajjasani commented Feb 24, 2022

Uh oh!

hadoop-yetus commented Feb 24, 2022

Uh oh!

aajisaka left a comment

Uh oh!

virajjasani commented Mar 3, 2022

Uh oh!

aajisaka commented Mar 3, 2022

Uh oh!

virajjasani commented Mar 4, 2022

Uh oh!

hadoop-yetus commented Mar 5, 2022

Uh oh!

hadoop-yetus commented Mar 5, 2022

Uh oh!

aajisaka left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

YARN-11081. TestYarnConfigurationFields consistently keeps failing #4027

YARN-11081. TestYarnConfigurationFields consistently keeps failing #4027

Uh oh!

Conversation

virajjasani commented Feb 24, 2022

Description of PR

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented Feb 24, 2022

Uh oh!

aajisaka left a comment

Choose a reason for hiding this comment

Uh oh!

virajjasani commented Mar 3, 2022

Uh oh!

aajisaka commented Mar 3, 2022

Uh oh!

virajjasani commented Mar 4, 2022

Uh oh!

hadoop-yetus commented Mar 5, 2022

Uh oh!

hadoop-yetus commented Mar 5, 2022

Uh oh!

aajisaka left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants