[SPARK-27180][BUILD][YARN] Fix testing issues with yarn module in Hadoop-3 #24115

wangyum · 2019-03-16T07:28:37Z

What changes were proposed in this pull request?

Fix testing issues with yarn module in Hadoop-3:

Upgrade jersey-1 to 1.19 to fix Cause: java.lang.NoClassDefFoundError: com/sun/jersey/spi/container/servlet/ServletContainer.
Copy ServerSocketUtil from hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/ServerSocketUtil.java to fix java.lang.NoClassDefFoundError: org/apache/hadoop/net/ServerSocketUtil.
Adapte SessionHandler from jetty-9.3.25.v20180904/jetty-server/src/main/java/org/eclipse/jetty/server/session/SessionHandler.java to fix java.lang.NoSuchMethodError: org.eclipse.jetty.server.session.SessionHandler.getSessionManager()Lorg/eclipse/jetty/server/SessionManager.

How was this patch tested?

manual tests:

build/sbt yarn/test -Pyarn
build/sbt yarn/test -Phadoop-3.2 -Pyarn

build/mvn -Dtest=none -DwildcardSuites=org.apache.spark.deploy.yarn.YarnClusterSuite -pl resource-managers/yarn test -Pyarn
build/mvn -Dtest=none -DwildcardSuites=org.apache.spark.deploy.yarn.YarnClusterSuite -pl resource-managers/yarn test -Pyarn -Phadoop-3.2

wangyum · 2019-03-16T07:29:52Z

resource-managers/yarn/pom.xml

  <properties>
    <sbt.project.name>yarn</sbt.project.name>
-    <jersey-1.version>1.9</jersey-1.version>
+    <jersey-1.version>1.19</jersey-1.version>


Upgrade jersey to 1.19. otherwise:

[info] Cause: java.lang.NoClassDefFoundError: com/sun/jersey/spi/container/servlet/ServletContainer [info] at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1103) [info] at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1271) [info] at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)

wangyum · 2019-03-16T07:41:14Z

resource-managers/yarn/src/test/java/org/apache/hadoop/net/ServerSocketUtil.java

@@ -0,0 +1,132 @@
+/*


Add this class. otherwise:

[info] YarnClusterSuite: [info] org.apache.spark.deploy.yarn.YarnClusterSuite *** ABORTED *** (33 milliseconds) [info] java.lang.NoClassDefFoundError: org/apache/hadoop/net/ServerSocketUtil [info] at org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:260)

I try to do it by maven, but failed. It seems that sbt-pom-reader does not support test-jar very well.:

<dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <version>${hadoop.version}</version> <type>test-jar</type> <scope>test</scope> <exclusions> <exclusion> <groupId>*</groupId> <artifactId>*</artifactId> </exclusion> </exclusions> </dependency>

We do have other test-jar imports; some use type and some classifier. Have you tried the latter? e.g. we have

<dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-yarn-server-tests</artifactId> <classifier>tests</classifier> <scope>test</scope> </dependency>

Yes, I tried it, this can be successful on hadoop-3.2, but throws a compilation exception on hadoop-2.7(It seems exclude hadoop-common).

[error] /Users/yumwang/SPARK-20845/spark/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ApplicationMasterSuite.scala:34: Class org.apache.hadoop.conf.Configuration not found - continuing with a stub. [error] val yarnConf = new YarnConfiguration() [error]

wangyum · 2019-03-16T07:47:09Z

pom.xml

        <hadoop.version>3.1.0</hadoop.version>
        <curator.version>2.12.0</curator.version>
        <zookeeper.version>3.4.9</zookeeper.version>
+        <jetty.version>9.3.24.v20180605</jetty.version>


Since HADOOP-10075(Hadoop-3.0.0). Hadoop update it's jetty dependency to version 9.3.x. This version conflict with 9.4.x:

[info] YarnClusterSuite: [info] org.apache.spark.deploy.yarn.YarnClusterSuite *** ABORTED *** (177 milliseconds) [info] org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NoSuchMethodError: org.eclipse.jetty.server.session.SessionHandler.getSessionManager()Lorg/eclipse/jetty/server/SessionManager;

Furthermore, We have some discuss about this change:
https://issues.apache.org/jira/browse/HADOOP-16152

Hm, I'm afraid this might conflict with Java 11 compatibility, as this was updated for that reason: #22993
https://www.eclipse.org/lists/jetty-announce/msg00124.html

I wonder if this can be worked around on the Spark side, or whether it is indeed possible for Hadoop 3 to update?

We shouldn't change Spark's version of Jetty because a YARN test is failing. A Spark application will not hit the same code paths as the YARN tests (which run a YARN server).

Maybe change the version of jetty in the YARN module when testing with hadoop-3, if there's no other way to work around this.

I tried several ways but they all failed:

Use hadoop-client-minicluster instead of hadoop-yarn-server-tests: https://github.com/wangyum/spark-hadoop-client-minicluster

Change the version of jetty in the YARN module when testing with hadoop-3

Upgrade Eclipse Jetty version to 9.4.x for Hadoop: https://issues.apache.org/jira/browse/HADOOP-16152

The ideal outcome is updating Hadoop, as otherwise it seems like Hadoop 3 and Java 11 support are in conflict. Then again, if Hadoop 3.x isn't quite going to work with Java 11 for other reasons, we have larger problems. Hadoop 2.x is currently working OK with Java 11 tests (it's Hive that's the issue) so I'm kind of surprised.

How did you try to override the Jetty version in the YARN module? I'd expect that's entirely possible

I try to override the Jetty version in the YARN module by: 29e583f.
But +-org.eclipse.jetty:jetty-servlet:9.3.24.v20180605 (evicted by: 9.4.12.v20180830).

I found that adapted SessionHandler from jetty-9.3.25.v20180904 can test the YARN module in Hadoop 2 and Hadoop 3. I know this is not a good way. But it seems that this is the only way at the moment.

SparkQA · 2019-03-16T17:45:35Z

Test build #103561 has finished for PR 24115 at commit 757a09a.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class ServerSocketUtil

# Conflicts: # pom.xml

�[0m[�[0minfo�[0m] �[0m | | | | +-org.eclipse.jetty:jetty-servlet:9.3.24.v20180605 (evicted by: 9.4.12.v20180830)�[0m

SparkQA · 2019-03-30T07:05:02Z

Test build #104104 has finished for PR 24115 at commit 3c9bb7b.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class SessionHandler extends ScopedHandler

wangyum · 2019-03-30T08:36:19Z

retest this please

SparkQA · 2019-03-30T13:25:56Z

Test build #104108 has finished for PR 24115 at commit 3c9bb7b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class SessionHandler extends ScopedHandler

resource-managers/yarn/src/test/java/org/eclipse/jetty/server/SessionManager.java

SparkQA · 2019-03-31T01:33:58Z

Test build #104123 has finished for PR 24115 at commit a33fcef.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-03-31T07:05:01Z

Test build #104124 has finished for PR 24115 at commit 6a00f19.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-03-31T09:17:36Z

retest this please

SparkQA · 2019-03-31T14:14:21Z

Test build #104136 has finished for PR 24115 at commit 6a00f19.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

resource-managers/yarn/src/test/java/org/eclipse/jetty/server/SessionManager.java

srowen

Looks OK if that's what we have to do here.

srowen · 2019-04-01T03:08:05Z

dev/.rat-excludes

 kafka-source-initial-offset-version-2.1.0.bin
 kafka-source-initial-offset-future-version.bin
 vote.tmpl
+SessionManager.java


I guess we can't narrow this to the particular file in its particular directory? it's probably fine if not

Yes. Please see RAT-240 CLI: RAT excludes file does not allow to set a full path

SparkQA · 2019-04-01T04:39:12Z

Test build #104149 has finished for PR 24115 at commit 49f0d8a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2019-04-02T20:38:36Z

Merged to master

…oop-3 Fix testing issues with `yarn` module in Hadoop-3: 1. Upgrade jersey-1 to `1.19` to fix ```Cause: java.lang.NoClassDefFoundError: com/sun/jersey/spi/container/servlet/ServletContainer```. 2. Copy `ServerSocketUtil` from hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/ServerSocketUtil.java to fix ```java.lang.NoClassDefFoundError: org/apache/hadoop/net/ServerSocketUtil```. 3. Adapte `SessionHandler` from jetty-9.3.25.v20180904/jetty-server/src/main/java/org/eclipse/jetty/server/session/SessionHandler.java to fix ```java.lang.NoSuchMethodError: org.eclipse.jetty.server.session.SessionHandler.getSessionManager()Lorg/eclipse/jetty/server/SessionManager```. manual tests: ```shell build/sbt yarn/test -Pyarn build/sbt yarn/test -Phadoop-3.2 -Pyarn build/mvn -Dtest=none -DwildcardSuites=org.apache.spark.deploy.yarn.YarnClusterSuite -pl resource-managers/yarn test -Pyarn build/mvn -Dtest=none -DwildcardSuites=org.apache.spark.deploy.yarn.YarnClusterSuite -pl resource-managers/yarn test -Pyarn -Phadoop-3.2 ``` Closes apache#24115 from wangyum/hadoop3-yarn. Authored-by: Yuming Wang <[email protected]> Signed-off-by: Sean Owen <[email protected]> (cherry picked from commit 13c5c1f)

* SNAPSHOT to 3.2.2.0-1095 * [SPARK-27180][BUILD][YARN] Fix testing issues with yarn module in Hadoop-3 Fix testing issues with `yarn` module in Hadoop-3: 1. Upgrade jersey-1 to `1.19` to fix ```Cause: java.lang.NoClassDefFoundError: com/sun/jersey/spi/container/servlet/ServletContainer```. 2. Copy `ServerSocketUtil` from hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/ServerSocketUtil.java to fix ```java.lang.NoClassDefFoundError: org/apache/hadoop/net/ServerSocketUtil```. 3. Adapte `SessionHandler` from jetty-9.3.25.v20180904/jetty-server/src/main/java/org/eclipse/jetty/server/session/SessionHandler.java to fix ```java.lang.NoSuchMethodError: org.eclipse.jetty.server.session.SessionHandler.getSessionManager()Lorg/eclipse/jetty/server/SessionManager```. manual tests: ```shell build/sbt yarn/test -Pyarn build/sbt yarn/test -Phadoop-3.2 -Pyarn build/mvn -Dtest=none -DwildcardSuites=org.apache.spark.deploy.yarn.YarnClusterSuite -pl resource-managers/yarn test -Pyarn build/mvn -Dtest=none -DwildcardSuites=org.apache.spark.deploy.yarn.YarnClusterSuite -pl resource-managers/yarn test -Pyarn -Phadoop-3.2 ``` Closes apache#24115 from wangyum/hadoop3-yarn. Authored-by: Yuming Wang <[email protected]> Signed-off-by: Sean Owen <[email protected]> (cherry picked from commit 13c5c1f) * [SPARK-42463][SPARK-27180][YARN][TESTS] Clean up the third-party Java files copy introduced by SPARK-27180 introduced some third-party Java source code to solve Yarn module test failure, but maven and sbt can also test pass without them, so this pr remove these files. Clean up the third-party Java source code copy in Spark. No - Pass GitHub Actions - manual check: **Maven** ``` build/mvn clean build/mvn clean install -DskipTestes -pl resource-managers/yarn -am -Pyarn build/mvn -Dtest=none -DwildcardSuites=org.apache.spark.deploy.yarn.YarnClusterSuite -pl resource-managers/yarn test -Pyarn build/mvn test -pl resource-managers/yarn -Pyarn -Dtest.exclude.tags=org.apache.spark.tags.ExtendedLevelDBTest ``` Both `YarnClusterSuite` and full module test passed. **SBT** ``` build/sbt clean yarn/test -Pyarn -Dtest.exclude.tags=org.apache.spark.tags.ExtendedLevelDBTest ``` All tests passed. Closes apache#40052 from LuciferYang/SPARK-42463. Authored-by: yangjie01 <[email protected]> Signed-off-by: Sean Owen <[email protected]> (cherry picked from commit 64e5928) * ODP-1095: jettison 1.5.4 * Fixed version as per main across all poms --------- Co-authored-by: kravii <[email protected]> Co-authored-by: Yuming Wang <[email protected]> Co-authored-by: yangjie01 <[email protected]> Co-authored-by: Prabhjyot Singh <[email protected]>

Fix hadoop-3 yarn test

757a09a

wangyum commented Mar 16, 2019

View reviewed changes

wangyum added 4 commits March 30, 2019 09:55

Merge remote-tracking branch 'upstream/master' into hadoop3-yarn

5fb2e13

# Conflicts: # pom.xml

Add test jetty version

29e583f

Add test jetty version, bit it does not work:

4058b2a

�[0m[�[0minfo�[0m] �[0m | | | | +-org.eclipse.jetty:jetty-servlet:9.3.24.v20180605 (evicted by: 9.4.12.v20180830)�[0m

Adapted SessionHandler from jetty-9.3.25.v20180904

3c9bb7b

srowen reviewed Mar 30, 2019

View reviewed changes

resource-managers/yarn/src/test/java/org/eclipse/jetty/server/SessionManager.java Show resolved Hide resolved

Update License

a33fcef

revert to apache License

6a00f19

srowen requested changes Mar 31, 2019

View reviewed changes

resource-managers/yarn/src/test/java/org/eclipse/jetty/server/SessionManager.java Show resolved Hide resolved

Fix LICENSE

49f0d8a

srowen approved these changes Apr 1, 2019

View reviewed changes

srowen closed this in 13c5c1f Apr 2, 2019

[SPARK-27180][BUILD][YARN] Fix testing issues with yarn module in Hadoop-3 #24115

[SPARK-27180][BUILD][YARN] Fix testing issues with yarn module in Hadoop-3 #24115

Uh oh!

Conversation

wangyum commented Mar 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wangyum Mar 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 16, 2019

Uh oh!

SparkQA commented Mar 30, 2019

Uh oh!

wangyum commented Mar 30, 2019

Uh oh!

SparkQA commented Mar 30, 2019

Uh oh!

Uh oh!

SparkQA commented Mar 31, 2019

Uh oh!

SparkQA commented Mar 31, 2019

Uh oh!

wangyum commented Mar 31, 2019

Uh oh!

SparkQA commented Mar 31, 2019

Uh oh!

Uh oh!

srowen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 1, 2019

Uh oh!

srowen commented Apr 2, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

wangyum commented Mar 16, 2019 •

edited

Loading

wangyum Mar 16, 2019 •

edited

Loading