Skip to content

Commit ee96460

Browse files
witgopwendell
witgo
authored and
pwendell
committed
Improved build configuration
1, Fix SPARK-1441: compile spark core error with hadoop 0.23.x 2, Fix SPARK-1491: maven hadoop-provided profile fails to build 3, Fix org.scala-lang: * ,org.apache.avro:* inconsistent versions dependency 4, A modified on the sql/catalyst/pom.xml,sql/hive/pom.xml,sql/core/pom.xml (Four spaces formatted into two spaces) Author: witgo <[email protected]> Closes #480 from witgo/format_pom and squashes the following commits: 03f652f [witgo] review commit b452680 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom bee920d [witgo] revert fix SPARK-1629: Spark Core missing commons-lang dependence 7382a07 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom 6902c91 [witgo] fix SPARK-1629: Spark Core missing commons-lang dependence 0da4bc3 [witgo] merge master d1718ed [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom e345919 [witgo] add avro dependency to yarn-alpha 77fad08 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom 62d0862 [witgo] Fix org.scala-lang: * inconsistent versions dependency 1a162d7 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom 934f24d [witgo] review commit cf46edc [witgo] exclude jruby 06e7328 [witgo] Merge branch 'SparkBuild' into format_pom 99464d2 [witgo] fix maven hadoop-provided profile fails to build 0c6c1fc [witgo] Fix compile spark core error with hadoop 0.23.x 6851bec [witgo] Maintain consistent SparkBuild.scala, pom.xml (cherry picked from commit 030f2c2) Conflicts: sql/catalyst/pom.xml sql/core/pom.xml sql/hive/pom.xml
1 parent 42ba706 commit ee96460

File tree

23 files changed

+295
-466
lines changed

23 files changed

+295
-466
lines changed

‎bagel/pom.xml‎

Lines changed: 0 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -31,20 +31,6 @@
3131
<name>Spark Project Bagel</name>
3232
<url>http://spark.apache.org/</url>
3333

34-
<profiles>
35-
<profile>
36-
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
37-
a Hadoop 0.23.X issue -->
38-
<id>yarn-alpha</id>
39-
<dependencies>
40-
<dependency>
41-
<groupId>org.apache.avro</groupId>
42-
<artifactId>avro</artifactId>
43-
</dependency>
44-
</dependencies>
45-
</profile>
46-
</profiles>
47-
4834
<dependencies>
4935
<dependency>
5036
<groupId>org.apache.spark</groupId>

‎core/pom.xml‎

Lines changed: 0 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -30,19 +30,6 @@
3030
<packaging>jar</packaging>
3131
<name>Spark Project Core</name>
3232
<url>http://spark.apache.org/</url>
33-
<!-- SPARK-1121: Adds an explicit dependency on Avro to work around a Hadoop 0.23.X issue -->
34-
<profiles>
35-
<profile>
36-
<id>yarn-alpha</id>
37-
<dependencies>
38-
<dependency>
39-
<groupId>org.apache.avro</groupId>
40-
<artifactId>avro</artifactId>
41-
</dependency>
42-
</dependencies>
43-
</profile>
44-
</profiles>
45-
4633
<dependencies>
4734
<dependency>
4835
<groupId>org.apache.hadoop</groupId>
@@ -147,15 +134,6 @@
147134
<groupId>org.json4s</groupId>
148135
<artifactId>json4s-jackson_${scala.binary.version}</artifactId>
149136
<version>3.2.6</version>
150-
<!-- see also exclusion for lift-json; this is necessary since it depends on
151-
scala-library and scalap 2.10.0, but we use 2.10.4, and only override
152-
scala-library -->
153-
<exclusions>
154-
<exclusion>
155-
<groupId>org.scala-lang</groupId>
156-
<artifactId>scalap</artifactId>
157-
</exclusion>
158-
</exclusions>
159137
</dependency>
160138
<dependency>
161139
<groupId>colt</groupId>

‎docs/building-with-maven.md‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -39,17 +39,23 @@ For Apache Hadoop versions 1.x, Cloudera CDH MRv1, and other Hadoop versions wit
3939
# Cloudera CDH 4.2.0 with MapReduce v1
4040
$ mvn -Dhadoop.version=2.0.0-mr1-cdh4.2.0 -DskipTests clean package
4141

42-
For Apache Hadoop 2.x, 0.23.x, Cloudera CDH MRv2, and other Hadoop versions with YARN, you should enable the "yarn-alpha" or "yarn" profile and set the "hadoop.version", "yarn.version" property:
42+
# Apache Hadoop 0.23.x
43+
$ mvn -Phadoop-0.23 -Dhadoop.version=0.23.7 -DskipTests clean package
44+
45+
For Apache Hadoop 2.x, 0.23.x, Cloudera CDH MRv2, and other Hadoop versions with YARN, you can enable the "yarn-alpha" or "yarn" profile and set the "hadoop.version", "yarn.version" property. Note that Hadoop 0.23.X requires a special `-Phadoop-0.23` profile:
4346

4447
# Apache Hadoop 2.0.5-alpha
4548
$ mvn -Pyarn-alpha -Dhadoop.version=2.0.5-alpha -Dyarn.version=2.0.5-alpha -DskipTests clean package
4649

4750
# Cloudera CDH 4.2.0 with MapReduce v2
4851
$ mvn -Pyarn-alpha -Dhadoop.version=2.0.0-cdh4.2.0 -Dyarn.version=2.0.0-cdh4.2.0 -DskipTests clean package
4952

50-
# Apache Hadoop 2.2.X ( e.g. 2.2.0 as below ) and newer
53+
# Apache Hadoop 2.2.X (e.g. 2.2.0 as below) and newer
5154
$ mvn -Pyarn -Dhadoop.version=2.2.0 -Dyarn.version=2.2.0 -DskipTests clean package
5255

56+
# Apache Hadoop 0.23.x
57+
$ mvn -Pyarn-alpha -Phadoop-0.23 -Dhadoop.version=0.23.7 -Dyarn.version=0.23.7 -DskipTests clean package
58+
5359
## Spark Tests in Maven ##
5460

5561
Tests are run by default via the [ScalaTest Maven plugin](http://www.scalatest.org/user_guide/using_the_scalatest_maven_plugin). Some of the require Spark to be packaged first, so always run `mvn package` with `-DskipTests` the first time. You can then run the tests with `mvn -Dhadoop.version=... test`.

‎examples/pom.xml‎

Lines changed: 4 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -31,20 +31,6 @@
3131
<name>Spark Project Examples</name>
3232
<url>http://spark.apache.org/</url>
3333

34-
<profiles>
35-
<profile>
36-
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
37-
a Hadoop 0.23.X issue -->
38-
<id>yarn-alpha</id>
39-
<dependencies>
40-
<dependency>
41-
<groupId>org.apache.avro</groupId>
42-
<artifactId>avro</artifactId>
43-
</dependency>
44-
</dependencies>
45-
</profile>
46-
</profiles>
47-
4834
<dependencies>
4935
<dependency>
5036
<groupId>org.apache.spark</groupId>
@@ -124,6 +110,10 @@
124110
<groupId>commons-logging</groupId>
125111
<artifactId>commons-logging</artifactId>
126112
</exclusion>
113+
<exclusion>
114+
<groupId>org.jruby</groupId>
115+
<artifactId>jruby-complete</artifactId>
116+
</exclusion>
127117
</exclusions>
128118
</dependency>
129119
<dependency>

‎external/flume/pom.xml‎

Lines changed: 0 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -31,20 +31,6 @@
3131
<name>Spark Project External Flume</name>
3232
<url>http://spark.apache.org/</url>
3333

34-
<profiles>
35-
<profile>
36-
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
37-
a Hadoop 0.23.X issue -->
38-
<id>yarn-alpha</id>
39-
<dependencies>
40-
<dependency>
41-
<groupId>org.apache.avro</groupId>
42-
<artifactId>avro</artifactId>
43-
</dependency>
44-
</dependencies>
45-
</profile>
46-
</profiles>
47-
4834
<dependencies>
4935
<dependency>
5036
<groupId>org.apache.spark</groupId>

‎external/kafka/pom.xml‎

Lines changed: 0 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -31,20 +31,6 @@
3131
<name>Spark Project External Kafka</name>
3232
<url>http://spark.apache.org/</url>
3333

34-
<profiles>
35-
<profile>
36-
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
37-
a Hadoop 0.23.X issue -->
38-
<id>yarn-alpha</id>
39-
<dependencies>
40-
<dependency>
41-
<groupId>org.apache.avro</groupId>
42-
<artifactId>avro</artifactId>
43-
</dependency>
44-
</dependencies>
45-
</profile>
46-
</profiles>
47-
4834
<dependencies>
4935
<dependency>
5036
<groupId>org.apache.spark</groupId>

‎external/mqtt/pom.xml‎

Lines changed: 0 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -31,20 +31,6 @@
3131
<name>Spark Project External MQTT</name>
3232
<url>http://spark.apache.org/</url>
3333

34-
<profiles>
35-
<profile>
36-
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
37-
a Hadoop 0.23.X issue -->
38-
<id>yarn-alpha</id>
39-
<dependencies>
40-
<dependency>
41-
<groupId>org.apache.avro</groupId>
42-
<artifactId>avro</artifactId>
43-
</dependency>
44-
</dependencies>
45-
</profile>
46-
</profiles>
47-
4834
<dependencies>
4935
<dependency>
5036
<groupId>org.apache.spark</groupId>

‎external/twitter/pom.xml‎

Lines changed: 0 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -31,20 +31,6 @@
3131
<name>Spark Project External Twitter</name>
3232
<url>http://spark.apache.org/</url>
3333

34-
<profiles>
35-
<profile>
36-
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
37-
a Hadoop 0.23.X issue -->
38-
<id>yarn-alpha</id>
39-
<dependencies>
40-
<dependency>
41-
<groupId>org.apache.avro</groupId>
42-
<artifactId>avro</artifactId>
43-
</dependency>
44-
</dependencies>
45-
</profile>
46-
</profiles>
47-
4834
<dependencies>
4935
<dependency>
5036
<groupId>org.apache.spark</groupId>

‎external/zeromq/pom.xml‎

Lines changed: 0 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -31,20 +31,6 @@
3131
<name>Spark Project External ZeroMQ</name>
3232
<url>http://spark.apache.org/</url>
3333

34-
<profiles>
35-
<profile>
36-
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
37-
a Hadoop 0.23.X issue -->
38-
<id>yarn-alpha</id>
39-
<dependencies>
40-
<dependency>
41-
<groupId>org.apache.avro</groupId>
42-
<artifactId>avro</artifactId>
43-
</dependency>
44-
</dependencies>
45-
</profile>
46-
</profiles>
47-
4834
<dependencies>
4935
<dependency>
5036
<groupId>org.apache.spark</groupId>

‎graphx/pom.xml‎

Lines changed: 0 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -31,20 +31,6 @@
3131
<name>Spark Project GraphX</name>
3232
<url>http://spark.apache.org/</url>
3333

34-
<profiles>
35-
<profile>
36-
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
37-
a Hadoop 0.23.X issue -->
38-
<id>yarn-alpha</id>
39-
<dependencies>
40-
<dependency>
41-
<groupId>org.apache.avro</groupId>
42-
<artifactId>avro</artifactId>
43-
</dependency>
44-
</dependencies>
45-
</profile>
46-
</profiles>
47-
4834
<dependencies>
4935
<dependency>
5036
<groupId>org.apache.spark</groupId>

0 commit comments

Comments
 (0)