Skip to content

[SPARK-27467][BUILD] Upgrade Maven to 3.6.1 #24481

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dongjoon-hyun wants to merge 3 commits into from
Closed

[SPARK-27467][BUILD] Upgrade Maven to 3.6.1 #24481

dongjoon-hyun wants to merge 3 commits into from

Conversation

@dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Apr 28, 2019 •
edited
Loading

What changes were proposed in this pull request?

This PR aims to upgrade Maven to 3.6.1 to bring JDK9+ related patches like MNG-6506. For the full release note, please see the following.

This was committed and reverted due to AppVeyor failure. It turns out that the root cause is PATH issue. With the updated AppVeyor script, it passed.

https://ci.appveyor.com/project/ApacheSoftwareFoundation/spark/builds/24273412

How was this patch tested?

Pass the Jenkins and AppVoyer

dongjoon-hyun and others added 2 commits April 27, 2019 21:07
@dongjoon-hyun
[SPARK-27467][BUILD][TEST-MAVEN] Upgrade Maven to 3.6.1
47c25a4 
## What changes were proposed in this pull request?

This PR aims to upgrade Maven to 3.6.1 to bring JDK9+ related patches like [MNG-6506](https://issues.apache.org/jira/browse/MNG-6506). For the full release note, please see the following.
- https://maven.apache.org/docs/3.6.1/release-notes.html

## How was this patch tested?

Pass the Jenkins with `[test-maven]` tag.

Closes #24377 from dongjoon-hyun/SPARK-27467.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
@wangyum @dongjoon-hyun
[SPARK-27467][FOLLOW-UP][BUILD] Upgrade Maven to 3.6.1 in AppVeyor an…
1eeeb5e 
…d Doc

## What changes were proposed in this pull request?

Update the `docs/building-spark.md`. Otherwise:
```
mvn package -DskipTests=true
...
[INFO] --- maven-enforcer-plugin:3.0.0-M2:enforce (enforce-versions)  spark-parent_2.12 ---
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.RequireMavenVersion failed with message:
Detected Maven Version: 3.6.0 is not in the allowed range 3.6.1.
...
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M2:enforce (enforce-versions) on project spark-parent_2.12: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]
[ERROR]
...
```

## How was this patch tested?
Just test `https://archive.apache.org/dist/maven/maven-3/3.6.1/binaries/apache-maven-3.6.1-bin.zip` is avilable.

Closes #24477 from wangyum/SPARK-27467.

Authored-by: Yuming Wang <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Apr 28, 2019

cc @HyukjinKwon , @wangyum

@dongjoon-hyun dongjoon-hyun changed the title [WIP][SPARK-27467][BUILD][R] Upgrade Maven to 3.6.1 [WIP][SPARK-27467][BUILD] Upgrade Maven to 3.6.1 Apr 28, 2019
@HyukjinKwon
Copy link
Member

HyukjinKwon commented Apr 28, 2019

I asked this case to AppVeyor - will share the updates if I get. I will probably be able to help take a look further in 2 days.

srowen
srowen approved these changes Apr 28, 2019
View reviewed changes
docs/building-spark.md

The Maven-based build is the build of reference for Apache Spark.
Building Spark using Maven requires Maven 3.6.0 and Java 8.
Building Spark using Maven requires Maven 3.6.1 and Java 8.
Copy link
Member

@srowen srowen Apr 28, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It probably doesn't actually require 3.6.1 over 3.6.0, but this is fine. I think it does require 3.6.x though.

Copy link
Member

@wangyum wangyum Apr 28, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will throw exception if it's not 3.6.1:

mvn package -DskipTests=true
...
[INFO] --- maven-enforcer-plugin:3.0.0-M2:enforce (enforce-versions) @ spark-parent_2.12 ---
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.RequireMavenVersion failed with message:
Detected Maven Version: 3.6.0 is not in the allowed range 3.6.1.
...
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M2:enforce (enforce-versions) on project spark-parent_2.12: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]
[ERROR]
...

Copy link
Member

@HyukjinKwon HyukjinKwon Apr 28, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea.. so problem is here. I don't know why it makes the build failed in Appveyor

@srowen
Copy link
Member

srowen commented Apr 28, 2019

Yeah you need to use ./build/mvn or else install 3.6.1 locally. That's always been the case.

@SparkQA
Copy link

SparkQA commented Apr 28, 2019

Test build #104974 has finished for PR 24481 at commit 1eeeb5e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun dongjoon-hyun reopened this May 2, 2019
@HyukjinKwon
Copy link
Member

HyukjinKwon commented May 3, 2019

Haven't got any information from AppVeyor - looks they don't know what's going on too ...

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented May 3, 2019

Thank you so much for informing that, @HyukjinKwon .

One strange thing in Today's run is that Maven 3.6.1 is downloaded and added into path, but 3.6.0 is used.

WARNING] Rule 0: org.apache.maven.plugins.enforcer.RequireMavenVersion failed with message:
884Detected Maven Version: 3.6.0 is not in the allowed range 3.6.1.
885[INFO] ------------------------------------------------------------------------
886[INFO] Reactor Summary for Spark Project Parent POM 3.0.0-SNAPSHOT:
887

@dongjoon-hyun dongjoon-hyun changed the title [WIP][SPARK-27467][BUILD] Upgrade Maven to 3.6.1 [SPARK-27467][BUILD] Upgrade Maven to 3.6.1 May 3, 2019
dongjoon-hyun
dongjoon-hyun commented May 3, 2019
View reviewed changes
dev/appveyor-install-dependencies.ps1

# add maven to environment variables
$env:Path += ";$tools\apache-maven-$mavenVer\bin"
$env:PATH = "$tools\apache-maven-$mavenVer\bin;" + $env:PATH
Copy link
Member Author

@dongjoon-hyun dongjoon-hyun May 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixes the AppVoyer failure.

@SparkQA
Copy link

SparkQA commented May 3, 2019

Test build #105094 has finished for PR 24481 at commit 2ba886f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented May 3, 2019 •
edited
Loading

Thank you, @HyukjinKwon , @wangyum , @srowen , @felixcheung .

Since this passed the AppVeyor and Jenkins, I merged this to master back.

@dongjoon-hyun dongjoon-hyun deleted the SPARK-R branch May 3, 2019 03:04
@HyukjinKwon
Copy link
Member

HyukjinKwon commented May 3, 2019

Thanks, @dongjoon-hyun

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented May 3, 2019

Thank YOU, @HyukjinKwon . Sorry for missing AppVeyor. :)

@HyukjinKwon HyukjinKwon mentioned this pull request Aug 19, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wangyum wangyum wangyum left review comments

@HyukjinKwon HyukjinKwon HyukjinKwon left review comments

@srowen srowen srowen approved these changes

@felixcheung felixcheung felixcheung approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@dongjoon-hyun @HyukjinKwon @srowen @SparkQA @wangyum @felixcheung