Skip to content

[HUDI-3195] optimize spark3 pom and modify build command#4538

Merged
xushiyan merged 1 commit intoapache:masterfrom
YannByron:master_3195
Jan 8, 2022
Merged

[HUDI-3195] optimize spark3 pom and modify build command#4538
xushiyan merged 1 commit intoapache:masterfrom
YannByron:master_3195

Conversation

@YannByron
Copy link
Copy Markdown
Contributor

with Spark3.0:
-- mvn clean install -DskipTests=true -P "scala-2.12,spark3.0.x"

with Spark3.1:
-- mvn clean install -DskipTests=true -P "scala-2.12,spark3.1.x"

with Spark3.2
-- mvn clean install -DskipTests=true -P "scala-2.12,spark3"

@YannByron
Copy link
Copy Markdown
Contributor Author

@leesf @xushiyan please review this with high priority.

Copy link
Copy Markdown
Member

@xushiyan xushiyan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Can you update the README as well? Also please remember need a separate PR for 0.10.1 where we don't have spark 3.2 support

@hudi-bot
Copy link
Copy Markdown
Collaborator

hudi-bot commented Jan 7, 2022

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@YannByron
Copy link
Copy Markdown
Contributor Author

YannByron commented Jan 8, 2022

LGTM. Can you update the README as well? Also please remember need a separate PR for 0.10.1 where we don't have spark 3.2 support

If you don't patch #4488 and this to 0.10.1, the previous commands for spark3.0.x and spark3.1.x works.

# Build against Spark 3.2.0 (default build shipped with the public jars)
mvn clean package -DskipTests -Dspark3

# Build against Spark 3.1.2
mvn clean package -DskipTests -Dspark3.1.x

# Build against Spark 3.0.3
mvn clean package -DskipTests -Dspark3.0.x

README is OK, not need to be changed.

@xushiyan xushiyan self-assigned this Jan 8, 2022
@xushiyan xushiyan merged commit 03a83ff into apache:master Jan 8, 2022
@vinishjail97 vinishjail97 mentioned this pull request Jan 24, 2022
5 tasks
vingov pushed a commit to vingov/hudi that referenced this pull request Jan 26, 2022
liusenhua pushed a commit to liusenhua/hudi that referenced this pull request Mar 1, 2022
vingov pushed a commit to vingov/hudi that referenced this pull request Apr 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants