Skip to content

[SPARK-33079][TESTS] Replace the existing Maven job for Scala 2.13 in Github Actions with SBT job #29958

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sarutak wants to merge 6 commits into from
Closed

[SPARK-33079][TESTS] Replace the existing Maven job for Scala 2.13 in Github Actions with SBT job #29958

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
8e45bc8
Added GitHub Actions job for building Spark with SBT for Scala 2.13.
sarutak Oct 6, 2020
5a22c38
Fix build_and_test.yml to consider .m2, test:compile and local cache …
sarutak Oct 6, 2020
5dec1ff
Merge branch 'master' of https://github.com/apache/spark into add-sbt…
sarutak Oct 6, 2020
47a4806
Replaced Maven test with SBT test for Scala 2.13.
sarutak Oct 9, 2020
91d1066
Merge branch 'master' of https://github.com/apache/spark into add-sbt…
sarutak Oct 9, 2020
30b2ddd
Changed the build goal to compile and test:compile.
sarutak Oct 9, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 6 additions & 10 deletions .github/workflows/build_and_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -300,22 +300,18 @@ jobs:
steps:
- name: Checkout Spark repository
uses: actions/checkout@v2
- name: Cache Maven local repository
- name: Cache Ivy local repository
uses: actions/cache@v2
with:
path: ~/.m2/repository
key: scala-213-maven-${{ hashFiles('**/pom.xml') }}
path: ~/.ivy2/cache
key: scala-213-ivy-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
scala-213-maven-
scala-213-ivy-
- name: Install Java 11
uses: actions/setup-java@v1
with:
java-version: 11
- name: Build with Maven
- name: Build with SBT
run: |
export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
export MAVEN_CLI_OPTS="--no-transfer-progress"
mkdir -p ~/.m2
./dev/change-scala-version.sh 2.13
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 -Pscala-2.13 install
rm -rf ~/.m2/repository/org/apache/spark
./build/sbt -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Djava.version=11 -Pscala-2.13 compile test:compile
Copy link
Member

@dongjoon-hyun dongjoon-hyun Oct 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, we are going to remove packaging and installation (Maven) here?

Copy link
Contributor

@LuciferYang LuciferYang Oct 10, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can Maven build and SBT build be executed serially in one GitHub Action？

Copy link
Member

@HyukjinKwon HyukjinKwon Oct 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it wouldn't matter too much. If a job is finished quicker the next job can be triggered

Copy link
Contributor

@LuciferYang LuciferYang Oct 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that 2.13 build action runs faster than other actions. Can we consider this ?

Copy link
Member

@HyukjinKwon HyukjinKwon Oct 10, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The number of job is limited in github actions instead of each entire workflow run IIRC.

2 changes: 1 addition & 1 deletion ...kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ class KinesisBackedBlockRDD[T: ClassTag](
require(_blockIds.length == arrayOfseqNumberRanges.length,
"Number of blockIds is not equal to the number of sequence number ranges")

override def isValid(): Boolean = true
override def isValid: Boolean = true
Copy link
Contributor

@LuciferYang LuciferYang Oct 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change related to the current pr?

Copy link
Member Author

@sarutak sarutak Oct 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. empty parameter list causes compilation error if we build for Scala 2.13 with SBT.
This is what I'd like to detect by CI.


override def getPartitions: Array[Partition] = {
Array.tabulate(_blockIds.length) { i =>
Expand Down