Skip to content

[SPARK-40893][INFRA] Upgrade to use setup-java v3 #38366

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
LuciferYang wants to merge 5 commits into from
Closed

[SPARK-40893][INFRA] Upgrade to use setup-java v3 #38366

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3b4cdb9
upgrade
LuciferYang Oct 21, 2022
25c27d3
add distribution: 'zulu'
LuciferYang Oct 21, 2022
47d63e5
add check-latest: true
LuciferYang Oct 21, 2022
eea16b9
Merge branch 'apache:master' into setup-java-v3
LuciferYang Oct 23, 2022
2ec30fb
Merge branch 'apache:master' into setup-java-v3
LuciferYang Oct 24, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions .github/workflows/benchmark.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,9 +105,11 @@ jobs:
run: cd tpcds-kit/tools && make OS=LINUX
- name: Install Java ${{ github.event.inputs.jdk }}
if: steps.cache-tpcds-sf-1.outputs.cache-hit != 'true'
uses: actions/setup-java@v1
uses: actions/setup-java@v3
with:
distribution: 'zulu'
java-version: ${{ github.event.inputs.jdk }}
check-latest: true
Copy link
Member

@Yikun Yikun Oct 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any specifc reason to use check-latest?

This might cause use latest java without cache, then build might slower (more time to download latest java).

https://github.com/actions/setup-java#check-latest

Copy link
Contributor Author

@LuciferYang LuciferYang Oct 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any specifc reason to use check-latest?

This might cause use latest java without cache, then build might slower (more time to download latest java).

https://github.com/actions/setup-java#check-latest

Ok, I will close this one, for check-latest: true because I have not finished reading the document, so it is temporarily set to true

- name: Generate TPC-DS (SF=1) table data
if: steps.cache-tpcds-sf-1.outputs.cache-hit != 'true'
run: build/sbt "sql/Test/runMain org.apache.spark.sql.GenTPCDSData --dsdgenDir `pwd`/tpcds-kit/tools --location `pwd`/tpcds-sf-1 --scaleFactor 1 --numPartitions 1 --overwrite"
Expand Down Expand Up @@ -156,9 +158,11 @@ jobs:
restore-keys: |
benchmark-coursier-${{ github.event.inputs.jdk }}
- name: Install Java ${{ github.event.inputs.jdk }}
uses: actions/setup-java@v1
uses: actions/setup-java@v3
with:
distribution: 'zulu'
java-version: ${{ github.event.inputs.jdk }}
check-latest: true
- name: Cache TPC-DS generated data
if: contains(github.event.inputs.class, 'TPCDSQueryBenchmark') || contains(github.event.inputs.class, '*')
id: cache-tpcds-sf-1
Expand Down
36 changes: 27 additions & 9 deletions .github/workflows/build_and_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,9 +227,11 @@ jobs:
restore-keys: |
${{ matrix.java }}-${{ matrix.hadoop }}-coursier-
- name: Install Java ${{ matrix.java }}
uses: actions/setup-java@v1
uses: actions/setup-java@v3
with:
distribution: 'zulu'
java-version: ${{ matrix.java }}
check-latest: true
- name: Install Python 3.8
uses: actions/setup-python@v2
# We should install one Python that is higher then 3+ for SQL and Yarn because:
Expand Down Expand Up @@ -384,9 +386,11 @@ jobs:
restore-keys: |
pyspark-coursier-
- name: Install Java ${{ matrix.java }}
uses: actions/setup-java@v1
uses: actions/setup-java@v3
with:
distribution: 'zulu'
java-version: ${{ matrix.java }}
check-latest: true
- name: List Python packages (Python 3.9, PyPy3)
run: |
python3.9 -m pip list
Expand Down Expand Up @@ -473,9 +477,11 @@ jobs:
restore-keys: |
sparkr-coursier-
- name: Install Java ${{ inputs.java }}
uses: actions/setup-java@v1
uses: actions/setup-java@v3
with:
distribution: 'zulu'
java-version: ${{ inputs.java }}
check-latest: true
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
run: |
Expand Down Expand Up @@ -597,9 +603,11 @@ jobs:
cd docs
bundle install
- name: Install Java 8
uses: actions/setup-java@v1
uses: actions/setup-java@v3
with:
distribution: 'zulu'
java-version: 8
check-latest: true
- name: Scala linter
run: ./dev/lint-scala
- name: Java linter
Expand Down Expand Up @@ -664,9 +672,11 @@ jobs:
restore-keys: |
java${{ matrix.java }}-maven-
- name: Install Java ${{ matrix.java }}
uses: actions/setup-java@v1
uses: actions/setup-java@v3
with:
distribution: 'zulu'
java-version: ${{ matrix.java }}
check-latest: true
- name: Build with Maven
run: |
export MAVEN_OPTS="-Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
Expand Down Expand Up @@ -713,9 +723,11 @@ jobs:
restore-keys: |
scala-213-coursier-
- name: Install Java 8
uses: actions/setup-java@v1
uses: actions/setup-java@v3
with:
distribution: 'zulu'
java-version: 8
check-latest: true
- name: Build with SBT
run: |
./dev/change-scala-version.sh 2.13
Expand Down Expand Up @@ -761,9 +773,11 @@ jobs:
restore-keys: |
tpcds-coursier-
- name: Install Java 8
uses: actions/setup-java@v1
uses: actions/setup-java@v3
with:
distribution: 'zulu'
java-version: 8
check-latest: true
- name: Cache TPC-DS generated data
id: cache-tpcds-sf-1
uses: actions/cache@v2
Expand Down Expand Up @@ -864,9 +878,11 @@ jobs:
restore-keys: |
docker-integration-coursier-
- name: Install Java 8
uses: actions/setup-java@v1
uses: actions/setup-java@v3
with:
distribution: 'zulu'
java-version: 8
check-latest: true
- name: Run tests
run: |
./dev/run-tests --parallelism 1 --modules docker-integration-tests --included-tags org.apache.spark.tags.DockerTest
Expand Down Expand Up @@ -921,9 +937,11 @@ jobs:
restore-keys: |
k8s-integration-coursier-
- name: Install Java ${{ inputs.java }}
uses: actions/setup-java@v1
uses: actions/setup-java@v3
with:
distribution: 'zulu'
java-version: ${{ inputs.java }}
check-latest: true
- name: start minikube
run: |
# See more in "Installation" https://minikube.sigs.k8s.io/docs/start/
Expand Down