Skip to content

Commit 64014d5

Browse files
dongjoon-hyunhuangxiaopingRD
dongjoon-hyun
authored and
huangxiaopingRD
committed
[SPARK-54143][INFRA] Add branch-4.1 Daily GitHub Action jobs
### What changes were proposed in this pull request? This PR aims to add `branch-4.1` Daily GitHub Action jobs. ### Why are the changes needed? Like `4.0.x`, we need to maintain the full test coverage for Apache Spark 4.1.0 preparation. The following jobs will be added. - https://github.com/apache/spark/actions/workflows/build_branch41_java21.yml - https://github.com/apache/spark/actions/workflows/build_branch41_non_ansi.yml - https://github.com/apache/spark/actions/workflows/build_branch41_maven.yml - https://github.com/apache/spark/actions/workflows/build_branch41_maven_java21.yml - https://github.com/apache/spark/actions/workflows/build_branch41_python.yml - https://github.com/apache/spark/actions/workflows/build_branch41_python_pypy3.10.yml ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#52842 from dongjoon-hyun/SPARK-54143. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent ddf0c06 commit 64014d5

File tree

8 files changed

+335
-0
lines changed

8 files changed

+335
-0
lines changed

‎.github/workflows/build_branch41.yml‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing,
13+
# software distributed under the License is distributed on an
14+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
# KIND, either express or implied. See the License for the
16+
# specific language governing permissions and limitations
17+
# under the License.
18+
#
19+
20+
name: "Build (branch-4.1, Scala 2.13, Hadoop 3, JDK 17)"
21+
22+
on:
23+
schedule:
24+
- cron: '0 12 * * *'
25+
workflow_dispatch:
26+
27+
jobs:
28+
run-build:
29+
permissions:
30+
packages: write
31+
name: Run
32+
uses: ./.github/workflows/build_and_test.yml
33+
if: github.repository == 'apache/spark'
34+
with:
35+
java: 17
36+
branch: branch-4.1
37+
hadoop: hadoop3
38+
envs: >-
39+
{
40+
"SCALA_PROFILE": "scala2.13",
41+
"PYSPARK_IMAGE_TO_TEST": "",
42+
"PYTHON_TO_TEST": "",
43+
"ORACLE_DOCKER_IMAGE_NAME": "gvenzl/oracle-free:23.7-slim"
44+
}
45+
jobs: >-
46+
{
47+
"build": "true",
48+
"sparkr": "true",
49+
"tpcds-1g": "true",
50+
"docker-integration-tests": "true",
51+
"k8s-integration-tests": "true",
52+
"lint" : "true"
53+
}

‎.github/workflows/build_branch41_java21.yml‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing,
13+
# software distributed under the License is distributed on an
14+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
# KIND, either express or implied. See the License for the
16+
# specific language governing permissions and limitations
17+
# under the License.
18+
#
19+
20+
name: "Build (branch-4.1, Scala 2.13, Hadoop 3, JDK 21)"
21+
22+
on:
23+
schedule:
24+
- cron: '0 5 * * *'
25+
workflow_dispatch:
26+
27+
jobs:
28+
run-build:
29+
permissions:
30+
packages: write
31+
name: Run
32+
uses: ./.github/workflows/build_and_test.yml
33+
if: github.repository == 'apache/spark'
34+
with:
35+
java: 21
36+
branch: branch-4.1
37+
hadoop: hadoop3
38+
envs: >-
39+
{
40+
"PYSPARK_IMAGE_TO_TEST": "python-311",
41+
"PYTHON_TO_TEST": "python3.11",
42+
"SKIP_MIMA": "true",
43+
"SKIP_UNIDOC": "true",
44+
"DEDICATED_JVM_SBT_TESTS": "org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormatV1Suite,org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormatV2Suite,org.apache.spark.sql.execution.datasources.orc.OrcSourceV1Suite,org.apache.spark.sql.execution.datasources.orc.OrcSourceV2Suite"
45+
}
46+
jobs: >-
47+
{
48+
"build": "true",
49+
"pyspark": "true",
50+
"sparkr": "true",
51+
"tpcds-1g": "true",
52+
"docker-integration-tests": "true",
53+
"yarn": "true",
54+
"k8s-integration-tests": "true",
55+
"buf": "true",
56+
"ui": "true"
57+
}

‎.github/workflows/build_branch41_maven.yml‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing,
13+
# software distributed under the License is distributed on an
14+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
# KIND, either express or implied. See the License for the
16+
# specific language governing permissions and limitations
17+
# under the License.
18+
#
19+
20+
name: "Build / Maven (branch-4.1, Scala 2.13, Hadoop 3, JDK 17)"
21+
22+
on:
23+
schedule:
24+
- cron: '0 14 * * *'
25+
workflow_dispatch:
26+
27+
jobs:
28+
run-build:
29+
permissions:
30+
packages: write
31+
name: Run
32+
uses: ./.github/workflows/maven_test.yml
33+
if: github.repository == 'apache/spark'
34+
with:
35+
branch: branch-4.1

‎.github/workflows/build_branch41_maven_java21.yml‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing,
13+
# software distributed under the License is distributed on an
14+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
# KIND, either express or implied. See the License for the
16+
# specific language governing permissions and limitations
17+
# under the License.
18+
#
19+
20+
name: "Build / Maven (branch-4.1, Scala 2.13, Hadoop 3, JDK 21)"
21+
22+
on:
23+
schedule:
24+
- cron: '0 14 * * *'
25+
workflow_dispatch:
26+
27+
jobs:
28+
run-build:
29+
permissions:
30+
packages: write
31+
name: Run
32+
uses: ./.github/workflows/maven_test.yml
33+
if: github.repository == 'apache/spark'
34+
with:
35+
branch: branch-4.1
36+
java: 21

‎.github/workflows/build_branch41_non_ansi.yml‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing,
13+
# software distributed under the License is distributed on an
14+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
# KIND, either express or implied. See the License for the
16+
# specific language governing permissions and limitations
17+
# under the License.
18+
#
19+
20+
name: "Build / Non-ANSI (branch-4.1, Hadoop 3, JDK 17, Scala 2.13)"
21+
22+
on:
23+
schedule:
24+
- cron: '0 2 * * *'
25+
workflow_dispatch:
26+
27+
jobs:
28+
run-build:
29+
permissions:
30+
packages: write
31+
name: Run
32+
uses: ./.github/workflows/build_and_test.yml
33+
if: github.repository == 'apache/spark'
34+
with:
35+
java: 17
36+
branch: branch-4.1
37+
hadoop: hadoop3
38+
envs: >-
39+
{
40+
"PYSPARK_IMAGE_TO_TEST": "python-311",
41+
"PYTHON_TO_TEST": "python3.11",
42+
"SPARK_ANSI_SQL_MODE": "false",
43+
}
44+
jobs: >-
45+
{
46+
"build": "true",
47+
"docs": "true",
48+
"pyspark": "true",
49+
"sparkr": "true",
50+
"tpcds-1g": "true",
51+
"docker-integration-tests": "true",
52+
"yarn": "true"
53+
}

‎.github/workflows/build_branch41_python.yml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing,
13+
# software distributed under the License is distributed on an
14+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
# KIND, either express or implied. See the License for the
16+
# specific language governing permissions and limitations
17+
# under the License.
18+
#
19+
20+
name: "Build / Python-only (branch-4.1)"
21+
22+
on:
23+
schedule:
24+
- cron: '0 12 * * *'
25+
workflow_dispatch:
26+
27+
jobs:
28+
run-build:
29+
permissions:
30+
packages: write
31+
name: Run
32+
uses: ./.github/workflows/build_and_test.yml
33+
if: github.repository == 'apache/spark'
34+
with:
35+
java: 17
36+
branch: branch-4.1
37+
hadoop: hadoop3
38+
envs: >-
39+
{
40+
"PYSPARK_IMAGE_TO_TEST": "python-311",
41+
"PYTHON_TO_TEST": "python3.11"
42+
}
43+
jobs: >-
44+
{
45+
"pyspark": "true",
46+
"pyspark-pandas": "true"
47+
}

‎.github/workflows/build_branch41_python_pypy3.10.yml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing,
13+
# software distributed under the License is distributed on an
14+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
# KIND, either express or implied. See the License for the
16+
# specific language governing permissions and limitations
17+
# under the License.
18+
#
19+
20+
name: "Build / Python-only (branch-4.1, PyPy 3.10)"
21+
22+
on:
23+
schedule:
24+
- cron: '0 16 * * *'
25+
workflow_dispatch:
26+
27+
jobs:
28+
run-build:
29+
permissions:
30+
packages: write
31+
name: Run
32+
uses: ./.github/workflows/build_and_test.yml
33+
if: github.repository == 'apache/spark'
34+
with:
35+
java: 17
36+
branch: branch-4.1
37+
hadoop: hadoop3
38+
envs: >-
39+
{
40+
"PYSPARK_IMAGE_TO_TEST": "pypy-310",
41+
"PYTHON_TO_TEST": "pypy3"
42+
}
43+
jobs: >-
44+
{
45+
"pyspark": "true",
46+
"pyspark-pandas": "true"
47+
}

‎README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,13 @@ This README file only contains basic setup instructions.
5252
| | [![GitHub Actions Build](https://github.com/apache/spark/actions/workflows/build_python_connect.yml/badge.svg)](https://github.com/apache/spark/actions/workflows/build_python_connect.yml) |
5353
| | [![GitHub Actions Build](https://github.com/apache/spark/actions/workflows/build_sparkr_window.yml/badge.svg)](https://github.com/apache/spark/actions/workflows/build_sparkr_window.yml) |
5454
| | [![GitHub Actions Build](https://github.com/apache/spark/actions/workflows/publish_snapshot.yml/badge.svg)](https://github.com/apache/spark/actions/workflows/publish_snapshot.yml) |
55+
| branch-4.1 | [![GitHub Actions Build](https://github.com/apache/spark/actions/workflows/build_branch41.yml/badge.svg)](https://github.com/apache/spark/actions/workflows/build_branch41.yml) |
56+
| | [![GitHub Actions Build](https://github.com/apache/spark/actions/workflows/build_branch41_java21.yml/badge.svg)](https://github.com/apache/spark/actions/workflows/build_branch41_java21.yml) |
57+
| | [![GitHub Actions Build](https://github.com/apache/spark/actions/workflows/build_branch41_non_ansi.yml/badge.svg)](https://github.com/apache/spark/actions/workflows/build_branch41_non_ansi.yml) |
58+
| | [![GitHub Actions Build](https://github.com/apache/spark/actions/workflows/build_branch41_maven.yml/badge.svg)](https://github.com/apache/spark/actions/workflows/build_branch41_maven.yml) |
59+
| | [![GitHub Actions Build](https://github.com/apache/spark/actions/workflows/build_branch41_maven_java21.yml/badge.svg)](https://github.com/apache/spark/actions/workflows/build_branch41_maven_java21.yml) |
60+
| | [![GitHub Actions Build](https://github.com/apache/spark/actions/workflows/build_branch41_python.yml/badge.svg)](https://github.com/apache/spark/actions/workflows/build_branch41_python.yml) |
61+
| | [![GitHub Actions Build](https://github.com/apache/spark/actions/workflows/build_branch41_python_pypy3.10.yml/badge.svg)](https://github.com/apache/spark/actions/workflows/build_branch41_python_pypy3.10.yml) |
5562
| branch-4.0 | [![GitHub Actions Build](https://github.com/apache/spark/actions/workflows/build_branch40.yml/badge.svg)](https://github.com/apache/spark/actions/workflows/build_branch40.yml) |
5663
| | [![GitHub Actions Build](https://github.com/apache/spark/actions/workflows/build_branch40_java21.yml/badge.svg)](https://github.com/apache/spark/actions/workflows/build_branch40_java21.yml) |
5764
| | [![GitHub Actions Build](https://github.com/apache/spark/actions/workflows/build_branch40_non_ansi.yml/badge.svg)](https://github.com/apache/spark/actions/workflows/build_branch40_non_ansi.yml) |

0 commit comments

Comments
 (0)