Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Spark runner] Removal of Spark 2 runner support #25263

Merged
merged 2 commits into from
Feb 8, 2023
Merged

[Spark runner] Removal of Spark 2 runner support #25263

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1e76728
[Spark runner] Removal of Spark 2 runner support (closes #25259)
Feb 1, 2023
86d464b
Merge branch 'master' into 25259_remove_spark_2
Feb 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 0 additions & 51 deletions .test-infra/jenkins/job_PostCommit_CrossLanguageValidatesRunner_Spark.groovy
View file
Open in desktop

This file was deleted.

1 change: 0 additions & 1 deletion .test-infra/jenkins/job_PostCommit_Java_Examples_Spark.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@ PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_Examples_Spark',
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
tasks(':runners:spark:2:examplesIntegrationTest')
tasks(':runners:spark:3:examplesIntegrationTest')
commonJobProperties.setGradleSwitches(delegate)
}
Expand Down
43 changes: 0 additions & 43 deletions .test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Spark2_Streaming.groovy
View file
Open in desktop

This file was deleted.

2 changes: 0 additions & 2 deletions .test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Spark_Batch.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,7 @@ PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_PVR_Spark_Batch',
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
tasks(':runners:spark:2:job-server:validatesPortableRunnerBatch')
tasks(':runners:spark:3:job-server:validatesPortableRunnerBatch')
tasks(':runners:spark:2:job-server:validatesPortableRunnerDocker')
tasks(':runners:spark:3:job-server:validatesPortableRunnerDocker')
commonJobProperties.setGradleSwitches(delegate)
}
Expand Down
1 change: 0 additions & 1 deletion .test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Spark.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,6 @@ PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_ValidatesRunner_Spark',
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
tasks(':runners:spark:2:validatesRunner')
tasks(':runners:spark:3:validatesRunner')
commonJobProperties.setGradleSwitches(delegate)
}
Expand Down
1 change: 0 additions & 1 deletion .test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@ PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_ValidatesRunner_SparkSt
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
tasks(':runners:spark:2:validatesStructuredStreamingRunnerBatch')
tasks(':runners:spark:3:validatesStructuredStreamingRunnerBatch')
commonJobProperties.setGradleSwitches(delegate)
}
Expand Down
1 change: 1 addition & 0 deletions CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@

## Breaking Changes

* The deprecated SparkRunner for Spark 2 (see [2.41.0](#2410---2022-08-23)) was removed ([#25263](https://github.com/apache/beam/pull/25263)).
* Python's BatchElements performs more aggressive batching in some cases,
capping at 10 second rather than 1 second batches by default and excluding
fixed cost in this computation to better handle cases where the fixed cost
Expand Down
5 changes: 0 additions & 5 deletions build.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -254,8 +254,6 @@ tasks.register("javaPreCommit") {
dependsOn(":runners:portability:java:build")
dependsOn(":runners:samza:build")
dependsOn(":runners:samza:job-server:build")
dependsOn(":runners:spark:2:build")
dependsOn(":runners:spark:2:job-server:build")
dependsOn(":runners:spark:3:build")
dependsOn(":runners:spark:3:job-server:build")
dependsOn(":runners:twister2:build")
Expand Down Expand Up @@ -340,7 +338,6 @@ tasks.register("javaPostCommitSickbay") {
dependsOn(":runners:flink:1.13:validatesRunnerSickbay")
dependsOn(":runners:flink:1.14:validatesRunnerSickbay")
dependsOn(":runners:flink:1.15:validatesRunnerSickbay")
dependsOn(":runners:spark:2:job-server:validatesRunnerSickbay")
dependsOn(":runners:spark:3:job-server:validatesRunnerSickbay")
dependsOn(":runners:direct-java:validatesRunnerSickbay")
dependsOn(":runners:portability:java:validatesRunnerSickbay")
Expand All @@ -353,7 +350,6 @@ tasks.register("javaHadoopVersionsTest") {
dependsOn(":sdks:java:io:hcatalog:hadoopVersionsTest")
dependsOn(":sdks:java:io:parquet:hadoopVersionsTest")
dependsOn(":sdks:java:extensions:sorter:hadoopVersionsTest")
dependsOn(":runners:spark:2:hadoopVersionsTest")
dependsOn(":runners:spark:3:hadoopVersionsTest")
}

Expand Down Expand Up @@ -548,7 +544,6 @@ tasks.register("typescriptPreCommit") {
}

tasks.register("pushAllDockerImages") {
dependsOn(":runners:spark:2:job-server:container:dockerPush")
dependsOn(":runners:spark:3:job-server:container:dockerPush")
dependsOn(":sdks:java:container:pushAll")
dependsOn(":sdks:python:container:pushAll")
Expand Down
3 changes: 0 additions & 3 deletions buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -505,7 +505,6 @@ class BeamModulePlugin implements Plugin<Project> {
def jmh_version = "1.34"

// Export Spark versions, so they are defined in a single place only
project.ext.spark2_version = spark2_version
project.ext.spark3_version = spark3_version

// A map of maps containing common libraries used per language. To use:
Expand Down Expand Up @@ -730,8 +729,6 @@ class BeamModulePlugin implements Plugin<Project> {
slf4j_jcl : "org.slf4j:slf4j-jcl:$slf4j_version",
snappy_java : "org.xerial.snappy:snappy-java:1.1.8.4",
spark_core : "org.apache.spark:spark-core_2.11:$spark2_version",
aromanenko-dev marked this conversation as resolved.
Show resolved Hide resolved
spark_network_common : "org.apache.spark:spark-network-common_2.11:$spark2_version",
spark_sql : "org.apache.spark:spark-sql_2.11:$spark2_version",
spark_streaming : "org.apache.spark:spark-streaming_2.11:$spark2_version",
spark3_core : "org.apache.spark:spark-core_2.12:$spark3_version",
spark3_network_common : "org.apache.spark:spark-network-common_2.12:$spark3_version",
Expand Down
33 changes: 0 additions & 33 deletions runners/spark/2/build.gradle
View file
Open in desktop

This file was deleted.

31 changes: 0 additions & 31 deletions runners/spark/2/job-server/build.gradle
View file
Open in desktop

This file was deleted.

27 changes: 0 additions & 27 deletions runners/spark/2/job-server/container/build.gradle
View file
Open in desktop

This file was deleted.

25 changes: 0 additions & 25 deletions ...rs/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/Constants.java
View file
Open in desktop

This file was deleted.

42 changes: 0 additions & 42 deletions ...pache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingPipelineOptions.java
View file
Open in desktop

This file was deleted.

Loading