Skip to content

Flink: Support configurable Scala version. #4157

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rdblue merged 1 commit into from
Feb 20, 2022
Merged

Flink: Support configurable Scala version. #4157

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 14 additions & 21 deletions flink/v1.12/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,18 +17,11 @@
* under the License.
*/

def flinkProjects = [
project(':iceberg-flink:iceberg-flink-1.12'),
project(':iceberg-flink:iceberg-flink-runtime-1.12')
]

configure(flinkProjects) {
project.ext {
flinkVersion = '1.12.5'
}
}
String flinkVersion = '1.12.5'
String flinkMajorVersion = '1.12'
String scalaVersion = System.getProperty("scalaVersion") != null ? System.getProperty("scalaVersion") : System.getProperty("defaultScalaVersion")

project(':iceberg-flink:iceberg-flink-1.12') {
project(":iceberg-flink:iceberg-flink-${flinkMajorVersion}_${scalaVersion}") {
Copy link
Member Author

@openinx openinx Feb 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rdblue @kbendick Seems I made a mistake here ? Since we apache iceberg flink & flink-runtime modules don't introduce any scala dependencies, so in theory the iceberg-flink-1.14.jar & iceberg-flink-runtime-1.14.jar can work fine on both flink-1.14.0-bin-scala_2.11 & flink-1.14.0-bin-scala_2.12 release. So we don't need to add any _${scalaVersion} suffix for iceberg-flink & iceberg-flink-runtime jar.

(But we still need this PR to make iceberg-flink/iceber-flink-runtime compile on different scala version ).

Copy link
Member Author

@openinx openinx Feb 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

➜ iceberg git:(master) ✗ ./gradlew :iceberg-flink:iceberg-flink-runtime-1.14_2.12:dependencies --configuration runtimeClasspath

> Task :iceberg-flink:iceberg-flink-runtime-1.14_2.12:dependencies

------------------------------------------------------------
Project ':iceberg-flink:iceberg-flink-runtime-1.14_2.12'
------------------------------------------------------------

runtimeClasspath - Runtime classpath of source set 'main'.
+--- com.github.stephenc.findbugs:findbugs-annotations -> 1.3.9-1
+--- project :iceberg-flink:iceberg-flink-1.14_2.12
|    +--- project :iceberg-api
|    |    +--- com.github.stephenc.findbugs:findbugs-annotations -> 1.3.9-1
|    |    \--- project :iceberg-bundled-guava
|    +--- project :iceberg-data
|    |    +--- project :iceberg-api (*)
|    |    +--- com.github.stephenc.findbugs:findbugs-annotations -> 1.3.9-1
|    |    +--- project :iceberg-bundled-guava
|    |    +--- project :iceberg-core
|    |    |    +--- project :iceberg-api (*)
|    |    |    +--- com.github.stephenc.findbugs:findbugs-annotations -> 1.3.9-1
|    |    |    +--- project :iceberg-common
|    |    |    |    +--- com.github.stephenc.findbugs:findbugs-annotations -> 1.3.9-1
|    |    |    |    \--- project :iceberg-bundled-guava
|    |    |    +--- project :iceberg-bundled-guava
|    |    |    +--- org.apache.avro:avro -> 1.10.1
|    |    |    |    +--- com.fasterxml.jackson.core:jackson-core:2.11.3 -> 2.11.4
|    |    |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.11.3 -> 2.11.4
|    |    |    |         +--- com.fasterxml.jackson.core:jackson-annotations:2.11.4
|    |    |    |         \--- com.fasterxml.jackson.core:jackson-core:2.11.4
|    |    |    +--- com.fasterxml.jackson.core:jackson-databind -> 2.11.4 (*)
|    |    |    +--- com.fasterxml.jackson.core:jackson-core -> 2.11.4
|    |    |    +--- com.github.ben-manes.caffeine:caffeine -> 2.8.4
|    |    |    |    +--- org.checkerframework:checker-qual:3.4.0
|    |    |    |    \--- com.google.errorprone:error_prone_annotations:2.3.4
|    |    |    \--- org.roaringbitmap:RoaringBitmap -> 0.9.22
|    |    |         \--- org.roaringbitmap:shims:0.9.22
|    |    +--- org.apache.orc:orc-core -> 1.7.3
|    |    |    +--- org.apache.orc:orc-shims:1.7.3
|    |    |    +--- io.airlift:aircompressor:0.21
|    |    |    +--- org.jetbrains:annotations:17.0.0
|    |    |    \--- org.threeten:threeten-extra:1.5.0
|    |    \--- org.apache.parquet:parquet-avro -> 1.12.2
|    |         +--- org.apache.parquet:parquet-column:1.12.2
|    |         |    +--- org.apache.parquet:parquet-common:1.12.2
|    |         |    |    +--- org.apache.parquet:parquet-format-structures:1.12.2
|    |         |    |    \--- org.apache.yetus:audience-annotations:0.12.0
|    |         |    \--- org.apache.parquet:parquet-encoding:1.12.2
|    |         |         \--- org.apache.parquet:parquet-common:1.12.2 (*)
|    |         +--- org.apache.parquet:parquet-hadoop:1.12.2
|    |         |    +--- org.apache.parquet:parquet-column:1.12.2 (*)
|    |         |    +--- org.apache.parquet:parquet-format-structures:1.12.2
|    |         |    +--- org.apache.parquet:parquet-jackson:1.12.2
|    |         |    \--- com.github.luben:zstd-jni:1.4.9-1
|    |         \--- org.apache.parquet:parquet-format-structures:1.12.2
|    +--- com.github.stephenc.findbugs:findbugs-annotations -> 1.3.9-1
|    +--- project :iceberg-bundled-guava
|    +--- project :iceberg-common (*)
|    +--- project :iceberg-core (*)
|    +--- project :iceberg-orc
|    |    +--- project :iceberg-api (*)
|    |    +--- com.github.stephenc.findbugs:findbugs-annotations -> 1.3.9-1
|    |    +--- project :iceberg-bundled-guava
|    |    +--- project :iceberg-core (*)
|    |    +--- org.apache.avro:avro -> 1.10.1 (*)
|    |    \--- org.apache.orc:orc-core -> 1.7.3 (*)
|    +--- project :iceberg-parquet
|    |    +--- project :iceberg-api (*)
|    |    +--- com.github.stephenc.findbugs:findbugs-annotations -> 1.3.9-1
|    |    +--- project :iceberg-bundled-guava
|    |    +--- project :iceberg-core (*)
|    |    +--- project :iceberg-common (*)
|    |    \--- org.apache.parquet:parquet-avro -> 1.12.2 (*)
|    +--- project :iceberg-hive-metastore
|    |    +--- project :iceberg-api (*)
|    |    +--- com.github.stephenc.findbugs:findbugs-annotations -> 1.3.9-1
|    |    +--- project :iceberg-bundled-guava
|    |    +--- project :iceberg-core (*)
|    |    +--- project :iceberg-common (*)
|    |    \--- com.github.ben-manes.caffeine:caffeine -> 2.8.4 (*)
|    +--- org.apache.parquet:parquet-avro -> 1.12.2 (*)
|    \--- org.apache.orc:orc-core -> 1.7.3 (*)
+--- project :iceberg-aws
|    +--- project :iceberg-api (*)
|    +--- com.github.stephenc.findbugs:findbugs-annotations -> 1.3.9-1
|    +--- project :iceberg-bundled-guava
|    +--- project :iceberg-common (*)
|    \--- project :iceberg-core (*)
+--- project :iceberg-aliyun
|    +--- project :iceberg-api (*)
|    +--- com.github.stephenc.findbugs:findbugs-annotations -> 1.3.9-1
|    +--- project :iceberg-bundled-guava
|    +--- project :iceberg-core (*)
|    \--- project :iceberg-common (*)
\--- project :iceberg-nessie
     +--- project :iceberg-api (*)
     +--- com.github.stephenc.findbugs:findbugs-annotations -> 1.3.9-1
     +--- project :iceberg-common (*)
     +--- project :iceberg-core (*)
     +--- project :iceberg-bundled-guava
     \--- org.projectnessie:nessie-client -> 0.20.1
          \--- org.projectnessie:nessie-model:0.20.1

(*) - dependencies omitted (listed previously)

A web-based, searchable dependency report is available by adding the --scan option.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 29s
1 actionable task: 1 executed

Copy link
Member Author

@openinx openinx Feb 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR is here: #4193


dependencies {
implementation project(path: ':iceberg-bundled-guava', configuration: 'shadow')
Expand All @@ -40,11 +33,11 @@ project(':iceberg-flink:iceberg-flink-1.12') {
implementation project(':iceberg-parquet')
implementation project(':iceberg-hive-metastore')

compileOnly "org.apache.flink:flink-streaming-java_2.12:${flinkVersion}"
compileOnly "org.apache.flink:flink-streaming-java_2.12:${flinkVersion}:tests"
compileOnly "org.apache.flink:flink-table-api-java-bridge_2.12:${flinkVersion}"
compileOnly "org.apache.flink:flink-table-planner-blink_2.12:${flinkVersion}"
compileOnly "org.apache.flink:flink-table-planner_2.12:${flinkVersion}"
compileOnly "org.apache.flink:flink-streaming-java_${scalaVersion}:${flinkVersion}"
compileOnly "org.apache.flink:flink-streaming-java_${scalaVersion}:${flinkVersion}:tests"
compileOnly "org.apache.flink:flink-table-api-java-bridge_${scalaVersion}:${flinkVersion}"
compileOnly "org.apache.flink:flink-table-planner-blink_${scalaVersion}:${flinkVersion}"
compileOnly "org.apache.flink:flink-table-planner_${scalaVersion}:${flinkVersion}"
compileOnly "org.apache.hadoop:hadoop-hdfs"
compileOnly "org.apache.hadoop:hadoop-common"
compileOnly("org.apache.hadoop:hadoop-minicluster") {
Expand All @@ -69,12 +62,12 @@ project(':iceberg-flink:iceberg-flink-1.12') {
}

testImplementation "org.apache.flink:flink-core:${flinkVersion}"
testImplementation "org.apache.flink:flink-runtime_2.12:${flinkVersion}"
testImplementation "org.apache.flink:flink-table-planner-blink_2.12:${flinkVersion}"
testImplementation "org.apache.flink:flink-runtime_${scalaVersion}:${flinkVersion}"
testImplementation "org.apache.flink:flink-table-planner-blink_${scalaVersion}:${flinkVersion}"
testImplementation("org.apache.flink:flink-test-utils-junit:${flinkVersion}") {
exclude group: 'junit'
}
testImplementation("org.apache.flink:flink-test-utils_2.12:${flinkVersion}") {
testImplementation("org.apache.flink:flink-test-utils_${scalaVersion}:${flinkVersion}") {
exclude group: "org.apache.curator", module: 'curator-test'
exclude group: 'junit'
}
Expand Down Expand Up @@ -118,7 +111,7 @@ project(':iceberg-flink:iceberg-flink-1.12') {
}
}

project(':iceberg-flink:iceberg-flink-runtime-1.12') {
project(":iceberg-flink:iceberg-flink-runtime-${flinkMajorVersion}_${scalaVersion}") {
apply plugin: 'com.github.johnrengelman.shadow'

tasks.jar.dependsOn tasks.shadowJar
Expand All @@ -138,7 +131,7 @@ project(':iceberg-flink:iceberg-flink-runtime-1.12') {
}

dependencies {
implementation project(':iceberg-flink:iceberg-flink-1.12')
implementation project(":iceberg-flink:iceberg-flink-${flinkMajorVersion}_${scalaVersion}")
implementation project(':iceberg-aws')
implementation(project(':iceberg-aliyun')) {
exclude group: 'edu.umd.cs.findbugs', module: 'findbugs'
Expand Down
35 changes: 14 additions & 21 deletions flink/v1.13/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,18 +17,11 @@
* under the License.
*/

def flinkProjects = [
project(':iceberg-flink:iceberg-flink-1.13'),
project(':iceberg-flink:iceberg-flink-runtime-1.13')
]

configure(flinkProjects) {
project.ext {
flinkVersion = '1.13.2'
}
}
String flinkVersion = '1.13.2'
String flinkMajorVersion = '1.13'
String scalaVersion = System.getProperty("scalaVersion") != null ? System.getProperty("scalaVersion") : System.getProperty("defaultScalaVersion")

project(':iceberg-flink:iceberg-flink-1.13') {
project(":iceberg-flink:iceberg-flink-${flinkMajorVersion}_${scalaVersion}") {

dependencies {
implementation project(path: ':iceberg-bundled-guava', configuration: 'shadow')
Expand All @@ -40,11 +33,11 @@ project(':iceberg-flink:iceberg-flink-1.13') {
implementation project(':iceberg-parquet')
implementation project(':iceberg-hive-metastore')

compileOnly "org.apache.flink:flink-streaming-java_2.12:${flinkVersion}"
compileOnly "org.apache.flink:flink-streaming-java_2.12:${flinkVersion}:tests"
compileOnly "org.apache.flink:flink-table-api-java-bridge_2.12:${flinkVersion}"
compileOnly "org.apache.flink:flink-table-planner-blink_2.12:${flinkVersion}"
compileOnly "org.apache.flink:flink-table-planner_2.12:${flinkVersion}"
compileOnly "org.apache.flink:flink-streaming-java_${scalaVersion}:${flinkVersion}"
compileOnly "org.apache.flink:flink-streaming-java_${scalaVersion}:${flinkVersion}:tests"
compileOnly "org.apache.flink:flink-table-api-java-bridge_${scalaVersion}:${flinkVersion}"
compileOnly "org.apache.flink:flink-table-planner-blink_${scalaVersion}:${flinkVersion}"
compileOnly "org.apache.flink:flink-table-planner_${scalaVersion}:${flinkVersion}"
compileOnly "org.apache.hadoop:hadoop-hdfs"
compileOnly "org.apache.hadoop:hadoop-common"
compileOnly("org.apache.hadoop:hadoop-minicluster") {
Expand All @@ -69,12 +62,12 @@ project(':iceberg-flink:iceberg-flink-1.13') {
}

testImplementation "org.apache.flink:flink-core:${flinkVersion}"
testImplementation "org.apache.flink:flink-runtime_2.12:${flinkVersion}"
testImplementation "org.apache.flink:flink-table-planner-blink_2.12:${flinkVersion}"
testImplementation "org.apache.flink:flink-runtime_${scalaVersion}:${flinkVersion}"
testImplementation "org.apache.flink:flink-table-planner-blink_${scalaVersion}:${flinkVersion}"
testImplementation ("org.apache.flink:flink-test-utils-junit:${flinkVersion}") {
exclude group: 'junit'
}
testImplementation("org.apache.flink:flink-test-utils_2.12:${flinkVersion}") {
testImplementation("org.apache.flink:flink-test-utils_${scalaVersion}:${flinkVersion}") {
exclude group: "org.apache.curator", module: 'curator-test'
exclude group: 'junit'
}
Expand Down Expand Up @@ -118,7 +111,7 @@ project(':iceberg-flink:iceberg-flink-1.13') {
}
}

project(':iceberg-flink:iceberg-flink-runtime-1.13') {
project(":iceberg-flink:iceberg-flink-runtime-${flinkMajorVersion}_${scalaVersion}") {
apply plugin: 'com.github.johnrengelman.shadow'

tasks.jar.dependsOn tasks.shadowJar
Expand All @@ -138,7 +131,7 @@ project(':iceberg-flink:iceberg-flink-runtime-1.13') {
}

dependencies {
implementation project(':iceberg-flink:iceberg-flink-1.13')
implementation project(":iceberg-flink:iceberg-flink-${flinkMajorVersion}_${scalaVersion}")
implementation project(':iceberg-aws')
implementation(project(':iceberg-aliyun')) {
exclude group: 'edu.umd.cs.findbugs', module: 'findbugs'
Expand Down
29 changes: 11 additions & 18 deletions flink/v1.14/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,18 +17,11 @@
* under the License.
*/

def flinkProjects = [
project(':iceberg-flink:iceberg-flink-1.14'),
project(':iceberg-flink:iceberg-flink-runtime-1.14')
]

configure(flinkProjects) {
project.ext {
flinkVersion = '1.14.0'
}
}
String flinkVersion = '1.14.0'
String flinkMajorVersion = '1.14'
String scalaVersion = System.getProperty("scalaVersion") != null ? System.getProperty("scalaVersion") : System.getProperty("defaultScalaVersion")
Copy link
Contributor

@kbendick kbendick Feb 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We’ve recently also had work for supporting Scala 2.13 in Spark, so it might be wise to name scalaVersion to something more Flink related to keep the two separated. Though this isn’t a list of all known Scala versions for Flink so might not be a concern here.

Copy link
Contributor

@kbendick kbendick Feb 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned, Spark is also using scalaVersion flag. So we’ll definitely need to define explicit known Flink Scala version combos to disallow say, somebody trying to build Flink with Scala 2.13 (as the name will probably be shared).

Very interested in people’s thoughts on how to handle this concern of scalaVersion and Flink / Spark versions and their different supported combinations.

Copy link
Member Author

@openinx openinx Feb 18, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question, I also considered this issue before. The current available scala version for flink are 2.11 and 2.12, for most of people as far I know, they mainly choose scala 2.11 for the production environment. But the available scala versions for spark are 2.11, 2.12, 2.13 ( Some major version only support 2.12).

So in fact, the supported scala version ranges are quite different between spark & flink. Besides, the supported scala version ranges are also different between different spark versions. Let's see the following table:

So even if we separate the scala version between spark & flink, the same scala version still could not be applied to build all the iceberg spark runtime jars I think. The correct approach is: try to package related artifacts in one build process as much as possible. If we can't build in one build, we can only choose to build again.

Copy link
Contributor

@rdblue rdblue Feb 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since there can only be one Scala version in the build, I think using the same scalaVersion flag should be fine.


project(':iceberg-flink:iceberg-flink-1.14') {
project(":iceberg-flink:iceberg-flink-${flinkMajorVersion}_${scalaVersion}") {

dependencies {
implementation project(path: ':iceberg-bundled-guava', configuration: 'shadow')
Expand All @@ -40,10 +33,10 @@ project(':iceberg-flink:iceberg-flink-1.14') {
implementation project(':iceberg-parquet')
implementation project(':iceberg-hive-metastore')

compileOnly "org.apache.flink:flink-streaming-java_2.12:${flinkVersion}"
compileOnly "org.apache.flink:flink-streaming-java_2.12:${flinkVersion}:tests"
compileOnly "org.apache.flink:flink-table-api-java-bridge_2.12:${flinkVersion}"
compileOnly "org.apache.flink:flink-table-planner_2.12:${flinkVersion}"
compileOnly "org.apache.flink:flink-streaming-java_${scalaVersion}:${flinkVersion}"
compileOnly "org.apache.flink:flink-streaming-java_${scalaVersion}:${flinkVersion}:tests"
compileOnly "org.apache.flink:flink-table-api-java-bridge_${scalaVersion}:${flinkVersion}"
compileOnly "org.apache.flink:flink-table-planner_${scalaVersion}:${flinkVersion}"
compileOnly "org.apache.hadoop:hadoop-hdfs"
compileOnly "org.apache.hadoop:hadoop-common"
compileOnly("org.apache.hadoop:hadoop-minicluster") {
Expand Down Expand Up @@ -72,7 +65,7 @@ project(':iceberg-flink:iceberg-flink-1.14') {
testImplementation ("org.apache.flink:flink-test-utils-junit:${flinkVersion}") {
exclude group: 'junit'
}
testImplementation("org.apache.flink:flink-test-utils_2.12:${flinkVersion}") {
testImplementation("org.apache.flink:flink-test-utils_${scalaVersion}:${flinkVersion}") {
exclude group: "org.apache.curator", module: 'curator-test'
exclude group: 'junit'
}
Expand Down Expand Up @@ -116,7 +109,7 @@ project(':iceberg-flink:iceberg-flink-1.14') {
}
}

project(':iceberg-flink:iceberg-flink-runtime-1.14') {
project(":iceberg-flink:iceberg-flink-runtime-${flinkMajorVersion}_${scalaVersion}") {
apply plugin: 'com.github.johnrengelman.shadow'

tasks.jar.dependsOn tasks.shadowJar
Expand All @@ -136,7 +129,7 @@ project(':iceberg-flink:iceberg-flink-runtime-1.14') {
}

dependencies {
implementation project(':iceberg-flink:iceberg-flink-1.14')
implementation project(":iceberg-flink:iceberg-flink-${flinkMajorVersion}_${scalaVersion}")
implementation project(':iceberg-aws')
implementation(project(':iceberg-aliyun')) {
exclude group: 'edu.umd.cs.findbugs', module: 'findbugs'
Expand Down
36 changes: 18 additions & 18 deletions settings.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,30 +87,30 @@ if (!flinkVersions.isEmpty()) {
}

if (flinkVersions.contains("1.12")) {
include ':iceberg-flink:flink-1.12'
include ':iceberg-flink:flink-runtime-1.12'
project(':iceberg-flink:flink-1.12').projectDir = file('flink/v1.12/flink')
project(':iceberg-flink:flink-1.12').name = 'iceberg-flink-1.12'
project(':iceberg-flink:flink-runtime-1.12').projectDir = file('flink/v1.12/flink-runtime')
project(':iceberg-flink:flink-runtime-1.12').name = 'iceberg-flink-runtime-1.12'
include ":iceberg-flink:flink-1.12_${scalaVersion}"
include ":iceberg-flink:flink-runtime-1.12_${scalaVersion}"
project(":iceberg-flink:flink-1.12_${scalaVersion}").projectDir = file('flink/v1.12/flink')
project(":iceberg-flink:flink-1.12_${scalaVersion}").name = "iceberg-flink-1.12_${scalaVersion}"
project(":iceberg-flink:flink-runtime-1.12_${scalaVersion}").projectDir = file('flink/v1.12/flink-runtime')
project(":iceberg-flink:flink-runtime-1.12_${scalaVersion}").name = "iceberg-flink-runtime-1.12_${scalaVersion}"
}

if (flinkVersions.contains("1.13")) {
include ':iceberg-flink:flink-1.13'
include ':iceberg-flink:flink-runtime-1.13'
project(':iceberg-flink:flink-1.13').projectDir = file('flink/v1.13/flink')
project(':iceberg-flink:flink-1.13').name = 'iceberg-flink-1.13'
project(':iceberg-flink:flink-runtime-1.13').projectDir = file('flink/v1.13/flink-runtime')
project(':iceberg-flink:flink-runtime-1.13').name = 'iceberg-flink-runtime-1.13'
include ":iceberg-flink:flink-1.13_${scalaVersion}"
include ":iceberg-flink:flink-runtime-1.13_${scalaVersion}"
project(":iceberg-flink:flink-1.13_${scalaVersion}").projectDir = file('flink/v1.13/flink')
project(":iceberg-flink:flink-1.13_${scalaVersion}").name = "iceberg-flink-1.13_${scalaVersion}"
project(":iceberg-flink:flink-runtime-1.13_${scalaVersion}").projectDir = file('flink/v1.13/flink-runtime')
project(":iceberg-flink:flink-runtime-1.13_${scalaVersion}").name = "iceberg-flink-runtime-1.13_${scalaVersion}"
}

if (flinkVersions.contains("1.14")) {
include ':iceberg-flink:flink-1.14'
include ':iceberg-flink:flink-runtime-1.14'
project(':iceberg-flink:flink-1.14').projectDir = file('flink/v1.14/flink')
project(':iceberg-flink:flink-1.14').name = 'iceberg-flink-1.14'
project(':iceberg-flink:flink-runtime-1.14').projectDir = file('flink/v1.14/flink-runtime')
project(':iceberg-flink:flink-runtime-1.14').name = 'iceberg-flink-runtime-1.14'
include ":iceberg-flink:flink-1.14_${scalaVersion}"
include ":iceberg-flink:flink-runtime-1.14_${scalaVersion}"
project(":iceberg-flink:flink-1.14_${scalaVersion}").projectDir = file('flink/v1.14/flink')
project(":iceberg-flink:flink-1.14_${scalaVersion}").name = "iceberg-flink-1.14_${scalaVersion}"
project(":iceberg-flink:flink-runtime-1.14_${scalaVersion}").projectDir = file('flink/v1.14/flink-runtime')
project(":iceberg-flink:flink-runtime-1.14_${scalaVersion}").name = "iceberg-flink-runtime-1.14_${scalaVersion}"
Copy link
Contributor

@kbendick kbendick Feb 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct me if I’m wrong, but I think we need to be able to publish / build all combinations of Flink / Scala versions here. For example, when publishing a new release to maven we need to build all of the artifacts.

Is it possible to do that in a way that doesn’t require two build calls (one for each Scala version)? I’m not a highly experienced Scala with gradle user, so I might be mistaken or it might be necessary to use two build calls.

But it makes the release process more cumbersome and particular to have to call build multiple times, which is something we’ve been trying to make more streamlined (the build / release process).

I’m ok with iterating on things and would not block on this, but it is a concern for me.

Copy link
Contributor

@kbendick kbendick Feb 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at the Spark on Scala 2.13 support PR, it looks like scalaVersion flag with two build calls is what is used.

I’m not sure if the release scripts have been updated to support this yet though.

Copy link
Member Author

@openinx openinx Feb 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to do that in a way that doesn’t require two build calls (one for each Scala version)?

As far as I know, the gradle command cannot deploy the projects with two different values for a given system property. I mean I think the two build calls to build two separate flink scala runtime jar is the thing we need to do. I see we usually deploy the artifacts to maven repository by executing this stage-binaries.sh. I think it's okay to add the two gradle build calls into this bash script.

(In general, I think it's a good question)

Copy link
Contributor

@rdblue rdblue Feb 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we can't have multiple Scala versions in the build without strange errors. We'll have to update the binary release script to publish Scala 2.13 modules.

}

if (sparkVersions.contains("3.0")) {
Expand Down