Flink: Remove the scala version suffix from artifact names. #4193
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Feb 22, 2022
Contributor
|
Should we revert #4157 if we aren't publishing artifacts that are specific to a Scala version? Looks like the main benefit of multi-version is testing? Are any of the Scala-specific artifacts included in transitive dependencies? |
Contributor
|
Jumping in to say that Flink 1.15 will not have Scala on the user's classpath, so there won't be a 2.11 / 2.12 version (or any specific Scala version) coming from upstream after that. |
Member
Author
|
@kbendick The flink community are pushing the scala-free work forward ( I mean the downstream users don't need to specific a scala version when they introduce a flink dependency, the flink is trying to create few dynamic class loader to load the existing flink modules implemented or depended on scala-lang ). Currently, I see the [1] https://issues.apache.org/jira/browse/FLINK-14105 |
Member
Author
|
@rdblue We iceberg flink 1.14 module indeed include scala related artifact in its transitive dependencies set, see the following: But in fact we iceberg flink modules don't add any scala code in iceberg-flink-1.14 and iceberg-flink-runtime-1.14 artifact packages, and we don't interact with any scala specific flink API. So the flink artifacts deployed from our apache iceberg side, then we don't need to deploy any scala specific jars to the maven repository.
I don't think we need to revert #4157 because we made the Scala version of Flink dependency dynamically configurable in the command line. This is very helpful for validating flink releases with different Scala versions, and indeed it also helps a lot for us to add scala versioned travis CI for flink integration. |
Member
Author
|
The failing unit test is still the flakey Let's get this re-run again. |
Contributor
|
If I understand correctly, the idea here is that because this is binary compatible using Flink's Java API, we don't need to care about what version of Scala was used in the Flink artifact. I don't think that I agree because if you rely on the Iceberg Flink Jar, it could transitively pull in I think that the runtime Jar could be published without the Scala suffix since it has no transitive dependencies. Or we could remove the transitive dependency and remove the Scala version suffix from all the Jars. But I don't think that Iceberg should public a Jar that has no Scala 2.12 suffix, but then pulls in a Scala 2.12 library. |
Member
Author
Correct.
It won't pull in any flink scala artifact actually because all of the flink related artifacts are introduced as
It's definitely the correction approach for our next flink 1.15, but not possible for flink 1.14 currently. As this #4193 (comment) said, flink 1.15 will remove all the scala version for all the user required artifacts, but flink 1.14 only clean part of the artifacts to remove scala version. Although we iceberg-flink modue introduce the flink artifact with scala versions, but it not require the iceberg flink modules to attach the scala version because the iceberg-flink module does not use any scala code or depends on scala-related APIs, nor does it introduce transitive flink scala dependencies. |
Contributor
|
Okay, if there is no Scala 2.12 dependency in the runtime or implementation classpath, then I'm fine removing these. |
Member
Author
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refer to the #4157 (comment)