Skip to content

Conversation

@Fokko
Copy link
Contributor

@Fokko Fokko commented Jan 13, 2021

Will solve issues in #1648 (comment)

Quoting @rdblue:

our Jackson version should be okay to update. We shade and relocate it in the runtime Jars to avoid conflicts with Spark.

@github-actions github-actions bot added the build label Jan 13, 2021
@Fokko
Copy link
Contributor Author

Fokko commented Jan 13, 2021

Same issue as #1648 (comment)

@rdblue
Copy link
Contributor

rdblue commented Jan 13, 2021

I think the test problem might be that we have some Jackson Scala dependencies forced in build.gradle:

      resolutionStrategy {
        force 'com.fasterxml.jackson.module:jackson-module-scala_2.11:2.10.2'
        force 'com.fasterxml.jackson.module:jackson-module-scala_2.12:2.10.2'
        force 'com.fasterxml.jackson.module:jackson-module-paranamer:2.10.2'
      }

@aokolnychyi, do you have more context on that?

I'm not sure that those are even used. I don't see dependencies on jackson-module-scala_* in iceberg-spark3-extensions.

kou pushed a commit to apache/arrow that referenced this pull request Jan 14, 2021
At Iceberg we're looking into consolidating the versions, see apache/iceberg#2084.
The latest Spark release is still on 2.10, and I wanted to bump everything to at least 2.11.

Closes #9189 from Fokko/ARROW-11236

Authored-by: Fokko Driesprong <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
@Fokko
Copy link
Contributor Author

Fokko commented Jan 18, 2021

@rdblue that did the trick, thanks!

@rdblue
Copy link
Contributor

rdblue commented Jan 18, 2021

I think this is okay, but I'd prefer to get it in after the 0.11.0 release is out so that we don't accidentally introduce a library incompatibility with little time to catch it. Is that okay with you, @Fokko?

@Fokko
Copy link
Contributor Author

Fokko commented Jan 18, 2021

@rdblue Yes, I think that's a great idea. Would be a pity to brick the 0.11 release. I would also postpone #2110 and #1648 in that case since they also pull in the latest version of Avro, and also the 2.11 Jackson version in a transitive way.

@rdblue
Copy link
Contributor

rdblue commented Jan 29, 2021

@Fokko, could you rebase this? Now that 0.11.0 is out, we can get it into master.

@Fokko
Copy link
Contributor Author

Fokko commented Jan 31, 2021

Awesome, sure thing. Pulled in latest master, and resolved the conflicts!

@rdblue rdblue merged commit 979575a into apache:master Feb 1, 2021
@rdblue
Copy link
Contributor

rdblue commented Feb 1, 2021

@aokolnychyi FYI

@rdblue
Copy link
Contributor

rdblue commented Feb 1, 2021

Thanks for the update, @Fokko!

kou pushed a commit to apache/arrow-java that referenced this pull request Nov 25, 2024
At Iceberg we're looking into consolidating the versions, see apache/iceberg#2084.
The latest Spark release is still on 2.10, and I wanted to bump everything to at least 2.11.

Closes #9189 from Fokko/ARROW-11236

Authored-by: Fokko Driesprong <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants