Make a Copy of StreamingOffset implementation from Spark2 into Spark3's version of Offset implementation #2615
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… to avoid naming conflicts
Collaborator
Author
|
@RussellSpitzer @rdblue @aokolnychyi - pl. see if you folks can (n)ack this. |
SreeramGarlapati
changed the title
Member
|
Mentioned on slack, Only thing I think we need here is to move the original StreamingOffset -> iceberg-spark2 |
RussellSpitzer
approved these changes
May 19, 2021
Member
|
Merged! Thanks for the first contribution @SreeramGarlapati |
SreeramGarlapati
added a commit
to SreeramGarlapati/iceberg
that referenced
this pull request
May 20, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a copy of #2092
Implementation of
StreamingOffsetin Iceberg-Spark project is derived fromorg.apache.spark.sql.sources.v2.reader.streaming.Offset- which is aSpark2class. As a result - this cannot be used for implementing Spark3 Structured Streaming (#2611) & had to be copied.So, moved the existing
StreamingOffsetimplementation iniceberg-sparktoiceberg-spark2.Made another copy of this with spark3 offset implementation in
iceberg-spark3.