Skip to content

MR: Use SerializableTable in IcebergSplit #2988

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
aokolnychyi merged 1 commit into from
Aug 17, 2021
Merged

MR: Use SerializableTable in IcebergSplit #2988

aokolnychyi merged 1 commit into from
Aug 17, 2021

Conversation

@aokolnychyi
Copy link
Contributor

@aokolnychyi aokolnychyi commented Aug 17, 2021

This PR migrates IcebergSplit to use SerializableTable to support changes in #2984.

aokolnychyi
aokolnychyi commented Aug 17, 2021
View reviewed changes
mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java
InputFormatConfig.InMemoryDataModel model = conf.getEnum(InputFormatConfig.IN_MEMORY_DATA_MODEL,
InputFormatConfig.InMemoryDataModel.GENERIC);
try (CloseableIterable<CombinedScanTask> tasksIterable = scan.planTasks()) {
Table serializableTable = SerializableTable.copyOf(table);
Copy link
Contributor Author

@aokolnychyi aokolnychyi Aug 17, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kbendick, I moved it outside the loop.

@github-actions github-actions bot added the MR label Aug 17, 2021
@aokolnychyi
Copy link
Contributor Author

aokolnychyi commented Aug 17, 2021

cc @chenjunjiedada @rdsr @shardulm94 @jackye1995 @flyrain @RussellSpitzer @karuppayya @kbendick

@aokolnychyi aokolnychyi mentioned this pull request Aug 17, 2021
Merged
jackye1995
jackye1995 approved these changes Aug 17, 2021
View reviewed changes
Copy link
Contributor

@jackye1995 jackye1995 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the MR related change is pretty straightforward, I don't have any additional concern. It also promotes the use of SerializableTable across engines, which is a good thing in general.

RussellSpitzer
RussellSpitzer approved these changes Aug 17, 2021
View reviewed changes
Copy link
Member

@RussellSpitzer RussellSpitzer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@aokolnychyi aokolnychyi merged commit 8181600 into apache:master Aug 17, 2021
@aokolnychyi
Copy link
Contributor Author

aokolnychyi commented Aug 17, 2021

Thanks for reviewing, @jackye1995 @RussellSpitzer!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@RussellSpitzer RussellSpitzer RussellSpitzer approved these changes

@jackye1995 jackye1995 jackye1995 approved these changes

Assignees

No one assigned

Labels

MR

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@aokolnychyi @RussellSpitzer @jackye1995