Skip to content

[HUDI-2903] get table schema from the last commit with data written #4180

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
nsivabalan merged 3 commits into from
Jan 18, 2022
Merged

[HUDI-2903] get table schema from the last commit with data written #4180

nsivabalan merged 3 commits into from
Jan 18, 2022

Conversation

@YannByron
Copy link
Contributor

@YannByron YannByron commented Dec 1, 2021

If the last operation is delete_partition, the HoodieCommitMetadata object from the last commit will has an empty getFileIdAndRelativePaths. So we can't get the table schema from it.

So, i wanna find the last commit which data is written to.

@leesf
Copy link
Contributor

leesf commented Dec 1, 2021

So, i wanna find the last commit which data is written to.

I think this should not be a minor and worth filing a jira ticket.

@YannByron
Copy link
Contributor Author

YannByron commented Dec 1, 2021

So, i wanna find the last commit which data is written to.

I think this should not be a minor and worth filing a jira ticket.

ok, a little lazy ~

@YannByron YannByron changed the title [MINOR] get table schema from the last commit with data written [HUDI-2903] get table schema from the last commit with data written Dec 1, 2021
@YannByron
Copy link
Contributor Author

YannByron commented Dec 1, 2021

@hudi-bot run azure

@xushiyan
Copy link
Member

xushiyan commented Dec 2, 2021

As discussed, let's hold this off.

@xushiyan xushiyan added the status:in-progress Work in progress label Dec 2, 2021
@vinothchandar vinothchandar self-assigned this Dec 25, 2021
@YannByron YannByron force-pushed the master_table_schema branch from 9f3823c to 74699cc Compare January 12, 2022 03:24
nsivabalan
nsivabalan requested changes Jan 12, 2022
View reviewed changes
hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java Outdated
HoodieInstant lastDeltaInstant = lastDeltaCommit.get();
// read from the log file wrote
commitMetadata = HoodieCommitMetadata.fromBytes(activeTimeline.getInstantDetails(lastDeltaInstant).get(),
HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(activeTimeline.getInstantDetails(lastDeltaInstant).get(),
Copy link
Contributor

@nsivabalan nsivabalan Jan 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is a chance that last delta commit is empty and does not have valid schema. so, the fix we have done for COW above, might also be required here.

Copy link
Contributor Author

@YannByron YannByron Jan 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have optimized the mor part, please help to review this again.

@nsivabalan
Copy link
Contributor

nsivabalan commented Jan 12, 2022

@xushiyan : may I know why it was moved to WIP. fix looks ok to me on a high level. have left some feedback.

nsivabalan
nsivabalan requested changes Jan 13, 2022
View reviewed changes
Copy link
Contributor

@nsivabalan nsivabalan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. just one ask for UT. can land then.

@YannByron YannByron force-pushed the master_table_schema branch from 6005512 to 92c0642 Compare January 14, 2022 12:12
@YannByron
Copy link
Contributor Author

YannByron commented Jan 17, 2022

@nsivabalan add some UT for this. have time to review this again?

@nsivabalan nsivabalan removed the status:in-progress Work in progress label Jan 17, 2022
@nsivabalan
Copy link
Contributor

nsivabalan commented Jan 18, 2022

sorry, looks like you need to rebase and resolve conflicts.

@nsivabalan
Copy link
Contributor

nsivabalan commented Jan 18, 2022

once rebased and CI is green, we can land this

@YannByron YannByron force-pushed the master_table_schema branch from 92c0642 to f4cec28 Compare January 18, 2022 13:13
@hudi-bot
Copy link
Collaborator

hudi-bot commented Jan 18, 2022

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@YannByron
Copy link
Contributor Author

YannByron commented Jan 18, 2022

sorry, looks like you need to rebase and resolve conflicts.

done for this.

@nsivabalan nsivabalan merged commit a09c231 into apache:master Jan 18, 2022
@vinishjail97 vinishjail97 mentioned this pull request Jan 24, 2022
Closed
5 tasks
vingov pushed a commit to vingov/hudi that referenced this pull request Jan 26, 2022
@YannByron
[HUDI-2903] get table schema from the last commit with data written (a…
ac0c648 
…pache#4180)
nsivabalan pushed a commit to onehouseinc/hudi that referenced this pull request Feb 9, 2022
@YannByron @nsivabalan
[HUDI-2903] get table schema from the last commit with data written (a…
a481f66 
…pache#4180)
liusenhua pushed a commit to liusenhua/hudi that referenced this pull request Mar 1, 2022
@YannByron @liusenhua
[HUDI-2903] get table schema from the last commit with data written (a…
2a5adff 
…pache#4180)
vingov pushed a commit to vingov/hudi that referenced this pull request Apr 3, 2022
@YannByron
[HUDI-2903] get table schema from the last commit with data written (a…
3e358fa 
…pache#4180)
yihua referenced this pull request Apr 12, 2022
@XuQianJin-Stars
[HUDI-3106] Fix HiveSyncTool not sync schema (#4452)
32505d5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nsivabalan nsivabalan nsivabalan approved these changes

Assignees

@nsivabalan nsivabalan

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@YannByron @leesf @xushiyan @nsivabalan @hudi-bot @vinothchandar