Skip to content

[HUDI-2757] Implement Hudi AWS Glue sync #5076

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
nsivabalan merged 7 commits into from
Mar 28, 2022
Merged

[HUDI-2757] Implement Hudi AWS Glue sync #5076

nsivabalan merged 7 commits into from
Mar 28, 2022

Conversation

@xushiyan
Copy link
Member

@xushiyan xushiyan commented Mar 20, 2022 •
edited
Loading

Add AWS Glue sync implementation to allow sync to AWS Glue catalog directly via AWS SDK APIs.

@xushiyan xushiyan force-pushed the HUDI-2757-glue-sync branch from 7c71f7c to 587f337 Compare March 20, 2022 13:56
@xushiyan xushiyan mentioned this pull request Mar 20, 2022
Closed
@xushiyan xushiyan force-pushed the HUDI-2757-glue-sync branch from 587f337 to 7bb94c0 Compare March 20, 2022 15:45
@xushiyan xushiyan added aws-support priority:blocker Production down; release blocker labels Mar 21, 2022
@xushiyan
Copy link
Member Author

xushiyan commented Mar 21, 2022

Screen Shot 2022-03-21 at 10 48 00 PM

Verified running meta sync with org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool and sync'ed to Glue table.

@xushiyan xushiyan force-pushed the HUDI-2757-glue-sync branch from b7d2409 to 7bb94c0 Compare March 21, 2022 17:28
@apache apache deleted a comment from hudi-bot Mar 21, 2022
nsivabalan
nsivabalan reviewed Mar 21, 2022
View reviewed changes
Copy link
Contributor

@nsivabalan nsivabalan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

few minor comments. looks good mostly.

@nsivabalan nsivabalan self-assigned this Mar 21, 2022
@xushiyan xushiyan force-pushed the HUDI-2757-glue-sync branch 2 times, most recently from 5b94cea to a115917 Compare March 26, 2022 11:54
@xushiyan xushiyan marked this pull request as ready for review March 26, 2022 11:54
@xushiyan xushiyan requested a review from nsivabalan March 26, 2022 11:54
nsivabalan
nsivabalan reviewed Mar 27, 2022
View reviewed changes
Copy link
Contributor

@nsivabalan nsivabalan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, we can't write any tests for AWSGlueCatalogSync is it. if we mock, we have to mock pretty much everything and there is no point in testing only. is my understanding right.

mostly minor comments. once addressed, we can land.

hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogClient.java
}

@Override
public void updateLastCommitTimeSynced(String tableName) {
Copy link
Contributor

@nsivabalan nsivabalan Mar 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A general question about updating HOODIE_LAST_COMMIT_TIME_SYNC. I see in HiveSyncTool, we call hoodieHiveClient.updateLastCommitTimeSynced(tableName)
in the end.
But updating partitions and updating the last commit time synced is not atomic. and within updateLastCommitTimeSynced, we get the latest commit time from activetimeline and update it as the lastCommitTimeSycned.
So, incase of multi-writers, there are chances that timeline has moved between updating partitions and calling updateLastCommitTimeSynced.

may be we should file a jira and do a follow up on the logic.

Copy link
Member Author

@xushiyan xushiyan Mar 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea, should be a follow up item

@xushiyan xushiyan mentioned this pull request Mar 28, 2022
Merged
@xushiyan xushiyan force-pushed the HUDI-2757-glue-sync branch from a115917 to 51879ca Compare March 28, 2022 14:28
nsivabalan
nsivabalan approved these changes Mar 28, 2022
View reviewed changes
Copy link
Contributor

@nsivabalan nsivabalan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@rmahindra123
Copy link
Contributor

rmahindra123 commented Mar 28, 2022

lgtm. Approved.

@hudi-bot
Copy link
Collaborator

hudi-bot commented Mar 28, 2022

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@nsivabalan nsivabalan merged commit 6ccbae4 into apache:master Mar 28, 2022
@xushiyan xushiyan deleted the HUDI-2757-glue-sync branch March 29, 2022 13:50
vingov pushed a commit to vingov/hudi that referenced this pull request Apr 3, 2022
@xushiyan
[HUDI-2757] Implement Hudi AWS Glue sync (apache#5076)
fe60f2c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nsivabalan nsivabalan nsivabalan approved these changes

Assignees

@nsivabalan nsivabalan

Labels

priority:blocker Production down; release blocker

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@xushiyan @rmahindra123 @hudi-bot @nsivabalan