Skip to content

[HUDI-3536] Add hudi-datahub-sync implementation #5155

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
xushiyan merged 4 commits into from
Mar 30, 2022
Merged

[HUDI-3536] Add hudi-datahub-sync implementation #5155

xushiyan merged 4 commits into from
Mar 30, 2022

Conversation

@xushiyan
Copy link
Member

@xushiyan xushiyan commented Mar 28, 2022 •
edited
Loading

Add MVP implementation for hudi-datahub-sync to sync table schema as a DataHub Dataset. See https://datahubproject.io/docs/generated/metamodel/entities/dataset

@xushiyan xushiyan force-pushed the HUDI-3536-datahub-sync-impl branch 3 times, most recently from d0f9d2b to c5f6f70 Compare March 30, 2022 08:52
@xushiyan xushiyan force-pushed the HUDI-3536-datahub-sync-impl branch from c5f6f70 to 8986554 Compare March 30, 2022 15:11
@xushiyan xushiyan marked this pull request as ready for review March 30, 2022 15:16
@xushiyan
Copy link
Member Author

xushiyan commented Mar 30, 2022 •
edited
Loading

Tested the setup with deltastreamer on EMR and set

--sync-tool-classes org.apache.hudi.sync.datahub.DataHubSyncTool \
--hoodie-conf hoodie.meta.sync.datahub.emitter.server=http://xxx.compute.amazonaws.com:8080 \

Able to sync table properties and schema

Screen Shot 2022-03-30 at 11 10 54 PM

Screen Shot 2022-03-30 at 11 10 48 PM

Screen Shot 2022-03-30 at 11 10 37 PM

nsivabalan
nsivabalan requested changes Mar 30, 2022
View reviewed changes
@xushiyan xushiyan requested a review from nsivabalan March 30, 2022 18:31
nsivabalan
nsivabalan approved these changes Mar 30, 2022
View reviewed changes
Copy link
Contributor

@nsivabalan nsivabalan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hudi-bot
Copy link
Collaborator

hudi-bot commented Mar 30, 2022

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@nsivabalan nsivabalan self-assigned this Mar 30, 2022
@nsivabalan nsivabalan added the priority:blocker Production down; release blocker label Mar 30, 2022
@xushiyan xushiyan merged commit 31d4a16 into apache:master Mar 30, 2022
@xushiyan xushiyan deleted the HUDI-3536-datahub-sync-impl branch March 30, 2022 21:38
@xushiyan
Copy link
Member Author

xushiyan commented Mar 30, 2022

CI flakiness failure is unrelated as this is an independent module. hence landing.

vingov pushed a commit to vingov/hudi that referenced this pull request Apr 3, 2022
@xushiyan
[HUDI-3536] Add hudi-datahub-sync implementation (apache#5155)
e533d4d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nsivabalan nsivabalan nsivabalan approved these changes

Assignees

@nsivabalan nsivabalan

Labels

priority:blocker Production down; release blocker

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@xushiyan @hudi-bot @nsivabalan