Skip to content

[HUDI-3810] Fixing lazy read for metadata log record readers #5241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
nsivabalan merged 1 commit into from
Apr 7, 2022
Merged

[HUDI-3810] Fixing lazy read for metadata log record readers #5241

nsivabalan merged 1 commit into from
Apr 7, 2022

Conversation

@nsivabalan
Copy link
Contributor

@nsivabalan nsivabalan commented Apr 6, 2022 •
edited
Loading

What is the purpose of the pull request

  • Even if point look ups are enabled for log record reader for metadata table, we do see an extra full scan is triggered. Triaged and root caused it to "read lazy" argument being always set to false. Likely these were written having FILES partition in mind where its always full scan.

Brief change log

  • Fixed "lazy read" to true for metadata log record readers in general.

Verify this pull request

  • manually verified the fix.
  • And existing tests should cover in general

Committer checklist

  • Has a corresponding JIRA in PR title & commit

  • Commit message is descriptive of the change

  • CI is green

  • Necessary doc changes done or have another open PR

  • For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@nsivabalan nsivabalan added the priority:blocker Production down; release blocker label Apr 6, 2022
yihua
yihua approved these changes Apr 6, 2022
View reviewed changes
Copy link
Contributor

@yihua yihua left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Good catch 👍

@nsivabalan nsivabalan force-pushed the fixReadLazyWithMetadataLogReader branch from f952f1b to 5e90efb Compare April 6, 2022 15:57
@nsivabalan
Copy link
Contributor Author

nsivabalan commented Apr 6, 2022

@prashantwason : Can you review this patch.

@nsivabalan nsivabalan force-pushed the fixReadLazyWithMetadataLogReader branch from 5e90efb to b6dd143 Compare April 6, 2022 19:33
@hudi-bot
Copy link
Collaborator

hudi-bot commented Apr 6, 2022

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

prashantwason
prashantwason approved these changes Apr 7, 2022
View reviewed changes
Copy link
Member

@prashantwason prashantwason left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nsivabalan nsivabalan merged commit ef06e4a into apache:master Apr 7, 2022
xushiyan pushed a commit that referenced this pull request Apr 14, 2022
@nsivabalan @xushiyan
[HUDI-3810] Fixing lazy read for metadata log record readers (#5241)
716ebe5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alexeykudinkin alexeykudinkin alexeykudinkin left review comments

@yihua yihua yihua approved these changes

@codope codope codope approved these changes

@prashantwason prashantwason prashantwason approved these changes

Assignees

@codope codope

Labels

priority:blocker Production down; release blocker

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@nsivabalan @hudi-bot @alexeykudinkin @yihua @codope @prashantwason