Skip to content

Support OR-ed condition in Delta checkpoint iterator#19240

Closed
ebyhr wants to merge 3 commits intotrinodb:masterfrom
ebyhr:ebi/delta-checkpoint-entry-or
Closed

Support OR-ed condition in Delta checkpoint iterator#19240
ebyhr wants to merge 3 commits intotrinodb:masterfrom
ebyhr:ebi/delta-checkpoint-entry-or

Conversation

@ebyhr
Copy link
Copy Markdown
Member

@ebyhr ebyhr commented Oct 3, 2023

Description

Relates to #19156

Release notes

(x) This is not user-visible or is docs only, and no release notes are required.

@ebyhr ebyhr self-assigned this Oct 3, 2023
@cla-bot cla-bot bot added the cla-signed label Oct 3, 2023
@github-actions github-actions bot added tests:hive hudi Hudi connector delta-lake Delta Lake connector hive Hive connector labels Oct 3, 2023
@ebyhr ebyhr changed the title Ebi/delta checkpoint entry or Support OR-ed condition in Delta checkpoint iterator Oct 3, 2023
@ebyhr ebyhr force-pushed the ebi/delta-checkpoint-entry-or branch from 3ef7456 to 644fb08 Compare October 3, 2023 04:04
@findinpath findinpath self-requested a review October 3, 2023 13:51
Comment on lines +537 to +539
if (parquetPredicate.isPresent() && options.isUseColumnIndex()) {
return parquetPredicate.get().toParquetFilter(timeZone);
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (parquetPredicate.isPresent() && options.isUseColumnIndex()) {
return parquetPredicate.get().toParquetFilter(timeZone);
}
if (options.isUseColumnIndex()) {
return parquetPredicate.flatMap(predicate -> predicate.toParquetFilter(timeZone));
}

nullableFilterPredicate = filter;
}
else {
nullableFilterPredicate = FilterApi.or(nullableFilterPredicate, filter);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a corresponding test in TestCheckpointEntryIterator

@ebyhr
Copy link
Copy Markdown
Member Author

ebyhr commented Oct 19, 2023

Superceded by #19439

@ebyhr ebyhr closed this Oct 19, 2023
@ebyhr ebyhr deleted the ebi/delta-checkpoint-entry-or branch October 19, 2023 01:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed delta-lake Delta Lake connector hive Hive connector hudi Hudi connector

Development

Successfully merging this pull request may close these issues.

2 participants