Skip to content

Enable predicate pushdown for TIMESTAMP type in Delta Lake#12344

Merged
ebyhr merged 1 commit intotrinodb:masterfrom
ebyhr:ebi/delta-timestamp-pushdown
May 19, 2022
Merged

Enable predicate pushdown for TIMESTAMP type in Delta Lake#12344
ebyhr merged 1 commit intotrinodb:masterfrom
ebyhr:ebi/delta-timestamp-pushdown

Conversation

@ebyhr
Copy link
Copy Markdown
Member

@ebyhr ebyhr commented May 12, 2022

Description

Documentation

(x) No documentation is needed.

Release notes

( ) Release notes entries required with the following suggested text:

# Delta Lake
* Support predicate pushdown for `timestamp with time zone` type. ({issue}`12007`)

@cla-bot cla-bot bot added the cla-signed label May 12, 2022
Comment on lines 185 to 186
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MiguelWeezardo i remember you originally added these lines

the TODO went with TestSplitPruning.testTimestampPruning
the condition went with some TestDeltaLakeReadTimestamps changes

please review

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move from smoke test to Delta Connector Test. keep smoke test lightweight

@ebyhr ebyhr force-pushed the ebi/delta-timestamp-pushdown branch from e4470a9 to 653378b Compare May 13, 2022 08:28
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add entries to test handling of gaps and ambiguous values in the JVM time zone. Note that these may appear as partition values as well.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MiguelWeezardo Added a below entry. Please take another look.

1970-01-01 01:13:42.000 America/Bahia_Banderas

@ebyhr ebyhr force-pushed the ebi/delta-timestamp-pushdown branch from 653378b to 1aed1c0 Compare May 17, 2022 00:24
@alexjo2144
Copy link
Copy Markdown
Member

I was just looking back through the commit history. We added this todo before implementing #4104 and I guess I never remembered to come back and enable it for Delta.

@ebyhr ebyhr requested a review from findepi May 19, 2022 12:19
@ebyhr ebyhr merged commit 21c6af1 into trinodb:master May 19, 2022
@ebyhr ebyhr deleted the ebi/delta-timestamp-pushdown branch May 19, 2022 14:26
@ebyhr ebyhr mentioned this pull request May 19, 2022
@github-actions github-actions bot added this to the 382 milestone May 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

Add Timestamp predicate pushdown to Parquet in Delta Lake

4 participants