Skip to content

Add support for writing timestamp_ntz type in Delta#18913

Merged
ebyhr merged 2 commits intomasterfrom
ebi/delta-timestamp-ntz-write
Sep 20, 2023
Merged

Add support for writing timestamp_ntz type in Delta#18913
ebyhr merged 2 commits intomasterfrom
ebi/delta-timestamp-ntz-write

Conversation

@ebyhr
Copy link
Copy Markdown
Member

@ebyhr ebyhr commented Sep 5, 2023

Description

Add support for writing timestamp_ntz type in Delta

Fixes #16927

Release notes

(x) Release notes are required, with the following suggested text:

# Delta Lake
* Add support for writing `timestamp` type. ({issue}`16927`)

@cla-bot cla-bot bot added the cla-signed label Sep 5, 2023
@github-actions github-actions bot added tests:hive delta-lake Delta Lake connector labels Sep 5, 2023
@ebyhr ebyhr self-assigned this Sep 5, 2023
@ebyhr ebyhr force-pushed the ebi/delta-timestamp-ntz-write branch from ce7a6dd to 967ae8c Compare September 5, 2023 02:01
@github-actions github-actions bot added the docs label Sep 5, 2023
@ebyhr ebyhr force-pushed the ebi/delta-timestamp-ntz-write branch 4 times, most recently from 0dfb632 to 4805c17 Compare September 11, 2023 08:54
@ebyhr ebyhr force-pushed the ebi/delta-timestamp-ntz-write branch 2 times, most recently from 7b1590a to 149d70a Compare September 19, 2023 00:49
@ebyhr ebyhr marked this pull request as ready for review September 19, 2023 00:50
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why discern between empty Set of features and no set of features?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ProtocolEntry has the following logic, so Optional.of(Set.of()) throws an exception for lower reader & writer versions. We could change the logic alternatively.

        if (minReaderVersion < MIN_VERSION_SUPPORTS_READER_FEATURES && readerFeatures.isPresent()) {
            throw new IllegalArgumentException("readerFeatures must not exist when minReaderVersion is less than " + MIN_VERSION_SUPPORTS_READER_FEATURES);
        }
        if (minWriterVersion < MIN_VERSION_SUPPORTS_WRITER_FEATURES && writerFeatures.isPresent()) {
            throw new IllegalArgumentException("writerFeatures must not exist when minWriterVersion is less than " + MIN_VERSION_SUPPORTS_WRITER_FEATURES);
        }

@ebyhr ebyhr force-pushed the ebi/delta-timestamp-ntz-write branch from 149d70a to 99bc09d Compare September 19, 2023 23:42
@ebyhr ebyhr force-pushed the ebi/delta-timestamp-ntz-write branch from 99bc09d to 1cbb6a2 Compare September 20, 2023 00:37
@ebyhr
Copy link
Copy Markdown
Member Author

ebyhr commented Sep 20, 2023

Rebased on master to resolve conflicts.

@ebyhr ebyhr merged commit 005e84b into master Sep 20, 2023
@ebyhr ebyhr deleted the ebi/delta-timestamp-ntz-write branch September 20, 2023 02:19
@github-actions github-actions bot added this to the 427 milestone Sep 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Development

Successfully merging this pull request may close these issues.

Add support for timestamp without time zone (TIMESTAMP_NTZ) type in Delta Lake connector

2 participants