Skip to content

Support type coercion for structural types in Iceberg and Delta Lake #21721

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
May 21, 2024
Merged

Support type coercion for structural types in Iceberg and Delta Lake #21721

merged 4 commits into from
May 21, 2024

Conversation

@ebyhr
Copy link
Member

@ebyhr ebyhr commented Apr 26, 2024 •
edited
Loading

Description

Fixes #21511
Supersedes #21055 with connector side approach

Release notes

(x) Release notes are required, with the following suggested text:

# Delta Lake, Iceberg
* Automatically use `varchar` in struct types as a type during table creation when `char` is specified. ({issue}`21511`)

# Iceberg
* Automatically use microsecond precision for temporal types in struct types as a type during table creation. ({issue}`21511`)

@ebyhr ebyhr requested review from Praveen2112 and mdesmet April 26, 2024 08:06
@cla-bot cla-bot bot added the cla-signed label Apr 26, 2024
@github-actions github-actions bot added iceberg Iceberg connector delta-lake Delta Lake connector labels Apr 26, 2024
@findinpath findinpath requested review from SemionPar and pajaks April 26, 2024 17:58
SemionPar
SemionPar approved these changes Apr 29, 2024
View reviewed changes
Copy link
Contributor

@SemionPar SemionPar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@SemionPar
Copy link
Contributor

SemionPar commented Apr 29, 2024

To avoid conflicts, basing char coercion in Delta (ebyhr#1) on this branch

@findinpath findinpath mentioned this pull request May 3, 2024
Closed
@ebyhr ebyhr requested a review from wendigo May 8, 2024 08:37
Praveen2112
Praveen2112 reviewed May 9, 2024
View reviewed changes
plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorTest.java Outdated
Copy link
Member

@Praveen2112 Praveen2112 May 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add test for nested structural types ?

Copy link
Member

@Praveen2112 Praveen2112 May 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to add them for delta lake ?

Copy link
Member Author

@ebyhr ebyhr May 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current helper class of TestDeltaLakeConnectorTest can't those types nicely. Let me handle in follow-up with TestSetup refactoring.

@ebyhr ebyhr force-pushed the ebi/iceberg-supported-type-complex branch from fb7dd29 to e47c5c7 Compare May 9, 2024 09:00
@ebyhr ebyhr marked this pull request as draft May 10, 2024 01:30
@ebyhr ebyhr force-pushed the ebi/iceberg-supported-type-complex branch from e47c5c7 to 32f4865 Compare May 17, 2024 23:03
@ebyhr ebyhr marked this pull request as ready for review May 18, 2024 01:39
@ebyhr ebyhr requested a review from Praveen2112 May 18, 2024 01:40
@ebyhr ebyhr force-pushed the ebi/iceberg-supported-type-complex branch from 32f4865 to 6aa2efc Compare May 18, 2024 11:59
@ebyhr ebyhr force-pushed the ebi/iceberg-supported-type-complex branch from 6aa2efc to 89610e0 Compare May 21, 2024 01:14
@ebyhr ebyhr merged commit debda27 into trinodb:master May 21, 2024
@github-actions github-actions bot added this to the 449 milestone May 21, 2024
@colebow colebow mentioned this pull request May 29, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Praveen2112 Praveen2112 Praveen2112 approved these changes

@mdesmet mdesmet Awaiting requested review from mdesmet

@pajaks pajaks Awaiting requested review from pajaks

@wendigo wendigo Awaiting requested review from wendigo

+1 more reviewer

@SemionPar SemionPar SemionPar approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

cla-signed delta-lake Delta Lake connector iceberg Iceberg connector

Milestone

449

Development

Successfully merging this pull request may close these issues.

Automatic type coercion doesn't work for structural types

3 participants

@ebyhr @SemionPar @Praveen2112