Skip to content

Support MERGE on cloned table in Delta Lake#24756

Merged
ebyhr merged 1 commit intotrinodb:masterfrom
chenjian2664:delta_clone
Mar 21, 2025
Merged

Support MERGE on cloned table in Delta Lake#24756
ebyhr merged 1 commit intotrinodb:masterfrom
chenjian2664:delta_clone

Conversation

@chenjian2664
Copy link
Copy Markdown
Contributor

@chenjian2664 chenjian2664 commented Jan 21, 2025

Description

Fix problem that fail update on cloned table, reproduce steps:

testing/bin/ptl env up --environment singlenode-delta-lake-oss

In Trino: create schema delta.tiny with (location='s3://test-bucket/tiny/');

In Spark-sql: CREATE TABLE tiny.t1 (id int, v string, part date) USING DELTA PARTITIONED BY (part);

In Trino: insert into delta.tiny.t1 values (1, 'A', TIMESTAMP '2024-01-01'), (2, 'B', TIMESTAMP '2024-01-01'), (3, 'C', TIMESTAMP '2024-02-02'), (4, 'D', TIMESTAMP '2024-02-02');

In Spark-sql: CREATE TABLE tiny.t1clone SHALLOW CLONE tiny.t1;

In Trino: update delta.tiny.t1clone set v = 'update1' where id in (1,3); It fails with:

Query 20240904_130833_00010_r4mc9 failed: path [s3://test-bucket/tiny/t1/part=2024-02-02/20240904_125421_00003_r4mc9_8e16bc1c-e33d-45da-b085-427d66e55960] must be a subdirectory of basePath [s3://test-bucket/tiny/t1clone]

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

## Delta Lake
* Fix failure for `MERGE` queries on [cloned](https://delta.io/blog/delta-lake-clone/) tables. ({issue}`24756`)

@cla-bot cla-bot bot added the cla-signed label Jan 21, 2025
@github-actions github-actions bot added the delta-lake Delta Lake connector label Jan 21, 2025
@chenjian2664 chenjian2664 force-pushed the delta_clone branch 6 times, most recently from 37ff93a to 60e9e73 Compare January 21, 2025 11:47
@ebyhr ebyhr requested review from ebyhr, pajaks and vinay-kl January 22, 2025 04:20
@chenjian2664 chenjian2664 force-pushed the delta_clone branch 2 times, most recently from 9bc68fc to 0cb664f Compare January 23, 2025 08:52
@chenjian2664 chenjian2664 force-pushed the delta_clone branch 2 times, most recently from 7ef5f5d to be52ef7 Compare January 26, 2025 07:59
@chenjian2664 chenjian2664 requested a review from pajaks January 27, 2025 07:30
@chenjian2664 chenjian2664 force-pushed the delta_clone branch 3 times, most recently from b583ce9 to e8d8d33 Compare February 4, 2025 14:20
@chenjian2664 chenjian2664 requested a review from pajaks February 4, 2025 14:30
@chenjian2664
Copy link
Copy Markdown
Contributor Author

Copy link
Copy Markdown
Member

@pajaks pajaks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ebyhr
Copy link
Copy Markdown
Member

ebyhr commented Feb 12, 2025

/test-with-secrets sha=0b2cb43ae47dea47a9f7fb86d745453699188cb3

@github-actions
Copy link
Copy Markdown

The CI workflow run with tests that require additional secrets has been started: https://github.com/trinodb/trino/actions/runs/13281480006

@ebyhr
Copy link
Copy Markdown
Member

ebyhr commented Feb 18, 2025

@chenjian2664 Please throw exceptions for unclear situations. This is initial merge support for cloned tables. We should start as strict as possible.

@chenjian2664 chenjian2664 force-pushed the delta_clone branch 3 times, most recently from 463ab8e to 0f5aa9c Compare February 18, 2025 10:45
@chenjian2664 chenjian2664 requested review from ebyhr and pajaks March 3, 2025 02:21
@ebyhr
Copy link
Copy Markdown
Member

ebyhr commented Mar 21, 2025

(Rebased on master with no changes)

@ebyhr ebyhr merged commit de86920 into trinodb:master Mar 21, 2025
46 checks passed
@github-actions github-actions bot added this to the 474 milestone Mar 21, 2025
@chenjian2664 chenjian2664 deleted the delta_clone branch March 27, 2025 12:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed delta-lake Delta Lake connector

Development

Successfully merging this pull request may close these issues.

4 participants