Skip to content

Adjust the drop table logic for Delta Lake managed tables#15880

Merged
ebyhr merged 1 commit intotrinodb:masterfrom
findinpath:findinpath/delta-drop-table
Jan 31, 2023
Merged

Adjust the drop table logic for Delta Lake managed tables#15880
ebyhr merged 1 commit intotrinodb:masterfrom
findinpath:findinpath/delta-drop-table

Conversation

@findinpath
Copy link
Copy Markdown
Contributor

Description

Advise the metastore to delete table data only when dealing with managed tables.

Additional context and related issues

Delta Lake same as Hive works with MANAGED and EXTERNAL table types. In case of dealing with MANAGED tables, the engine is responsible (either through delegating internally this responsibility to the metastore - HMS or doing this operation itself in case of AWS) to remove the content of the table when executing DROP TABLE table_name statements.

Release notes

(x) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text:

@cla-bot cla-bot bot added the cla-signed label Jan 27, 2023
@findinpath findinpath self-assigned this Jan 27, 2023
@findinpath findinpath added no-release-notes This pull request does not require release notes entry and removed cla-signed labels Jan 27, 2023
@findinpath findinpath changed the title Adjust the drop table logic for managed tables Adjust the drop table logic for Delta Lake managed tables Jan 27, 2023
@findinpath findinpath requested review from ebyhr and findepi January 31, 2023 05:41
Copy link
Copy Markdown
Member

@ebyhr ebyhr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you rename the argument in TestDeltaLakeSplitManager.dropTable method as well?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: How about renaming to deleteData? We need to guess the behavior from managedTable, but we can easily understand the behavior form deleteData.

@findinpath findinpath force-pushed the findinpath/delta-drop-table branch from bbeb664 to 0d8691a Compare January 31, 2023 06:18
@cla-bot cla-bot bot added the cla-signed label Jan 31, 2023
@ebyhr
Copy link
Copy Markdown
Member

ebyhr commented Jan 31, 2023

/test-with-secrets sha=0d8691a8d20e66144e90961587c2520df73170c6

@findinpath
Copy link
Copy Markdown
Contributor Author

findinpath commented Jan 31, 2023

Stumbled on

Error:  io.trino.plugin.deltalake.TestDeltaTaskFailureRecoveryTest.init  Time elapsed: 74.792 s  <<< FAILURE!
io.airlift.bootstrap.ApplicationConfigurationException: 
Configuration errors:

1) Error: Configuration property 'task-retry-attempts-overall' was not used

Seems to be caused by left-over stuff from 920d14c

I see that there's already a PR for this.

@github-actions
Copy link
Copy Markdown

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/4051475639

@findinpath
Copy link
Copy Markdown
Contributor Author

@ebyhr can you pls trigger again a build with secrets?

@ebyhr
Copy link
Copy Markdown
Member

ebyhr commented Jan 31, 2023

/test-with-secrets sha=0d8691a8d20e66144e90961587c2520df73170c6

@ebyhr
Copy link
Copy Markdown
Member

ebyhr commented Jan 31, 2023

All Databricks product tests succeeded in https://github.com/trinodb/trino/actions/runs/4054233976/jobs/6975812918. Let me merge now.

@ebyhr ebyhr merged commit de429cf into trinodb:master Jan 31, 2023
@github-actions github-actions bot added this to the 407 milestone Feb 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed no-release-notes This pull request does not require release notes entry

Development

Successfully merging this pull request may close these issues.

2 participants