Add delta history metadata table by findinpath · Pull Request #15556 · trinodb/trino

findinpath · 2022-12-29T16:12:26Z

Description

Provide the ability to query the history of changes performed on a Delta Lake table.

The documentation from https://docs.databricks.com/delta/history.html DESCRIBE HISTORY command was used as reference.

Same as in case of Iceberg tables, the history content can be retrieved by doing the query:

SELECT * from "table_name$history";

Fixes #15683

Additional context and related issues

Release notes

(x) Release notes are required, with the following suggested text:

# Delta Lake
* Add `$history` system table which can be queried to inspect Delta Lake table history. ({issue}`15683`)

findinpath · 2022-12-29T16:23:52Z

The content of this class has been copied with small adaptations from trino-iceberg.

I'd be inclined to move the class in trino-spi , but I'm missing a builder class for pages collection.

findinpath · 2022-12-30T09:00:06Z

Although this information is not currently used in Trino, there is some extra info that could be added here:

operationMetrics

userMetadata

engineInfo

Also operationParameters seems to be a bit more complex than only a map of `<string, string> entries.

Why the capital B Boolean? If it may not be present, use Optional<Boolean>

ebyhr

Could you update the docs and add a product test?

findinpath · 2023-01-19T06:47:50Z

Probably job (same as notebook ) is not relevant in the context of Trino.

See https://docs.databricks.com/delta/history.html

Let's remove this column. We can add it later if needed.

findinpath · 2023-01-28T07:23:57Z

It would be nice if this had the same top level schema as the Iceberg one, putting all the connector specific metadata in a nested row type field.
@ebyhr @findepi do you think it's worth the work? Would probably mean changing the schema of the Iceberg one too.

@ebyhr please don't merge this yet.

alexjo2144 · 2023-02-03T18:09:00Z

please don't merge this yet.

I think my comment on the schema is something we can do as follow up

findinpath · 2023-02-07T13:04:33Z

Rebasing on master to deal with the code conflicts.

findinpath · 2023-02-08T14:29:25Z

Rebasing on master to deal with the code conflicts.

alexjo2144 · 2023-02-08T15:06:02Z

Similarly to what @krvikash is working on for Iceberg, we should avoid materializing all the entries here and do a streaming approach instead.

We could do this eventually as a follow-up - in case it is needed.

alexjo2144 · 2023-02-08T15:08:59Z

Could be follow up but being able to push down a predicate on WHERE version > x seems pretty helpful.

ebyhr · 2023-02-10T09:23:06Z

/test-with-secrets sha=a6ae7a7ef5cb89e17fc24f955924f60c3531cfbb

github-actions · 2023-02-10T16:10:52Z

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/4143883144

findinpath · 2023-02-12T09:21:30Z

https://github.com/trinodb/trino/pull/15556/checks?check_run_id=11257287142

GitHub Actions / hive-tests (config-hdp3) with secrets
failed yesterday in 0s

I don't know what happened here.

@ebyhr could you please run the build again (to be on the safe side) ?

ebyhr · 2023-02-12T22:45:36Z

/test-with-secrets sha=a6ae7a7ef5cb89e17fc24f955924f60c3531cfbb

github-actions · 2023-02-12T22:47:01Z

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/4158604825

The transaction log files can be written by query engines (Trino included at the time of this writing) which do not fill this property

Previously this property was being serialized by using `blindAppend` name.

ebyhr · 2023-02-13T08:40:12Z

/test-with-secrets sha=96c80020b151ae66b2c05bd2d64779aeb7b27bf0

github-actions · 2023-02-13T13:21:36Z

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/4162331409

cla-bot Bot added the cla-signed label Dec 29, 2022

findinpath force-pushed the findinpath/delta-history-table branch from 1e50935 to ec53b61 Compare December 29, 2022 16:13

findinpath added release-notes needs-docs This pull request requires changes to the documentation labels Dec 29, 2022

findinpath force-pushed the findinpath/delta-history-table branch from ec53b61 to 150865f Compare December 29, 2022 16:21

findinpath commented Dec 29, 2022

View reviewed changes

findinpath force-pushed the findinpath/delta-history-table branch from 150865f to 5a43b14 Compare December 29, 2022 16:26

findinpath commented Dec 30, 2022

View reviewed changes

findinpath force-pushed the findinpath/delta-history-table branch from 5a43b14 to 9d596c6 Compare December 30, 2022 09:59

krvikash reviewed Dec 30, 2022

View reviewed changes

findinpath force-pushed the findinpath/delta-history-table branch from 9d596c6 to 4993652 Compare December 31, 2022 13:08

github-actions Bot added the tests:hive label Dec 31, 2022

findinpath force-pushed the findinpath/delta-history-table branch 3 times, most recently from 5a2a8ab to bbe9c15 Compare January 4, 2023 10:53

findinpath requested review from findepi and homar January 4, 2023 10:53

homar reviewed Jan 11, 2023

View reviewed changes

Comment thread plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeHistoryTable.java Outdated

homar reviewed Jan 11, 2023

View reviewed changes

Comment thread plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeHistoryTable.java Outdated

homar approved these changes Jan 11, 2023

View reviewed changes

ebyhr reviewed Jan 12, 2023

View reviewed changes

findinpath force-pushed the findinpath/delta-history-table branch 4 times, most recently from 25f3552 to 3dedaa5 Compare January 19, 2023 06:46

findinpath commented Jan 19, 2023

View reviewed changes

github-actions Bot added the docs label Jan 19, 2023

findinpath force-pushed the findinpath/delta-history-table branch 2 times, most recently from c1e8a3c to 95299e2 Compare January 19, 2023 16:01

findinpath requested a review from ebyhr January 20, 2023 05:01

findinpath force-pushed the findinpath/delta-history-table branch from b5189d1 to 7e07120 Compare February 7, 2023 13:11

findinpath requested a review from ebyhr February 7, 2023 13:21

findinpath force-pushed the findinpath/delta-history-table branch from 7e07120 to a6ae7a7 Compare February 8, 2023 14:38

alexjo2144 reviewed Feb 8, 2023

View reviewed changes

findinpath mentioned this pull request Feb 9, 2023

Add WHERE version > x predicate pushdown on Delta $history table #16046

Open

findinpath added 3 commits February 13, 2023 09:38

Mark isBlindAppend as optional

3070622

The transaction log files can be written by query engines (Trino included at the time of this writing) which do not fill this property

Enforce using isBlindAppend property name for JSON serialization

4691b37

Previously this property was being serialized by using `blindAppend` name.

Use the version corresponding to log file name for the log entry

2ea5c51

findinpath force-pushed the findinpath/delta-history-table branch from a6ae7a7 to 96c8002 Compare February 13, 2023 08:39

ebyhr mentioned this pull request Feb 13, 2023

Test: Add delta history metadata table #16084

Closed

ebyhr approved these changes Feb 13, 2023

View reviewed changes

Comment thread plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeHistoryTable.java Outdated

Add history metadata table for Delta Lake

2f05ab6

findinpath force-pushed the findinpath/delta-history-table branch from 96c8002 to 2f05ab6 Compare February 14, 2023 11:47

ebyhr merged commit 0bc4292 into trinodb:master Feb 15, 2023

ebyhr mentioned this pull request Feb 15, 2023

Release notes for 407 #15854

Closed

github-actions Bot added this to the 407 milestone Feb 15, 2023

colebow mentioned this pull request Feb 16, 2023

Add Trino 407 release notes #15919

Merged

findinpath mentioned this pull request Feb 28, 2023

Retrieve the table handle only when dealing with a system table #15568

Merged

Conversation

findinpath commented Dec 29, 2022 • edited by ebyhr Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Additional context and related issues

Release notes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ebyhr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

findinpath commented Jan 28, 2023

Uh oh!

alexjo2144 commented Feb 3, 2023

Uh oh!

findinpath commented Feb 7, 2023

Uh oh!

findinpath commented Feb 8, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ebyhr commented Feb 10, 2023

Uh oh!

github-actions Bot commented Feb 10, 2023

Uh oh!

findinpath commented Feb 12, 2023

Uh oh!

ebyhr commented Feb 12, 2023

Uh oh!

github-actions Bot commented Feb 12, 2023

Uh oh!

ebyhr commented Feb 13, 2023

Uh oh!

github-actions Bot commented Feb 13, 2023

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

findinpath commented Dec 29, 2022 •

edited by ebyhr

Loading