Add metadata_log_entries table to Iceberg #20410

oneonestar · 2024-01-18T13:23:36Z

Description

Add metadata_log_entries table to Iceberg

SELECT * FROM "test_table$metadata_log_entries"

             timestamp                 |                                                              file                                                          | latest_snapshot_id  | latest_schema_id | latest_sequence_number
---------------------------------------+----------------------------------------------------------------------------------------------------------------------------+---------------------+------------------+------------------------
 2024-01-16 15:55:31.172 Europe/Vienna | hdfs://hadoop-master:9000/user/hive/warehouse/test_table/metadata/00000-39174715-be2a-48fa-9949-35413b8b736e.metadata.json | 1221802298419195590 |                0 |                      1
 2024-01-16 17:19:56.118 Europe/Vienna | hdfs://hadoop-master:9000/user/hive/warehouse/test_table/metadata/00001-e40178c9-271f-4a96-ad29-eed5e7aef9b0.metadata.json | 7124386610209126943 |                0 |                      2

Additional context and related issues

Reference:
https://iceberg.apache.org/docs/latest/spark-queries/#metadata-log-entries
https://www.dremio.com/blog/getting-started-with-flink-sql-and-apache-iceberg/#h-metadata-log-entries

Release notes

(x) Release notes are required, with the following suggested text:

# Iceberg
* Add support for `metadata_log_entries` system table. ({issue}`20410`)

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/MetadataLogEntriesTable.java

jkylling

Thanks a lot for adding this!

...rino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergMetastoreAccessOperations.java

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorTest.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/MetadataLogEntriesTable.java

...c/test/java/io/trino/plugin/iceberg/catalog/glue/TestIcebergGlueCatalogAccessOperations.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/MetadataLogEntriesTable.java

findinpath

LGTM % comments

findinpath · 2024-02-20T10:42:35Z

@ebyhr could you please run this PR with secrets ?
I see changes in TestIcebergGlueCatalogAccessOperations.

ebyhr · 2024-02-20T10:49:10Z

/test-with-secrets sha=328415474d00cd4b90a29150616b84bde9f445f2

github-actions · 2024-02-20T12:40:45Z

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/7972684667

ebyhr · 2024-02-20T13:30:39Z

/test-with-secrets sha=8aa32a6a8bdb63d586c9fe3179a6ab0980c928e6

ebyhr · 2024-02-20T13:41:26Z

Rebased on master to fix CI failure with secrets.

github-actions · 2024-02-20T15:30:54Z

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/7975058140

oneonestar · 2024-03-11T07:52:52Z

Rebased to master and resolved conflicts.

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorTest.java

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergSystemTables.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/MetadataLogEntriesTable.java

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorTest.java

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergSystemTables.java

...rino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergMetastoreAccessOperations.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/BaseSystemTable.java

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorTest.java

oneonestar · 2024-03-12T06:53:58Z

OPTIMIZE creates two metadata log entries since 947f972

Updated the test accordingly at https://github.com/trinodb/trino/pull/20410/files#diff-2c77e347c613ee81e28953f6b552f710e44bb21e3f75b140770226ac7a1647ceR239-R251

ebyhr · 2024-03-18T01:15:11Z

@oneonestar Could you rebase on master as #21034 (Iceberg 1.5.0 bump) landed. Sorry, I missed merging this PR.

oneonestar · 2024-03-18T02:18:31Z

Rebased and updated tests. latest_snapshot_id is no longer null after RTAS.

oneonestar · 2024-03-18T02:53:41Z

The error seems unrelated.

cla-bot bot added the cla-signed label Jan 18, 2024

github-actions bot added docs iceberg Iceberg connector labels Jan 18, 2024

oneonestar force-pushed the ICEBERG_METADATA_LOG_ENTRIES branch 3 times, most recently from b0f67c5 to 4a04f0e Compare January 19, 2024 04:52

findinpath self-requested a review January 22, 2024 05:28

findinpath reviewed Jan 22, 2024

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/MetadataLogEntriesTable.java Outdated Show resolved Hide resolved

oneonestar requested a review from findinpath January 29, 2024 01:31

oneonestar mentioned this pull request Feb 5, 2024

Add mechanism to get metadata file location of Iceberg table #20584

Closed

jkylling self-requested a review February 5, 2024 20:23

jkylling reviewed Feb 6, 2024

View reviewed changes

oneonestar force-pushed the ICEBERG_METADATA_LOG_ENTRIES branch 2 times, most recently from 02d37e7 to 63c7633 Compare February 7, 2024 13:27

oneonestar requested a review from jkylling February 8, 2024 00:11

jkylling approved these changes Feb 12, 2024

View reviewed changes

oneonestar force-pushed the ICEBERG_METADATA_LOG_ENTRIES branch 2 times, most recently from 575ba33 to 3284154 Compare February 17, 2024 23:58

findinpath mentioned this pull request Feb 19, 2024

Null values in metadata_log_entries after replacing Table apache/iceberg#9723

Closed

findinpath reviewed Feb 20, 2024

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/MetadataLogEntriesTable.java Outdated Show resolved Hide resolved

findinpath approved these changes Feb 20, 2024

View reviewed changes

ebyhr force-pushed the ICEBERG_METADATA_LOG_ENTRIES branch from 3284154 to 8aa32a6 Compare February 20, 2024 13:30

ebyhr self-requested a review February 20, 2024 22:43

oneonestar force-pushed the ICEBERG_METADATA_LOG_ENTRIES branch from 8aa32a6 to 464b7db Compare February 22, 2024 02:01

oneonestar force-pushed the ICEBERG_METADATA_LOG_ENTRIES branch 2 times, most recently from 8d3c144 to 44bb855 Compare March 11, 2024 07:33

ebyhr reviewed Mar 11, 2024

View reviewed changes

oneonestar force-pushed the ICEBERG_METADATA_LOG_ENTRIES branch from 7dc18a1 to d7547f1 Compare March 12, 2024 01:16

oneonestar requested a review from ebyhr March 12, 2024 01:24

ebyhr force-pushed the ICEBERG_METADATA_LOG_ENTRIES branch from d7547f1 to c9aecce Compare March 12, 2024 01:28

ebyhr reviewed Mar 12, 2024

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/BaseSystemTable.java Outdated Show resolved Hide resolved

ebyhr reviewed Mar 12, 2024

View reviewed changes

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorTest.java Outdated Show resolved Hide resolved

oneonestar force-pushed the ICEBERG_METADATA_LOG_ENTRIES branch 5 times, most recently from f41d0e8 to 1f8851a Compare March 12, 2024 06:34

oneonestar requested a review from ebyhr March 12, 2024 07:30

ebyhr approved these changes Mar 14, 2024

View reviewed changes

oneonestar force-pushed the ICEBERG_METADATA_LOG_ENTRIES branch from 1f8851a to f4337bb Compare March 18, 2024 01:43

oneonestar added 2 commits March 25, 2024 11:50

Extract base class from SnapshotsTable

ad40089

Add metadata_log_entries table to Iceberg

d673332

ebyhr force-pushed the ICEBERG_METADATA_LOG_ENTRIES branch from f4337bb to d673332 Compare March 25, 2024 02:51

ebyhr merged commit 8cec8cd into trinodb:master Mar 25, 2024

github-actions bot added this to the 444 milestone Mar 25, 2024

mosabua mentioned this pull request Mar 25, 2024

Add Trino 444 release notes #21212

Merged

colebow mentioned this pull request Mar 27, 2024

Add Trino 444 release notes #21293

Closed

findinpath mentioned this pull request Mar 29, 2024

Make Iceberg to Spark compatibility tests more resilient to retries #21310

Merged

oneonestar deleted the ICEBERG_METADATA_LOG_ENTRIES branch January 7, 2025 07:26

Add metadata_log_entries table to Iceberg #20410

Add metadata_log_entries table to Iceberg #20410

Uh oh!

Conversation

oneonestar commented Jan 18, 2024 • edited by ebyhr Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Additional context and related issues

Release notes

Uh oh!

Uh oh!

jkylling left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

findinpath left a comment

Choose a reason for hiding this comment

Uh oh!

findinpath commented Feb 20, 2024

Uh oh!

ebyhr commented Feb 20, 2024

Uh oh!

github-actions bot commented Feb 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ebyhr commented Feb 20, 2024

Uh oh!

ebyhr commented Feb 20, 2024

Uh oh!

github-actions bot commented Feb 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oneonestar commented Mar 11, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

oneonestar commented Mar 12, 2024

Uh oh!

ebyhr commented Mar 18, 2024

Uh oh!

oneonestar commented Mar 18, 2024

Uh oh!

oneonestar commented Mar 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

oneonestar commented Jan 18, 2024 •

edited by ebyhr

Loading

github-actions bot commented Feb 20, 2024 •

edited

Loading

github-actions bot commented Feb 20, 2024 •

edited

Loading