Skip to content

Support snapshot queries on Iceberg system tables - IcebergSystemTableHandle is a ConnectorTableHandle with custom instanceof handling in IcebergMetadata#13497

Closed
findinpath wants to merge 1 commit intotrinodb:masterfrom
findinpath:snapshot-versioned-queries-v2
Closed

Conversation

@findinpath
Copy link
Copy Markdown
Contributor

@findinpath findinpath commented Aug 4, 2022

Description

Use IcebergSystemTableHandle for dealing with system tables
because it encodes for the table being queried also:

  • snapshot id
  • table type

NOTE that this change reuses IcebergMetadata columns for performing queries related to system tables.

Is this change a fix, improvement, new feature, refactoring, or other?

Bugfix

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

Iceberg connector

How would you describe this change to a non-technical end user or system administrator?

The metadata tables exposed by the Iceberg connector for its tables should support snapshot/versioned queries in the same fashion as the Iceberg tables.

SELECT * FROM table1$files FOR VERSION AS OF 1242141
SELECT * FROM table1$files FOR TIMESTAMP AS OF TIMESTAMP '2022-01-01'

Related issues, pull requests, and links

Fixes #12736

This PR is built on top of #12737 and uses IcebergSystemTableHandle for dealing with system tables, instead of IcebergTableHandle.

This PR could pave the way to the functionality of having distributed $files calculation. See #4840

Having a distributed retrieval for $files could be beneficial in debugging scenarios when dealing with tables with a lot of files. Probably other scenarios could follow.

Documentation

(x) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

() No release notes entries required.
(x) Release notes entries required with the following suggested text:

# Iceberg
* Support snapshot queries on Iceberg system tables

@cla-bot cla-bot bot added the cla-signed label Aug 4, 2022
@findinpath findinpath force-pushed the snapshot-versioned-queries-v2 branch 2 times, most recently from 70701a4 to 8879436 Compare August 4, 2022 13:25
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In io.trino.connector.system.SystemTablesMetadata#applyFilter there is made the call

SystemTable systemTable = checkAndGetTable(session, table);

to verify whether the table has disappeared in the meantime.

Should we do it here as well?

@findinpath findinpath force-pushed the snapshot-versioned-queries-v2 branch 2 times, most recently from 77dfbbe to fb3cab1 Compare August 4, 2022 18:16
@findinpath findinpath requested review from electrum and findepi August 5, 2022 03:30
@findinpath findinpath force-pushed the snapshot-versioned-queries-v2 branch from fb3cab1 to c47291e Compare August 8, 2022 09:55
@findinpath findinpath changed the title Support snapshot queries on Iceberg system tables v2 Support snapshot queries on Iceberg system tables Aug 22, 2022
@findinpath findinpath force-pushed the snapshot-versioned-queries-v2 branch from c47291e to e651e40 Compare August 22, 2022 14:07
Use `IcebergSystemTableHandle` for dealing with system tables
because it encodes for the table being queried also:

- snapshot id
- table type

Previously there were supported versioned queries on system
tables because the snapshot id could be encoded within the
table name:

```
SELECT * FROM table1$files@1242141
```

The syntax mentioned previously is now deprecated and
replaced by a richer syntax:

```
SELECT * FROM table1$files FOR VERSION AS OF 1242141
SELECT * FROM table1$files FOR TIMESTAMP AS OF TIMESTAMP '2022-01-01'
```
@findinpath findinpath force-pushed the snapshot-versioned-queries-v2 branch from e651e40 to 80ff57c Compare August 22, 2022 14:08
@findinpath findinpath requested review from alexjo2144 and ebyhr August 22, 2022 14:14
@findinpath findinpath changed the title Support snapshot queries on Iceberg system tables Support snapshot queries on Iceberg system tables - IcebergSystemTableHandle in IcebergMetadata Aug 25, 2022
@findinpath findinpath changed the title Support snapshot queries on Iceberg system tables - IcebergSystemTableHandle in IcebergMetadata Support snapshot queries on Iceberg system tables - IcebergSystemTableHandle is a ConnectorTableHandle with custom handling in IcebergMetadata Aug 25, 2022
@findinpath findinpath changed the title Support snapshot queries on Iceberg system tables - IcebergSystemTableHandle is a ConnectorTableHandle with custom handling in IcebergMetadata Support snapshot queries on Iceberg system tables - IcebergSystemTableHandle is a ConnectorTableHandle with custom instanceof handling in IcebergMetadata Aug 25, 2022
@findinpath findinpath closed this Oct 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

Support time travel on Iceberg system tables

1 participant