Support Iceberg time travel by jackye1995 · Pull Request #10258 · trinodb/trino

jackye1995 · 2021-12-10T07:28:02Z

Support AS OF TIMESTAMP and AS OF VERSION queries for Iceberg.

@electrum @findepi @losipiuk @alexjo2144

jackye1995 · 2021-12-10T07:31:08Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

It's quite difficult to test DATE type, I'm still thinking about how to simulate that... meanwhile I am just supporting the 2 timestamp types.

It's also unclear to me if we should take the beginning of a day or end of a day for time travel, so I think it's better to just not support it until explicit user request.

I agree, using dates would be problematic as Iceberg stores the snapshot timestamp in millis.

It's also unclear to me if we should take the beginning of a day or end of a day

TIMESTAMP 2021-01-02 03:04:05 means TIMESTAMP 2021-01-02 03:04:05.000000000
TIMESTAMP 2021-01-02 03:04 means TIMESTAMP 2021-01-02 03:04:00.000000000
TIMESTAMP 2021-01-02 means TIMESTAMP 2021-01-02 00:00:00.000000000
DATE 2021-01-02 means TIMESTAMP 2021-01-02

i.e. beginning of the day

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergReadVersionedTable.java

losipiuk

Skimmed

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java

alexjo2144

Overall, looks good to me.

One more general testing question, what happens if you have the FOR TIMESTAMP AS OF syntax in a DML statement like an insert? Does that parse?

alexjo2144 · 2022-01-11T16:22:08Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergReadVersionedTable.java

Any reason not to reuse the code in IcebergQueryRunner#createIcebergQueryRunner?

because I need the catalog object to perform operations like loadTable and inspect information in the table.

Please do use a helper method like the following

private Optional<Long> getLatestSnapshotId(String tableName) { MaterializedResult result = computeActual(format("SELECT snapshot_id FROM \"%s$snapshots\" ORDER BY committed_at DESC LIMIT 1", tableName)); return result.getRowCount() > 0 ? Optional.of((Long) result.getOnlyValue()) : Optional.empty(); }

and replace the createQueryRunner() method with:

@Override protected DistributedQueryRunner createQueryRunner() throws Exception { return createIcebergQueryRunner(); }

In this fashion you can avoid the low-level details of building from scratch the IcebergQueryRunner and working with Iceberg specific API and the maintenance of the test can be made easier.

alexjo2144 · 2022-01-11T16:28:34Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java

More of a semantic question for the AS OF syntax, but should this fail or should it return an empty set of data?

I think it should fail, returning an empty set of data should mean the table snapshot at that point of time actually contains no data.

And this will also match the Spark SQL behavior in Iceberg.

I agree with @jackye1995 that the query should fail instead of returning an empty set of data.

jackye1995 · 2022-01-11T16:51:33Z

@alexjo2144 thanks for the review!

what happens if you have the FOR TIMESTAMP AS OF syntax in a DML statement like an insert

if you are talking about INSERT INTO SELECT or CREATE TABLE AS SELECT it should work.

If you are talking about INSERT INTO table FOR TIMESTAMP AS OF xxx it does not parse:

Caused by: io.trino.sql.parser.ParsingException: line 1:59: mismatched input 'FOR'. Expecting: '(', <query>
	at io.trino.sql.parser.ErrorHandler.syntaxError(ErrorHandler.java:109)
	at org.antlr.v4.runtime.ProxyErrorListener.syntaxError(ProxyErrorListener.java:41)
	at org.antlr.v4.runtime.Parser.notifyErrorListeners(Parser.java:544)
	at org.antlr.v4.runtime.DefaultErrorStrategy.reportInputMismatch(DefaultErrorStrategy.java:327)
	at org.antlr.v4.runtime.DefaultErrorStrategy.reportError(DefaultErrorStrategy.java:139)
	at io.trino.sql.parser.SqlBaseParser.statement(SqlBaseParser.java:4909)
	at io.trino.sql.parser.SqlBaseParser.singleStatement(SqlBaseParser.java:290)
	at io.trino.sql.parser.SqlParser.invokeParser(SqlParser.java:145)
	at io.trino.sql.parser.SqlParser.createStatement(SqlParser.java:85)
	at io.trino.execution.QueryPreparer.prepareQuery(QueryPreparer.java:55)

findinpath · 2022-01-13T09:48:19Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java

Please do check whether the timestamp is in the future.
I think we shouldn't allow future timestamps here.

why not? I think there could be a valid use case where a process provide a fixed timestamp for query to get consistent query result, and the timestamp could be in the future at the beginning.

@findinpath i think this is a really important question.
The time travel syntax is supposed to provide a consistent, time-related view of a table.
Allowing timestamps in the future doesn't provide this, since future state naturally still may change.
i think this should be handled on the engine level though.

findinpath · 2022-01-13T10:05:06Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java

From a user perspective, if the oldest history timestamp is shown, I think it should be formatted to be easily readable.

Use for reference (in case you want to show the timestamp to the user) :

https://github.com/trinodb/trino/blob/master/plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/HistoryTable.java#L85

https://github.com/trinodb/trino/blob/master/core/trino-spi/src/main/java/io/trino/spi/type/Timestamps.java#L147 (this corresponds to the what is being seen in the trino cli when retrieving select current_timestamp;)

Just as an alternative, Snowflake uses the following message in such situations:
Time travel data is not available for table %s. The requested time is either beyond the allowed time travel period or before the object creation time.

Thanks, I have made the changes (similar to the Snowflake's message).

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

cla-bot · 2022-01-24T12:31:05Z

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please submit the signed CLA to cla@trino.io. For more information, see https://github.com/trinodb/cla.

rajarshisarkar · 2022-05-25T11:54:49Z

I have rebased the changes.

findepi · 2022-05-25T12:46:13Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergReadVersionedTable.java

Please refactor the test to sleep at most few millis.

findepi · 2022-05-25T12:47:14Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergReadVersionedTable.java

"unixtime" can be misunderstood.

call it "epochSeconds"

(same below)

findepi · 2022-05-25T12:48:32Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergReadVersionedTable.java

You don't need a new schema. Use the default schema provided by the query runner.

(i.e. don't qualify table references with schema at all)

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

findepi · 2022-05-25T13:20:07Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

Avoid string concatenation when condition holds

verify(versionType == BIGINT, "Iceberg target ID time travel only supports type Bigint, but got %s", versionType);

findepi · 2022-05-25T13:26:18Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

I am not sure that the interpretation in UTC is the right one.
This is user-provided, so interpreting in session zone is probably appropriate.

cc @martint

bump.

for now you can remove timestamp support from here, to unblock the PR

Sure, I have removed the timestamp support.

findepi · 2022-05-25T13:28:39Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

It's also unclear to me if we should take the beginning of a day or end of a day

TIMESTAMP 2021-01-02 03:04:05 means TIMESTAMP 2021-01-02 03:04:05.000000000
TIMESTAMP 2021-01-02 03:04 means TIMESTAMP 2021-01-02 03:04:00.000000000
TIMESTAMP 2021-01-02 means TIMESTAMP 2021-01-02 00:00:00.000000000
DATE 2021-01-02 means TIMESTAMP 2021-01-02

i.e. beginning of the day

findepi · 2022-05-25T13:29:40Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java

What is the order of table.history() elements?
is it guaranteed?

(the API doesn't seem to document this and this is being relied on)

In the increasing order of timestamp. Yes, it is guaranteed (it's being used in SnapshotUtil class of Iceberg as well).

Yes, it is guaranteed

I can read the implementation myself, and the usages.
Still, to me "guaranteed" means something more than that.

It's not "guaranteed" unless documentation says it is guaranteed.
So for now we should do filter + max.

Or, make a PR to Iceberg to update table.history() documentation.

Was going through this, just wanted to mention I don't think table.history() ordered is guaranteed due to staged commits. See apache/iceberg#3891 (comment) for more details.

It's part the reason why for time travel lookups of snapshots, we don't leverage binary search

Thanks @amogh-jahagirdar for the reference

@rajarshisarkar can you add that link as an explanatory code comment?

Ah, I see. Sure, let me use the filter + max logic here and add the link as an explanatory code comment.

findepi · 2022-05-25T13:30:19Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java

nit: import format statically

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java

findinpath · 2022-05-25T14:00:35Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergReadVersionedTable.java

Use the pattern from io.trino.plugin.iceberg.BaseIcebergConnectorTest#getLatestSnapshotId or extract an utility method

findepi · 2022-05-26T14:09:45Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

Suggested change

if (table.snapshot((long) version.getVersion()) == null) {

throw new TrinoException(GENERIC_USER_ERROR, "Iceberg target ID does not exists: " + version.getVersion());

}

return (long) version.getVersion();

ilong snapshotId = (long) version.getVersion();

if (table.snapshot(snapshotId) == null) {

throw new TrinoException(GENERIC_USER_ERROR, "Iceberg snapshot ID does not exists: " + snapshotId);

}

return snapshotId;

findepi · 2022-05-26T14:11:09Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

bump.

for now you can remove timestamp support from here, to unblock the PR

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

findepi · 2022-05-26T14:13:33Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java

Suggested change

Long snapshotId = null;

for (HistoryEntry logEntry : table.history()) {

if (logEntry.timestampMillis() <= timestampMillis) {

snapshotId = logEntry.snapshotId();

}

}

if (snapshotId == null) {

throw new TrinoException(GENERIC_USER_ERROR, format("Time travel data is not available for table " +

"%s. The requested time is beyond the allowed time travel period.", table.name()));

}

return snapshotId;

return table.history().stream()

.filter(logEntry -> logEntry.timestampMillis() <= timestampMillis)

.max(comparing(HistoryEntry::timestampMillis))

.orElseThrow(() -> new TrinoException(GENERIC_USER_ERROR, format(

"Time travel data is not available for table %s. The requested time is beyond the allowed time travel period.",

table.name())))

.snapshotId();

findepi · 2022-05-26T14:14:04Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergReadVersionedTable.java

unixtime -> epochMillis

findepi · 2022-05-26T14:14:28Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergReadVersionedTable.java

Please refactor the test to sleep at most few millis.

bump

findepi · 2022-05-26T14:15:46Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergReadVersionedTable.java

v1SnapshotId is the ID of the v1 snapshot
v1Unixtime should be timestamp of the v1 snapshot, not "some timestamp after that"
read actual snapshot's timestamp (committed_at) instead of assuming (otherwise code requires explanation)

Makes sense to me, I have made the change. Thanks, for the suggestion.

findepi · 2022-05-26T14:16:13Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergReadVersionedTable.java

redundant, query runner cleanup will do that

findepi · 2022-05-26T14:17:16Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergReadVersionedTable.java

inline timestamp(1, 8)

just use some fixed timestamp that's before today, as a SQL literal value

Sure, I have used TIMESTAMP '1970-01-01 00:00:00.001000000 Z' now

findepi · 2022-05-26T14:22:22Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergReadVersionedTable.java

from_unixtime semantics are never obvious to me (eg is it session zone dependent?)

Make the code more explicit

private String timestampLiteral(long epochSeconds, int precision) { return DateTimeFormatter.ofPattern("'TIMESTAMP '''uuuu-MM-dd HH:mm:ss." + "S".repeat(precision) + " VV''") .format(Instant.ofEpochMilli(epochSeconds).atZone(UTC)); }

this also frees you from needing to wrap the values in CAST and differentiate between "short" and "long".

This looks a lot cleaner, I have made the change. Thanks, for the suggestion.

findepi · 2022-05-26T14:30:54Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergReadVersionedTable.java

"throws" should be on next line

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

findepi · 2022-05-31T12:56:40Z

@rajarshisarkar did you maybe have time to apply the comments?
or, i can help if you prefer.

rajarshisarkar · 2022-05-31T13:29:47Z

@findepi I will update the PR this week.

rajarshisarkar · 2022-06-02T04:43:31Z

@findepi I have addressed the review comments + rebased the changes.

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergReadVersionedTable.java

findepi · 2022-06-07T11:21:50Z

@rajarshisarkar i've applied some changes myself.
will merge when green

findepi · 2022-06-07T19:31:37Z

@rajarshisarkar @jackye1995 thank you!

cla-bot bot added the cla-signed label Dec 10, 2021

jackye1995 requested a review from findepi December 10, 2021 07:28

jackye1995 commented Dec 10, 2021

View reviewed changes

jackye1995 requested review from alexjo2144, jacob-stein and losipiuk December 10, 2021 18:41

losipiuk reviewed Dec 13, 2021

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java Outdated Show resolved Hide resolved

losipiuk reviewed Dec 13, 2021

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java Outdated Show resolved Hide resolved

losipiuk reviewed Dec 13, 2021

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java Outdated Show resolved Hide resolved

losipiuk reviewed Dec 13, 2021

View reviewed changes

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergReadVersionedTable.java Outdated Show resolved Hide resolved

losipiuk reviewed Dec 13, 2021

View reviewed changes

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergReadVersionedTable.java Outdated Show resolved Hide resolved

losipiuk reviewed Dec 13, 2021

View reviewed changes

jackye1995 mentioned this pull request Dec 13, 2021

Spark: Use snapshot schema when reading snapshot apache/iceberg#3722

Merged

jackye1995 force-pushed the time-travel branch 3 times, most recently from 2124cdb to 891221f Compare December 15, 2021 04:19

jackye1995 requested a review from losipiuk December 15, 2021 04:37

jackye1995 force-pushed the time-travel branch from 891221f to d47082f Compare December 15, 2021 04:41

alexjo2144 reviewed Dec 20, 2021

View reviewed changes

jackye1995 force-pushed the time-travel branch from d47082f to 82af1cc Compare January 11, 2022 07:07

jackye1995 requested a review from alexjo2144 January 11, 2022 07:07

jackye1995 force-pushed the time-travel branch from 82af1cc to aaf1cd9 Compare January 11, 2022 07:12

alexjo2144 reviewed Jan 11, 2022

View reviewed changes

jackye1995 requested a review from alexjo2144 January 11, 2022 16:59

alexjo2144 approved these changes Jan 11, 2022

View reviewed changes

phd3 mentioned this pull request Jan 12, 2022

Document Iceberg time travel & snapshot queries #10515

Merged

findinpath reviewed Jan 13, 2022

View reviewed changes

cla-bot bot removed the cla-signed label Jan 24, 2022

findepi requested changes May 25, 2022

View reviewed changes

findinpath reviewed May 25, 2022

View reviewed changes

rajarshisarkar force-pushed the time-travel branch 2 times, most recently from b5e412b to 935996e Compare May 26, 2022 13:33

findepi reviewed May 26, 2022

View reviewed changes

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergReadVersionedTable.java Outdated

Copy link

Member

findepi May 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"throws" should be on next line

alexjo2144 reviewed May 26, 2022

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java Outdated Show resolved Hide resolved

rajarshisarkar force-pushed the time-travel branch from 935996e to 2008a31 Compare June 1, 2022 11:54

alexjo2144 mentioned this pull request Jun 1, 2022

Test AS OF syntax with table redirections #12634

Closed

findepi reviewed Jun 7, 2022

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java Outdated Show resolved Hide resolved

Support Iceberg time travel

1212853

findepi reviewed Jun 7, 2022

View reviewed changes

findepi force-pushed the time-travel branch from 2008a31 to 1212853 Compare June 7, 2022 11:20

findepi approved these changes Jun 7, 2022

View reviewed changes

findepi merged commit 4a0f84f into trinodb:master Jun 7, 2022

findepi mentioned this pull request Jun 7, 2022

Deprecate @ based syntax for accessing iceberg snapshots #10768

Merged

This was referenced Jun 7, 2022

Iceberg Connector #1324

Closed

Release notes for 385 #12686

Closed

Time travel engine improvements #12542

Merged

github-actions bot added this to the 385 milestone Jun 7, 2022

This was referenced Jun 8, 2022

Add Trino 385 release notes #12703

Closed

Document Iceberg time travel #12745

Closed

javrod87 mentioned this pull request Aug 25, 2022

Time Travel (querying the historical versions of data) ability for Hudi Table prestodb/presto#18229

Open

findepi mentioned this pull request Sep 13, 2022

Fix incorrect metadata for Iceberg table after rollback #13704

Closed

-                if (table.snapshot((long) version.getVersion()) == null) {
-                    throw new TrinoException(GENERIC_USER_ERROR, "Iceberg target ID does not exists: " + version.getVersion());
-                }
-                return (long) version.getVersion();
+                ilong snapshotId = (long) version.getVersion();
+                if (table.snapshot(snapshotId) == null) {
+                    throw new TrinoException(GENERIC_USER_ERROR, "Iceberg snapshot ID does not exists: " + snapshotId);
+                }
+                return snapshotId;

-        Long snapshotId = null;
-        for (HistoryEntry logEntry : table.history()) {
-            if (logEntry.timestampMillis() <= timestampMillis) {
-                snapshotId = logEntry.snapshotId();
-            }
-        }
-        if (snapshotId == null) {
-            throw new TrinoException(GENERIC_USER_ERROR, format("Time travel data is not available for table " +
-                    "%s. The requested time is beyond the allowed time travel period.", table.name()));
-        }
-        return snapshotId;
+        return table.history().stream()
+                .filter(logEntry -> logEntry.timestampMillis() <= timestampMillis)
+                .max(comparing(HistoryEntry::timestampMillis))
+                .orElseThrow(() -> new TrinoException(GENERIC_USER_ERROR, format(
+                        "Time travel data is not available for table %s. The requested time is beyond the allowed time travel period.",
+                        table.name())))
+                .snapshotId();

Conversation

jackye1995 commented Dec 10, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

losipiuk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexjo2144 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jackye1995 commented Jan 11, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

findinpath Jan 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cla-bot bot commented Jan 24, 2022

Uh oh!

rajarshisarkar commented May 25, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

findinpath Jan 13, 2022 •

edited

Loading

amogh-jahagirdar May 31, 2022 •

edited

Loading