Skip to content

Document Iceberg time travel & snapshot queries #10515

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
losipiuk merged 3 commits into from
Sep 13, 2022
Merged

Document Iceberg time travel & snapshot queries #10515

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7c82190
Break query into multiple lines
findinpath Sep 8, 2022
2a5b0b4
Extract introduction for snapshots
findinpath Jan 11, 2022
4210d73
Document time travel queries
findinpath Jan 10, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 41 additions & 7 deletions docs/src/main/sphinx/connector/iceberg.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -510,17 +510,51 @@ Row level deletion
Tables using v2 of the Iceberg specification support deletion of individual rows
by writing position delete files.

Rolling back to a previous snapshot
-----------------------------------
Snapshots
---------

Iceberg supports a "snapshot" model of data, where table snapshots are
identified by an snapshot IDs.
identified by a snapshot ID.

The connector provides a system table exposing snapshot information for every
Iceberg table. Snapshots are identified by ``BIGINT`` snapshot IDs.
For example, you could find the snapshot IDs for the ``customer_orders`` table
by running the following query::

SELECT snapshot_id
FROM iceberg.testdb."customer_orders$snapshots"
ORDER BY committed_at DESC

Time travel queries
Comment thread
findinpath marked this conversation as resolved.
Outdated
^^^^^^^^^^^^^^^^^^^

The connector offers the ability to query historical data.
This allows you to query the table as it was when a previous snapshot
of the table was taken, even if the data has since been modified or deleted.

The historical data of the table can be retrieved by specifying the
snapshot identifier corresponding to the version of the table that
needs to be retrieved::

SELECT *
FROM iceberg.testdb.customer_orders FOR VERSION AS OF 8954597067493422955

A different approach of retrieving historical data is to specify
a point in time in the past, such as a day or week ago. The latest snapshot
of the table taken before or at the specified timestamp in the query is
internally used for providing the previous state of the table::

SELECT *
FROM iceberg.testdb.customer_orders FOR TIMESTAMP AS OF TIMESTAMP '2022-03-23 09:59:29.803 Europe/Vienna'

Rolling back to a previous snapshot
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The connector provides a system snapshots table for each Iceberg table. Snapshots are
identified by BIGINT snapshot IDs. You can find the latest snapshot ID for table
``customer_orders`` by running the following command::
Use the ``$snapshots`` metadata table to determine the latest snapshot ID of the table like in the following query::

SELECT snapshot_id FROM iceberg.testdb."customer_orders$snapshots" ORDER BY committed_at DESC LIMIT 1
SELECT snapshot_id
FROM iceberg.testdb."customer_orders$snapshots"
ORDER BY committed_at DESC LIMIT 1

A SQL procedure ``system.rollback_to_snapshot`` allows the caller to roll back
the state of the table to a previous snapshot id::
Expand Down