Skip to content

Spec: Update row lineage requirements for upgrading tables #12781

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rdblue merged 5 commits into from
Apr 20, 2025
Merged

Spec: Update row lineage requirements for upgrading tables #12781

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d46c6f9
Spec: Update row lineage requirements for upgrading tables.
rdblue Apr 12, 2025
0db277a
Update for review comments.
rdblue Apr 14, 2025
46527d4
Use "increase by" wording rather than strict first_row_id assignment.
rdblue Apr 16, 2025
d0c9704
Fix typo.
rdblue Apr 16, 2025
261e8f1
Doc: Make next-row-id required in v3
ebyhr Apr 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 20 additions & 20 deletions format/spec.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -404,8 +404,6 @@ On read, if `_last_updated_sequence_number` is `null` it is assigned the `sequen

When `null`, a row's `_row_id` field is assigned to the `first_row_id` from its containing data file plus the row position in that data file (`_pos`). A data file's `first_row_id` field is assigned using inheritance and is documented in [First Row ID Inheritance](#first-row-id-inheritance). A manifest's `first_row_id` is assigned when writing the manifest list for a snapshot and is documented in [First Row ID Assignment](#first-row-id-assignment). A snapshot's `first-row-id` is set to the table's `next-row-id` and is documented in [Snapshot Row IDs](#snapshot-row-ids).

Values for `_row_id` and `_last_updated_sequence_number` are either read from the data file or assigned at read time. As a result on read, rows in a table always have non-null values for these fields when lineage is enabled.

When an existing row is moved to a different data file for any reason, writers should write `_row_id` and `_last_updated_sequence_number` according to the following rules:

1. The row's existing non-null `_row_id` must be copied into the new data file
Expand All @@ -428,17 +426,17 @@ Writing a new append snapshot would create snapshot metadata with `first-row-id`
}
```

The snapshot's manifest list would contain existing manifests, plus new manifests with an assigned `first_row_id` based on the `added_rows_count` of previously listed added manifests:
The snapshot's manifest list would contain existing manifests, plus new manifests with an assigned `first_row_id` based on the `added_rows_count` and `existing_rows_count` of previously listed added manifests:

| `manifest_path` | `added_rows_count` | `existing_rows_count` | `first_row_id` |
|-----------------|--------------------|-----------------------|--------------------|
| ... | ... | ... | ... |
| existing | 75 | 0 | 925 |
| added1 | 100 | 25 | 1000 |
| added2 | 0 | 100 | 1100 |
| added3 | 125 | 25 | 1100 |
| added2 | 0 | 100 | 1125 |
| added3 | 125 | 25 | 1225 |

The first added file, `added1`, is assigned the same `first_row_id` as the snapshot and the following manifests are assigned `first_row_id` based on the number of rows added by the previously listed manifests. The second file, `added2`, does not change the `first_row_id` of the next manifest because it contains no added data files.
The first added file, `added1`, is assigned the same `first_row_id` as the snapshot and the following manifests are assigned `first_row_id` based on the number of rows in the previously listed manifests. The second file, `added2`, changes the `first_row_id` of the next manifest even though it contains no added data files because any data file without a `first_row_id` will be assigned one.

Within `added1`, the first added manifest, each data file's `first_row_id` follows a similar pattern:

Expand All @@ -450,21 +448,24 @@ Within `added1`, the first added manifest, each data file's `first_row_id` follo

The `first_row_id` of the EXISTING file `data1` was already assigned, so the file metadata was copied into manifest `added1`.

Files `data2` and `data3` are written with `null` for `first_row_id` and are assigned `first_row_id` at read time based on the manifest's `first_row_id` and the `record_count` of previously listed ADDED files in this manifest: (1,000 + 0) and (1,000 + 50).
Files `data2` and `data3` are written with `null` for `first_row_id` and are assigned `first_row_id` at read time based on the manifest's `first_row_id` and the `record_count` of previously files without `first_row_id` in this manifest: (1,000 + 0) and (1,000 + 50).

The snapshot then populates the total number of `added-rows` based on the sum of all added rows in the manifests: 100 (50 + 50)

When the new snapshot is committed, the table's `next-row-id` must also be updated (even if the new snapshot is not in the main branch). Because 225 rows were added (`added1`: 100 + `added2`: 0 + `added3`: 125), the new value is 1,000 + 225 = 1,225:
When the new snapshot is committed, the table's `next-row-id` must also be updated (even if the new snapshot is not in the main branch). Because 375 rows were in data files in manifests that were assigned a `first_row_id` (`added1` 100+25, `added2` 0+100, `added3` 125+25) the new value is 1,000 + 375 = 1,375.
Copy link
Member

@RussellSpitzer RussellSpitzer Apr 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, but we have spaces before and after + in all the other examples



##### Row Lineage for Upgraded Tables

Any snapshot without the field `first-row-id` does not have any lineage information and values for `_row_id` and `_last_updated_sequence_number` cannot be assigned accurately.
When a table is upgraded to v3, existing snapshots are not modified and do not have `first-row-id` set. For such snapshots without `first-row-id`, `first_row_id` values for data files and data manifests are null, and values for `_row_id` are read as null for all rows. When `first_row_id` is null, inherited row ID values are also null.

Snapshots that are created after upgrading to v3 must set the snapshot's `first-row-id` and assign row IDs to existing and added files in the snapshot. When writing the manifest list, all data manifests must be assigned a `first_row_id`, which assigns a `first_row_id` to all data files via inheritance.

Note that:

All files that were added before upgrading to v3 must propagate null for all row-lineage related
fields. The values for `_row_id` and `_last_updated_sequence_number` must always return null and when these rows are copied,
null must be explicitly written. After this point, rows are treated as if they were just created
and assigned `row_id` and `_last_updated_sequence_number` as if they were new rows.
* Snapshots from before upgrading to v3 do not have row IDs.
Copy link
Member

@RussellSpitzer RussellSpitzer Apr 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So does this mean that we aren't going to do any work on upgrade to assign new row Id's to existing snapshots? We'll only do this on the creation of future snapshots created after the upgrade?

I feel like we can handle this now by just setting next-row-id on all snapshots using the logic below. Then all previous snapshots can get row_id

Copy link
Contributor

@amogh-jahagirdar amogh-jahagirdar Apr 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think how existing snapshots are handled can probably can be left up to implementations? But I agree that in the reference implementation it makes more sense to just do it on upgrade

Copy link
Member

@RussellSpitzer RussellSpitzer Apr 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's fine, I realize that actually adding post-facto row lineage does require a manifest list rewrite for past snapshots in order to differentiate between existing and added manifests

Copy link
Contributor Author

@rdblue rdblue Apr 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I don't think that we should add row lineage for older snapshots because it creates a lot of problems that we don't have good ways to solve and doesn't create much, if any, value.

Here are some of the problems:

  • To add row IDs to older snapshots, we would need to change the snapshot metadata. This could be cached so changing it may cause strange issues, but the larger problem is that modifying it would require coordination (to avoid losing the changes) and changes to the REST protocol to replace an older snapshot
  • If we were to add row IDs to older snapshots, then those row IDs would not be very useful without an expensive operation that rewrites the whole metadata tree.
    • If we assigned new row IDs independently in each snapshot, then a data file would have different row IDs across versions -- making the IDs useless.
    • That means we would need to track data files and update the metadata tree with consistent first_row_id values.
    • Even with the expensive rewrite of the whole metadata tree, the _row_id column would be missing from data files so the lineage of individual rows is not available.

I think the best approach is to not modify older snapshots. The goal with this update is to ensure that we have good row lineage from the next snapshot after upgrade and forward, when we can write _row_id. That's why I updated the first_row_id assignment to include any unassigned data file or data manifest, and changed the assignment strategy to leave space for existing rows. With those changes we have created an invariant: new snapshots in v3 tables always have IDs assigned to all rows.

Branching is still a little difficult because we don't want to analyze branch history (which may be lost) and attempt to assign consistently. With this update, branches get separate IDs but it leave open the possibility to do some external analysis and assign the same IDs for the same data files.

Copy link
Contributor Author

@rdblue rdblue Apr 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I replied here before I saw a comment on the implementation PR that was similar. Here's the link to my response there: #12672 (comment)

I think I summarized the issues better on that one:

I think the argument against filling in historical row IDs is that it is extra work to think through the issues and build an algorithm to modify old data. And when we've done that extra work, we don't actually gain anything: you still can't reason about the IDs of records between branches or even within a branch.

My conclusion is that we should just expose the data that we know, which is that the row ID was null until the row was assigned an ID in a branch.

Copy link
Contributor

@amogh-jahagirdar amogh-jahagirdar Apr 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the detailed explanation, yeah I'm convinced now that rewriting the older snapshots adds questionable value at the cost of a lot of complexity.

* After upgrading, new snapshots in different branches will assign disjoint ID ranges to existing data files, based on the table's `next-row-id` when the snapshot is committed. For a data file in multiple branches, a writer may write the `first_row_id` from another branch or may assign a new `first_row_id` to the data file (to avoid large metadata rewrites).
Copy link
Member

@szehon-ho szehon-ho Apr 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if i understand, this means if same data file exists on different branch, can have same first_row_id? It seems the two sentence contradict (first sentence specifies disjoint Id ranges). Would it be more clear:

After upgrading, new snapshots in different branches will assign disjoint ID ranges to existing distinct data files, based on the table's next-row-id when the snapshot is committed. For the same data file in multiple branches...

Copy link
Member

@RussellSpitzer RussellSpitzer Apr 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

During upgrade it is possible that an existing row can exist on two different branches with different row ids, after upgrade this will not be possible for new rows.

Copy link
Contributor Author

@rdblue rdblue Apr 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What this is saying is that when another branch is updated, all of the files in that branch must be assigned IDs by the v3 snapshot, and unless the writer does some additional work to find and write the row IDs for the same data file in other branches, the IDs will be assigned for the branch.

This also says (the last sentence) that the writer can choose to do that extra work, find the first_row_id used in another branch for the file, and use it to have consistent IDs across the branches. This is optional.

Copy link
Member

@szehon-ho szehon-ho Apr 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think makes sense. I still read the two sentence as contradict (as the first sentence specifies 'disjoint', but second sentence says we can optionally re-use the other branch first_row_id), hence my suggestion if it makes sense

* Existing rows will inherit `_last_updated_sequence_number` from their containing data file.


### Partitioning
Expand Down Expand Up @@ -689,9 +690,11 @@ When reading v1 manifests with no sequence number column, sequence numbers for a

When adding a new data file, its `first_row_id` field is set to `null` because it is not assigned until the snapshot is successfully committed.

When reading, the `first_row_id` is assigned by replacing `null` with the manifest's `first_row_id` plus the sum of `record_count` for all added data files that preceded the file in the manifest.
When reading, the `first_row_id` is assigned by replacing `null` with the manifest's `first_row_id` plus the sum of `record_count` for all data files that preceded the file in the manifest that had a null `first_row_id`.

The `first_row_id` is only inherited for added data files. The inherited value must be written into the data file metadata for existing and deleted entries. The value of `first_row_id` for delete files is always `null`.
The inherited value of `first_row_id` must be written into the data file metadata when creating existing and deleted entries. The value of `first_row_id` for delete files is always `null`.

In most cases, only added files will be assigned a new `first_row_id` via inheritance, but any unassigned `first_row_id` must be assigned to handle manifests in upgraded tables that have not yet assigned `first_row_id` for existing entries.
Copy link
Member

@RussellSpitzer RussellSpitzer Apr 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence is a little difficult, I think there is a redundancy which is making a it a little confusing with "any unassigned ... that have not yet assigned"

Maybe we can shorten this to.:

Suggested change
In most cases, only added files will be assigned a new `first_row_id` via inheritance, but any unassigned `first_row_id` must be assigned to handle manifests in upgraded tables that have not yet assigned `first_row_id` for existing entries.
Assignment of `first_row_id` usually only applies to newly added files but during table format version upgrades existing files will also have a null value for `first_row_id` and must also be assigned.

?

Copy link
Contributor Author

@rdblue rdblue Apr 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated this to:

Any null (unassigned) first_row_id must be assigned via inheritance, even if the data file is existing. This ensures that row IDs are assigned to existing data files in upgraded tables in the first commit after upgrading to v3.

There's no need to talk about "usually" and make assumptions.


### Snapshots

Expand All @@ -708,7 +711,6 @@ A snapshot consists of the following fields:
| _optional_ | _required_ | _required_ | **`summary`** | A string map that summarizes the snapshot changes, including `operation` as a _required_ field (see below) |
| _optional_ | _optional_ | _optional_ | **`schema-id`** | ID of the table's current schema when the snapshot was created |
| | | _required_ | **`first-row-id`** | The first `_row_id` assigned to the first row in the first data file in the first manifest, see [Row Lineage](#row-lineage) |
| | | _required_ | **`added-rows`** | Sum of the [`added_rows_count`](#manifest-lists) from all manifests added in this snapshot. |


The snapshot summary's `operation` field is used by some operations, like snapshot expiration, to skip processing certain snapshots. Possible `operation` values are:
Expand Down Expand Up @@ -736,8 +738,6 @@ A snapshot's `first-row-id` is assigned to the table's current `next-row-id` on

The snapshot's `first-row-id` is the starting `first_row_id` assigned to manifests in the snapshot's manifest list.

The snapshot's `added-rows` is the sum of all the [`added_rows_count`](#manifest-lists) in all added manifests.
Copy link
Contributor Author

@rdblue rdblue Apr 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RussellSpitzer, I've removed added-rows because I think that it is misleading and makes row ID range assignment more complicated.

In this PR, row ID range assignment is based on the total number of existing or added rows in new manifests. That leaves room for any data files that are missing first_row_id even if the files are existing and not added. That may not be the total number of added rows so I don't think it makes sense to have added-rows in the snapshot.

Copy link
Member

@RussellSpitzer RussellSpitzer Apr 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's fine, we only had this added in order to pass the information from the snapshot into the table metadata. I believe now that logic has moved we don't have that issue.



### Manifest Lists

Expand Down Expand Up @@ -786,9 +786,9 @@ Notes:

#### First Row ID Assignment

When adding a new data manifest file, its `first_row_id` field is assigned the value of the snapshot's `first_row_id` plus the sum of `added_rows_count` for all data manifests that preceded the manifest in the manifest list.
When adding a new data manifest file, its `first_row_id` field is assigned the value of the snapshot's `first_row_id` plus the sum of `added_rows_count` and `existing_rows_count` for all data manifests that preceded the manifest in the manifest list and were assigned a `first_row_id`.
Copy link
Member

@RussellSpitzer RussellSpitzer Apr 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Above you said "had a null ..." and i think that's a little bit clearer than "were assigned a"

Copy link
Contributor Author

@rdblue rdblue Apr 14, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated, but this is a bit tricky because these are simply new manifests. Having a null first_row_id is an implementation detail that is not written into metadata. I've updated it to say both:

When adding a new data manifest file, its first_row_id field is assigned the value of the snapshot's first_row_id plus the sum of added_rows_count and existing_rows_count for all new data manifests that preceded it in the manifest list; that is, those that had a null first_row_id and were assigned one.

I also reformatted this section a little.


The `first_row_id` is only assigned for new data manifests. Values for existing manifests must be preserved when writing a new manifest list. The value of `first_row_id` for delete manifests is always `null`.
The `first_row_id` is only assigned for data manifests that do not have a non-null `first_row_id`. Values for existing manifests must be preserved when writing a new manifest list. The value of `first_row_id` for delete manifests is always `null`.
Copy link
Contributor

@amogh-jahagirdar amogh-jahagirdar Apr 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we change that do not have a non-null to that have a null? Feels easier to read

Copy link
Contributor Author

@rdblue rdblue Apr 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated this to be more clear:

The first_row_id is only assigned for new data manifests that do not have a first_row_id.

That avoids assuming null because null isn't written for those manifests.


### Scan Planning

Expand Down