Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 12 additions & 12 deletions model/src/main/java/org/projectnessie/model/IcebergTable.java
Original file line number Diff line number Diff line change
Expand Up @@ -28,14 +28,14 @@
* Represents the state of an Iceberg table in Nessie. An Iceberg table is globally identified via
* its {@link Contents#getId() unique ID}.
*
* <p>The Iceberg-table-state consists of the location to the table-metadata and the snapshot-ID.
* <p>The Iceberg-table-state consists of the location to the table-metadata and the state of the ID
* generators using the serialized version of Iceberg's {@code TableIdGenerators} object.
*
* <p>The table-metadata-location is managed globally within Nessie, which means that all versions
* of the same table share the table-metadata across all named-references (branches and tags). There
* is only one version, the current version, of Iceberg's table-metadata.
* <p>The table-metadata-location is managed on each Nessie reference, which means that all versions
* of the same table have distinct table-metadata across all named-references (branches and tags).
*
* <p>Within each named-reference (branch or tag), the (current) Iceberg-snapshot-ID will be
* different. In other words: changes to an Iceberg table update the snapshot-ID.
* <p>The information needed to generate IDs for an Iceberg table that need to be globally unique,
* for example the last-column-ID, is managed globally within Nessie.
*
* <p>When adding a new table (aka contents-object identified by a contents-id), use a {@link
* org.projectnessie.model.Operation.Put} without an expected-value. In all other cases (updating an
Expand Down Expand Up @@ -69,20 +69,20 @@ public abstract class IcebergTable extends Contents {
@NotBlank
public abstract String getMetadataLocation();

/** ID of the current Iceberg snapshot. This value is not present, if there is no snapshot. */
public abstract long getSnapshotId();
/** Opaque representation of Iceberg's {@code TableIdGenerators}. */
public abstract String getIdGenerators();

public static IcebergTable of(String metadataLocation, long snapshotId) {
public static IcebergTable of(String metadataLocation, String idGenerators) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happened to snapshotId do we no longer track it in Nessie?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we handle going back to an old snapshot now?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either you refer to a specific Nessie commit (as before) or you use Iceberg's time-travel functionality.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The table-metadata's tracked now (again) - so it's currentSnapshotId is the one

return ImmutableIcebergTable.builder()
.metadataLocation(metadataLocation)
.snapshotId(snapshotId)
.idGenerators(idGenerators)
.build();
}

public static IcebergTable of(String metadataLocation, long snapshotId, String contentsId) {
public static IcebergTable of(String metadataLocation, String idGenerators, String contentsId) {
return ImmutableIcebergTable.builder()
.metadataLocation(metadataLocation)
.snapshotId(snapshotId)
.idGenerators(idGenerators)
.id(contentsId)
.build();
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ class CommitToBranchSimulation extends Simulation {
val contentsId = tableName

val tableMeta = IcebergTable
.of(s"path_on_disk_${tableName}_$commitNum", commitNum, contentsId)
.of(s"path_on_disk_${tableName}_$commitNum", "x", contentsId)

// TODO the expectedContents is wrong!! commitNum is for the session, but we need the actual global state!!
val op =
Expand All @@ -65,7 +65,7 @@ class CommitToBranchSimulation extends Simulation {
tableMeta,
IcebergTable.of(
s"path_on_disk_${tableName}_${commitNum - 1}",
commitNum - 1,
"x",
contentsId
)
)
Expand Down
4 changes: 2 additions & 2 deletions python/pynessie/model.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,15 +30,15 @@ class IcebergTable(Contents):
"""Dataclass for Nessie Contents."""

metadata_location: str = desert.ib(fields.Str(data_key="metadataLocation"))
snapshot_id: int = desert.ib(fields.Int(data_key="snapshotId"))
id_generators: str = desert.ib(fields.Str(data_key="idGenerators"))

def requires_expected_state(self: "IcebergTable") -> bool:
"""Returns True - expected state should be provided for Put operations on Iceberg tables."""
return True

def pretty_print(self: "IcebergTable") -> str:
"""Print out for cli."""
return "Iceberg table:\n\tmetadata-location:{}\n\tsnapshot-id:{}".format(self.metadata_location, self.snapshot_id)
return "Iceberg table:\n\tmetadata-location:{}\n\tid-generators:{}".format(self.metadata_location, self.id_generators)


IcebergTableSchema = desert.schema_class(IcebergTable)
Expand Down
Loading