Skip to content

Move table-metadata-pointer back to global state, use snapshot,schema,spec,sort-order ids as on-ref state #2626

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
snazy merged 4 commits into from
Nov 12, 2021
Merged

Move table-metadata-pointer back to global state, use snapshot,schema,spec,sort-order ids as on-ref state #2626

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
59bb906
Move table-metadata-pointer back to global state, use snapshot,schema…
snazy Nov 9, 2021
c5d686e
review updates
snazy Nov 10, 2021
cf082c2
review comments
snazy Nov 11, 2021
529860b
re-record after merge-conflict
snazy Nov 11, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions model/src/main/java/org/projectnessie/api/ContentApi.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,8 @@ public interface ContentApi {
*
* <p>If the table-metadata is tracked globally (Iceberg), Nessie returns a {@link Content}
* object, that contains the most up-to-date part for the globally tracked part (Iceberg:
* table-metadata) plus the per-Nessie-reference/hash specific part (Iceberg: snapshot-ID).
* table-metadata) plus the per-Nessie-reference/hash specific part (Iceberg: snapshot-ID,
* schema-ID, partition-spec-ID, default-sort-order-ID).
Copy link
Contributor

@ajantha-bhat ajantha-bhat Nov 11, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: In the descriptions, If we are naming this variable(default-sort-order-ID) based on iceberg spec, then other three variable has to be default-spec-id, current-schema-id, current-snapshot-id

or may be instead of mentioning each field, we just link it to IcebergRefState

Copy link
Contributor

@rymurr rymurr Nov 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: i thikn ID shjould be id

*
* @param key the {@link ContentKey}s to retrieve
* @param ref named-reference to retrieve the content for
Expand All @@ -64,7 +65,8 @@ Content getContent(
*
* <p>If the table-metadata is tracked globally (Iceberg), Nessie returns a {@link Content}
* object, that contains the most up-to-date part for the globally tracked part (Iceberg:
* table-metadata) plus the per-Nessie-reference/hash specific part (Iceberg: snapshot-ID).
* table-metadata) plus the per-Nessie-reference/hash specific part (Iceberg: snapshot-id,
* schema-id, partition-spec-id, default-sort-order-id).
*
* @param ref named-reference to retrieve the content for
* @param hashOnRef hash on {@code ref} to retrieve the content for, translates to {@code HEAD},
Expand Down
6 changes: 4 additions & 2 deletions model/src/main/java/org/projectnessie/api/http/HttpContentApi.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,8 @@ public interface HttpContentApi extends ContentApi {
+ "If the table-metadata is tracked globally (Iceberg), "
+ "Nessie returns a 'Content' object, that contains the most up-to-date part for "
+ "the globally tracked part (Iceberg: table-metadata) plus the "
+ "per-Nessie-reference/hash specific part (Iceberg: snapshot-ID).")
+ "per-Nessie-reference/hash specific part (Iceberg: snapshot-id, "
+ "schema-id, partition-spec-id, default-sort-order-id).")
@APIResponses({
@APIResponse(
responseCode = "200",
Expand Down Expand Up @@ -103,7 +104,8 @@ Content getContent(
+ "If the table-metadata is tracked globally (Iceberg), "
+ "Nessie returns a 'Content' object, that contains the most up-to-date part for "
+ "the globally tracked part (Iceberg: table-metadata) plus the "
+ "per-Nessie-reference/hash specific part (Iceberg: snapshot-ID).")
+ "per-Nessie-reference/hash specific part (Iceberg: snapshot-ID,"
+ "schema-ID, partition-spec-ID, default-sort-order-ID).")
@APIResponses({
@APIResponse(
responseCode = "200",
Expand Down
40 changes: 26 additions & 14 deletions model/src/main/java/org/projectnessie/model/IcebergTable.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,14 +28,8 @@
* Represents the state of an Iceberg table in Nessie. An Iceberg table is globally identified via
* its {@link Content#getId() unique ID}.
*
* <p>The Iceberg-table-state consists of the location to the table-metadata and the state of the ID
* generators using the serialized version of Iceberg's {@code TableIdGenerators} object.
*
* <p>The table-metadata-location is managed on each Nessie reference, which means that all versions
* of the same table have distinct table-metadata across all named-references (branches and tags).
*
* <p>The information needed to generate IDs for an Iceberg table that need to be globally unique,
* for example the last-column-ID, is managed globally within Nessie.
* <p>The Iceberg-table-state consists of the location to the table-metadata and the state of
* relevant IDs using a serialized version of those.
*
* <p>When adding a new table (aka content-object identified by a content-id), use a {@link
* org.projectnessie.model.Operation.Put} without an expected-value. In all other cases (updating an
Expand Down Expand Up @@ -69,20 +63,38 @@ public abstract class IcebergTable extends Content {
@NotBlank
public abstract String getMetadataLocation();

/** Opaque representation of Iceberg's {@code TableIdGenerators}. */
public abstract String getIdGenerators();
public abstract long getSnapshotId();
Copy link
Contributor

@rymurr rymurr Nov 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do these need default values? particularly the shcema/specid etc? Or do they have to be specified every time

Copy link
Member Author

@snazy snazy Nov 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't think those should have default values. It uses what Iceberg's TableMetadata says and not rely on a default value.


public abstract int getSchemaId();

public abstract int getSpecId();

public abstract int getSortOrderId();

public static IcebergTable of(String metadataLocation, String idGenerators) {
public static IcebergTable of(
String metadataLocation, long snapshotId, int schemaId, int specId, int sortOrderId) {
return ImmutableIcebergTable.builder()
.metadataLocation(metadataLocation)
.idGenerators(idGenerators)
.snapshotId(snapshotId)
.schemaId(schemaId)
.specId(specId)
.sortOrderId(sortOrderId)
.build();
}

public static IcebergTable of(String metadataLocation, String idGenerators, String contentId) {
public static IcebergTable of(
String metadataLocation,
long snapshotId,
int schemaId,
int specId,
int sortOrderId,
String contentId) {
return ImmutableIcebergTable.builder()
.metadataLocation(metadataLocation)
.idGenerators(idGenerators)
.snapshotId(snapshotId)
.schemaId(schemaId)
.specId(specId)
.sortOrderId(sortOrderId)
.id(contentId)
.build();
}
Expand Down
14 changes: 12 additions & 2 deletions ...ulations/src/test/scala/org/projectnessie/perftest/gatling/CommitToBranchSimulation.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,14 @@ class CommitToBranchSimulation extends Simulation {
val contentId = tableName

val tableMeta = IcebergTable
.of(s"path_on_disk_${tableName}_$commitNum", "x", contentId)
.of(
s"path_on_disk_${tableName}_$commitNum",
42,
43,
44,
45,
contentId
)

// TODO the expectedContent is wrong!! commitNum is for the session, but we need the actual global state!!
val op =
Expand All @@ -65,7 +72,10 @@ class CommitToBranchSimulation extends Simulation {
tableMeta,
IcebergTable.of(
s"path_on_disk_${tableName}_${commitNum - 1}",
"x",
42,
43,
44,
45,
contentId
)
)
Expand Down
11 changes: 9 additions & 2 deletions python/pynessie/model.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,10 @@ class IcebergTable(Content):
"""Dataclass for Nessie Content."""

metadata_location: str = desert.ib(fields.Str(data_key="metadataLocation"))
id_generators: str = desert.ib(fields.Str(data_key="idGenerators"))
snapshot_id: int = desert.ib(fields.Int(data_key="snapshotId"))
schema_id: int = desert.ib(fields.Int(data_key="schemaId"))
spec_id: int = desert.ib(fields.Int(data_key="specId"))
sort_order_id: int = desert.ib(fields.Int(data_key="sortOrderId"))

@staticmethod
def requires_expected_state() -> bool:
Expand All @@ -56,7 +59,11 @@ def requires_expected_state() -> bool:

def pretty_print(self: "IcebergTable") -> str:
"""Print out for cli."""
return "Iceberg table:\n\tmetadata-location:{}\n\tid-generators:{}".format(self.metadata_location, self.id_generators)
return (
f"Iceberg table:\n\tmetadata-location: {self.metadata_location}\n\tsnapshot-id: {self.snapshot_id}"
f"\n\tschema-id: {self.schema_id}"
f"\n\tpartition-spec-id: {self.spec_id}\n\tdefault-sort-order-id: {self.sort_order_id}"
)


IcebergTableSchema = desert.schema_class(IcebergTable)
Expand Down
67 changes: 34 additions & 33 deletions python/tests/cassettes/test_nessie_cli/test_assign.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,8 @@ interactions:
code: 200
message: OK
- request:
body: '{"name": "dev", "hash": "2e1cfa82b035c26cbbbdae632cea070514eb8b773f616aaeaf668e2f0be8f10d",
"type": "BRANCH"}'
body: '{"hash": "2e1cfa82b035c26cbbbdae632cea070514eb8b773f616aaeaf668e2f0be8f10d",
"name": "dev", "type": "BRANCH"}'
headers:
Accept:
- '*/*'
Expand Down Expand Up @@ -131,11 +131,12 @@ interactions:
code: 404
message: Not Found
- request:
body: '{"commitMeta": {"commitTime": null, "author": "nessie test", "authorTime":
null, "hash": null, "properties": null, "committer": null, "message": "test
message", "signedOffBy": null}, "operations": [{"expectedContent": null, "key":
{"elements": ["assign", "foo", "bar"]}, "content": {"idGenerators": "xyz", "metadataLocation":
"/a/b/c", "id": "test_assign", "type": "ICEBERG_TABLE"}, "type": "PUT"}]}'
body: '{"commitMeta": {"hash": null, "authorTime": null, "signedOffBy": null,
"message": "test message", "commitTime": null, "properties": null, "committer":
null, "author": "nessie test"}, "operations": [{"expectedContent": null, "key":
{"elements": ["assign", "foo", "bar"]}, "content": {"specId": 44, "id": "test_assign",
"schemaId": 43, "snapshotId": 42, "sortOrderId": 45, "metadataLocation": "/a/b/c",
"type": "ICEBERG_TABLE"}, "type": "PUT"}]}'
headers:
Accept:
- '*/*'
Expand All @@ -144,7 +145,7 @@ interactions:
Connection:
- keep-alive
Content-Length:
- '399'
- '443'
Content-Type:
- application/json
User-Agent:
Expand All @@ -153,7 +154,7 @@ interactions:
uri: http://localhost:19120/api/v1/trees/branch/dev/commit?expectedHash=2e1cfa82b035c26cbbbdae632cea070514eb8b773f616aaeaf668e2f0be8f10d
response:
body:
string: "{\n \"type\" : \"BRANCH\",\n \"name\" : \"dev\",\n \"hash\" : \"9f983acf8888862652e98bb2528fc835afe1a119ad2f193e5158720438679e95\"\n}"
string: "{\n \"type\" : \"BRANCH\",\n \"name\" : \"dev\",\n \"hash\" : \"291dd61cb415c8a23d3ea17fca9a55eaf2e7b5d259ec77469de82ffdc496c455\"\n}"
headers:
Content-Type:
- application/json
Expand All @@ -177,7 +178,7 @@ interactions:
uri: http://localhost:19120/api/v1/trees/tree/dev
response:
body:
string: "{\n \"type\" : \"BRANCH\",\n \"name\" : \"dev\",\n \"hash\" : \"9f983acf8888862652e98bb2528fc835afe1a119ad2f193e5158720438679e95\"\n}"
string: "{\n \"type\" : \"BRANCH\",\n \"name\" : \"dev\",\n \"hash\" : \"291dd61cb415c8a23d3ea17fca9a55eaf2e7b5d259ec77469de82ffdc496c455\"\n}"
headers:
Content-Type:
- application/json
Expand All @@ -187,8 +188,8 @@ interactions:
code: 200
message: OK
- request:
body: '{"name": "main", "hash": "9f983acf8888862652e98bb2528fc835afe1a119ad2f193e5158720438679e95",
"type": "BRANCH"}'
body: '{"hash": "291dd61cb415c8a23d3ea17fca9a55eaf2e7b5d259ec77469de82ffdc496c455",
"name": "main", "type": "BRANCH"}'
headers:
Accept:
- '*/*'
Expand Down Expand Up @@ -242,8 +243,8 @@ interactions:
code: 200
message: OK
- request:
body: '{"name": "dev", "hash": "9f983acf8888862652e98bb2528fc835afe1a119ad2f193e5158720438679e95",
"type": "BRANCH"}'
body: '{"hash": "291dd61cb415c8a23d3ea17fca9a55eaf2e7b5d259ec77469de82ffdc496c455",
"name": "dev", "type": "BRANCH"}'
headers:
Accept:
- '*/*'
Expand Down Expand Up @@ -282,9 +283,9 @@ interactions:
response:
body:
string: "{\n \"token\" : null,\n \"references\" : [ {\n \"type\" : \"BRANCH\",\n
\ \"name\" : \"main\",\n \"hash\" : \"9f983acf8888862652e98bb2528fc835afe1a119ad2f193e5158720438679e95\"\n
\ \"name\" : \"main\",\n \"hash\" : \"291dd61cb415c8a23d3ea17fca9a55eaf2e7b5d259ec77469de82ffdc496c455\"\n
\ }, {\n \"type\" : \"BRANCH\",\n \"name\" : \"dev\",\n \"hash\"
: \"9f983acf8888862652e98bb2528fc835afe1a119ad2f193e5158720438679e95\"\n }
: \"291dd61cb415c8a23d3ea17fca9a55eaf2e7b5d259ec77469de82ffdc496c455\"\n }
],\n \"hasMore\" : false\n}"
headers:
Content-Type:
Expand All @@ -309,7 +310,7 @@ interactions:
uri: http://localhost:19120/api/v1/trees/tree/main
response:
body:
string: "{\n \"type\" : \"BRANCH\",\n \"name\" : \"main\",\n \"hash\" : \"9f983acf8888862652e98bb2528fc835afe1a119ad2f193e5158720438679e95\"\n}"
string: "{\n \"type\" : \"BRANCH\",\n \"name\" : \"main\",\n \"hash\" : \"291dd61cb415c8a23d3ea17fca9a55eaf2e7b5d259ec77469de82ffdc496c455\"\n}"
headers:
Content-Type:
- application/json
Expand All @@ -319,8 +320,8 @@ interactions:
code: 200
message: OK
- request:
body: '{"name": "v1.0", "hash": "9f983acf8888862652e98bb2528fc835afe1a119ad2f193e5158720438679e95",
"type": "TAG"}'
body: '{"hash": "291dd61cb415c8a23d3ea17fca9a55eaf2e7b5d259ec77469de82ffdc496c455",
"name": "v1.0", "type": "TAG"}'
headers:
Accept:
- '*/*'
Expand All @@ -338,7 +339,7 @@ interactions:
uri: http://localhost:19120/api/v1/trees/tree?sourceRefName=main
response:
body:
string: "{\n \"type\" : \"TAG\",\n \"name\" : \"v1.0\",\n \"hash\" : \"9f983acf8888862652e98bb2528fc835afe1a119ad2f193e5158720438679e95\"\n}"
string: "{\n \"type\" : \"TAG\",\n \"name\" : \"v1.0\",\n \"hash\" : \"291dd61cb415c8a23d3ea17fca9a55eaf2e7b5d259ec77469de82ffdc496c455\"\n}"
headers:
Content-Type:
- application/json
Expand All @@ -363,10 +364,10 @@ interactions:
response:
body:
string: "{\n \"token\" : null,\n \"references\" : [ {\n \"type\" : \"BRANCH\",\n
\ \"name\" : \"main\",\n \"hash\" : \"9f983acf8888862652e98bb2528fc835afe1a119ad2f193e5158720438679e95\"\n
\ \"name\" : \"main\",\n \"hash\" : \"291dd61cb415c8a23d3ea17fca9a55eaf2e7b5d259ec77469de82ffdc496c455\"\n
\ }, {\n \"type\" : \"BRANCH\",\n \"name\" : \"dev\",\n \"hash\"
: \"9f983acf8888862652e98bb2528fc835afe1a119ad2f193e5158720438679e95\"\n },
{\n \"type\" : \"TAG\",\n \"name\" : \"v1.0\",\n \"hash\" : \"9f983acf8888862652e98bb2528fc835afe1a119ad2f193e5158720438679e95\"\n
: \"291dd61cb415c8a23d3ea17fca9a55eaf2e7b5d259ec77469de82ffdc496c455\"\n },
{\n \"type\" : \"TAG\",\n \"name\" : \"v1.0\",\n \"hash\" : \"291dd61cb415c8a23d3ea17fca9a55eaf2e7b5d259ec77469de82ffdc496c455\"\n
\ } ],\n \"hasMore\" : false\n}"
headers:
Content-Type:
Expand All @@ -391,7 +392,7 @@ interactions:
uri: http://localhost:19120/api/v1/trees/tree/dev
response:
body:
string: "{\n \"type\" : \"BRANCH\",\n \"name\" : \"dev\",\n \"hash\" : \"9f983acf8888862652e98bb2528fc835afe1a119ad2f193e5158720438679e95\"\n}"
string: "{\n \"type\" : \"BRANCH\",\n \"name\" : \"dev\",\n \"hash\" : \"291dd61cb415c8a23d3ea17fca9a55eaf2e7b5d259ec77469de82ffdc496c455\"\n}"
headers:
Content-Type:
- application/json
Expand All @@ -401,8 +402,8 @@ interactions:
code: 200
message: OK
- request:
body: '{"name": "v1.0", "hash": "9f983acf8888862652e98bb2528fc835afe1a119ad2f193e5158720438679e95",
"type": "TAG"}'
body: '{"hash": "291dd61cb415c8a23d3ea17fca9a55eaf2e7b5d259ec77469de82ffdc496c455",
"name": "v1.0", "type": "TAG"}'
headers:
Accept:
- '*/*'
Expand Down Expand Up @@ -446,7 +447,7 @@ interactions:
uri: http://localhost:19120/api/v1/trees/tree/v1.0
response:
body:
string: "{\n \"type\" : \"TAG\",\n \"name\" : \"v1.0\",\n \"hash\" : \"9f983acf8888862652e98bb2528fc835afe1a119ad2f193e5158720438679e95\"\n}"
string: "{\n \"type\" : \"TAG\",\n \"name\" : \"v1.0\",\n \"hash\" : \"291dd61cb415c8a23d3ea17fca9a55eaf2e7b5d259ec77469de82ffdc496c455\"\n}"
headers:
Content-Type:
- application/json
Expand All @@ -456,8 +457,8 @@ interactions:
code: 200
message: OK
- request:
body: '{"name": "dev", "hash": "9f983acf8888862652e98bb2528fc835afe1a119ad2f193e5158720438679e95",
"type": "TAG"}'
body: '{"hash": "291dd61cb415c8a23d3ea17fca9a55eaf2e7b5d259ec77469de82ffdc496c455",
"name": "dev", "type": "TAG"}'
headers:
Accept:
- '*/*'
Expand All @@ -472,7 +473,7 @@ interactions:
User-Agent:
- python-requests/2.26.0
method: PUT
uri: http://localhost:19120/api/v1/trees/tag/v1.0?expectedHash=9f983acf8888862652e98bb2528fc835afe1a119ad2f193e5158720438679e95
uri: http://localhost:19120/api/v1/trees/tag/v1.0?expectedHash=291dd61cb415c8a23d3ea17fca9a55eaf2e7b5d259ec77469de82ffdc496c455
response:
body:
string: ''
Expand All @@ -496,10 +497,10 @@ interactions:
response:
body:
string: "{\n \"token\" : null,\n \"references\" : [ {\n \"type\" : \"BRANCH\",\n
\ \"name\" : \"main\",\n \"hash\" : \"9f983acf8888862652e98bb2528fc835afe1a119ad2f193e5158720438679e95\"\n
\ \"name\" : \"main\",\n \"hash\" : \"291dd61cb415c8a23d3ea17fca9a55eaf2e7b5d259ec77469de82ffdc496c455\"\n
\ }, {\n \"type\" : \"BRANCH\",\n \"name\" : \"dev\",\n \"hash\"
: \"9f983acf8888862652e98bb2528fc835afe1a119ad2f193e5158720438679e95\"\n },
{\n \"type\" : \"TAG\",\n \"name\" : \"v1.0\",\n \"hash\" : \"9f983acf8888862652e98bb2528fc835afe1a119ad2f193e5158720438679e95\"\n
: \"291dd61cb415c8a23d3ea17fca9a55eaf2e7b5d259ec77469de82ffdc496c455\"\n },
{\n \"type\" : \"TAG\",\n \"name\" : \"v1.0\",\n \"hash\" : \"291dd61cb415c8a23d3ea17fca9a55eaf2e7b5d259ec77469de82ffdc496c455\"\n
\ } ],\n \"hasMore\" : false\n}"
headers:
Content-Type:
Expand Down
8 changes: 4 additions & 4 deletions python/tests/cassettes/test_nessie_cli/test_branch.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,8 @@ interactions:
code: 200
message: OK
- request:
body: '{"name": "dev", "hash": "2e1cfa82b035c26cbbbdae632cea070514eb8b773f616aaeaf668e2f0be8f10d",
"type": "BRANCH"}'
body: '{"hash": "2e1cfa82b035c26cbbbdae632cea070514eb8b773f616aaeaf668e2f0be8f10d",
"name": "dev", "type": "BRANCH"}'
headers:
Accept:
- '*/*'
Expand Down Expand Up @@ -155,8 +155,8 @@ interactions:
code: 200
message: OK
- request:
body: '{"name": "etl", "hash": "2e1cfa82b035c26cbbbdae632cea070514eb8b773f616aaeaf668e2f0be8f10d",
"type": "BRANCH"}'
body: '{"hash": "2e1cfa82b035c26cbbbdae632cea070514eb8b773f616aaeaf668e2f0be8f10d",
"name": "etl", "type": "BRANCH"}'
headers:
Accept:
- '*/*'
Expand Down
Loading