diff --git a/site/docs/spec.md b/site/docs/spec.md index 97e391a6b9b8..6bcfd379d7f8 100644 --- a/site/docs/spec.md +++ b/site/docs/spec.md @@ -498,6 +498,7 @@ Table metadata consists of the following fields: | _required_ | | **`partition-spec`**| The table’s current partition spec, stored as only fields. Note that this is used by writers to partition data, but is not used when reading because reads use the specs stored in manifest files. (**Deprecated**: use `partition-specs` and `default-spec-id`instead ) | | _optional_ | _required_ | **`partition-specs`**| A list of partition specs, stored as full partition spec objects. | | _optional_ | _required_ | **`default-spec-id`**| ID of the “current” spec that writers should use by default. | +| _optional_ | _required_ | **`last-partition-id`**| An integer; the highest assigned partition field ID across all partition specs for the table. This is used to ensure partition fields are always assigned an unused ID when evolving specs. | | _optional_ | _optional_ | **`properties`**| A string to string map of table properties. This is used to control settings that affect reading and writing and is not intended to be used for arbitrary metadata. For example, `commit.retry.num-retries` is used to control the number of commit retries. | | _optional_ | _optional_ | **`current-snapshot-id`**| `long` ID of the current table snapshot. | | _optional_ | _optional_ | **`snapshots`**| A list of valid snapshots. Valid snapshots are snapshots for which all data files exist in the file system. A data file must not be deleted from the file system until the last snapshot in which it was listed is garbage collected. | @@ -894,6 +895,7 @@ Table metadata is serialized as a JSON object according to the following table. |**`partition-spec`**|`JSON partition fields (list)`|`See above, read partition-specs instead`| |**`partition-specs`**|`JSON partition specs (list of objects)`|`See above`| |**`default-spec-id`**|`JSON int`|`0`| +|**`last-partition-id`**|`JSON int`|`1000`| |**`properties`**|`JSON object: {`
  `"": "",`
  `...`
`}`|`{`
  `"write.format.default": "avro",`
  `"commit.retry.num-retries": "4"`
`}`| |**`current-snapshot-id`**|`JSON long`|`3051729675574597004`| |**`snapshots`**|`JSON list of objects: [ {`
  `"snapshot-id": ,`
  `"timestamp-ms": ,`
  `"summary": {`
    `"operation": ,`
    `... },`
  `"manifest-list": ""`
  `},`
  `...`
`]`|`[ {`
  `"snapshot-id": 3051729675574597004,`
  `"timestamp-ms": 1515100955770,`
  `"summary": {`
    `"operation": "append"`
  `},`
  `"manifest-list": "s3://b/wh/.../s1.avro"`
`} ]`| @@ -948,6 +950,7 @@ Writing v2 metadata: * Table metadata now requires field `table-uuid`. * Table metadata now requires field `partition-specs`. * Table metadata now requires field `default-spec-id`. +* Table metadata now requires field `last-partition-id`. * Table metadata field `partition-spec` is no longer required and may be omitted. * Snapshot added required field field `sequence-number`. * Snapshot now requires field `manifest-list`. diff --git a/spark2/src/test/java/org/apache/iceberg/examples/README.md b/spark2/src/test/java/org/apache/iceberg/examples/README.md index 4dc9a2a5f04e..5f02061b6226 100644 --- a/spark2/src/test/java/org/apache/iceberg/examples/README.md +++ b/spark2/src/test/java/org/apache/iceberg/examples/README.md @@ -111,6 +111,7 @@ The metadata file will contain things like the table location, the schema and th ... } ] } ], + "last-partition-id" : 1000, "properties" : { }, "current-snapshot-id" : -1, "snapshots" : [ ],