From aa09063b956cbab983f7a9319be17cbfd5f27572 Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Thu, 12 Jun 2025 22:19:12 -0400 Subject: [PATCH 01/25] docs: Add `Polaris Evolution` page --- site/content/in-dev/unreleased/evolution.md | 61 +++++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 site/content/in-dev/unreleased/evolution.md diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md new file mode 100644 index 0000000000..6814fd5a0a --- /dev/null +++ b/site/content/in-dev/unreleased/evolution.md @@ -0,0 +1,61 @@ +--- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +title: Polaris Evolution +type: docs +weight: 1000 +--- + +This page discusses what can be expected from Apache Polaris as the project evolves. + +## Polaris as a Catalog + +Polaris is primarily intended to be used as a Catalog of Tables and Views. As such, +it implements the Iceberg REST API, Polaris Management API and the Generic Tables API. + +Revisions of the Iceberg REST API are controlled by the [Apache Iceberg](https://iceberg.apache.org/) +community. Polaris attempts to accurately implement the most recent version of this specification, +which is available at the time of Polaris releases. + +The Polaris Management API is maintained as a versioned REST API. New releases of Polaris may +include changes to the current version of the Management API. When that happens those changes +are intended to be compatible with prior versions of Polaris clients. + +In case a major change is required to the Management API that cannot be implemented in a +backward-compatible way, a new version of the API will be introduced. Polaris servers will support +old API version for some transition period to allow clients to migrate. Note that API versions +are not meant to be 1:1 with Polaris releases. + +The Generic Tables API is an experimental API at this time. Refer to its documentation page +for more details about compatibility expectations. + +## Polaris as a Library + +Polaris produces several jars. These jars or custom builds of Polaris code may be used in +downstream projects according to the terms of the license included into Polaris distributions. + +However, Polaris' focus as a project is not on ensuring ideal binary compatibility at the java +class level between releases. Maintainers try to keep binary compatibility on the "best effort" +basis. The main goal being the implementation of Polaris capabilities for access via REST APIs. + +This approach is not meant to discourage the use of Polaris code in downstream projects, but +to allow more flexibility in evolving the codebase to support new Catalog-level features +and improve code efficiency. Maintainers of downstream projects are encouraged to join Polaris +mailing lists to monitor project changes, suggest improvements, and engage with the Polaris +community in case of specific compatibility concerns. From 1cf431098f79b75164695d39d569f741c4f086a2 Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Fri, 13 Jun 2025 20:53:29 -0400 Subject: [PATCH 02/25] review: clarify Iceberg REST API version --- site/content/in-dev/unreleased/evolution.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index 6814fd5a0a..fb37d5c5e5 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -30,8 +30,9 @@ Polaris is primarily intended to be used as a Catalog of Tables and Views. As su it implements the Iceberg REST API, Polaris Management API and the Generic Tables API. Revisions of the Iceberg REST API are controlled by the [Apache Iceberg](https://iceberg.apache.org/) -community. Polaris attempts to accurately implement the most recent version of this specification, -which is available at the time of Polaris releases. +community. Polaris attempts to accurately implement this specification. However, there +is no guarantee that Polaris releases always implement the latest version of the Iceberg +REST Catalog API. The Polaris Management API is maintained as a versioned REST API. New releases of Polaris may include changes to the current version of the Management API. When that happens those changes From 28838b51204ad07e1afa78f53d8a8f0069292610 Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Fri, 13 Jun 2025 21:04:07 -0400 Subject: [PATCH 03/25] review: mention API deprecation --- site/content/in-dev/unreleased/evolution.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index fb37d5c5e5..8dddef78e5 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -36,12 +36,19 @@ REST Catalog API. The Polaris Management API is maintained as a versioned REST API. New releases of Polaris may include changes to the current version of the Management API. When that happens those changes -are intended to be compatible with prior versions of Polaris clients. +are intended to be compatible with prior versions of Polaris clients. Certain endpoints and +parameters may be deprecated. In case a major change is required to the Management API that cannot be implemented in a -backward-compatible way, a new version of the API will be introduced. Polaris servers will support -old API version for some transition period to allow clients to migrate. Note that API versions -are not meant to be 1:1 with Polaris releases. +backward-compatible way, new endpoints (URI paths) may be introduced. New URI "roots" may +be introduced too (e.g. `catalog/v2`). + +Note that those "v1", "v2", etc. URI path segments are not meant to be 1:1 with Polaris +releases or Polaris project version numbers (e.g. a "v2" path segment does not mean that +it is added in Polaris 2.0). + +Polaris servers will support deprecated API endpoints / parameters / versions / etc. +for some transition period to allow clients to migrate. The Generic Tables API is an experimental API at this time. Refer to its documentation page for more details about compatibility expectations. From 82b57b4224deb1b2548a7a892dec3b79be19a45b Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Fri, 13 Jun 2025 21:07:55 -0400 Subject: [PATCH 04/25] review: remove Generic Tables paragraph --- site/content/in-dev/unreleased/evolution.md | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index 8dddef78e5..874bba2d52 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -34,14 +34,14 @@ community. Polaris attempts to accurately implement this specification. However, is no guarantee that Polaris releases always implement the latest version of the Iceberg REST Catalog API. -The Polaris Management API is maintained as a versioned REST API. New releases of Polaris may -include changes to the current version of the Management API. When that happens those changes -are intended to be compatible with prior versions of Polaris clients. Certain endpoints and -parameters may be deprecated. +Any API under Polaris control (e.g. the Management API) is maintained as a versioned REST +API. New releases of Polaris may include changes to the current version of the API. When +that happens those changes are intended to be compatible with prior versions of Polaris +clients. Certain endpoints and parameters may be deprecated. In case a major change is required to the Management API that cannot be implemented in a backward-compatible way, new endpoints (URI paths) may be introduced. New URI "roots" may -be introduced too (e.g. `catalog/v2`). +be introduced too (e.g. `api/catalog/v2`). Note that those "v1", "v2", etc. URI path segments are not meant to be 1:1 with Polaris releases or Polaris project version numbers (e.g. a "v2" path segment does not mean that @@ -50,9 +50,6 @@ it is added in Polaris 2.0). Polaris servers will support deprecated API endpoints / parameters / versions / etc. for some transition period to allow clients to migrate. -The Generic Tables API is an experimental API at this time. Refer to its documentation page -for more details about compatibility expectations. - ## Polaris as a Library Polaris produces several jars. These jars or custom builds of Polaris code may be used in From a8cb09b7e04f3581d794b46f6d9896b830689117 Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Fri, 13 Jun 2025 21:09:46 -0400 Subject: [PATCH 05/25] review: update titles --- site/content/in-dev/unreleased/evolution.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index 874bba2d52..dfc050eb0d 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -24,7 +24,7 @@ weight: 1000 This page discusses what can be expected from Apache Polaris as the project evolves. -## Polaris as a Catalog +## Using Polaris as a Catalog Polaris is primarily intended to be used as a Catalog of Tables and Views. As such, it implements the Iceberg REST API, Polaris Management API and the Generic Tables API. @@ -50,7 +50,7 @@ it is added in Polaris 2.0). Polaris servers will support deprecated API endpoints / parameters / versions / etc. for some transition period to allow clients to migrate. -## Polaris as a Library +## Using Polaris as a Build-Time Dependency Polaris produces several jars. These jars or custom builds of Polaris code may be used in downstream projects according to the terms of the license included into Polaris distributions. From fe8511b2cae5f3c7dced176823e0d29dd1200124 Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Fri, 13 Jun 2025 21:25:39 -0400 Subject: [PATCH 06/25] review: use semver --- site/content/in-dev/unreleased/evolution.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index dfc050eb0d..d99cae63fa 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -55,9 +55,19 @@ for some transition period to allow clients to migrate. Polaris produces several jars. These jars or custom builds of Polaris code may be used in downstream projects according to the terms of the license included into Polaris distributions. +Polaris strives to follow [Semantic Versioning](https://semver.org/) conventions both with +respect to java code and REST APIs. + However, Polaris' focus as a project is not on ensuring ideal binary compatibility at the java class level between releases. Maintainers try to keep binary compatibility on the "best effort" basis. The main goal being the implementation of Polaris capabilities for access via REST APIs. +Even modules that have words like "core" or "api" in their names still follow the "best effort" +strategy and may have non-backward compatible changes between releases. + +This means that major version increments should be expected often (in the SemVer sense). + +The minimal version of the JRE required by Polaris code (compilation target) may be updated in +any release. Different Polaris jars may have different minimal JRE version requirements. This approach is not meant to discourage the use of Polaris code in downstream projects, but to allow more flexibility in evolving the codebase to support new Catalog-level features From 588eccc7302847359d57dad843a478c264434916 Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Fri, 13 Jun 2025 21:38:31 -0400 Subject: [PATCH 07/25] add: Managing Polaris Database --- site/content/in-dev/unreleased/evolution.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index d99cae63fa..b6fa310cc5 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -50,6 +50,23 @@ it is added in Polaris 2.0). Polaris servers will support deprecated API endpoints / parameters / versions / etc. for some transition period to allow clients to migrate. +### Managing Polaris Database + +Polaris stores its data in a database, which is sometimes referred to as "Meta Store" or +"Persistence" in other docs. + +Several Persistence implementations may be supported by Polaris in each release. +For example: "EclipseLink" (deprecated) and "JDBC" (current). + +Each type of Persistence evolves individually. Within each Persistence type, Polaris +attempts to support rolling upgrades (both version X and X + 1 servers running at the +same time). + +However, users should not expect that a transition from "EclipseLink" to "JDBC" can be +done in a rolling upgrade manner. Polaris provides tools for migrating between different +catalogs and those tools may be used to migrate between different Persistence types. +Service interruption (downtime) should be expected in those cases. + ## Using Polaris as a Build-Time Dependency Polaris produces several jars. These jars or custom builds of Polaris code may be used in From 51231fd7c9a2d68daa1de2f1454d00fbd266126e Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Mon, 16 Jun 2025 12:36:29 -0400 Subject: [PATCH 08/25] review: remove "binary compat" statements --- site/content/in-dev/unreleased/evolution.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index b6fa310cc5..3add7b4c41 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -75,12 +75,6 @@ downstream projects according to the terms of the license included into Polaris Polaris strives to follow [Semantic Versioning](https://semver.org/) conventions both with respect to java code and REST APIs. -However, Polaris' focus as a project is not on ensuring ideal binary compatibility at the java -class level between releases. Maintainers try to keep binary compatibility on the "best effort" -basis. The main goal being the implementation of Polaris capabilities for access via REST APIs. -Even modules that have words like "core" or "api" in their names still follow the "best effort" -strategy and may have non-backward compatible changes between releases. - This means that major version increments should be expected often (in the SemVer sense). The minimal version of the JRE required by Polaris code (compilation target) may be updated in From 166f48a7ef048d6249defb0eb074d6b825328f1a Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Mon, 16 Jun 2025 12:39:06 -0400 Subject: [PATCH 09/25] review: remove "attempts" from the REST API statement --- site/content/in-dev/unreleased/evolution.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index 3add7b4c41..519c33781d 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -30,7 +30,7 @@ Polaris is primarily intended to be used as a Catalog of Tables and Views. As su it implements the Iceberg REST API, Polaris Management API and the Generic Tables API. Revisions of the Iceberg REST API are controlled by the [Apache Iceberg](https://iceberg.apache.org/) -community. Polaris attempts to accurately implement this specification. However, there +community. Polaris accurately implements this specification. However, there is no guarantee that Polaris releases always implement the latest version of the Iceberg REST Catalog API. From e4f07af8b1dc85fca019fe2bdb0a4265a42ae47f Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Mon, 16 Jun 2025 12:48:23 -0400 Subject: [PATCH 10/25] review: rephrase persistence-related text --- site/content/in-dev/unreleased/evolution.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index 519c33781d..ff33479f3d 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -55,17 +55,18 @@ for some transition period to allow clients to migrate. Polaris stores its data in a database, which is sometimes referred to as "Meta Store" or "Persistence" in other docs. -Several Persistence implementations may be supported by Polaris in each release. -For example: "EclipseLink" (deprecated) and "JDBC" (current). +Each Polaris release may support multiple Persistence implementations, +for example, "EclipseLink" (deprecated) and "JDBC" (current). Each type of Persistence evolves individually. Within each Persistence type, Polaris attempts to support rolling upgrades (both version X and X + 1 servers running at the same time). -However, users should not expect that a transition from "EclipseLink" to "JDBC" can be -done in a rolling upgrade manner. Polaris provides tools for migrating between different -catalogs and those tools may be used to migrate between different Persistence types. -Service interruption (downtime) should be expected in those cases. +However, migrating between different Persistence types is not supported in a rolling +upgrade manner (for example, migrating from "EclipseLink" to "JDBC"). Polaris provides +[tools](https://github.com/apache/polaris-tools/) for migrating between different +catalogs and those tools may be used to migrate between different Persistence types +as well. Service interruption (downtime) should be expected in those cases. ## Using Polaris as a Build-Time Dependency From c71dacc04cb8133162894864ad8dbd430f937951 Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Mon, 16 Jun 2025 14:16:28 -0400 Subject: [PATCH 11/25] review: fix capitalization --- site/content/in-dev/unreleased/evolution.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index ff33479f3d..5038be0878 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -74,7 +74,7 @@ Polaris produces several jars. These jars or custom builds of Polaris code may b downstream projects according to the terms of the license included into Polaris distributions. Polaris strives to follow [Semantic Versioning](https://semver.org/) conventions both with -respect to java code and REST APIs. +respect to Java code and REST APIs. This means that major version increments should be expected often (in the SemVer sense). @@ -82,7 +82,7 @@ The minimal version of the JRE required by Polaris code (compilation target) may any release. Different Polaris jars may have different minimal JRE version requirements. This approach is not meant to discourage the use of Polaris code in downstream projects, but -to allow more flexibility in evolving the codebase to support new Catalog-level features +to allow more flexibility in evolving the codebase to support new catalog-level features and improve code efficiency. Maintainers of downstream projects are encouraged to join Polaris mailing lists to monitor project changes, suggest improvements, and engage with the Polaris community in case of specific compatibility concerns. From cd0800d07ef7c1c233357654b1fcf3b918eb3384 Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Mon, 16 Jun 2025 14:25:42 -0400 Subject: [PATCH 12/25] review: link to ../metastores --- site/content/in-dev/unreleased/evolution.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index 5038be0878..cf7af0cf32 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -52,10 +52,10 @@ for some transition period to allow clients to migrate. ### Managing Polaris Database -Polaris stores its data in a database, which is sometimes referred to as "Meta Store" or +Polaris stores its data in a database, which is sometimes referred to as "Metastore" or "Persistence" in other docs. -Each Polaris release may support multiple Persistence implementations, +Each Polaris release may support multiple Persistence [implementations](../metastores), for example, "EclipseLink" (deprecated) and "JDBC" (current). Each type of Persistence evolves individually. Within each Persistence type, Polaris From 4deb83c23b31b29e7b14ef03ee9b9f4c43ec483c Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Mon, 16 Jun 2025 14:30:42 -0400 Subject: [PATCH 13/25] review: link from prod config --- .../in-dev/unreleased/configuring-polaris-for-production.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/site/content/in-dev/unreleased/configuring-polaris-for-production.md b/site/content/in-dev/unreleased/configuring-polaris-for-production.md index 3d0bfd232c..fac51b40f9 100644 --- a/site/content/in-dev/unreleased/configuring-polaris-for-production.md +++ b/site/content/in-dev/unreleased/configuring-polaris-for-production.md @@ -215,3 +215,8 @@ polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES" = [ "S3", "Azure" ] ``` Leave out `FILE` to prevent its use. Only include the storage types your setup needs. +### Upgrade Considerations + +The [Polaris Evolution](../evolution) page discusses backward compatibility and +upgrade concerns. + From 65678350990faa5e15b8326f2af0e362f0fd1f85 Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Mon, 16 Jun 2025 17:47:46 -0400 Subject: [PATCH 14/25] review: rephrase Iceberg REST support paragraph --- site/content/in-dev/unreleased/evolution.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index cf7af0cf32..fdb90d5473 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -30,9 +30,10 @@ Polaris is primarily intended to be used as a Catalog of Tables and Views. As su it implements the Iceberg REST API, Polaris Management API and the Generic Tables API. Revisions of the Iceberg REST API are controlled by the [Apache Iceberg](https://iceberg.apache.org/) -community. Polaris accurately implements this specification. However, there -is no guarantee that Polaris releases always implement the latest version of the Iceberg -REST Catalog API. +community. Polaris attempts to accurately implement this specification. Nonetheless, +optional REST Catalog features may or may not be supported immediately. In general, +there is no guarantee that Polaris releases always implement the latest version of +the Iceberg REST Catalog API. Any API under Polaris control (e.g. the Management API) is maintained as a versioned REST API. New releases of Polaris may include changes to the current version of the API. When From a9686662a604c2462dd2ab1ae70656eb8fc8e625 Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Mon, 16 Jun 2025 18:17:13 -0400 Subject: [PATCH 15/25] review: expand SemVer handling --- site/content/in-dev/unreleased/evolution.md | 39 ++++++++++++++++++--- 1 file changed, 35 insertions(+), 4 deletions(-) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index fdb90d5473..fe0f03ceb8 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -74,10 +74,8 @@ as well. Service interruption (downtime) should be expected in those cases. Polaris produces several jars. These jars or custom builds of Polaris code may be used in downstream projects according to the terms of the license included into Polaris distributions. -Polaris strives to follow [Semantic Versioning](https://semver.org/) conventions both with -respect to Java code and REST APIs. - -This means that major version increments should be expected often (in the SemVer sense). +This means that major version increments should be expected often (in the SemVer sense, see +below). The minimal version of the JRE required by Polaris code (compilation target) may be updated in any release. Different Polaris jars may have different minimal JRE version requirements. @@ -87,3 +85,36 @@ to allow more flexibility in evolving the codebase to support new catalog-level and improve code efficiency. Maintainers of downstream projects are encouraged to join Polaris mailing lists to monitor project changes, suggest improvements, and engage with the Polaris community in case of specific compatibility concerns. + +## Semantic Versioning + +Polaris strives to follow [Semantic Versioning](https://semver.org/) conventions both with +respect to Java code and REST APIs. + +The API surface for the purpose of Semantic Versioning is defined as follows. + +**Client-Facing API** + +* Iceberg REST Catalog API and Generic Tables API (refer to this [link](../polaris-catalog-service/) +for their combined Open API definition). + * Note: Polaris implementing an optional Iceberg REST Catalog feature that was unimplemented + in the previous release is not considered a major change. + * Supporting a new revision of the Iceberg REST Catalog spec in a backward-compatible way + is not considered a major change. + * Changing the implementation of an Iceberg REST Catalog feature / endpoint in a non-backward + compatible way is a major change. +* [Polaris Management API](../polaris-management-service/) +* [Polaris Policies](http://localhost:1313/in-dev/unreleased/policy/) +* Configuration + * Public (not internal) properties inside Polaris [Entities](../entities/) that affect Polaris behaviour. + * Runtime [configuration](../configuration/) properties with the `polaris.` name prefix. + +**Build-Time Dependencies** + +* The `polaris-core` module (`public` classes and interfaces) + +**Non-API Modules** + +Build-time dependencies other than the `polaris-core` module are considered non-API +and are not subject to tracking in semantic version number changes. Their `public` +Java interfaces may change without notice even in minor releases. From eb54ece4f828c7d0b032ec122ef75a3380f7d7e3 Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Mon, 16 Jun 2025 18:23:34 -0400 Subject: [PATCH 16/25] review: add OpenAPI links --- site/content/in-dev/unreleased/evolution.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index fe0f03ceb8..d4a378a9f4 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -27,7 +27,8 @@ This page discusses what can be expected from Apache Polaris as the project evol ## Using Polaris as a Catalog Polaris is primarily intended to be used as a Catalog of Tables and Views. As such, -it implements the Iceberg REST API, Polaris Management API and the Generic Tables API. +it implements the [Iceberg REST API, Generic Tables API](../polaris-catalog-service/), +and [Polaris Management API](../polaris-management-service/). Revisions of the Iceberg REST API are controlled by the [Apache Iceberg](https://iceberg.apache.org/) community. Polaris attempts to accurately implement this specification. Nonetheless, From 018f203f5995d0c22c6c69bd7080b99c6c6d29cd Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Mon, 16 Jun 2025 21:53:45 -0400 Subject: [PATCH 17/25] review: add OpenAPI links --- site/content/in-dev/unreleased/evolution.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index d4a378a9f4..e384b3ba10 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -105,7 +105,7 @@ for their combined Open API definition). * Changing the implementation of an Iceberg REST Catalog feature / endpoint in a non-backward compatible way is a major change. * [Polaris Management API](../polaris-management-service/) -* [Polaris Policies](http://localhost:1313/in-dev/unreleased/policy/) +* [Polaris Policies](../policy/) * Configuration * Public (not internal) properties inside Polaris [Entities](../entities/) that affect Polaris behaviour. * Runtime [configuration](../configuration/) properties with the `polaris.` name prefix. From 269aa1b90720d8105bb99428c820583dbe89cb49 Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Tue, 17 Jun 2025 10:40:32 -0400 Subject: [PATCH 18/25] rework SemVer text --- site/content/in-dev/unreleased/evolution.md | 44 +++++++++------------ 1 file changed, 19 insertions(+), 25 deletions(-) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index e384b3ba10..754f9dead8 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -75,12 +75,12 @@ as well. Service interruption (downtime) should be expected in those cases. Polaris produces several jars. These jars or custom builds of Polaris code may be used in downstream projects according to the terms of the license included into Polaris distributions. -This means that major version increments should be expected often (in the SemVer sense, see -below). - The minimal version of the JRE required by Polaris code (compilation target) may be updated in any release. Different Polaris jars may have different minimal JRE version requirements. +Changes in Java class should be expected at any time regardless of the module name or +whether the class / method is `public` or `private`. + This approach is not meant to discourage the use of Polaris code in downstream projects, but to allow more flexibility in evolving the codebase to support new catalog-level features and improve code efficiency. Maintainers of downstream projects are encouraged to join Polaris @@ -90,32 +90,26 @@ community in case of specific compatibility concerns. ## Semantic Versioning Polaris strives to follow [Semantic Versioning](https://semver.org/) conventions both with -respect to Java code and REST APIs. +respect REST APIs (beta and experimental APIs excepted), [Polaris Policies](../policy/) +and user-facing [configuration](../configuration/). -The API surface for the purpose of Semantic Versioning is defined as follows. +The following are some examples of Polaris approach to SemVer in REST APIs / configuration. +These examples are for illustration purposes and should not be considered to be +exhaustive. -**Client-Facing API** +Polaris implementing an optional Iceberg REST Catalog feature that was unimplemented +in the previous release is not considered a major change. -* Iceberg REST Catalog API and Generic Tables API (refer to this [link](../polaris-catalog-service/) -for their combined Open API definition). - * Note: Polaris implementing an optional Iceberg REST Catalog feature that was unimplemented - in the previous release is not considered a major change. - * Supporting a new revision of the Iceberg REST Catalog spec in a backward-compatible way - is not considered a major change. - * Changing the implementation of an Iceberg REST Catalog feature / endpoint in a non-backward - compatible way is a major change. -* [Polaris Management API](../polaris-management-service/) -* [Polaris Policies](../policy/) -* Configuration - * Public (not internal) properties inside Polaris [Entities](../entities/) that affect Polaris behaviour. - * Runtime [configuration](../configuration/) properties with the `polaris.` name prefix. +Supporting a new revision of the Iceberg REST Catalog spec in a backward-compatible way +is not considered a major change. Specifically, supporting new REST API prefixes (e.g. `v2`) +is not a mojor change because it does not affect older clients. -**Build-Time Dependencies** +Changing the implementation of an Iceberg REST Catalog feature / endpoint in a non-backward +compatible way (e.g. removing or renaming a request parameter) is a major change. -* The `polaris-core` module (`public` classes and interfaces) +Dropping support for a configuration property with the `polaris.` name prefix is a major change. -**Non-API Modules** +Dropping support for any previously defined [Policy](../policy/) type or property is a major change. -Build-time dependencies other than the `polaris-core` module are considered non-API -and are not subject to tracking in semantic version number changes. Their `public` -Java interfaces may change without notice even in minor releases. +Upgrading Quarkus Runtime to its next major version is a major change (because +Quarkus-managed configuration may change). From 41e6d92f465076708048ab468703364f8d386a3d Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Tue, 17 Jun 2025 10:46:02 -0400 Subject: [PATCH 19/25] reword REST API statements, mention "beta" --- site/content/in-dev/unreleased/evolution.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index 754f9dead8..a707e64b6e 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -27,19 +27,19 @@ This page discusses what can be expected from Apache Polaris as the project evol ## Using Polaris as a Catalog Polaris is primarily intended to be used as a Catalog of Tables and Views. As such, -it implements the [Iceberg REST API, Generic Tables API](../polaris-catalog-service/), -and [Polaris Management API](../polaris-management-service/). +it implements the Iceberg REST Catalog API and its own REST APIs -Revisions of the Iceberg REST API are controlled by the [Apache Iceberg](https://iceberg.apache.org/) +Revisions of the Iceberg REST Catalog API are controlled by the [Apache Iceberg](https://iceberg.apache.org/) community. Polaris attempts to accurately implement this specification. Nonetheless, optional REST Catalog features may or may not be supported immediately. In general, there is no guarantee that Polaris releases always implement the latest version of the Iceberg REST Catalog API. -Any API under Polaris control (e.g. the Management API) is maintained as a versioned REST -API. New releases of Polaris may include changes to the current version of the API. When -that happens those changes are intended to be compatible with prior versions of Polaris -clients. Certain endpoints and parameters may be deprecated. +Any API under Polaris control that is not in an "experimental" or "beta" state +(e.g. the Management API) is maintained as a versioned REST API. New releases of Polaris +may include changes to the current version of the API. When that happens those changes +are intended to be compatible with prior versions of Polaris clients. Certain endpoints +and parameters may be deprecated. In case a major change is required to the Management API that cannot be implemented in a backward-compatible way, new endpoints (URI paths) may be introduced. New URI "roots" may From 1edd258b5d227074d76bbf3936b95ab0cb51f217 Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Tue, 17 Jun 2025 10:57:03 -0400 Subject: [PATCH 20/25] review: typos --- site/content/in-dev/unreleased/evolution.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index a707e64b6e..244aa04941 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -27,7 +27,7 @@ This page discusses what can be expected from Apache Polaris as the project evol ## Using Polaris as a Catalog Polaris is primarily intended to be used as a Catalog of Tables and Views. As such, -it implements the Iceberg REST Catalog API and its own REST APIs +it implements the Iceberg REST Catalog API and its own REST APIs. Revisions of the Iceberg REST Catalog API are controlled by the [Apache Iceberg](https://iceberg.apache.org/) community. Polaris attempts to accurately implement this specification. Nonetheless, @@ -36,7 +36,7 @@ there is no guarantee that Polaris releases always implement the latest version the Iceberg REST Catalog API. Any API under Polaris control that is not in an "experimental" or "beta" state -(e.g. the Management API) is maintained as a versioned REST API. New releases of Polaris +(e.g. the Management API) is maintained as a versioned REST API. New releases of Polaris may include changes to the current version of the API. When that happens those changes are intended to be compatible with prior versions of Polaris clients. Certain endpoints and parameters may be deprecated. From 7a4062abde4b127f8bcfa8986b01e2a481e15e54 Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Tue, 17 Jun 2025 12:42:09 -0400 Subject: [PATCH 21/25] review: style --- site/content/in-dev/unreleased/evolution.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index 244aa04941..8d46635489 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -79,7 +79,7 @@ The minimal version of the JRE required by Polaris code (compilation target) may any release. Different Polaris jars may have different minimal JRE version requirements. Changes in Java class should be expected at any time regardless of the module name or -whether the class / method is `public` or `private`. +whether the class / method is `public` or not. This approach is not meant to discourage the use of Polaris code in downstream projects, but to allow more flexibility in evolving the codebase to support new catalog-level features From 281c117c0181c6ea3a27edb859e55eeb851fdc7a Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Tue, 17 Jun 2025 12:43:36 -0400 Subject: [PATCH 22/25] review: avoid specifics --- site/content/in-dev/unreleased/evolution.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index 8d46635489..561833e751 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -41,7 +41,7 @@ may include changes to the current version of the API. When that happens those c are intended to be compatible with prior versions of Polaris clients. Certain endpoints and parameters may be deprecated. -In case a major change is required to the Management API that cannot be implemented in a +In case a major change is required to an API that cannot be implemented in a backward-compatible way, new endpoints (URI paths) may be introduced. New URI "roots" may be introduced too (e.g. `api/catalog/v2`). From 6951cc17b2a58310d9448f6e08baf3bac505593b Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Tue, 17 Jun 2025 12:44:04 -0400 Subject: [PATCH 23/25] review: grammar --- site/content/in-dev/unreleased/evolution.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index 561833e751..ccd74d764a 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -90,7 +90,7 @@ community in case of specific compatibility concerns. ## Semantic Versioning Polaris strives to follow [Semantic Versioning](https://semver.org/) conventions both with -respect REST APIs (beta and experimental APIs excepted), [Polaris Policies](../policy/) +respect to REST APIs (beta and experimental APIs excepted), [Polaris Policies](../policy/) and user-facing [configuration](../configuration/). The following are some examples of Polaris approach to SemVer in REST APIs / configuration. From 41df096b4085b57272ae1eb0422a3b478d8d24ef Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Tue, 17 Jun 2025 12:45:21 -0400 Subject: [PATCH 24/25] review: format --- site/content/in-dev/unreleased/evolution.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index ccd74d764a..b17ea7f542 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -97,19 +97,19 @@ The following are some examples of Polaris approach to SemVer in REST APIs / con These examples are for illustration purposes and should not be considered to be exhaustive. -Polaris implementing an optional Iceberg REST Catalog feature that was unimplemented +* Polaris implementing an optional Iceberg REST Catalog feature that was unimplemented in the previous release is not considered a major change. -Supporting a new revision of the Iceberg REST Catalog spec in a backward-compatible way +* Supporting a new revision of the Iceberg REST Catalog spec in a backward-compatible way is not considered a major change. Specifically, supporting new REST API prefixes (e.g. `v2`) is not a mojor change because it does not affect older clients. -Changing the implementation of an Iceberg REST Catalog feature / endpoint in a non-backward +* Changing the implementation of an Iceberg REST Catalog feature / endpoint in a non-backward compatible way (e.g. removing or renaming a request parameter) is a major change. -Dropping support for a configuration property with the `polaris.` name prefix is a major change. +* Dropping support for a configuration property with the `polaris.` name prefix is a major change. -Dropping support for any previously defined [Policy](../policy/) type or property is a major change. +* Dropping support for any previously defined [Policy](../policy/) type or property is a major change. -Upgrading Quarkus Runtime to its next major version is a major change (because +* Upgrading Quarkus Runtime to its next major version is a major change (because Quarkus-managed configuration may change). From 746cf3d73d292b76e9e78011a00623e4c9e4e0cd Mon Sep 17 00:00:00 2001 From: Dmitri Bourlatchkov Date: Tue, 17 Jun 2025 13:45:23 -0400 Subject: [PATCH 25/25] review: typo Co-authored-by: Eric Maynard --- site/content/in-dev/unreleased/evolution.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/content/in-dev/unreleased/evolution.md b/site/content/in-dev/unreleased/evolution.md index b17ea7f542..ea29badc84 100644 --- a/site/content/in-dev/unreleased/evolution.md +++ b/site/content/in-dev/unreleased/evolution.md @@ -102,7 +102,7 @@ in the previous release is not considered a major change. * Supporting a new revision of the Iceberg REST Catalog spec in a backward-compatible way is not considered a major change. Specifically, supporting new REST API prefixes (e.g. `v2`) -is not a mojor change because it does not affect older clients. +is not a major change because it does not affect older clients. * Changing the implementation of an Iceberg REST Catalog feature / endpoint in a non-backward compatible way (e.g. removing or renaming a request parameter) is a major change.