For this tutorial, we'll launch an instance of Polaris that stores entities only in-memory. This means that any entities that you define will be destroyed when Polaris is shut down. It also means that Polaris will automatically bootstrap itself with root credentials. For more information on how to configure Polaris for production usage, see the docs.
+">For this tutorial, we'll launch an instance of Polaris that stores entities only in-memory. This means that any entities that you define will be destroyed when Polaris is shut down. It also means that Polaris will automatically bootstrap itself with root credentials. For more information on how to configure Polaris for production usage, see the docs.
When Polaris is launched using in-memory mode the root principal credentials can be found in stdout on initial startup. For example:
realm: default-realm root principal credentials: <client-id>:<client-secret>
Building Polaris
export CLIENT_ID=<client-id>
export CLIENT_SECRET=<client-secret>
Building Polaris
<pre><code><span class="token punctuation">.</span><span class="token operator">/</span>polaris <span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span> principals create example <span class="token punctuation">{</span><span class="token string">"clientId"</span><span class="token punctuation">:</span> <span class="token string">"XXXX"</span><span class="token punctuation">,</span> <span class="token string">"clientSecret"</span><span class="token punctuation">:</span> <span class="token string">"YYYY"</span><span class="token punctuation">}</span> </code></pre> -<p>Now, we grant the principal the <a href="./entities.md#principal-role">principal role</a> we created, and grant the <a href="./entities.md#catalog-role">catalog role</a> the principal role we created. For more information on these entities, please refer to the linked documentation.</p> +<p>Now, we grant the principal the <a href="#tag/Polaris-Entities/Principal-Role">principal role</a> we created, and grant the <a href="#tag/Polaris-Entities/Catalog-Role">catalog role</a> the principal role we created. For more information on these entities, please refer to the linked documentation.</p> <pre><code class="language-shell">./polaris <span class="token punctuation">\</span> --client-id <span class="token variable">${CLIENT_ID}</span> <span class="token punctuation">\</span> --client-secret <span class="token variable">${CLIENT_SECRET}</span> <span class="token punctuation">\</span> @@ -674,7 +674,7 @@Building Polaris
</code></pre> <p>Now, we’ve linked our principal to the catalog via roles like so:</p> <p><img src="./img/quickstart/privilege-illustration-1.png" alt="Principal to Catalog" title="Principal to Catalog"></p> -<p>In order to give this principal the ability to interact with the catalog, we must assign some <a href="./entities.md#privilege">privileges</a>. For the time being, we will give this principal the ability to fully manage content in our new catalog. We can do this with the CLI like so:</p> +<p>In order to give this principal the ability to interact with the catalog, we must assign some <a href="#tag/Polaris-Entities/Privilege">privileges</a>. For the time being, we will give this principal the ability to fully manage content in our new catalog. We can do this with the CLI like so:</p> <pre><code class="language-shell">./polaris <span class="token punctuation">\</span> --client-id <span class="token variable">${CLIENT_ID}</span> <span class="token punctuation">\</span> --client-secret <span class="token variable">${CLIENT_SECRET}</span> <span class="token punctuation">\</span> @@ -685,10 +685,10 @@Building Polaris
--catalog-role quickstart_catalog_role <span class="token punctuation">\</span> CATALOG_MANAGE_CONTENT </code></pre> -<p>This grants the <a href="./entities.md#privilege">catalog privileges</a> <code>CATALOG_MANAGE_CONTENT</code> to our catalog role, linking everything together like so:</p> +<p>This grants the <a href="#tag/Polaris-Entities/Privilege">catalog privileges</a> <code>CATALOG_MANAGE_CONTENT</code> to our catalog role, linking everything together like so:</p> <p><img src="./img/quickstart/privilege-illustration-2.png" alt="Principal to Catalog with Catalog Role" title="Principal to Catalog with Catalog Role"></p> <p><code>CATALOG_MANAGE_CONTENT</code> has create/list/read/write privileges on all entities within the catalog. The same privilege could be granted to a namespace, in which case the principal could create/list/read/write any entity under that namespace.</p> -">In Polaris, the catalog is the top-level entity that objects like tables and views are organized under. With a Polaris service running, you can create a catalog like so:
+">In Polaris, the catalog is the top-level entity that objects like tables and views are organized under. With a Polaris service running, you can create a catalog like so:
cd ~/polaris
./polaris \
@@ -703,10 +703,10 @@ Building Polaris
This will create a new catalog called quickstart_catalog.
The DEFAULT_BASE_LOCATION you provide will be the default location that objects in this catalog should be stored in, and the ROLE_ARN you provide should be a Role ARN with access to read and write data in that location. These credentials will be provided to engines reading data from the catalog once they have authenticated with Polaris using credentials that have access to those resources.
If you’re using a storage type other than S3, such as Azure, you’ll provide a different type of credential than a Role ARN. For more details on supported storage types, see the docs.
-Additionally, if Polaris is running somewhere other than localhost:8181, you can specify the correct hostname and port by providing --host and --port flags. For the full set of options supported by the CLI, please refer to the docs.
If you’re using a storage type other than S3, such as Azure, you’ll provide a different type of credential than a Role ARN. For more details on supported storage types, see the docs.
+Additionally, if Polaris is running somewhere other than localhost:8181, you can specify the correct hostname and port by providing --host and --port flags. For the full set of options supported by the CLI, please refer to the docs.
Creating a Principal and Assigning it Privileges
-With a catalog created, we can create a principal that has access to manage that catalog. For details on how to configure the Polaris CLI, see the section above or refer to the docs.
+With a catalog created, we can create a principal that has access to manage that catalog. For details on how to configure the Polaris CLI, see the section above or refer to the docs.
./polaris \
--client-id ${CLIENT_ID} \
--client-secret ${CLIENT_SECRET} \
@@ -734,7 +734,7 @@ Creating a Principal a
./polaris ... principals create example
{"clientId": "XXXX", "clientSecret": "YYYY"}
-
Now, we grant the principal the principal role we created, and grant the catalog role the principal role we created. For more information on these entities, please refer to the linked documentation.
+Now, we grant the principal the principal role we created, and grant the catalog role the principal role we created. For more information on these entities, please refer to the linked documentation.
./polaris \
--client-id ${CLIENT_ID} \
--client-secret ${CLIENT_SECRET} \
@@ -754,7 +754,7 @@ Creating a Principal a
Now, we’ve linked our principal to the catalog via roles like so:
-In order to give this principal the ability to interact with the catalog, we must assign some privileges. For the time being, we will give this principal the ability to fully manage content in our new catalog. We can do this with the CLI like so:
+In order to give this principal the ability to interact with the catalog, we must assign some privileges. For the time being, we will give this principal the ability to fully manage content in our new catalog. We can do this with the CLI like so:
./polaris \
--client-id ${CLIENT_ID} \
--client-secret ${CLIENT_SECRET} \
@@ -765,7 +765,7 @@ Creating a Principal a
--catalog-role quickstart_catalog_role \
CATALOG_MANAGE_CONTENT
-This grants the catalog privileges CATALOG_MANAGE_CONTENT to our catalog role, linking everything together like so:
+This grants the catalog privileges CATALOG_MANAGE_CONTENT to our catalog role, linking everything together like so:
CATALOG_MANAGE_CONTENT has create/list/read/write privileges on all entities within the catalog. The same privilege could be granted to a namespace, in which case the principal could create/list/read/write any entity under that namespace.
With Polaris, you can provide centralized, secure read and write access to your Iceberg tables across different REST-compatible query engines.
Connecting with Spark
</tr> </tbody></table> ">This section introduces key concepts associated with using Apache Polaris (Incubating).
-In the following diagram, a sample Apache Polaris (Incubating) structure with nested namespaces is shown for Catalog1. No tables +
In the following diagram, a sample Apache Polaris (Incubating) structure with nested namespaces is shown for Catalog1. No tables or namespaces have been created yet for Catalog2 or Catalog3.
Catalog
@@ -1154,16 +1154,16 @@Service connection
A service connection represents a REST-compatible engine (such as Apache Spark™, Apache Flink®, or Trino) that can read from and write to Polaris Catalog. When creating a new service connection, the Polaris administrator grants the service principal that is created with the new service connection either a new or existing principal role. A principal role is a resource in Polaris that you can use to logically group Polaris -service principals together and grant privileges on securable objects. For more information, see Principal role. Polaris uses a role-based access control (RBAC) model to grant service principals access to resources. For more information, -see Access control. For a diagram of this model, see RBAC model.
+service principals together and grant privileges on securable objects. For more information, see Principal role. Polaris uses a role-based access control (RBAC) model to grant service principals access to resources. For more information, +see Access control. For a diagram of this model, see RBAC model.If the Polaris administrator grants the service principal for the new service connection a new principal role, the service principal doesn't have any privileges granted to it yet. When securing the catalog that the new service connection will connect to, the Polaris administrator grants privileges to catalog roles and then grants these catalog roles to the new principal role. As a result, the service -principal for the new service connection has these privileges. For more information about catalog roles, see Catalog role.
+principal for the new service connection has these privileges. For more information about catalog roles, see Catalog role.If the Polaris administrator grants an existing principal role to the service principal for the new service connection, the service principal has the same privileges granted to the catalog roles that are granted to the existing principal role. If needed, the Polaris administrator can grant additional catalog roles to the existing principal role or remove catalog roles from it to adjust the privileges -bestowed to the service principal. For an example of how RBAC works in Polaris, see RBAC example.
+bestowed to the service principal. For an example of how RBAC works in Polaris, see RBAC example.Storage configuration
A storage configuration stores a generated identity and access management (IAM) entity for your external cloud storage and is created when you create a catalog. The storage configuration is used to set the values to connect Polaris to your cloud storage. During the @@ -1246,7 +1246,7 @@
Storage configuration
catalog resources and granted to principal roles.</p> </li> </ul> -<p>For more information, see <a href="access-control.md" title="Access control">Access control</a>.</p> +<p>For more information, see <a href="#tag/Access-Control" title="Access control">Access control</a>.</p> ">This section describes security and access control.
Credential vending
To secure interactions with service connections, Polaris vends temporary storage credentials to the query engine during query @@ -1271,7 +1271,7 @@
Access control
catalog resources and granted to principal roles. -For more information, see Access control.
+For more information, see Access control.
Apache®, Apache Iceberg™, Apache Spark™, Apache Flink®, and Flink® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
This page documents various entities that can be managed in Apache Polaris (Incubating).
A catalog is a top-level entity in Polaris that may contain other entities like namespaces and tables. These map directly to Apache Iceberg catalogs.
-For information on managing catalogs with the REST API or for more information on what data can be associated with a catalog, see the API docs.
+For information on managing catalogs with the REST API or for more information on what data can be associated with a catalog, see the API docs.
Storage Type
All catalogs in Polaris are associated with a storage type. Valid Storage Types are S3, Azure, and GCS. The FILE type is also additionally available for testing. Each of these types relates to a different storage provider where data within the catalog may reside. Depending on the storage type, various other configurations may be set for a catalog including credentials to be used when accessing data inside the catalog.
For details on how to use Storage Types in the REST API, see the API docs.
+For details on how to use Storage Types in the REST API, see the API docs.
A namespace is a logical entity that resides within a catalog and can contain other entities such as tables or views. Some other systems may refer to namespaces as schemas or databases.
In Polaris, namespaces can be nested. For example, a.b.c.d.e.f.g is a valid namespace. b is said to reside within a, and so on.
For information on managing namespaces with the REST API or for more information on what data can be associated with a namespace, see the API docs.
+For information on managing namespaces with the REST API or for more information on what data can be associated with a namespace, see the API docs.
Polaris tables are entities that map to Apache Iceberg tables.
-For information on managing tables with the REST API or for more information on what data can be associated with a table, see the API docs.
+For information on managing tables with the REST API or for more information on what data can be associated with a table, see the API docs.
Polaris views are entities that map to Apache Iceberg views.
-For information on managing views with the REST API or for more information on what data can be associated with a view, see the API docs.
+For information on managing views with the REST API or for more information on what data can be associated with a view, see the API docs.
Polaris principals are unique identities that can be used to represent users or services. Each principal may have one or more principal roles assigned to it for the purpose of accessing catalogs and the entities within them.
-For information on managing principals with the REST API or for more information on what data can be associated with a principal, see the API docs.
+For information on managing principals with the REST API or for more information on what data can be associated with a principal, see the API docs.
Polaris principal roles are labels that may be granted to principals. Each principal may have one or more principal roles, and the same principal role may be granted to multiple principals. Principal roles may be assigned based on the persona or responsibilities of a given principal, or on how that principal will need to access different entities within Polaris.
-For information on managing principal roles with the REST API or for more information on what data can be associated with a principal role, see the API docs.
+For information on managing principal roles with the REST API or for more information on what data can be associated with a principal role, see the API docs.
Polaris catalog roles are labels that may be granted to catalogs. Each catalog may have one or more catalog roles, and the same catalog role may be granted to multiple catalogs. Catalog roles may be assigned based on the nature of data that will reside in a catalog, or by the groups of users and services that might need to access that data.
@@ -1354,19 +1354,19 @@Storage Type
<p>A privilege can be scoped to any entity inside a catalog, including the catalog itself.</p> <p>For a list of supported privileges for each privilege class, see the API docs:</p> <ul> -<li><a href="../regtests/client/python/docs/TablePrivilege.md">Table Privileges</a></li> -<li><a href="../regtests/client/python/docs/ViewPrivilege.md">View Privileges</a></li> -<li><a href="../regtests/client/python/docs/NamespacePrivilege.md">Namespace Privileges</a></li> -<li><a href="../regtests/client/python/docs/CatalogPrivilege.md">Catalog Privileges</a></li> +<li><a href="https://github.com/apache/polaris/blob/main/regtests/client/python/docs/TablePrivilege.md">Table Privileges</a></li> +<li><a href="https://github.com/apache/polaris/blob/main/regtests/client/python/docs/ViewPrivilege.md">View Privileges</a></li> +<li><a href="https://github.com/apache/polaris/blob/main/regtests/client/python/docs/NamespacePrivilege.md">Namespace Privileges</a></li> +<li><a href="https://github.com/apache/polaris/blob/main/regtests/client/python/docs/CatalogPrivilege.md">Catalog Privileges</a></li> </ul> ">Polaris privileges are granted to catalog roles in order to grant principals with a given principal role some degree of access to catalogs with a given catalog role. When a privilege is granted to a catalog role, any principal roles granted that catalog role receive the privilege. In turn, any principals who are granted that principal role receive it.
A privilege can be scoped to any entity inside a catalog, including the catalog itself.
For a list of supported privileges for each privilege class, see the API docs:
Request samples
- Payload
{- "source": {
- "namespace": [
- "accounting",
- "tax"
], - "name": "paid-view"
}, - "destination": {
- "namespace": [
- "accounting",
- "tax"
], - "name": "owed-view"
}
}Response samples
- 400
- 401
- 403
- 404
- 406
- 409
- 419
- 503
- 5XX
{- "error": {
- "message": "Malformed request",
- "type": "BadRequestException",
- "code": 400
}
}