Skip to content

Conversation

@MonkeyCanCode
Copy link
Contributor

Synced usage from latest CLI.

Checklist

  • 🛡️ Don't disclose security issues! (contact security@apache.org)
  • 🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
  • 🧪 Added/updated tests with good coverage, or manually tested (and explained how)
  • 💡 Added comments for complex logic
  • 🧾 Updated CHANGELOG.md (if needed)
  • 📚 Updated documentation in site/content/in-dev/unreleased (if needed)

--service-account (Only for GCS) The service account to use when connecting to GCS
--property A key/value pair such as: tag=value. Multiple can be provided by specifying this option more than once
--catalog-connection-type The type of external catalog in [iceberg-rest, hadoop, hive].
--catalog-connection-type The type of external catalog in [ICEBERG, HADOOP, HIVE].
Copy link
Contributor

@flyrain flyrain Dec 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

@HonahX HonahX Dec 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think it should be the lower cased version, since they are the actual value of those enum type:

class CatalogConnectionType(Enum):
"""
Represents a ConnectionType for an EXTERNAL catalog -- see ConnectionConfigInfo in the spec
"""
HADOOP = "hadoop"
ICEBERG = "iceberg-rest"
HIVE = "hive"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

K. let me fix the tool doc.

Copy link
Contributor Author

@MonkeyCanCode MonkeyCanCode Dec 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so it doesn't appear it matters:

        Argument(Arguments.CATALOG_CONNECTION_TYPE, str,
                 Hints.Catalogs.External.CATALOG_CONNECTION_TYPE, lower=True,
                 choices=[ct.value for ct in CatalogConnectionType]),

as we are applying lower=True anyway. Also, in other places, we are refers to upper case ones as well:

# Allows both REST and HIVE connection type
polaris.features."SUPPORTED_CATALOG_CONNECTION_TYPES"=["ICEBERG_REST","HIVE"]

Also, as all other arguments with enum are pass in as upper case, should we keep them as upper case then. For example:

            CATALOG_AUTHENTICATION_TYPE = (
                "The type of authentication in [OAUTH, BEARER, SIGV4, IMPLICIT]"
            )
...
        Argument(Arguments.CATALOG_AUTHENTICATION_TYPE, str,
                 Hints.Catalogs.External.CATALOG_AUTHENTICATION_TYPE, lower=True,
                 choices=[at.value for at in AuthenticationType]),

Another thing is we have it as ICEBERG but the actual value is iceberg-rest and in the java server side, we have it as ICEBERG_REST, should we keep them consistent?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with [ICEBERG-REST, HADOOP, HIVE] in that case.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for diving deep into this!
Upper case SGTM. As for the iceberg-rest to iceberg change that would be a breaking change for current users. It's worth to mention in the changelog if we modify it.

IMHO, ICEBERG-REST should be fine.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for review. As the current update is correct (the upper case ones). Should we proceed with this change first? I will create another PR for changing from ICEBERG_REST to ICEBERG-REST (have deprecation for ICEBERG_REST then we can remove it in the next release?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this against the main branch. Passing --catalog-connection-type ICEBERG currently fails with the following error:

yufei@Yufeis-MacBook-Pro-2 polaris % ./polaris --profile dev catalogs create --type external --catalog-connection-type ICEBERG --default-base-location file://temp/c1 --storage-type file --catalog-authentication-type BEARER --catalog-bearer-token my-token-xxx c1
usage: ...
polaris catalogs create: error: argument --catalog-connection-type: invalid choice: 'iceberg' (choose from 'hadoop', 'iceberg-rest', 'hive')

Given that we’ve already been using ICEBERG_REST on the server side, I think it makes sense to keep it as-is to avoid introducing a breaking change. Sorry for the confusion earlier, I didn’t mean to suggest changing it to ICEBERG-REST.

For the CLI, one option could be to standardize on uppercase enum-style values (e.g. ICEBERG_REST, HADOOP, HIVE) for consistency with other arguments.

What do you think? @MonkeyCanCode @HonahX

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. Thanks for testing. Let me look into this tomorrow and fix this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should now be resolved as the issue is in client/python/apache_polaris/cli/constants.py. On the code base, it is enum as what @HonahX was mentioning earlier but we only ref by value in the helper function. Thus, I updated the helper used by options parser to use the right value now.

@MonkeyCanCode
Copy link
Contributor Author

@flyrain mind take another look? Then I do the needed change for ICEBERG_REST to ICEBERG-REST (be backward compatible)

Copy link
Contributor

@flyrain flyrain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @MonkeyCanCode !

@github-project-automation github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Dec 16, 2025
@flyrain flyrain closed this Dec 16, 2025
@github-project-automation github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Dec 16, 2025
@flyrain flyrain reopened this Dec 16, 2025
@github-project-automation github-project-automation bot moved this from Done to PRs In Progress in Basic Kanban Board Dec 16, 2025
@MonkeyCanCode MonkeyCanCode merged commit eb62065 into apache:main Dec 16, 2025
29 of 31 checks passed
@github-project-automation github-project-automation bot moved this from PRs In Progress to Done in Basic Kanban Board Dec 16, 2025
snazy added a commit to snazy/polaris that referenced this pull request Feb 11, 2026
* Doc cleanup for local deployment (apache#3213)

* Doc cleanup for admin tool (apache#3214)

* Bump version from 1.0.0 to 1.2.0 and fix health port (apache#3211)

* fix(deps): update dependency io.opentelemetry:opentelemetry-bom to v1.57.0 (apache#3223)

* fix(deps): update dependency org.apache.commons:commons-text to v1.15.0 (apache#3233)

* fix(deps): update dependency software.amazon.awssdk:bom to v2.40.3 (apache#3234)

* Core: Add timeout and retry logic to Azure token fetch (apache#3113)

* update markdown lint check (apache#3187)

use tcort/github-action-markdown-link-check, gaurav-nelson/github-action-markdown-link-check is deprecated

* NoSQL: Add metastore types and mappings (apache#3207)

Add the NoSQL specific metastore persistence types including the mapping from and to `*Polaris*Entity`.

* NoSQL/nit: fix javadoc for `Realms` (apache#3229)

* Fix build issue for docker not found when using latest docker desktop (apache#3227)

* fix(deps): update dependency org.mongodb:mongodb-driver-sync to v5.6.2 (apache#3238)

* fix(deps): update immutables to v2.12.0 (apache#3240)

* fix(deps): update dependency io.micrometer:micrometer-bom to v1.16.1 (apache#3239)

* [Core, Bug] CreateEntitiesIfNotExist/CreatePrincipal not return the same entity persisted. (apache#3219)

The PR fixes the issue, "CreateEntitiesIfNotExist/CreatePrincipal not return the same entity persisted", by letting persistEntity return the entity persisted and include that in the EntityResult. The PR also include new unit tests to verify the behavior

* (feat) doc: Update Makefile to fix admonitions in helm doc and remove redundant sections (apache#3232)

* Change org.testcontainers:<dep> to org.testcontainers:testcontainers-<dep> (apache#3225)

* Helm: add support for topologySpreadConstraints (apache#3216)

* chore(deps): update registry.access.redhat.com/ubi9/openjdk-21-runtime docker tag to v1.23-6.1764764731 (apache#3241)

* NoSQL correctness tests: add missing `logback-test.xml` files (apache#3230)

* Add Docker-based Ceph + Polaris cluster setup (apache#3022)


---------

Co-authored-by: sarunas.svegzda <ssv@infrahub.io>

* Service: Remove *CommitTableEvent, Add *UpdateTableEvent to Transactions (apache#3195)

* Update dependency pydantic to >=2.12.5,<2.13.0 (apache#2807)

* fix(deps): update dependency com.github.dasniko:testcontainers-keycloak to v4.0.1 (apache#3244)

* fix(deps): update mockito monorepo to v5.21.0 (apache#3245)

* Allow retrieving a config directly from a `Map` (apache#3220)

The current implementation deserializes the catalog configuration properties for each invocation of `getConfig*()` taking a `CatalogEntity`.

This change adds another `getConfig*()` variant that takes a `Map` to allow call sites to memoize the properties, where possible.

* Runtime/service: move getConfig() down to `IcebergCatalogHandler` (apache#3231)

All catalog specific functionality is implemented in `IcebergCatalogHandler`, whereas `IcebergCatalogAdapter` is meant to act as a "REST wrapper" to it.

This change moves the implementation of `getConfig` down to the handler, no functional changes.

* chore(deps): update quay.io/ceph/ceph docker tag to v20 (apache#3242)

* fix(deps): update quarkus platform and group to v3.30.3 (apache#3247)

* NoSQL: Prepare for NoSQL tests (apache#3235)

* Add an optional `bootstrapRealm()` implementation to `PolarisAuthzTestBase`
* Allow extending `IcebergCatalogHandlerAuthzTest`, move tests to `AbstractIcebergCatalogHandlerAuthzTest`
* No functional changes

* Shell script to verify staged release candidate artifacts (apache#2824)

Performs a bunch of verifications against a proposed (staged) release candidate using the new `tools/verify-release/verify-release.sh` script against Maven artifacts, main distributions and Helm chart.

Checks:
* GPG signature and checksum verifications
* All expected artifacts are present
* Build artifacts are reproducible (minus known exceptions)
  * jar files
  * Main distribution zip/tarball
  * Helm chart
* Build passes.
* DISCLAIMER/LICENSE/NOTICE files are present in artifacts that require those

More information in the added web site page.

Fixes apache#2822

---------

Co-authored-by: Pierre Laporte <pierre@pingtimeout.fr>

* Core: Add GCP service account impersonation for credentials. (apache#3246)

* fix(deps): update dependency ch.qos.logback:logback-classic to v1.5.22 (apache#3253)

* fix(deps): update dependency com.google.cloud:google-cloud-iamcredentials to v2.80.0 (apache#3254)

* feat: pass principal name as part of aws subscoped credentials session (apache#3224)

* feat: pass principal name as part of aws subscoped credentials session name

* feat: resolve principal from CurrentIdentityAssociation

* fix: handle principal injection for async tasks

* add feature flag for principal name include

* add changelog, address comments

* handle null identity, refactor tests

* Added user token to the PolarisPrincipal (apache#3236)

* Added user token to the PolarisPrincipal

* added redacted

* Fix compilation failures in GcpCredentialsStorageIntegrationTest (apache#3257)

* chore(deps): update github artifact actions (apache#3260)

* chore(deps): update medyagh/setup-minikube action to v0.0.21 (apache#3264)

* NoSQL: Metastore implementation (apache#3237)

* Fix typo in nosql (apache#3263)

* Corrected a typo in a key configuration parameter in the 1.2.0 release notes (apache#3262)

* fix(deps): update dependency software.amazon.awssdk:bom to v2.40.8 (apache#3271)

* Add NOTES.txt to Helm chart with installation instructions (apache#3173)

* Add NOTES.txt to Helm chart with installation instructions

Provides port-forward commands, health check endpoint, and log viewing for users after installation.

* Fix helm unittest for GH action (apache#3279)

* [doc]: Doc fix for CLI usage (apache#3215)

* [doc]: Add doc for helm prod deployment (apache#3265)

* chore(deps): update docker.io/prom/prometheus docker tag to v3.8.1 (apache#3282)

* chore(deps): update dependency jupyterlab to v4.5.1 (apache#3275)

* fix(deps): update dependency com.google.cloud:google-cloud-storage-bom to v2.61.0 (apache#3274)

* chore(deps): update dependency mypy to >=1.19, <=1.19.1 (apache#3272)

* Bump to 1.4.0-incubating-SNAPSHOT (apache#3181)

* Bump to 1.4.0-incubating-SNAPSHOT

* Update Python client version

* Add exclude check note in the release guide (apache#3182)

* Add exclude check note in the release guide

* Update site/content/release-guide.md

Co-authored-by: Robert Stupp <snazy@snazy.de>

---------

Co-authored-by: Robert Stupp <snazy@snazy.de>

* docs(tools): Create the Tools Section in the Docs (apache#3189)

* fix(deps): update dependency org.apache.logging.log4j:log4j-core to v2.25.3 (apache#3283)

* Add Polaris Community Meeting 20251211 (apache#3284)

* chore(deps): update dependency pre-commit to v4.5.1 (apache#3286)

* fix(deps): update dependency com.google.cloud:google-cloud-iamcredentials to v2.81.0 (apache#3287)

* ensure AddressResolver supports localhost even if ipv6 is disabled in sysctl but not /etc/hosts (apache#3285)

* Migrate to Jackson mapper builder pattern (apache#3269)

Mappers and factories are fully immutable objects in Jackson 3. This change is rather a no-op, but migrates the code to use the builder-pattern.

This is only a little building-block for "real" Jackson 3 support, there's more to do and more that's required from other frameworks.

* fix(deps): update quarkus platform and group to v3.30.4 (apache#3291)

* Rework release guide to include workflows (apache#3273)

* Add a release guides section
* Rename current release guide to manual (deprecated)
* Add new semi-automated release guide
* Move release verification guide under release guides section
* Add scss style for better screenshot separation
* Add redirection from old pages to new ones

Co-authored-by: Robert Stupp <snazy@snazy.de>

* Site: Fix typos in release guide (apache#3296)

* [chore]: Match openapi-generator-cli version in build system to dependency (apache#3266)

* Fix openapi-generator-cli version in build system

* Fix openapi-generator-cli version in build system

* chore(deps): update registry.access.redhat.com/ubi9/openjdk-21-runtime docker tag to v1.24-1 (apache#3297)

* chore(deps): update dependency openapi-generator-cli to v7.17.0 (apache#3298)

* chore(deps): update docker.io/mongo docker tag to v8.2.3 (apache#3299)

* chore(deps): update mongo docker tag to v8.2.3 (apache#3300)

* fix(deps): update dependency io.smallrye.config:smallrye-config-core to v3.15.0 (apache#3302)

* fix(deps): update dependency org.apache.httpcomponents.client5:httpclient5 to v5.6 (apache#3301)

* chore(deps): update plugin com.gradle.develocity to v4.3 (apache#3248)

* Unify mongo image ref (apache#3303)

To prevent duplicate version-bump PRs like apache#3299 and apache#3300

* fix(deps): update dependency org.testcontainers:testcontainers-bom to v2.0.3 (apache#3277)

* Disable sectionPagesMenu (apache#3312)

* Remove docker-java.properties (apache#3307)

* Ensure release can only run from specific SHA (apache#3295)

* Ensure release publish workflow can only run from last RC (apache#3290)
* Enable use of second release workflow for RC>0
* Patch 3rd workflow to support commits with multiple RC tags
* Force 4th workflow to only run from a release branch
* Update release guide to match new workflows

* fix(deps): update dependency ch.qos.logback:logback-classic to v1.5.23 (apache#3308)

* fix(deps): update dependency software.amazon.awssdk:bom to v2.40.13 (apache#3309)

* chore(deps): update registry.access.redhat.com/ubi9/openjdk-21-runtime docker tag to v1.24-2 (apache#3313)

* NoSQL: reduce heap pressure when running tests

Some tests generate a lot of realms, likely one realm per test case. While the amount of data per realm is not much, it is nontheless nice to remove that data immediately (for tests).

The maintenance service, which purges data of eligible realms, cannot be run against the in-memory backend (different JVM).

This change adds a rather "test only" workaround to purge the realm data in the in-memory backend immediately.

* NoSQL: Metastore maintenance

Implementation of the NoSQL meta-store maintenance implementation. It adds the meta-store specific handling to the existing NoSQL maintenance service to purge unreferenced and unneeded data from the database.

* NoSQL: Add to runtime-service

* NoSQL: Add metastore-maintenance to admin tool

* NoSQL: revert LICENSE file change

* Last merged commit 62d774f

---------

Co-authored-by: Yong Zheng <yongzheng0809@gmail.com>
Co-authored-by: Mend Renovate <bot@renovateapp.com>
Co-authored-by: fivetran-rahulprakash <127075959+fivetran-rahulprakash@users.noreply.github.com>
Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>
Co-authored-by: Honah (Jonas) J. <honahx@apache.org>
Co-authored-by: Šarūnas Švėgžda <39830362+sharas2050@users.noreply.github.com>
Co-authored-by: sarunas.svegzda <ssv@infrahub.io>
Co-authored-by: Adnan Hemani <adnan.h@berkeley.edu>
Co-authored-by: Pierre Laporte <pierre@pingtimeout.fr>
Co-authored-by: Talat UYARER <talat@uyarer.com>
Co-authored-by: Tornike Gurgenidze <tokoko96@gmail.com>
Co-authored-by: cccs-cat001 <56204545+cccs-cat001@users.noreply.github.com>
Co-authored-by: Alexandre Dutra <adutra@apache.org>
Co-authored-by: zgxme <u143@qq.com>
Co-authored-by: Tamas Mate <50709850+tmater@users.noreply.github.com>
Co-authored-by: JB Onofré <jbonofre@apache.org>
Co-authored-by: Adam Christian <105929021+adam-christian-software@users.noreply.github.com>
Co-authored-by: Romain Manni-Bucau <rmannibucau@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants