Require explicit user-consent to enable HadoopFileIO #1532

snazy · 2025-05-06T12:14:32Z

Using HadoopFileIO in Polaris can enable "hidden features" that users are likely not aware of. This change requires users to manually update the configuration to be able to use HadoopFileIO in way that highlights the consequences of enabling it.

This PR updates Polaris in multiple ways:

The default of SUPPORTED_CATALOG_STORAGE_TYPES is changed to not include the FILE storage type.
Respect the ALLOW_SPECIFYING_FILE_IO_IMPL configuration on namespaces, tables and views to prevent setting an io-impl value for anything but one of the configured, supported storage-types.
Unify validation code in a new class IcebergPropertiesValidation.
Using FILE or HadoopFileIO now also requires the explicit configuration ALLOW_INSECURE_STORAGE_TYPES_ACCEPTING_SECURITY_RISKS=true.
Added production readiness checks that trigger when ALLOW_INSECURE_STORAGE_TYPES_ACCEPTING_SECURITY_RISKS is true or SUPPORTED_CATALOG_STORAGE_TYPES contains FILE (defaults and per-realm).
The two new readiness checks are considered severe. Severe readiness-errors prevent the server from starting up - unless the user explicitly configured polaris.readiness.ignore-security-issues=true.

Log messages and configuration options explicitly use "clear" phrases highlighting the consequences.

With these changes it is intentionally extremely difficult to start Polaris with HadoopFileIO. People who work around all these safety nets must have realized that what they are doing.

A lot of the test code relies on FILE/HadoopFileIO, those tests got all the configurations to let those tests continue to work as they are, bypassing the added security safeguards.

dimas-b · 2025-05-06T12:59:28Z

...kus/service/src/test/java/org/apache/polaris/service/quarkus/catalog/IcebergCatalogTest.java

WDYT about putting this into a global quarkus test config profile?

Yea - but I guess all Quarkus profiles should either be special per-test or unified/composable or moved to a central place.

composition works by default... one can have application.properties and profile classes at the same time and their properties are merged, right?

Yea - but let's defer to a separate PR to clean all those up.

eric-maynard · 2025-05-06T17:13:46Z

This is not a bug.

eric-maynard

There's a lot going on with this PR and I think it might be good to separate out some of these changes. For example, just removing FILE as a default supported storage type would be valuable.

polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java

eric-maynard · 2025-05-06T17:26:53Z

polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java

I don't see how this isn't completely redundant with SUPPORTED_CATALOG_STORAGE_TYPES

It's there to make the fact mentioned in the description explicit.

If you put FILE in SUPPORTED_CATALOG_STORAGE_TYPES that seems quite explicit no?

@eric-maynard You are correct in that this flag is redundant. However, I believe the intent here is to make the consequences 100% explicit.

I can see how some newcomers may overlook the SUPPORTED_CATALOG_STORAGE_TYPES=[..., "FILE"] during a configuration review and not realize the consequences it could have. Whereas an environment variable that contains "insecure" and "security risks" will definitely raise eyebrows.

Think about deleting a Github repository where you have to go to the Danger Zone, click "Delete, confirm, then confirm a second time, then type the repository name and confirm a third time". Here, given the consequences, I think it is worth having a double-confirmation of the FILE storage type.

I have a hard time believing we can't incrementalize this change the evaluate the necessity of the individual components of it. We're going from this being in the default config for multiple releases to requiring multiple flags to enable it.

AFAICT, because HadoopFileIO is considered insecure, we want to make it hard for users to do the wrong thing. Hence the addition of multiple flags. With the current way, it is trivial to accidentally overlook the activation of HadoopFileIO.

There could be other proposals for this implementation. But then they will be evaluated against this one goal: make it hard for users to do the wrong thing.

I think you are (and maybe this PR is) conflating two things -- FILE storage type and HadoopFileIO. They are not always the same.

SUPPORTED_CATALOG_STORAGE_TYPES is probably sufficient to disallow the FILE storage type. FILE shouldn't be in the default, but I think it's too great a leap to go from having it in the default to requiring multiple flags to enable it.

For disallowing HadoopFileIO, we need a different approach than what's done in this PR.

Concrete suggestions for improvements are always welcome.

However, this PR addresses some issue that should really be mitigated quickly.

eric-maynard · 2025-05-06T17:28:36Z

...rvice/src/main/java/org/apache/polaris/service/quarkus/config/ProductionReadinessChecks.java

These changes seem pretty unnecessary

I do think that there are nuances in what's a warning and what's severe.

That may be so, but I don't think this change is necessary to Require explict user-consent to enable HadoopFileIO

@eric-maynard I am not sure what the ask is here. Would you rather have the UTF-8 icon and/or the log message implemented in a separate PR? Or do you mean that the PR title should be a bit more general?

Sure, I think a separate PR for these changes could be a good idea. If this PR really is a bugfix, let's not couple unnecessary changes with it.

Let's be mindful on the overhead that too fine-grained PR would cause though...

The changes in this PR are cohesive to one another. And it is quite small. So it makes sense to submit the changes in a single PR. Splitting a PR into many tiny PRs would prevent reviewers from understanding the big picture and would not have any functional merit.

As I think the changes are "related", I think it's fair to do the changes in one PR (even if it's just "display/rendering" messages).

It's definitely fair, it's only a question of necessity. If there really is an urgent 1.0-blocking bugfix to be made, must we include these changes here? Are these logging changes really 1.0-blocking?

eric-maynard · 2025-05-06T17:31:11Z

...src/main/java/org/apache/polaris/service/catalog/validation/IcebergPropertiesValidation.java

Storage types and File IO implementations are not necessarily the same thing; you could have any number of FileIO implementations that can talk to S3

Yes? That's what fromFileIoImplementation() figures out, no?

For the hard-coded FileIO implementations (assuming the implementation you have in mind is actually the one on the classpath), yes. For other implementations, no.

@eric-maynard could you provide an example of what you mean?

There's one in the repo here. It uses Azure, but it's not represented in StorageTypeFileIO. How is that meant to behave?

eric-maynard · 2025-05-06T17:31:29Z

...src/main/java/org/apache/polaris/service/catalog/validation/IcebergPropertiesValidation.java

What makes a custom File IO insecure?

What makes it secure?

Nothing. That's why there's nothing in our current documentation or code that promises that it is secure.

snazy · 2025-05-07T15:33:42Z

This is not a bug.

It is. And it is very severe.

eric-maynard · 2025-05-07T17:21:31Z

How exactly is this a bug?

flyrain · 2025-05-07T18:31:50Z

  S3("org.apache.iceberg.aws.s3.S3FileIO", true),
  GCS("org.apache.iceberg.gcp.gcs.GCSFileIO", true),
  AZURE("org.apache.iceberg.azure.adlsv2.ADLSFileIO", true),
  FILE("org.apache.iceberg.hadoop.HadoopFileIO", false),

Thanks for bring this up. It's a good point to clarify whether the HadoopFileIO is suitable for production. However, there is no guarantee any of above strings ensure a secure FileIO. They can be easily injected by any customized build or even completely different implementations with the same name. I believe it is ultimately Polaris admin's responsibility to make sure their Polaris deployment as secure as possible. We(Polaris developer/community) can provide suggestions to them for sure. With that, I'm +1 on well documented behavior on HadoopFileIO, but I don't think these checking and new configurations are necessary. SUPPORTED_CATALOG_STORAGE_TYPES should be good enough to let Polaris admin make the decision.

snazy · 2025-05-07T19:03:42Z

whether the HadoopFileIO is suitable for production

@flyrain you should really know why it's not

Using `HadoopFileIO` in Polaris can enable "hidden features" that users are likely not aware of. This change requires users to manually update the configuration to be able to use `HadoopFileIO` in way that highlights the consequences of enabling it. This PR updates Polaris in multiple ways: * The default of `SUPPORTED_CATALOG_STORAGE_TYPES` is changed to not include the `FILE` storage type. * Respect the `ALLOW_SPECIFYING_FILE_IO_IMPL` configuration on namespaces, tables and views to prevent setting an `io-impl` value for anything but one of the configured, supported storage-types. * Unify validation code in a new class `IcebergPropertiesValidation`. * Using `FILE` or `HadoopFileIO` now _also_ requires the explicit configuration `ALLOW_INSECURE_STORAGE_TYPES_ACCEPTING_SECURITY_RISKS=true`. * Added production readiness checks that trigger when `ALLOW_INSECURE_STORAGE_TYPES_ACCEPTING_SECURITY_RISKS` is `true` or `SUPPORTED_CATALOG_STORAGE_TYPES` contains `FILE` (defaults and per-realm). * The two new readiness checks are considered _severe_. Severe readiness-errors prevent the server from starting up - unless the user explicitly configured `polaris.readiness.ignore-security-issues=true`. Log messages and configuration options explicitly use "clear" phrases highlighting the consequences. With these changes it is intentionally extremely difficult to start Polaris with HadoopFileIO. People who work around all these safety nets must have realized that what they are doing. A lot of the test code relies on `FILE`/`HadoopFileIO`, those tests got all the configurations to let those tests continue to work as they are, bypassing the added security safeguards.

flyrain · 2025-05-20T18:01:45Z

service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java

+  static final FeatureConfiguration<Boolean> INITIALIZE_DEFAULT_CATALOG_FILEIO_FOR_TEST =
+      PolarisConfiguration.<Boolean>builder()
+          .key("INITIALIZE_DEFAULT_CATALOG_FILEIO_FOR_TEST")
+          .defaultValue(false)
+          .description("")
+          .buildFeatureConfiguration();


I thought this was reverted. Also #1624 is going to remove it completely. These changes seems unnecessary.

This change should have reverted. Why do you still do this, given that you are aware of PR #1624, and reviewed it?

Not sure what you're asking here?
Inconvenience due to some merge conflict on a PR that's not yet ready to be merged?

This part of the change is completely unnecessary in this PR and makes more conflicts for follow-ups like #1624.

Having no conflicts ever is a tough goal in an active project.
I can't even count the number of rebase/merge conflicts I have to handle every day - did I complain?

I take that as a personal offense. /cc @jbonofre

Lets stop this thread here. Please no further responses. If necessary please escalate to the PPMC list

Deleted my last comment. The part of reason to remove my -1 on this PR is based on the reverting of INITIALIZE_DEFAULT_CATALOG_FILEIO_FOR_TEST related change in this PR, which @snazy agreed on in ML. Obviously, it was still here. Anyway, we can fix forward. I truly hope the community works as a team with good will and help each other.

Deleting a comment is not the right way - it completely changes the history of these messages!

You asked to move it back out of FeaturesConfiguration into the class where it was before - and that is EXACTLY what happened. So I seriously ask you to not bring up wrong accusations & personal offenses!

I agree. Let's please stop this thread. We are here to collaborate on the project and move it forward.

(see next comment)

snazy · 2025-05-21T07:38:24Z

Merging this one based on approvals from @dimas-b and @jbonofre and the +0 from @flyrain.
The underlying issue is, as some know, too serious to further delay this change.
Improvements can always be made in follow-ups.

* main: Update docker.io/prom/prometheus Docker tag to v3.4.0 (apache#1602) * Site: Update production configuration page (apache#1606) * main: Update dependency com.google.cloud:google-cloud-storage-bom to v2.52.3 (apache#1623) * main: Update dependency boto3 to v1.38.19 (apache#1622) * Remove Bouncy Castle dependency usage from PemUtils (apache#1318) - Added PEM format parsing in PemUtils - Added unit test for PemUtils for empty file and multiple PEM objects - Removed Bouncy Castle Provider dependency from service common module - Removed Bouncy Castle Provider dependency from quarkus service module * Site: Add a page for policy management (apache#1600) * [Policy Store | Management Spec] Add policy privileges to spec and update admin service impl (apache#1529) This PR adds new policy related privileges to polaris-management-api.yml and update PolarisAdminService to allow granting new privileges * Spec: Add SigV4 Auth Support for Catalog Federation (apache#1506) * Spec changes for SigV4 Auth Support for Catalog Federation * Extract service identity info as a nested object * nit: fix admin tool log level and comments (apache#1626) The previous WARNING log levels seems to work, but WARN aligns better with standard Quarkus log levels. Fixes apache#1612 * Doc: switch to use iceberg-aws-bundle jar (apache#1609) * main: Update dependency org.mockito:mockito-core to v5.18.0 (apache#1630) * main: Update dependency boto3 to v1.38.20 (apache#1631) * Require explicit user-consent to enable HadoopFileIO (apache#1532) Using `HadoopFileIO` in Polaris can enable "hidden features" that users are likely not aware of. This change requires users to manually update the configuration to be able to use `HadoopFileIO` in way that highlights the consequences of enabling it. This PR updates Polaris in multiple ways: * The default of `SUPPORTED_CATALOG_STORAGE_TYPES` is changed to not include the `FILE` storage type. * Respect the `ALLOW_SPECIFYING_FILE_IO_IMPL` configuration on namespaces, tables and views to prevent setting an `io-impl` value for anything but one of the configured, supported storage-types. * Unify validation code in a new class `IcebergPropertiesValidation`. * Using `FILE` or `HadoopFileIO` now _also_ requires the explicit configuration `ALLOW_INSECURE_STORAGE_TYPES_ACCEPTING_SECURITY_RISKS=true`. * Added production readiness checks that trigger when `ALLOW_INSECURE_STORAGE_TYPES_ACCEPTING_SECURITY_RISKS` is `true` or `SUPPORTED_CATALOG_STORAGE_TYPES` contains `FILE` (defaults and per-realm). * The two new readiness checks are considered _severe_. Severe readiness-errors prevent the server from starting up - unless the user explicitly configured `polaris.readiness.ignore-security-issues=true`. Log messages and configuration options explicitly use "clear" phrases highlighting the consequences. With these changes it is intentionally extremely difficult to start Polaris with HadoopFileIO. People who work around all these safety nets must have realized that what they are doing. A lot of the test code relies on `FILE`/`HadoopFileIO`, those tests got all the configurations to let those tests continue to work as they are, bypassing the added security safeguards. --------- Co-authored-by: Dmitri Bourlatchkov <[email protected]> --------- Co-authored-by: Mend Renovate <[email protected]> Co-authored-by: Yufei Gu <[email protected]> Co-authored-by: David Handermann <[email protected]> Co-authored-by: Honah (Jonas) J. <[email protected]> Co-authored-by: Rulin Xing <[email protected]> Co-authored-by: Dmitri Bourlatchkov <[email protected]> Co-authored-by: MonkeyCanCode <[email protected]>

* fix: Remove duplicated code in IcebergCatalog (apache#1681) * Fix SparkClient listGenericTable to use ListGenericTablesRESTResponse ListTableResponse is the class used by iceberg endpoint, which is the same as ListGenericTablesRESTResponse. However, we are suppose to use ListGenericTablesRESTResponse to be correct * fix: Remove info log about deprecated internal method from PolarisConfiguration (apache#1672) Remove the INFO log about calls to this method in Polaris, because users cannot do anything about these messages. Phasing out old property names requires coordination with users (e.g. release notes), so it is not a matter of merely avoiding calls to that method in Polaris code. Fixes apache#1666 * Create a single binary distribution bundle (apache#1589) * fix(quickstart): Correct Quickstart Instructions (apache#1673) * Remove Java URI validations for Blob Storage providers (apache#1604) This is a retry at apache#1586, which I'll close if this PR is on the right direction instead. Java URI does not actually apply any normalization to URIs if we do not call URI.normalize() (which we currently do not). Additionally, blob storage providers like S3 and GCS can provide ".." and "." as valid fragments in URLs - which Java URI would attempt to normalize incorrectly. As a result, attempting to validate and/or normalize URIs for blob storage providers using the Java URI class is the incorrect behavior. While we may want to add location validation via regex later, removing it first should at least unblock the bug we see in apache#1545. * Improve test coverage for invalid inputs in Policy APIs (apache#1665) * Fix getting-started docker start by PR apache#1532 (apache#1687) * Fix the manual test broken by PR apache#1532 (apache#1688) * Fix credentials printing twice (apache#1682) * main: Update dependency com.diffplug.spotless:spotless-plugin-gradle to v7.0.4 (apache#1690) * INFO: last merged commit 493de03 --------- Co-authored-by: Alexandre Dutra <[email protected]> Co-authored-by: gh-yzou <[email protected]> Co-authored-by: Dmitri Bourlatchkov <[email protected]> Co-authored-by: Yufei Gu <[email protected]> Co-authored-by: Adnan Hemani <[email protected]> Co-authored-by: William Hyun <[email protected]> Co-authored-by: Christopher Lambert <[email protected]> Co-authored-by: Mend Renovate <[email protected]>

@eric-maynard

* Use config-file to define errorprone rule (apache#1233) Also enabled a couple more simple rules, and adding suppressions/fixes for/to the code. The two rules `EqualsGetClass` and `UnusedMethod`, which I think are useful, are not enabled yet, because that would mean actual code changes, which I do not want to do in this PR. The rule `PatternMatchingInstanceof`, introduced in apache#393, is disabled in this PR. It does not work before errorrpone 2.37.0 (via apache#1213) - requires additional changes to enable the rule (see apache#1215). * Add Yun as a contributor (apache#1310) * Refactor CatalogHandler to comply with ErrorProne rules (apache#1312) Fix the CI error after apache#1233 * Implement PolicyCatalog Stage 1: CRUD + ListPolicies (apache#1294) * main: Update dependency io.opentelemetry:opentelemetry-bom to v1.49.0 (apache#1316) * main: Update docker.io/jaegertracing/all-in-one Docker tag to v1.68.0 (apache#1317) * main: Update dependency boto3 to v1.37.28 (apache#1328) * main: Update dependency software.amazon.awssdk:bom to v2.31.16 (apache#1329) * Make `BasePolaritsMetaStoreManagerTest` and `(Base)ResolverTest` reusable (apache#1308) Moves the test cases into the `Base*` classes and make sure the classes can be reused by other persistence implementations. * main: Update dependency io.opentelemetry.semconv:opentelemetry-semconv to v1.32.0 (apache#1293) * main: Update mockito monorepo to v5.17.0 (apache#1311) * PySpark Update AWS Region (apache#1302) Co-authored-by: Travis Bowen <[email protected]> * main: Update dependency com.nimbusds:nimbus-jose-jwt to v10.2 (apache#1334) * main: Update dependency com.diffplug.spotless:spotless-plugin-gradle to v7.0.3 (apache#1335) * Maven publication: Produce correct `<scm><tag>` in `pom.xml` (apache#1330) `project.scm.tag` in a Maven pom is intended to refer to the SCM (Git) tag. We currently publish `main`, which is incorrect. This change omits the SCM tag for snapshot builds, but emits the Git tag for releases. * Remove `@StaticInitSafe` annotation (apache#1331) There was an issue around mapped configurations having the `@StaticInitSafe` annotation that led to _two_ instances (a "static" one and a "somewhet application-scoped" one) - this was fixed in Quarkus 3.21. One bug in smallrye-config is fixed for Quarkus > 3.21.0, another issue however remains. Since `@StaticInitSafe` annotated configs seem to cause some weird issues, it seems legit to remote that annotation altogether. This approach was [taken in Nessie](projectnessie/nessie#10606) as well. Investigations (via practical experiments) have proven that there's no measurable impact (runtime + heap) when doing this - and that's also been confirmed by Quarkus + Smallrye-config maintainers. Hence this change remotes that annotation from the code base. * Build/Release: Add a "generate digest" task and use for source tarball and Quarkus distributables (apache#1271) * Ensure that digest and signature are generated for both Polaris-Server and admin tar/zip distribution * Move "generate digest" functionality to a Gradle task * main: Update dependency com.google.errorprone:error_prone_core to v2.37.0 (apache#1213) * main: Update Quarkus Platform and Group to v3.21.1 (apache#1291) * main: Update dependency io.netty:netty-codec-http2 to v4.2.0.Final (apache#1301) * Remove unnecessary `clean` and `--no-build-cache` from Gradle invocations (apache#1338) `quarkusAppPartsBuild --rerun` is the right way to force a Docker image build. * Generalize bootstrapping in servers (apache#1313) * Remove `instanceof` checks from `QuarkusProducers`. * Remove the now unused `onStartup` method from `InMemoryPolarisMetaStoreManagerFactory`. * Instead, call the good old `bootstrapRealms` method from `QuarkusProducers`. * Add new config property to control which MetaStore types are bootstrapped automatically (defaults to `in-memory` as before). * There is no bootstrap behaviour change in this PR, only refactorings to simplify code. * Add info log message to indicate when a realm is bootstrapped in runtime using preset credentials. Future enhancements may include pulling preset credentials from a secret manager like Vault for bootstrapping (s discussed in comments on apache#1228). * main: Update actions/stale digest to 816d9db (apache#1341) * main: Update dependency com.adobe.testing:s3mock-testcontainers to v4 (apache#1342) * main: Update dependency org.eclipse.persistence:eclipselink to v4.0.6 (apache#1343) * main: Update dependency io.quarkus to v3.21.2 (apache#1344) * main: Update dependency com.google.guava:guava to v33.4.7-jre (apache#1340) Co-authored-by: Robert Stupp <[email protected]> * Spark: Add Namespaces and View support for SparkCatalog (apache#1332) * Demote technical log messages to DEBUG in PolarisCallContextCatalogFactory (apache#1346) These messages appear to be logging low-level technical details about what is going on in the factory and are not likely to be of interest to most users on a daily basis. * Core/Service: Implement PolicyCatalog Stage 2: detach/attach/getApplicablePolicies (apache#1314) * Spec: Add 'inherited' and 'namespace' Fields to GetApplicablePolicies API Response (apache#1277) * Properly track bootstrappedRealms in InMemoryPolarisMetaStoreManagerFactory (apache#1352) Fixes apache#1351 * Implement GenericTableCatalogAdapter; admin-related fixes (apache#1298) * initial commit: * debugging * some polish * autolint * spec change * bugfix * bugfix * various fixes * another missing admin location * autolint * false by default * fixes per review * autolint * more fixes * DRY * revert small change for a better error * integration test * extra test * autolint * stable * wip * rework subtypes a bit * stable again * autolint * apply new lint rule * errorprone again * adjustments per review * update golden files * add another test * clean up logic in PolarisAdminService * autolint * more fixes per review * format * Update versions in distribution LICENSE and NOTICE (apache#1350) * Spark: Add CreateTable and LoadTable implementation for SparkCatalog (apache#1303) * Add a weigher to the EntityCache based on approximate entity size (apache#490) * initial commit * autolint * resolve conflicts * autolint * pull main * Add multiplier * account for name, too * adjust multiplier * add config * autolint * remove old cast * more tests, fixes per review * add precise weight test * autolint * populate credentials field for loadTableResponse (apache#1225) * populate credentials field for loadTableResponse * spotless * spotless * remove unused hashset * fix merge * fix empty credential case * spotlessApply --------- Co-authored-by: David Lu <[email protected]> * main: Update dependency io.smallrye.common:smallrye-common-annotation to v2.12.0 (apache#1355) * Build: Avoid adding duplicated projects for Intelij IDE usage (apache#1333) * main: Update dependency org.junit:junit-bom to v5.12.2 (apache#1354) * main: Update dependency org.apache.commons:commons-text to v1.13.1 (apache#1358) * main: Update dependency boto3 to v1.37.33 (apache#1360) * main: Update dependency software.amazon.awssdk:bom to v2.31.21 (apache#1361) * main: Update dependency io.micrometer:micrometer-bom to v1.14.6 (apache#1362) * main: Update dependency com.google.guava:guava to v33.4.8-jre (apache#1366) * Update LICENSE/NOTICE with latest versions (apache#1364) * Use "clean" LICENSE and NOTICE in published jar artifacts (apache#1292) * main: Update dependency io.projectreactor.netty:reactor-netty-http to v1.2.5 (apache#1372) * Add `Varint` type for variable-length integer encoding (apache#1229) * main: Update docker.io/prom/prometheus Docker tag to v3.3.0 (apache#1375) * Set version to 0.10.0-beta in prepaaration for the next release (apache#1370) * Update the link to OpenAPI in the documentation (apache#1379) * Integration test for Spark Client (apache#1349) * add integration test * add change * add comments * rebase main * update class comments * add base integration * clean up comments * main: Update dependency net.ltgt.gradle:gradle-errorprone-plugin to v4.2.0 (apache#1392) * Add generic table documentations (apache#1374) * add generic table documentation (incomplete) * fix table and spacing * remove documentation in client api since there is no implementation yet * remove spacing * minor fix - proof read * review fix, wording * add generic table documentation (incomplete) * fix table and spacing * remove documentation in client api since there is no implementation yet * remove spacing * minor fix - proof read * review fix, wording * proof read - punctuation fix * change table privilege reference * Unblock test `listNamespacesWithEmptyNamespace` (apache#1289) * Unblock test `listNamespacesWithEmptyNamespace` * Use `containsExactly` to simplify the test * Fix empty namespace behavior * Address comments * Block dropping empty namespace * Improve error messages * Revamp the Quick Start page (apache#1367) * First Draft with AWS * try again * try again * try again * try again * try again * try now * should work * AWS First Draft Complete * ensure file changed * Azure First Draft Complete * Azure First Draft, pt. 2 * Azure Completed * GCP First Draft * GCP Verified * File structure fixed * Remove Trino-specific tutorial * Restructured Quick Start * Addresses minor comments from @eric-maynard * Added reference to Deploying Polaris in Production * Fix MD Link Checker --------- Co-authored-by: Adnan Hemani <[email protected]> * Update README with links to new Quickstart experience (apache#1393) * Update the StorageConfiguration to invoke singleton client objects, a… (apache#1386) * Update the StorageConfiguration to invoke singleton client objects, and add a test * Fix formatting * using guava suppliers * Add aws region * Cleanup and mock test * Spark: Add rest table operations (drop, list, purge and rename etc) for Spark Client (apache#1368) * Initial MVP implementation of Catalog Federation to remote Iceberg REST Catalogs (apache#1305) * Initial prototype of catalog federation just passing special properties into internal properties. Make Resolver federation-aware to properly handle "best-effort" resolution of passthrough facade entities. Targets will automatically reflect the longest-path that we happen to have stored locally and resolve grants against that path (including the degenerate case where the longest-path is just the catalog itself). This provides Catalog-level RBAC for passthrough federation. Sketch out persistence-layer flow for how connection secrets might be pushed down into a secrets-management layer. * Defined internal representation classes for connection config * Construct and initialize federated iceberg catalog based on connection config * Apply the same spec renames to the internal ConnectionConfiguration representations. * Manually pick @XJDKC fixes for integration tests and omittign secrets in response objects * Fix internal connection structs with updated naming from spec PR * Push CreateCatalogRequest down to PolarisAdminService::createCatalog just like UpdateCatalogRequest in updateCatalog. This is needed if we're going to make PolarisAdminService handle secrets management without ever putting the secrets into a CatalogEntity. * Add new interface UserSecretsManager along with a default implementation The default UnsafeInMemorySecretsManager just uses an inmemory ConcurrentHashMap to store secrets, but structurally illustrates the full flow of intended implementations. For mutual protection against a compromise of a secret store or the core persistence store, the default implementation demonstrates storing only an encrypted secret in the secret store, and a one-time-pad key in the returned referencePayload; other implementations using standard crypto protocols may choose to instead only utilize the remote secret store as the encryption keystore while storing the ciphertext in the referencePayload (e.g. using a KMS engine with Vault vs using a KV engine). Additionally, it demonstrates the use of an integrity check by storing a basic hashCode in the referencePayload as well. * Wire in UserSecretsManager to createCatalog and federated Iceberg API handlers Update the internal DPOs corresponding to the various ConnectionConfigInfo API objects to no longer contain any possible fields for inline secrets, instead holding the JSON-serializable UserSecretReference corresponding to external/offloaded secrets. CreateCatalog for federated catalogs containing secrets will now first extract UserSecretReferences from the CreateCatalogRequest, and the CatalogEntity will populate the DPOs corresponding to ConnectionConfigInfos in a secondary pass by pulling out the relevant extracted UserSecretReferences. For federated catalog requests, when reconstituting the actual sensitive secret configs, the UserSecretsManager will be used to obtain the secrets by using the stored UserSecretReferences. Remove vestigial internal properties from earlier prototypes. * Since we already use commons-codec DigestUtils.sha256Hex, use that for the hash in UnsafeInMemorySecretsManager just for consistency and to illustrate a typical scenario using a cryptographic hash. * Rename the persistence-objects corresponding to API model objects with a new naming convention that just takes the API model object name and appends "Dpo" as a suffix; * Use UserSecretsManagerFactory to Produce the UserSecretsManager (#1) * Move PolarisAuthenticationParameters to a top-level property according to the latest spec * Create a Factory for UserSecretsManager * Fix a typo in UnsafeInMemorySecretsManagerFactory * Gate all federation logic behind a new FeatureConfiguration - ENABLE_CATALOG_FEDERATION * Also rename some variables and method names to be consistent with prior rename to ConnectionConfigInfoDpo * Change ConnectionType and AuthenticationType to be stored as int codes in persistence objects. Address PR feedback for various nits and javadoc comments. * Add javadoc comment to IcebergCatalogPropertiesProvider * Add some constraints on the expected format of the URN in UserSecretReference and placeholders for next steps where we'd provide a ResolvingUserSecretsManager for example if the runtime ever needs to delegate to two different implementations of UserSecretsManager for different entities. Reduce the `forEntity` argument to just PolarisEntityCore to make it more clear that the implementation is supposed to extract the necessary identifier info from forEntity for backend cleanup and tracking purposes. --------- Co-authored-by: Rulin Xing <[email protected]> Co-authored-by: Rulin Xing <[email protected]> * Add Adnan and Neelesh to collaborators list (apache#1396) * Replace authentication filters with Quarkus Security (apache#1373) * Implement PolicyCatalogHandler and Add Policy Privileges Stage 1: CRUD + ListPolicies (apache#1357) * Add PolicyCatalogHandler and tests * Fix style * Address review comments * Address review comments 2 * fix nit * Remove CallContext.getAuthenticatedPrincipal() (apache#1400) * main: Update dependency info.picocli:picocli-codegen to v4.7.7 (apache#1408) * main: Update dependency com.google.errorprone:error_prone_core to v2.38.0 (apache#1404) * Add Polaris Community Meeting 2025-04-17 (apache#1409) * main: Update dependency boto3 to v1.37.37 (apache#1412) * EclipseLink: add PrimaryKey to policy mapping records JPA model (apache#1403) * Re-instate dependencies between Docker Compose services (apache#1407) * Do not rotate bootstrapped root credentials (apache#1414) * Add Getting Started Button to the Apache Polaris Webshite Homepage (apache#1406) * Core: change to return ApplicablePolicies (apache#1415) * Rename the Snapshot Retention policy (apache#1284) * Rename the Snapshot Retention policy * Resolve comments * Resolve comments --------- Co-authored-by: Yufei Gu <yufei.apache.org> * main: Update dependency com.adobe.testing:s3mock-testcontainers to v4.1.0 (apache#1419) * rename snapshotRetention to snashotExpiry (apache#1420) * main: Update registry.access.redhat.com/ubi9/openjdk-21-runtime Docker tag to v1.22-1.1744796716 (apache#1394) * main: Update dependency software.amazon.awssdk:bom to v2.31.26 (apache#1413) * main: Update dependency com.adobe.testing:s3mock-testcontainers to v4.1.1 (apache#1425) * Fix releaseEmailTemplate task (apache#1384) * Update distributions LICENSE and NOTICE with AWS SDK 2.31.26 update (apache#1423) * Support snapshots=refs (apache#1405) * initial commit * autolint * small revert * rebase * autolint * simpler * autolint * tests * autolint * stable * fix leak * ready for review * improved test * autolint * logic flip again * Update service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java Co-authored-by: Alexandre Dutra <[email protected]> * Update integration-tests/src/main/java/org/apache/polaris/service/it/env/CatalogApi.java Co-authored-by: Alexandre Dutra <[email protected]> * adjustments for committed suggestions * autolint --------- Co-authored-by: Alexandre Dutra <[email protected]> * Remove activatedPrincipalRoles property from AuthenticatedPolarisPrincipal (apache#1410) This seems to be a leftover from when ActiveRolesProvider was introduced. The setter was still used, but the getter wasn't, which hints at the fact that this property can be safely removed. As a bonus, AuthenticatedPolarisPrincipal now becomes immutable, which is imho a very good thing. * Implement PolicyCatalogHandler and Add Policy Privileges Stage 2: AttachPolicy + DetachPolicy (apache#1416) * add auth test for attach/detach * apply formatter * refactor authorizePolicyAttachmentOperation * address comment * better naming * Ship eclipselink and PostgreSQL JDBC driver by default in Polaris distribution (apache#1411) * Fix Connection Config DPOs (apache#1422) * Fix connection config dpos * Run spotlessApply * Doc: Fix the issue that html tags are not working in Hugo (apache#1382) * Implement PolicyCatalogHandler Stage 3: GetApplicablePolicies (apache#1421) * [JDBC] Part2: Add Relational JDBC module (apache#1287) * Bump version to 0.11.0-beta-incubating-SNAPSHOT (apache#1429) * Make entity lookups by id honor the specified entity type (apache#1401) * Make entity lookups by id honor the specified entity type All implementations of `TransactionalPersistence.lookupEntityInCurrentTxn()` are currently ignoring the `typeCode` parameter completely and could potentially return an entity of the wrong type. This can become very concerning during authentication, since a principal lookup could return some entity that is not a principal, and that would be considered a successful authentication. * review * Remove "test" Authenticator (apache#1399) * Propagate SQLException as "caused by" (apache#1430) * Remove logging for DbOps (apache#1433) * Spark: Add regtests for Spark client to test built jars (apache#1402) * main: Update dependency com.google.cloud:google-cloud-storage-bom to v2.51.0 (apache#1436) * main: Update dependency org.testcontainers:testcontainers-bom to v1.21.0 (apache#1437) * main: Update actions/setup-python digest to a26af69 (apache#1440) * Spark-IT: use correct configurations (apache#1444) ... do not let Spark leak into Quarkus * PolarisRestCatalogIntegrationTest: Always purge generic tables (apache#1443) * Add missing Postgresql dependency (apache#1447) * Add Request Timeouts (apache#1431) * add timeout * add iceberg exception mapping * dont use quarkus bom, disable timeout * nits * Fix sparks sql regtests with up to date config (apache#1454) * Refactor BasePolarisTableOperations & BasePolarisViewOperations (apache#1426) * initial copy paste * Reorder * view copy paste * fixes, polish * stable * yank * CODE_COPIED_TO_POLARIS comments * autolint * update license * typofix * update comments * autolint * Use .sha512 extension instead of -sha512 (apache#1449) * main: Update dependency org.eclipse.microprofile.fault-tolerance:microprofile-fault-tolerance-api to v4.1.2 (apache#1451) * Doc: Update Local Root Principal Credentials in Quickstart (apache#1452) * Update the Getting Started Workflow with each Cloud Provider's Blob Storage (apache#1435) * AWS First Draft * Debug * revert typo * Add JQ to docker runtime * Debug, pt2 * debug * debug * Allow Instance Profile Roles * change random suffix * change instance profile to regular IAM roles * AWS Final Draft * Azure First Draft * debug * Azure First Draft * debug * typo * GCP First Try * GCP Complete * GCP Final * add all jars to Spark * refactor * Implement PolicyCatalogAdapter (apache#1438) * Generic Table/Policy Store: Move feature config check to Adapter and some small refactoring (apache#1465) * update refs (apache#1464) * [JDBC] Part3: Plumb JDBC module to Quarkus (apache#1371) * Allow BasePolarisTableOperations to skip refreshing metadata after a commit (apache#1456) * initial commit * fix another test * changes per comments * visibility * changes per review * autolint * oops * main: Update dependency com.fasterxml.jackson:jackson-bom to v2.19.0 (apache#1455) * Doc: Added set custom credentials instruction in README (apache#1461) * Doc: Add policy documentation (apache#1460) * main: Update dependency software.amazon.awssdk:bom to v2.31.30 (apache#1475) * main: Update dependency gradle to v8.14 (apache#1459) * main: Update dependency gradle to v8.14 * fix PR --------- Co-authored-by: Robert Stupp <[email protected]> * Remove unused class TokenInfoExchangeResponse (apache#1479) This is an oversight from apache#1399. * Upgrade Polaris to Iceberg 1.9.0 (apache#1309) * Doc: Update on access-control policy docs (apache#1472) * main: Update Quarkus Platform and Group (apache#1381) * Added link to the Spark-Jupyter Notebook Getting Started from the main Getting Started Page (apache#1453) * Added link to the Spark-Jupyter Notebook Getting Started from the main Quickstart page * Typo Co-authored-by: Eric Maynard <[email protected]> * Suggestions as per @eric-maynard's review * Fix Typo --------- Co-authored-by: Eric Maynard <[email protected]> * [JDBC] Support Policy (apache#1468) * Refactor EntityCache into an interface (apache#1193) * Refactor EntityCache to an interface * fix * spotless * Remove unused PolarisCredentialVendor.validateAccessToLocations() (apache#1480) * Remove unused PolarisCredentialVendor.validateAccessToLocations() * review: remove ValidateAccessResult and comments * Policy Store: Check whether Policy is in use before dropping and support `detach-all` flag (apache#1467) * fix error (apache#1492) * Ensure writeToPolicyMappingRecord update existing record if primary key equals in EclipseLink Persistence Impl (apache#1469) * update PolicyMappingRecord if not exists * update test * add TODO * Eliminate getCurrentContext() call in PolarisAuthorizerImpl (apache#1494) * Add getting-started for Polaris Spark Client with Delta tables (apache#1488) * Fix: Pull Postgres image automatically (apache#1495) * Fix Outdated Information and add Information regarding `docker compose down` to Quickstart (apache#1497) * Fix Outdated Information and Add Information regarding docker compose down to Quickstart * Revision 2 * Remove shutdown from README * typo * Upgrade Iceberg REST Spec to match Iceberg 1.8 (apache#1283) * prep for review * reset * more changes * fixes * github action change * another build change * try api revert * re-all * custom type mappings, rebuild * autolint * polish * yank custom types * update * autolint * wip * Revert build changes * example * autolint * Fix FileIOExceptionsTest to conform to new Iceberg 1.8 API (apache#1501) It looks like after apache#1283, this test no longer compiles as the Iceberg API has changed. I'm not sure how this wasn't caught by CI on that PR itself. * JDBC: Optimize writeEntity calls (apache#1496) * Remove transaction from atomic writes * remove if-else * main: Update registry.access.redhat.com/ubi9/openjdk-21-runtime Docker tag to v1.22-1.1745840590 (apache#1499) * Support for external identity providers (apache#1397) * JDBC: create objects without reflection (apache#1434) * Include quarkus-container-image and README in the binary distributions (apache#1493) * Site: Fix Management and Catalog Spec links (apache#1507) * Lazy iteration over JDBC ResultSet (apache#1487) * refactor * autolint * polish * autolint * changes per review * autolint * unwrapping caller * changes per review * Update distributions LICENSE and NOTICE with artifacts and versions sync (apache#1509) * Avoid using deprecated `NestedField.of()` (apache#1514) * Fix compile warning: unknown enum constant Id.NAME (apache#1513) * Doc: Add getting started with JDBC source (apache#1470) * Site: Add Polaris Spark client webpage under unreleased (apache#1503) * Add new committers (apache#1518) * Docs: Fix the wrong catalog name in `using polaris` page (apache#1471) * fix Signed-off-by: owenowenisme <[email protected]> * update docker compose Signed-off-by: owenowenisme <[email protected]> --------- Signed-off-by: owenowenisme <[email protected]> * main: Update dependency org.apache.commons:commons-configuration2 to v2.12.0 (apache#1481) * main: Update dependency com.google.cloud:google-cloud-storage-bom to v2.52.1 (apache#1485) * main: Update dependency com.azure:azure-sdk-bom to v1.2.34 (apache#1490) * main: Update docker.io/prom/prometheus Docker tag to v3.3.1 (apache#1510) * Add new committers on website (apache#1521) * main: Update dependency software.amazon.awssdk:bom to v2.31.35 (apache#1524) * fix overlapping menu item on the nav bar (apache#1520) * fix overlapping menu item on the nav bar * prevent dropdowns expanding inside the navbar * Additional refs update for iceberg 1.9.0 (apache#1491) * Additional refs update for iceberg 1.9.0 * Additional refs update for iceberg 1.9.0 * Additional refs update for iceberg 1.9.0 * Fix typo on Pierre's github URL (apache#1527) * Refactor storage access configuration handling (apache#1504) * Refactor storage access configuration handling This is a step towards supporting non-AWS S3 storage, but this refactoring is relevant to all storage backends. There is no change to existing behaviours. * Rename PolarisCredentialProperty to StorageAccessProperty and introduce non-credential properties (as an example for now) * StorageAccessProperty values are ultimately meant to be produced by PolarisStorageIntegration implementations * Some previous entries in StorageAccessProperty are not really credential properties, but their treatment is not changed in this PR to maintain exactly the same bahaviour as before. * Add AccessConfig to represent both credential and non-credential properties related to storage access. * [JDBC] : Deprecate EclipseLink (apache#1515) * Auto-bootstrap: add verbose logging (apache#1376) Log explicit messages around auto-bootstrapping and unnecessary/left-over secrets that are (still) available. * Add nightly build GH action to publish SNAPSHOT on Nexus (apache#1383) * Add nightly build GH action to publish SNAPSHOT on Nexus (apache#1383) * Build: Fix `fetchAsfProjectName` and make the publishing extension more flexible (apache#1442) The added flexibility is intended to be ported to the multiple project in the polaris-tools repository. (Follow up of apache#1384) * Poetry v2 (apache#898) * PEP 621 and Poetry v2 * PEP 621 and Poetry v2 * Update min python to 3.9 * Add back flask8 for apache#1096 * Add Integration tests for Delta tables for Spark Client (apache#1500) * main: Update dependency com.google.cloud:google-cloud-storage-bom to v2.52.2 (apache#1536) * main: Update dependency poetry to v2.1.3 (apache#1534) * main: Update dependency io.netty:netty-codec-http2 to v4.2.1.Final (apache#1533) * main: Update dependency boto3 to v1.38.10 (apache#1525) * Fix test failure (apache#1541) * Fix the URL of the KEYS file in the release vote email template (apache#1538) * Event Listeners (apache#922) Implementation of event listeners discussed [here](https://lists.apache.org/thread/03yz5wolkvy8l7rbcwjnqdq1bl8p065v). I decided to keep this implementation generic and not take a dependency on Jakarta Events nor Vertx busses. It's easy to extend this, either within Polaris or in an external PolarisEventListener, and handle events however one wishes. Some high level notes: - PolarisEventListener is the main interface with all the event methods such as `onBeforeRequestRateLimited` - DefaultPolarisEventListener is an empty implementation which allows users to only partially implement event handlers - `polaris.events.type` is the config that lets you specify your event listener implementation * Update metastores.md (apache#1537) * Update metastores.md * Resolve comment. * Resolve comment. --------- Co-authored-by: Yufei Gu <yufei.apache.org> * Doc: Document the Concept of realm (apache#1478) * main: Update dependency boto3 to v1.38.11 (apache#1542) * Fix compile warning: [unchecked] unchecked cast (apache#1544) Use `Class.cast()` instead of implicit cast. * Doc: Add release guide on the website (apache#1539) * main: Update actions/stale digest to f78de97 (apache#1547) * main: Update dependency boto3 to v1.38.12 (apache#1548) * main: Update postgres Docker tag to v17.5 (apache#1549) * main: Update dependency com.adobe.testing:s3mock-testcontainers to v4.2.0 (apache#1551) * main: Update dependency com.nimbusds:nimbus-jose-jwt to v10.3 (apache#1552) * Interface changes for pagination (apache#1528) * add missing apis * more tests, fixes * clean up drop * autolint * changes per review * revert iceberg messages to comply with oss tests * another revert * more iceberg catalog changes * autolint * dependency issues * more wiring * continuing rebase * remaining issues are related to task loading * re-add tests * debugging * fix failing tests * fix another test * changes per review * autolint * some fixes * stable * updates for new persistence * fix * continuing work * more reverts * continue reverts * more reverts * yank tests * autolint * test reverts * try to support limit without real page tokens * autolint * Stable * change comment * autolint * remove catalog config for now * changes per review * more tweaks * simplify types per review * Stable, about to refactor more * re-stable * polish * autolint * more changes per review * stable * Introduce reserved-properties setting; reserve "polaris." by default (apache#1417) * initial commit * initial commit * try to test * quarkus fixes * fix a bunch of callsites * Start applying changes * autolint * chase todos * autolint * bugfix * stable * add one test * stable with more tests * autolint * more tests * autolint * stable tests * clean up * oops * stabilize on main * autolint * more changes per review * Add cleanup support for partition-level statistics files when `DROP TABLE PURGE` (apache#1508) * cleaning up partition stats * update partition stat file extension * update test partition stat write impl * Implement federation to HadoopCatalog (apache#1466) * wip * quarkus fixes * autolint * hadoop impl * autolint * Refactors * refactored * autolint * add config * autolint * stable * Remove breakpoint anchor * add line to application.properties * yank HADOOP * autolint * Spark: Use builder for CreateGenericTableRequest instead of constructor for easier API spec update (apache#1546) * main: Update docker.io/jaegertracing/all-in-one Docker tag to v1.69.0 (apache#1559) * main: Update dependency io.opentelemetry:opentelemetry-bom to v1.50.0 (apache#1558) * main: Update dependency software.amazon.awssdk:bom to v2.31.40 (apache#1567) * main: Update dependency boto3 to v1.38.13 (apache#1556) * Include DISCLAIMER in binary distributions (apache#1568) * main: Update dependency io.micrometer:micrometer-bom to v1.14.7 (apache#1570) * JDBC: Simplify JDBC entity conversion (apache#1564) * fix(Catalog): Add List PolarisStorageAction for all metadata read operations (apache#1391) * fix(Catalog): Add List PolarisStorageAction for all metadata read operations * Site : Update cloud providers quickstart to use (apache#1554) * [JDBC] Add retries with delay (apache#1517) [JDBC] Add retries with delay This change adds retries in the JDBC persistence layer, these retries are with jitter and are tunable in the following ways : a. max_retries : Total number of retries we expect the persistence to do on Connection Reset exception and serializable error exceptions, before giving up. b. max_duaration_in_ms : Time in ms since the first attempt this retries should be done. For ex on configured 500 ms the total time spent in retrying should not exceed 500ms (optimistically) c. initial_delay_in_ms : initial delay before the first attempt * [Docs] Add JDBC retry properties (apache#1550) * Use env var in spark container (apache#1522) * added Signed-off-by: owenowenisme <[email protected]> * fix Signed-off-by: owenowenisme <[email protected]> * add export Signed-off-by: owenowenisme <[email protected]> * update docs using .env Signed-off-by: owenowenisme <[email protected]> * update docs Signed-off-by: owenowenisme <[email protected]> * change back from using .env to export Signed-off-by: owenowenisme <[email protected]> * Apply suggestions from code review Co-authored-by: Adnan Hemani <[email protected]> --------- Signed-off-by: owenowenisme <[email protected]> Co-authored-by: Adnan Hemani <[email protected]> * Migrate catalog configs to the new reserved prefix (apache#1557) * rewrite * rewrite * stable * changes per comments * Remove unused javadoc parameter in BasePersistence (apache#1580) * Site: Publish table maintenance policies (apache#1581) * Add schema symlinks to static site directory Co-authored-by: Yufei Gu <yufei.apache.org> * Remove `defaults` / `overrides` from feature configurations (apache#1572) * double WithParentName * autolint * Revert some * autolint * add to BCconfigs * autolint * yank * copy yuns test * autolint * remove defaults * repair test * autolint * stablize test * stable * autolint * configmap change * copypaste * regen helm docs * autolint * no dots in props * remove accidental file * small changes per review * clean out defaults * BCC fix * autolint * typefix * autolint * main: Update dependency io.prometheus:prometheus-metrics-exporter-servlet-jakarta to v1.3.7 (apache#1578) * main: Update dependency io.micrometer:micrometer-bom to v1.15.0 (apache#1575) * main: Update dependency io.projectreactor.netty:reactor-netty-http to v1.2.6 (apache#1577) * main: Update dependency boto3 to v1.38.15 (apache#1574) * Fix header in the gradle-wrapper.properties (apache#1587) * Refactor GenericTableCatalog to support generic table federation (apache#1579) * initial commit * stable * javadocs * autolint * changes per review * Update headers for files copied from non ASF projects and update LICENSE (apache#1562) * [Policy Store] Enforce regex requirement for policy name (apache#1583) * Enforce regex requirement for policy name * enable validation in open api builder * FIx nit * Include LICENSE, NOTICE, DISCLAIMER files in the docker images (apache#1588) * main: Update swagger to v1.6.16 (apache#1594) * Eliminate usage of CatalogHandlers; introduce CatalogHandlerUtils (apache#1576) * compiles * wire it up * De-static, add config * autolint * license entry * cut over * pull * revert helm doc * autolint * Allow realm context resolution to execute blocking calls (apache#1591) * main: Update dependency com.adobe.testing:s3mock-testcontainers to v4.3.0 (apache#1597) * Fix header in helm chart (apache#1560) * Following vote/discussion on the Incubator general mailing list, NOTICE now contains part of Nessie NOTICE. (apache#1565) * In binary distributions, now we group the artifacts and we add BSD/MIT licenses inline (apache#1573) * Better null-safety for check against unsafe catalog configs (apache#1584) Better null-safety for check against unsafe catalog configs * Add support for federated principal and role with block for manual role assignment (apache#1353) * Add support for federated principal and role with block for manual role assignment * Update spec to distinguish federated and non-federated entities * Changed builder to allow setting federated status twice * Revert spec changes - add 'federated' property back to Principal entity * Fixed builder to remove federated property * Removed unnecessary openapi config flags * Fix compilation issue in test * Remove federated flag from principal entity * Fixed builder oversight * Fix compilation failures and rebase on main * Remove verbose terminal output for admin tool (apache#1598) * Site: Use the overview page as the home page (apache#1605) * Include only relevant information from Picocli NOTICE as it doesn't apply in Polaris (as we use binary package, not source package) (apache#1561) * Site: Simplified doc for spark download (apache#1608) * main: Update dependency software.amazon.awssdk:bom to v2.31.45 (apache#1607) * main: Update registry.access.redhat.com/ubi9/openjdk-21-runtime Docker tag to v1.22-1.1747241886 (apache#1603) * main: Update dependency boto3 to v1.38.18 (apache#1592) * main: Update docker.io/prom/prometheus Docker tag to v3.4.0 (apache#1602) * Site: Update production configuration page (apache#1606) * main: Update dependency com.google.cloud:google-cloud-storage-bom to v2.52.3 (apache#1623) * main: Update dependency boto3 to v1.38.19 (apache#1622) * Remove Bouncy Castle dependency usage from PemUtils (apache#1318) - Added PEM format parsing in PemUtils - Added unit test for PemUtils for empty file and multiple PEM objects - Removed Bouncy Castle Provider dependency from service common module - Removed Bouncy Castle Provider dependency from quarkus service module * Site: Add a page for policy management (apache#1600) * [Policy Store | Management Spec] Add policy privileges to spec and update admin service impl (apache#1529) This PR adds new policy related privileges to polaris-management-api.yml and update PolarisAdminService to allow granting new privileges * Spec: Add SigV4 Auth Support for Catalog Federation (apache#1506) * Spec changes for SigV4 Auth Support for Catalog Federation * Extract service identity info as a nested object * nit: fix admin tool log level and comments (apache#1626) The previous WARNING log levels seems to work, but WARN aligns better with standard Quarkus log levels. Fixes apache#1612 * Doc: switch to use iceberg-aws-bundle jar (apache#1609) * main: Update dependency org.mockito:mockito-core to v5.18.0 (apache#1630) * main: Update dependency boto3 to v1.38.20 (apache#1631) * Require explicit user-consent to enable HadoopFileIO (apache#1532) Using `HadoopFileIO` in Polaris can enable "hidden features" that users are likely not aware of. This change requires users to manually update the configuration to be able to use `HadoopFileIO` in way that highlights the consequences of enabling it. This PR updates Polaris in multiple ways: * The default of `SUPPORTED_CATALOG_STORAGE_TYPES` is changed to not include the `FILE` storage type. * Respect the `ALLOW_SPECIFYING_FILE_IO_IMPL` configuration on namespaces, tables and views to prevent setting an `io-impl` value for anything but one of the configured, supported storage-types. * Unify validation code in a new class `IcebergPropertiesValidation`. * Using `FILE` or `HadoopFileIO` now _also_ requires the explicit configuration `ALLOW_INSECURE_STORAGE_TYPES_ACCEPTING_SECURITY_RISKS=true`. * Added production readiness checks that trigger when `ALLOW_INSECURE_STORAGE_TYPES_ACCEPTING_SECURITY_RISKS` is `true` or `SUPPORTED_CATALOG_STORAGE_TYPES` contains `FILE` (defaults and per-realm). * The two new readiness checks are considered _severe_. Severe readiness-errors prevent the server from starting up - unless the user explicitly configured `polaris.readiness.ignore-security-issues=true`. Log messages and configuration options explicitly use "clear" phrases highlighting the consequences. With these changes it is intentionally extremely difficult to start Polaris with HadoopFileIO. People who work around all these safety nets must have realized that what they are doing. A lot of the test code relies on `FILE`/`HadoopFileIO`, those tests got all the configurations to let those tests continue to work as they are, bypassing the added security safeguards. --------- Co-authored-by: Dmitri Bourlatchkov <[email protected]> * CI: Make helm-CI run for all changes (apache#1611) Whether helm-charts work (or not) doesn't only depend on what's on the `helm/` directory, but also the production code. It's hard to figure out which changes could break the helm-charts or what the helm-charts rely on. Therefore it's likely better to run Helm-CI for all changes. * fix(helm): Fix features configuration section & enhance docs (apache#1638) Fixes apache#1618 . Fixes apache#1634. This change adapts the `Values.features` section to the changes from apache#1572, also fixing incorrect configmap entries produced by the template. It also re-syncs the README.md file with README.md.gotmpl (the sync was lost a few commits ago), and enhances the instructions for running Helm tests locally. * Add DefaultConfigurationStoreDecorator to support inject callContext for DefaultConfigurationStore (apache#1505) * Set the default value of DROP WITH PURGE to false (apache#1619) * Core: Remove configure: INITIALIZE_DEFAULT_CATALOG_FILEIO_FOR_TEST (apache#1624) * remove duplicated files --------- Signed-off-by: owenowenisme <[email protected]> Co-authored-by: Robert Stupp <[email protected]> Co-authored-by: Yufei Gu <[email protected]> Co-authored-by: Honah (Jonas) J. <[email protected]> Co-authored-by: Mend Renovate <[email protected]> Co-authored-by: Travis Bowen <[email protected]> Co-authored-by: Travis Bowen <[email protected]> Co-authored-by: Dmitri Bourlatchkov <[email protected]> Co-authored-by: gh-yzou <[email protected]> Co-authored-by: Eric Maynard <[email protected]> Co-authored-by: JB Onofré <[email protected]> Co-authored-by: Juichang Lu <[email protected]> Co-authored-by: David Lu <[email protected]> Co-authored-by: gfakbar20 <[email protected]> Co-authored-by: Liam Bao <[email protected]> Co-authored-by: Adnan Hemani <[email protected]> Co-authored-by: Adnan Hemani <[email protected]> Co-authored-by: Neelesh Salian <[email protected]> Co-authored-by: Dennis Huo <[email protected]> Co-authored-by: Rulin Xing <[email protected]> Co-authored-by: Rulin Xing <[email protected]> Co-authored-by: Alexandre Dutra <[email protected]> Co-authored-by: fabio-rizzo-01 <[email protected]> Co-authored-by: Prashant Singh <[email protected]> Co-authored-by: Pierre Laporte <[email protected]> Co-authored-by: Richard Liu <[email protected]> Co-authored-by: Michael Collado <[email protected]> Co-authored-by: Owen Lin (You-Cheng Lin) <[email protected]> Co-authored-by: Eric Maynard <[email protected]> Co-authored-by: Andrew Guterman <[email protected]> Co-authored-by: MonkeyCanCode <[email protected]> Co-authored-by: danielhumanmod <[email protected]> Co-authored-by: fivetran-ashokborra <[email protected]> Co-authored-by: David Handermann <[email protected]> Co-authored-by: William Hyun <[email protected]>

snazy requested review from adutra, ashvina, collado-mike, dennishuo, dimas-b, eric-maynard, jackye1995, jbonofre and vvcephei as code owners May 6, 2025 12:14

github-project-automation bot added this to Basic Kanban Board May 6, 2025

snazy requested review from HonahX, MonkeyCanCode, RussellSpitzer, ebyhr, flyrain and takidau as code owners May 6, 2025 12:14

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board May 6, 2025

dimas-b reviewed May 6, 2025

View reviewed changes

snazy force-pushed the explicit-hadoop-fio branch from 0a748ba to 3f65378 Compare May 6, 2025 13:42

snazy added bug Something isn't working 1.0-blocker labels May 6, 2025

eric-maynard removed the bug Something isn't working label May 6, 2025

eric-maynard previously requested changes May 6, 2025

View reviewed changes

snazy added the bug Something isn't working label May 7, 2025

snazy force-pushed the explicit-hadoop-fio branch from 3f65378 to c149a7f Compare May 8, 2025 14:11

dimas-b and others added 9 commits May 20, 2025 17:51

phrasing

2a3b06d

rename one property

dd3fb6f

rename one mth

cabc890

property

a8d1e36

move

a30f6a5

update test

cf7abcb

"defaults"

2beceb8

review

6e4ffa6

snazy force-pushed the explicit-hadoop-fio branch from e951904 to 6e4ffa6 Compare May 20, 2025 15:58

snazy requested review from dimas-b and jbonofre May 20, 2025 15:58

dimas-b approved these changes May 20, 2025

View reviewed changes

flyrain reviewed May 20, 2025

View reviewed changes

snazy merged commit e1494bc into apache:main May 21, 2025
6 checks passed

github-project-automation bot moved this from PRs In Progress to Done in Basic Kanban Board May 21, 2025

snazy deleted the explicit-hadoop-fio branch May 21, 2025 07:36

singhpk234 mentioned this pull request May 21, 2025

[HOTFIX] FIX QUICKSTART #1646

Merged

flyrain mentioned this pull request May 22, 2025

./regtests/run_spark_sql.sh broken #1649

Closed

Tishj mentioned this pull request May 22, 2025

Polaris CI is failing duckdb/duckdb-iceberg#256

Closed

flyrain pushed a commit to flyrain/polaris that referenced this pull request May 28, 2025

Fix the manual test broken by PR apache#1532

59b797e

flyrain pushed a commit that referenced this pull request May 28, 2025

Fix getting-started docker start by PR #1532 (#1687)

408389f

flyrain added a commit that referenced this pull request May 28, 2025

Fix the manual test broken by PR #1532 (#1688)

7961fdd

flyrain mentioned this pull request May 28, 2025

Rename polaris-quarkus-* projects as polaris-* and renaming quarkus folder as runtime #1695

Merged

Require explicit user-consent to enable HadoopFileIO #1532

Require explicit user-consent to enable HadoopFileIO #1532

Uh oh!

Conversation

snazy commented May 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eric-maynard commented May 6, 2025

Uh oh!

eric-maynard left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eric-maynard May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snazy commented May 7, 2025

Uh oh!

eric-maynard commented May 7, 2025

Uh oh!

flyrain commented May 7, 2025

Uh oh!

dimas-b May 6, 2025 •

edited

Loading

eric-maynard May 13, 2025 •

edited

Loading