-
Notifications
You must be signed in to change notification settings - Fork 364
Modularize federation (Option 2) #2332
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
If @snazy is okay with using this approach for HMS federation, then I think it's preferable to option 1 |
snazy
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea, thanks for removing the conditional build parameter. One comment though. But otherwise I'm fine with it.
| public static final String ICEBERG_REST_FACTORY_IDENTIFIER = "ICEBERG_REST"; | ||
| public static final String HADOOP_FACTORY_IDENTIFIER = "HADOOP"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
THose are superfluous, as those are the enum names, and can therefore be removed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We still need these because we can't use ConnnectionType.ICEBERG_REST.name() or ConnectionType.HADOOP.name(). The annotations needs to be constant expressions. Here's the error for your reference.
The value for annotation attribute Identifier.value must be a constant expressionJava(536871525)
| public String getFactoryIdentifier() { | ||
| switch (this) { | ||
| case ICEBERG_REST: | ||
| return ICEBERG_REST_FACTORY_IDENTIFIER; | ||
| case HADOOP: | ||
| return HADOOP_FACTORY_IDENTIFIER; | ||
| default: | ||
| throw new UnsupportedOperationException( | ||
| "No factory identifier for connection type: " + this); | ||
| } | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You may want to just return name().toLowerCase(Locale.ROOT) – since most CDI identifiers in Polaris (if not all) are lower-case.
| public String getFactoryIdentifier() { | |
| switch (this) { | |
| case ICEBERG_REST: | |
| return ICEBERG_REST_FACTORY_IDENTIFIER; | |
| case HADOOP: | |
| return HADOOP_FACTORY_IDENTIFIER; | |
| default: | |
| throw new UnsupportedOperationException( | |
| "No factory identifier for connection type: " + this); | |
| } | |
| } | |
| public String getFactoryIdentifier() { | |
| return name().toLowerCase(Locale.ROOT); | |
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can't use the name().toLowerCase() directly because of two reasons:
- We need the annotations to be constants. (Please see my comment above)
- We need to explicitly handle the NULL_TYPE/default case.
To make things consistent, I've changed the declarations to use lower case.
dennishuo
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
|
Looks like other comments have also been addressed, merging now. |
* Fix Keycloak getting-started example (apache#2349) The `polaris-setup` container was erroneously including a non-existent scope when fetching a token from Keycloak. * fix(deps): update dependency com.nimbusds:nimbus-jose-jwt to v10.4.2 (apache#2350) * Use PolarisTaskConstants (apache#2346) * Add a regression test for Catalog Federation (apache#2286) * Add a regression test for Catalog Federation * Install jq dependency * Fix token issues * Update regtests/README.md Co-authored-by: Eric Maynard <[email protected]> * Update README.md --------- Co-authored-by: Eric Maynard <[email protected]> * Modularize federation (Option 2) (apache#2332) * Modularize federation (Option 2) * Move polaris-extensions-federation-hadoop dependency * Change identifier to lowerCase * Change identifiers to constants * Replace CallContext with RealmConfig in enforceFeatureEnabledOrThrow (apache#2348) * Replace CallContext with RealmConfig in CatalogEntity (apache#2336) * chore(deps): update registry.access.redhat.com/ubi9/openjdk-21-runtime docker tag to v1.23-6 (apache#2353) * fix(deps): update dependency com.gradleup.shadow:shadow-gradle-plugin to v9.0.2 (apache#2358) * chore(deps): update postgres docker tag to v17.6 (apache#2354) * Add integration tests with Keycloak (apache#2343) * Fix REST responses for failed Admin operations (apache#2291) * Fix REST responses for failed Admin operations the `boolean` return values of many methods in `PolarisAdminService` were often simply not getting used at all, thus the REST api returned success in those cases even though the `PrivilegeResult` was marked as failed. due to this fix a silently failing test now needs to be adjusted. we return the `PrivilegeResult` instead of a `boolean` to give the client at least some indication of what has gone wrong on the server side. note that some of the other operations were throwing Expcetions already, which are already reported back correctly to the client. * review: use http 400 BAD_REQUEST * Make PolarisAuthorizer RequestScoped (apache#2340) all methods in `PolarisAuthorizer` currently have a `CallContext` parameter. in its only implementation only `CallContext.getRealmConfig` is getting used. so since `PolarisAuthorizer` cant be used outside a request, we can simply make it request-scoped and inject the request-scoped `RealmConfig` directly. * fix(deps): update mockito monorepo to v5.19.0 (apache#2360) * Fix soft-merge conflict on `main` (apache#2364) * feat(docs): Add Getting Stated guide for MinIO (apache#2227) * feat(docs): Add Getting Stated guide for MinIO A simple page of step-by-step instructions for setting up a local environment with Polaris, MinIO and Spark. Closes apache#1530 * IntelliJ: fix project icon in IJ project list (apache#2366) ... copy source has changed * Use asMap property helpers (apache#2347) seems like these helpers existed for a long time but were just not getting used consistently * SigV4 Auth Support for Catalog Federation - Part 2: Connection Config Persistence (apache#2190) * Add SigV4 related DPOs * Rename UserSecretReference to SecretReference and fix some small issues * fix(deps): update dependency software.amazon.awssdk:bom to v2.32.24 (apache#2371) * Rat-check: exclude venv, cleanup excludes, include .svg (apache#2363) * `.svg` files are XML files and can contain a license header * Re-grouped the exclusion rat patterns * Added exclude for `.venv` * Added exclude for `.ruff_cache` * NoSQL: Async-impls: add some safeguards + javadoc spelling * NoSQL: spelling * NoSQL: dependency updates * Last merged commit: 5a7686b --------- Co-authored-by: Alexandre Dutra <[email protected]> Co-authored-by: Mend Renovate <[email protected]> Co-authored-by: Christopher Lambert <[email protected]> Co-authored-by: Pooja Nilangekar <[email protected]> Co-authored-by: Eric Maynard <[email protected]> Co-authored-by: Dmitri Bourlatchkov <[email protected]> Co-authored-by: Rulin Xing <[email protected]>
Based on the discussion in the community sync, this PR modularizes federation calls to a separate factory. The specific instances of the NonRESTCatalogFactory is loaded only when a non-REST catalog is initialized and accessed. Beyond the catalog initialization, all the calls are identical for internal and federated catalogs.
Based on the discussion in #2301, this PR removes the compile-time flag used to build the
HadoopFederatedCatalogFactory.Testing:
Applied the regtest in PR #2286.