Skip to content

Make StorageAccessConfigProvider request-scoped #2974

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dimas-b merged 1 commit into from
Nov 11, 2025
Merged

Make StorageAccessConfigProvider request-scoped #2974

dimas-b merged 1 commit into from
Nov 11, 2025

Conversation

@XN137
Copy link
Contributor

@XN137 XN137 commented Nov 4, 2025 •
edited
Loading

  • add StorageCredentialsVendor as request-scoped wrapper around PolarisCredentialVendor
  • make FileIOFactory request-scoped
  • make TaskFileIOSupplier request-scoped

@github-project-automation github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Nov 4, 2025
@XN137 XN137 force-pushed the request-scoped-AccessConfigProvider branch from 662a47e to 95cd2cf Compare November 7, 2025 16:20
@XN137 XN137 marked this pull request as ready for review November 7, 2025 16:23
dimas-b
dimas-b reviewed Nov 7, 2025
View reviewed changes
Copy link
Contributor

@dimas-b dimas-b left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR LGTM as it stands 👍 but I wonder if we could take it one step further (as commented below) 🤔 WDYT?

...service/src/main/java/org/apache/polaris/service/catalog/io/StorageAccessConfigProvider.java
? tableLocations
: Set.of();
AccessConfig accessConfig =
storageCredentialCache.getOrGenerateSubScopeCreds(
Copy link
Contributor

@dimas-b dimas-b Nov 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WDYT about making AccessConfigProvider and interface and hiding StorageCredentialCache in one of the implementations?

StorageCredentialCache will remain application-scoped, of course?

Copy link
Contributor Author

@XN137 XN137 Nov 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i agree with splitting out an interface, however i think we should do it in a follow-up PR as its not directly related to the goal of this PR, wdyt?

Copy link
Contributor

@dimas-b dimas-b Nov 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. A follow-up PR for this SGTM 👍

dimas-b
dimas-b previously approved these changes Nov 10, 2025
View reviewed changes
@github-project-automation github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Nov 10, 2025
@XN137
Make StorageAccessConfigProvider request-scoped
ada4731 
- add `StorageCredentialsVendor` as request-scoped wrapper around `PolarisCredentialVendor`
- make `FileIOFactory` request-scoped
- make `TaskFileIOSupplier` request-scoped
@XN137 XN137 force-pushed the request-scoped-AccessConfigProvider branch from 95cd2cf to ada4731 Compare November 11, 2025 08:24
@XN137 XN137 changed the title Make AccessConfigProvider request-scoped Make StorageAccessConfigProvider request-scoped Nov 11, 2025
@XN137
Copy link
Contributor Author

XN137 commented Nov 11, 2025

rebased after conflict with aa72157

@dimas-b dimas-b merged commit 308b103 into apache:main Nov 11, 2025
15 checks passed
@github-project-automation github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Nov 11, 2025
MonkeyCanCode pushed a commit to MonkeyCanCode/polaris that referenced this pull request Nov 12, 2025
@XN137 @MonkeyCanCode
Make StorageAccessConfigProvider request-scoped (apache#2974)
f857e29 
- add `StorageCredentialsVendor` as request-scoped wrapper around `PolarisCredentialVendor`
- make `FileIOFactory` request-scoped
- make `TaskFileIOSupplier` request-scoped
@XN137 XN137 deleted the request-scoped-AccessConfigProvider branch November 12, 2025 07:37
@dimas-b dimas-b mentioned this pull request Dec 3, 2025
Closed
snazy added a commit to snazy/polaris that referenced this pull request Feb 11, 2026
@snazy @collado-mike
@cccs-cat001 @dimas-b @adutra @renovate-bot @CodingBangboo @MonkeyCanCode @XN137 @jbonofre @flyrain @sungwy @pingtimeout
Dremio merge 2025 11 18 18 27 (apache#161)
222dfce 
* Add loadEntities batch call and rename listFullEntities (apache#2508)

* Add loadEntities batch call and rename listFullEntities

* Changed batch call to implement loadResolvedEntities instead

* Add loadResolvedEntities by id and entity cache support

* Add additional test for loadResolvedEntities by id

* Added additional test and updated comments in EntityCache interface

* Add additional constructor to ResolvedEntitiesResult

* Fixed unused method reference

* Removed loadResolvedEntities method with lookup record param

* Pulled out toResolvedPolarisEntity method per PR comment

* Core: made the ARN role regex more generic (apache#3005)

* fix(docs): Generify S3 index page (apache#2997)

* Remove the mention of "cloud" since not all possible storage options are provided in "cloud".

* Avoid listing specific child pages in the doc test. Rely on Hugo-general index (on the left-hand pane).

---------

Co-authored-by: Alexandre Dutra <[email protected]>

* fix(deps): update dependency io.prometheus:prometheus-metrics-exporter-servlet-jakarta to v1.4.3 (apache#3009)

* fix(deps): update dependency com.google.cloud:google-cloud-storage-bom to v2.60.0 (apache#3011)

* fix(deps): update dependency io.opentelemetry:opentelemetry-bom to v1.56.0 (apache#3012)

* fix(deps): update dependency com.adobe.testing:s3mock-testcontainers to v4.10.0 (apache#3010)

* fix(deps): update dependency org.agrona:agrona to v2.3.2 (apache#3014)

* fix(deps): update quarkus platform and group to v3.29.2 (apache#3013)

* chore(deps): update dependency pre-commit to v4.4.0 (apache#3015)

* fix(deps): update dependency software.amazon.awssdk:bom to v2.38.2 (apache#3019)

* Add test for TracingFilter (apache#2847)

* NoSQL: Add (micro-ish) benchmarks (apache#3006)

A project for JMH based benchmarks against NoSQL persistence.

* Helm chart: include configmap checksum in deployment annotations (apache#3023)

* fix(deps): update dependency ch.qos.logback:logback-classic to v1.5.21 (apache#3025)

* NoSQL: Realms handling (apache#3007)

Introduces handling for realms including realm-state management/transition.

The `RealmStore` implementation for NoSQL depends on CDI components, coming in a follo-up PR.

* Rename AccessConfig and AccessConfigProvider for clarity (apache#2883)

* rename AccessConfig for clarity

* rename getStorageAccessConfig() and add javadoc

* Refactor: improve and clean up Dockerfiles (apache#2957)

* Refactor: improve and clean up Dockerfiles

* Refactor: improve and clean up Dockerfiles

* Refactor: improve and clean up Dockerfiles

* Refactor: improve and clean up Dockerfiles

* Refactor: improve and clean up Dockerfiles

* Refactor: improve and clean up Dockerfiles

* Make StorageAccessConfigProvider request-scoped (apache#2974)

- add `StorageCredentialsVendor` as request-scoped wrapper around `PolarisCredentialVendor`
- make `FileIOFactory` request-scoped
- make `TaskFileIOSupplier` request-scoped

* Increase javadoc visibility in `nosql/realms` (apache#3029)

This is to fix javadoc error: `No public or protected classes found to document`

* NoSQL: Add correctness tests (apache#3027)

Verifies the correctness of concurrent commits, and big index handling.

These tests are intentionally _not_ part of the base-backend test suite for two reasons:
1. These tests do not run against the `Backend` interface but the `Persistence` interface, including commit and index logic.
2. These tests are intended to be runnable against a custom provisioned database cluster, not just tiny-ish test containers.

* NoSQL: Add maintenance API, SPI (apache#3028)

Maintenance operations include a bunch of tasks that are regularly executed against a backend database.

Types of maintenance operations include:
* Purging unreferenced objects and references within a catalog
* Purging whole catalogs that are marked to be purged
* Purging whole realms that are marked to be purged

Implementation added in a follow-up PR.

* Embrace request-scoped TokenBroker (apache#3024)

* Embrace request-scoped TokenBroker

`TokenBroker` and `CallContext` are both request-scoped, so instead of
passing the former into the latter, we can do this via the
`TokenBrokerFactory` and thus simplify the `TokenBroker` interface.

* fix(deps): update dependency io.smallrye:jandex to v3.5.2 (apache#3032)

* Fix monkey patching (apache#3016)

* chore(deps): update quay.io/keycloak/keycloak docker tag to v26.4.5 (apache#3034)

* chore(deps): update registry.access.redhat.com/ubi9/openjdk-21-runtime docker tag to v1.23-6.1762870925 (apache#3053)

* fix(deps): update dependency org.testcontainers:testcontainers-bom to v2.0.2 (apache#3054)

* chore(deps): update postgres docker tag to v18.1 (apache#3055)

* Add Polaris Community Meeting 2025-11-13 (apache#3060)

* Site: Rename menu "downloads" to "releases" (apache#2928)

* Update dependency software.amazon.awssdk:bom to v2.38.7 (apache#3065)

* Test-fix: Cleanup OPA test container on stop (apache#3041)

Quarkus takes care of reusing a test-resource across tests. The current behavior leaves the container around.

Plus some nit-fixes (deprecation + local var)

* Update dependency org.apache.commons:commons-lang3 to v3.20.0 (apache#3063)

* Build: ensure LICENSE/NOTICE is in all jars, always add pom-files to all jars (apache#3057)

There are a some inconsistencies between the different kinds of jars and the included information:
* LICENSE/NOTICE files are present in the "main" jar and in the sources jar, but not in the javadoc jar.
* The Maven pom.xml and pom.properties files are only present for release builds or when explicitly requested.
* "Additional" jar-manifest attributes that are only present in release builds.

This change fixes the three mentioned issues:
* Always include pom.xml and pom.properties in the built jar files.
* Always include the additional jar-manifest attributes, except the Git information, which would otherwise render the Gradle build cache ineffective.
* Include pom.xml + pom.properties + license/notice in literally all jar files.

The Gradle logic to include the license+notice+pom files has been simplified as well.

* Remove unused polarisEventListener field from IcebergCatalogHandler (apache#3045)

it was added in c3f5001 but then its
only usage was removed in d03c717

* fix(deps): update quarkus platform and group to v3.29.3 (apache#3052)

* Site: Add Open Policy Agent (OPA) as External Policy Decision Point (apache#3030)

Doc PR following up the introduction of OpaPolarisAuthorizer: apache#2680

* OPA: Tackle deprecation warnings (apache#3042)

Instead of suppressing the deprecations, this change updates the code a little bit to remove the mocks (except to create a non-nullable parameter).

* Use POJOs for OPA JSON schema construction and publish schema (apache#3031)

Co-authored-by: Robert Stupp <[email protected]>

* Use CDI for more test setups (apache#3040)

this avoids a bunch of redundant manual setup.

the important parts are establishing a `RealmContext` by calling
`QuarkusMock.installMockForType` and then populating `polarisContext`
from the injected `CallContext`.

* fix(deps): update dependency com.github.dasniko:testcontainers-keycloak to v4 (apache#3070)

* chore(deps): update actions/checkout digest to 93cb6ef (apache#3068)

* OPA: Fail fast when OPA bearer token file is unreadable (apache#3062)

* fix(deps): update immutables to v2.11.7 (apache#3072)

* Skip Hugo Site workflow on forks (apache#3056)

Forks usually don't have the "versioned-docs" tag and thus PRs against forks or rebasing the main branch on a fork currently always causes workflow failures.

* Fix warnings around TransactionWorkspaceMetaStoreManager (apache#3044)

- dont return `null` for interface methods that are `@Nonnull`
- fix wrong method name parameters
- dont annotate void methods as `@Nonnull`

* NoSQL: Add CDI/common+testing + necessary nosql-store implementations (apache#3035)

Adds common and test-specific CDI functionality. Requires the NoSQL store implementations `:polaris-persistence-nosql-realms-store-nosql` and `:polaris-nodes-store-nosql`.

Those modules have cross-project dependencies for test purposes, hence those are all contained in this PR.

CDI for Quarkus will be added in a follow-up.

* Automate the release guide - Take 2 - Github workflows (apache#2383)

The release automation is simplified to four GitHub workflows that just require the really mandatory user input: the version number.
1. workflow: Trigger the creation of the release branch
2. workflow: Upgrade the release branch with the version and build the the final change-log for that version
3. workflow: Build the RC artifacts from the release branch and push those to the various staging repositories
4. workflow: Eventually release the artifacts.

See also the [email announcement](https://lists.apache.org/thread/d0smz07gnr509yj5dc6omo3cvkf1pnh7).

---------

Co-authored-by: Robert Stupp <[email protected]>

* Update actions/checkout digest to 93cb6ef (apache#3082)

* NoSQL: adapt to conflicting changes in main

* Last merged commit 8ccddc5

---------

Co-authored-by: Michael Collado <[email protected]>
Co-authored-by: cccs-cat001 <[email protected]>
Co-authored-by: Dmitri Bourlatchkov <[email protected]>
Co-authored-by: Alexandre Dutra <[email protected]>
Co-authored-by: Mend Renovate <[email protected]>
Co-authored-by: Nuoya Jiang <[email protected]>
Co-authored-by: Yong Zheng <[email protected]>
Co-authored-by: Christopher Lambert <[email protected]>
Co-authored-by: JB Onofré <[email protected]>
Co-authored-by: Yufei Gu <[email protected]>
Co-authored-by: Sung Yun <[email protected]>
Co-authored-by: Pierre Laporte <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dimas-b dimas-b dimas-b approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@XN137 @dimas-b