AWS S3: Support STS endpoint, WebIdentity, RoleArn, RoleSession configuration #480
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mbrobbel
approved these changes
Sep 5, 2025
…WebIdentity auth flow
|
There are a lot more changes now than just being able to set the endpoint? |
@kylebarron I've updated the description of the PR to mention including the other env vars in the config |
alamb
changed the title
alamb
approved these changes
Sep 17, 2025
There was a problem hiding this comment.
Looks good to me -- thank you @Friede80 @mbrobbel and @kylebarron
Can someone resolve the conflicts so we can merge this PR?
|
I took the liberty of merging up to resolve the conflicts and plan to merge this PR when CI passes |
|
The clippy failure appears to be due to the new rust release: https://github.com/apache/arrow-rs-object-store/actions/runs/17860194133/job/50788677121?pr=480 I will make a PR to fix |
6 tasks
|
CI is failing due to #492 I fixed that and merged up again and hopefully we get a clean CI run now |
|
🚀 |
|
Thanks again @Friede80 @kylebarron and @mbrobbel |
Simon-3008-Simon
added a commit
to sap-contributions/arrow-rs-object-store
that referenced
this pull request
Oct 14, 2025
* Improve documentation for http client timeout (apache#390) * chore: fix some clippy 1.89 warnings and ignore some doctests on wasm32 (apache#468) * chore: fix some clippy 1.89 warnings * fix another warning * Skip some doctests for wasm32 * build(deps): bump actions/checkout from 4 to 5 (apache#463) Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 5. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v4...v5) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Allow "application_credentials" in `impl FromStr for GoogleConfigKey` (apache#467) * build(deps): bump actions/setup-python from 5 to 6 (apache#476) Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5 to 6. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v5...v6) --- updated-dependencies: - dependency-name: actions/setup-python dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump actions/setup-node from 4 to 5 (apache#477) Bumps [actions/setup-node](https://github.com/actions/setup-node) from 4 to 5. - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](actions/setup-node@v4...v5) --- updated-dependencies: - dependency-name: actions/setup-node dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump actions/github-script from 7 to 8 (apache#478) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 8. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](actions/github-script@v7...v8) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '8' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(client/retry): include error info in logs when retry occurs (apache#487) On a request retry, it logs an info message stating that an error was encountered and information about the retry process but it hasn't included any details about the error that is causing the retry. This PR updates the logging to include the status if it is a server error and the http error kind if a transport error occurred. While the last error when retries are exhausted is returned up the call stack, the intermediate errors need not be exactly the same. It is helpful to include some minimum information about what error triggered a retry each time it happens. * aws: downgrade credential provider info! log messages to debug! (apache#436) These log messages are very noisy. * Add storage class for aws, gcp, and azure (apache#456) * Add storage class for aws and gcp * Add azure storage class attribute * Update attribute docs * Update http client * Add version 0.12.4 release plan to README (apache#490) * Fix for clippy 1.90 (apache#492) * AWS S3: Support STS endpoint, WebIdentity, RoleArn, RoleSession configuration (apache#480) * Allow setting STS endpoint via env var * Properly use AmazonS3Builder::credentials_from_env for AssumeRoleWithWebIdentity auth flow --------- Co-authored-by: Andrew Lamb <[email protected]> * Revert "refactor: remove AWS dynamo integration (apache#407)" (apache#493) This reverts commit 034733f. * Update version to 0.12.4 and add changelog (apache#491) * Update version to 0.12.4 * Update update_changelog.sh script * Update changelog * Last touchups * Update changelog * Reapply "refactor: remove AWS dynamo integration (apache#407)" (apache#494) This reverts commit bebd53b. * Add Content_length header to S3 create_multipart (apache#496) * minor: Fix MSRV CI workflow (apache#502) * ci: Fix MSRV CI workflow * Update .github/workflows/rust.yml * Update .github/workflows/rust.yml --------- Co-authored-by: Matthijs Brobbel <[email protected]> * All changes to support SAP HDLFS * add hdlfs * FOS-9911: support delta path version 2.0 * FOS-10349: fix object_url (#2) * FOS-10349: fix object_url * FOS-11081: sync code --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Andrew Lamb <[email protected]> Co-authored-by: Matthijs Brobbel <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Kyle Barron <[email protected]> Co-authored-by: Phil Bracikowski <[email protected]> Co-authored-by: Alfonso Subiotto Marqués <[email protected]> Co-authored-by: Matthew Turner <[email protected]> Co-authored-by: Matt Friede <[email protected]> Co-authored-by: Yongming Ding <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #283
Rationale for this change
I am using a self-hosted S3 store and want to be able to use the AssumeRoleWithWebIdentity auth flow.
What changes are included in this PR?
The endpoint used for STS can now be sourced from the AWS_ENDPOINT_URL_STS env var instead of always being hardcoded to
https://sts.{region}.amazonaws.com.Additionally, all env vars used for the AssumeRoleWithWebIdentity auth flow were previously not included as
AmazonS3ConfigKeys, so I've added these to document the usage.Are there any user-facing changes?