Skip to content

Add storage class for aws, gcp, and azure #456

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Sep 13, 2025
Merged

Add storage class for aws, gcp, and azure #456

merged 4 commits into from
Sep 13, 2025

Conversation

@matthewmturner
Copy link
Contributor

@matthewmturner matthewmturner commented Aug 2, 2025

Which issue does this PR close?

Closes #330

Rationale for this change

I would like to set the storage class on objects without a custom HTTP client

What changes are included in this PR?

  1. Add StorageClass variant to Attribute
  2. Add implementations for aws and gcp clients

I started looking into Azure blob tiers but I have never used Azure before and it wasn't immediately obvious to me that it works the same way as AWS and GCP so I did not include anything for that.

Are there any user-facing changes?

A new Attribute variant is introduced

@tustvold
Copy link
Contributor

tustvold commented Aug 6, 2025

CI appears to be failing on Azure

@tustvold
Copy link
Contributor

tustvold commented Aug 6, 2025

I think this may also tie into #116 as a mechanism to change the storage tier of an object

@matthewmturner matthewmturner changed the title Add storage class for aws and gcp Add storage class for aws, gcp, and azure Sep 4, 2025
@matthewmturner
Copy link
Contributor Author

matthewmturner commented Sep 4, 2025

@tustvold Sorry for the delay following up on this. Ive updated azure and the http client.

@alamb
Copy link
Contributor

alamb commented Sep 13, 2025

@tustvold are you happy with merging this?

tustvold
tustvold approved these changes Sep 13, 2025
View reviewed changes
Copy link
Contributor

@tustvold tustvold left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine to me, although I haven't tested it

alamb
alamb reviewed Sep 13, 2025
View reviewed changes
src/attributes.rs
/// [Azure](https://learn.microsoft.com/en-us/rest/api/storageservices/set-blob-tier).
/// `StorageClass` is used as the name for this attribute because 2 of the 3 storage providers
/// use that name
StorageClass,
Copy link
Contributor

@alamb alamb Sep 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I verified that adding a new enum variant is not a breaking API change as this enum is marked #[non_exhaustive] ✅

@alamb alamb merged commit da88a75 into apache:main Sep 13, 2025
8 checks passed
@alamb
Copy link
Contributor

alamb commented Sep 13, 2025

Thanks everyone!

@kylebarron kylebarron mentioned this pull request Sep 13, 2025
Open
Simon-3008-Simon added a commit to sap-contributions/arrow-rs-object-store that referenced this pull request Oct 14, 2025
@Simon-3008-Simon @alamb
@mbrobbel @dependabot @kylebarron @philjb @asubiotto @matthewmturner @Friede80 @dreamtalen
FOS-11081: sync code (#3)
c40000b 
* Improve documentation for http client timeout (apache#390)

* chore: fix some clippy 1.89 warnings and ignore some doctests on wasm32 (apache#468)

* chore: fix some clippy 1.89 warnings

* fix another warning

* Skip some doctests for wasm32

* build(deps): bump actions/checkout from 4 to 5 (apache#463)

Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 5.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v4...v5)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Allow "application_credentials" in `impl FromStr for GoogleConfigKey` (apache#467)

* build(deps): bump actions/setup-python from 5 to 6 (apache#476)

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5 to 6.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](actions/setup-python@v5...v6)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump actions/setup-node from 4 to 5 (apache#477)

Bumps [actions/setup-node](https://github.com/actions/setup-node) from 4 to 5.
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](actions/setup-node@v4...v5)

---
updated-dependencies:
- dependency-name: actions/setup-node
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump actions/github-script from 7 to 8 (apache#478)

Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 8.
- [Release notes](https://github.com/actions/github-script/releases)
- [Commits](actions/github-script@v7...v8)

---
updated-dependencies:
- dependency-name: actions/github-script
  dependency-version: '8'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(client/retry): include error info in logs when retry occurs (apache#487)

On a request retry, it logs an info message stating that an error was
encountered and information about the retry process but it hasn't
included any details about the error that is causing the retry. This PR
updates the logging to include the status if it is a server error and
the http error kind if a transport error occurred. While the last error
when retries are exhausted is returned up the call stack, the
intermediate errors need not be exactly the same. It is helpful to
include some minimum information about what error triggered a retry each
time it happens.

* aws: downgrade credential provider info! log messages to debug! (apache#436)

These log messages are very noisy.

* Add storage class for aws, gcp, and azure (apache#456)

* Add storage class for aws and gcp

* Add azure storage class attribute

* Update attribute docs

* Update http client

* Add version 0.12.4 release plan to README (apache#490)

* Fix for clippy 1.90 (apache#492)

* AWS S3: Support STS endpoint, WebIdentity, RoleArn, RoleSession configuration (apache#480)

* Allow setting STS endpoint via env var

* Properly use AmazonS3Builder::credentials_from_env for AssumeRoleWithWebIdentity auth flow

---------

Co-authored-by: Andrew Lamb <[email protected]>

* Revert "refactor: remove AWS dynamo integration (apache#407)" (apache#493)

This reverts commit 034733f.

* Update version to 0.12.4 and add changelog (apache#491)

* Update version to 0.12.4

* Update update_changelog.sh script

* Update changelog

* Last touchups

* Update changelog

* Reapply "refactor: remove AWS dynamo integration (apache#407)" (apache#494)

This reverts commit bebd53b.

* Add Content_length header to S3 create_multipart (apache#496)

* minor: Fix MSRV CI workflow (apache#502)

* ci: Fix MSRV CI workflow

* Update .github/workflows/rust.yml

* Update .github/workflows/rust.yml

---------

Co-authored-by: Matthijs Brobbel <[email protected]>

* All changes to support SAP HDLFS

* add hdlfs

* FOS-9911: support delta path version 2.0

* FOS-10349:  fix object_url (#2)

* FOS-10349: fix object_url

* FOS-11081: sync code

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Andrew Lamb <[email protected]>
Co-authored-by: Matthijs Brobbel <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Kyle Barron <[email protected]>
Co-authored-by: Phil Bracikowski <[email protected]>
Co-authored-by: Alfonso Subiotto Marqués <[email protected]>
Co-authored-by: Matthew Turner <[email protected]>
Co-authored-by: Matt Friede <[email protected]>
Co-authored-by: Yongming Ding <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alamb alamb alamb left review comments

@tustvold tustvold tustvold approved these changes

@mbrobbel mbrobbel mbrobbel approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Support setting storage class when objects are written

4 participants

@matthewmturner @tustvold @alamb @mbrobbel