Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions doc/manual/rl-next/s3-storage-class.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
---
synopsis: "S3 binary cache stores now support storage class configuration"
prs: [14464]
issues: [7015]
---

S3 binary cache stores now support configuring the storage class for uploaded objects via the `storage-class` parameter. This allows users to optimize costs by selecting appropriate storage tiers based on access patterns.

Example usage:

```bash
# Use Glacier storage for long-term archival
nix copy --to 's3://my-bucket?storage-class=GLACIER' /nix/store/...

# Use Intelligent Tiering for automatic cost optimization
nix copy --to 's3://my-bucket?storage-class=INTELLIGENT_TIERING' /nix/store/...
```

The storage class applies to both regular uploads and multipart uploads. When not specified, objects use the bucket's default storage class.

See the [S3 storage classes documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html) for available storage classes and their characteristics.
18 changes: 18 additions & 0 deletions src/libstore-tests/s3-binary-cache-store.cc
Original file line number Diff line number Diff line change
Expand Up @@ -122,4 +122,22 @@ TEST(S3BinaryCacheStore, parameterFiltering)
EXPECT_EQ(ref.params["priority"], "10");
}

/**
* Test storage class configuration
*/
TEST(S3BinaryCacheStore, storageClassDefault)
{
S3BinaryCacheStoreConfig config{"s3", "test-bucket", {}};
EXPECT_EQ(config.storageClass.get(), std::nullopt);
}

TEST(S3BinaryCacheStore, storageClassConfiguration)
{
StringMap params;
params["storage-class"] = "GLACIER";

S3BinaryCacheStoreConfig config("s3", "test-bucket", params);
EXPECT_EQ(config.storageClass.get(), std::optional<std::string>("GLACIER"));
}

} // namespace nix
20 changes: 20 additions & 0 deletions src/libstore/include/nix/store/s3-binary-cache-store.hh
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,26 @@ struct S3BinaryCacheStoreConfig : HttpBinaryCacheStoreConfig
Default is 100 MiB. Only takes effect when multipart-upload is enabled.
)"};

const Setting<std::optional<std::string>> storageClass{
this,
std::nullopt,
"storage-class",
R"(
The S3 storage class to use for uploaded objects. When not set (default),
uses the bucket's default storage class. Valid values include:
- STANDARD (default, frequently accessed data)
- REDUCED_REDUNDANCY (less frequently accessed data)
- STANDARD_IA (infrequent access)
- ONEZONE_IA (infrequent access, single AZ)
- INTELLIGENT_TIERING (automatic cost optimization)
- GLACIER (archival with retrieval times in minutes to hours)
- DEEP_ARCHIVE (long-term archival with 12-hour retrieval)
- GLACIER_IR (instant retrieval archival)

See AWS S3 documentation for detailed storage class descriptions and pricing:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html
)"};

/**
* Set of settings that are part of the S3 URI itself.
* These are needed for region specification and other S3-specific settings.
Expand Down
8 changes: 6 additions & 2 deletions src/libstore/s3-binary-cache-store.cc
Original file line number Diff line number Diff line change
Expand Up @@ -134,10 +134,14 @@ void S3BinaryCacheStore::upsertFile(
const std::string & path, RestartableSource & source, const std::string & mimeType, uint64_t sizeHint)
{
auto doUpload = [&](RestartableSource & src, uint64_t size, std::optional<Headers> headers) {
Headers uploadHeaders = headers.value_or(Headers());
if (auto storageClass = s3Config->storageClass.get()) {
uploadHeaders.emplace_back("x-amz-storage-class", *storageClass);
}
if (s3Config->multipartUpload && size > s3Config->multipartThreshold) {
uploadMultipart(path, src, size, mimeType, std::move(headers));
uploadMultipart(path, src, size, mimeType, std::move(uploadHeaders));
} else {
upload(path, src, size, mimeType, std::move(headers));
upload(path, src, size, mimeType, std::move(uploadHeaders));
}
};

Expand Down
Loading