Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .version
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2.33.2
2.33.3
32 changes: 32 additions & 0 deletions doc/manual/source/release-notes/rl-2.33.md
Original file line number Diff line number Diff line change
Expand Up @@ -279,3 +279,35 @@ This release was made possible by the following 33 contributors:
- Henry [**(@cootshk)**](https://github.com/cootshk)
- Martin Joerg [**(@mjoerg)**](https://github.com/mjoerg)
- Farid Zakaria [**(@fzakaria)**](https://github.com/fzakaria)
# Release 2.33.3 (2026-02-13)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks odd... But if upstream has it then I guess let's keep it.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's correct (though maybe the formatting won't look great).


- S3 binary caches now use virtual-hosted-style addressing by default [#15208](https://github.com/NixOS/nix/issues/15208)

S3 binary caches now use virtual-hosted-style URLs
(`https://bucket.s3.region.amazonaws.com/key`) instead of path-style URLs
(`https://s3.region.amazonaws.com/bucket/key`) when connecting to standard AWS
S3 endpoints. This enables HTTP/2 multiplexing and fixes TCP connection
exhaustion (TIME_WAIT socket accumulation) under high-concurrency workloads.

A new `addressing-style` store option controls this behavior:

- `auto` (default): virtual-hosted-style for standard AWS endpoints, path-style
for custom endpoints.
- `path`: forces path-style addressing (deprecated by AWS).
- `virtual`: forces virtual-hosted-style addressing (bucket names must not
contain dots).

Bucket names containing dots (e.g., `my.bucket.name`) automatically fall back
to path-style addressing in `auto` mode, because dotted names create
multi-level subdomains that break TLS wildcard certificate validation.

Example using path-style for backwards compatibility:

```
s3://my-bucket/key?region=us-east-1&addressing-style=path
```
Comment on lines +304 to +308
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Add a language tag to the fenced code block.
This avoids MD040 warnings and keeps markdownlint happy.

📝 Suggested fix
-  ```
+  ```text
   s3://my-bucket/key?region=us-east-1&addressing-style=path
</details>

<!-- suggestion_start -->

<details>
<summary>📝 Committable suggestion</summary>

> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

```suggestion
Example using path-style for backwards compatibility:
🧰 Tools
🪛 markdownlint-cli2 (0.20.0)

[warning] 306-306: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

🤖 Prompt for AI Agents
In `@doc/manual/source/release-notes/rl-2.33.md` around lines 304 - 308, The
fenced code block containing the example URL
"s3://my-bucket/key?region=us-east-1&addressing-style=path" needs a language tag
to satisfy markdownlint (MD040); update the triple-backtick fence to include a
tag such as ```text (or ```bash) before the URL so the block reads like ```text
followed by the URL and then closing backticks.


Additionally, TCP keep-alive is now enabled on all HTTP connections, preventing
idle connections from being silently dropped by intermediate network devices
(NATs, firewalls, load balancers).

252 changes: 237 additions & 15 deletions src/libstore-tests/s3-url.cc
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,33 @@ INSTANTIATE_TEST_SUITE_P(
},
},
"with_absolute_endpoint_uri",
},
ParsedS3URLTestCase{
"s3://bucket/key?addressing-style=virtual",
{
.bucket = "bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
},
"with_addressing_style_virtual",
},
ParsedS3URLTestCase{
"s3://bucket/key?addressing-style=path",
{
.bucket = "bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Path,
},
"with_addressing_style_path",
},
ParsedS3URLTestCase{
"s3://bucket/key?addressing-style=auto",
{
.bucket = "bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Auto,
},
"with_addressing_style_auto",
}),
[](const ::testing::TestParamInfo<ParsedS3URLTestCase> & info) { return info.param.description; });

Expand Down Expand Up @@ -138,6 +165,26 @@ INSTANTIATE_TEST_SUITE_P(
InvalidS3URLTestCase{"s3://bucket", "error: URI has a missing or invalid key", "missing_key"}),
[](const ::testing::TestParamInfo<InvalidS3URLTestCase> & info) { return info.param.description; });

TEST(ParsedS3URLTest, invalidAddressingStyleThrows)
{
ASSERT_THROW(ParsedS3URL::parse(parseURL("s3://bucket/key?addressing-style=bogus")), InvalidS3AddressingStyle);
}

TEST(ParsedS3URLTest, virtualStyleWithAuthoritylessEndpointThrows)
{
ParsedS3URL input{
.bucket = "bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
.endpoint =
ParsedURL{
.scheme = "file",
.path = {"", "some", "path"},
},
};
ASSERT_THROW(input.toHttpsUrl(), nix::Error);
}

// =============================================================================
// S3 URL to HTTPS Conversion Tests
// =============================================================================
Expand Down Expand Up @@ -166,17 +213,18 @@ INSTANTIATE_TEST_SUITE_P(
S3ToHttpsConversion,
S3ToHttpsConversionTest,
::testing::Values(
// Default (auto) addressing style: virtual-hosted for standard AWS endpoints
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "my-bucket",
.key = {"my-key.txt"},
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.us-east-1.amazonaws.com"},
.path = {"", "my-bucket", "my-key.txt"},
.authority = ParsedURL::Authority{.host = "my-bucket.s3.us-east-1.amazonaws.com"},
.path = {"", "my-key.txt"},
},
"https://s3.us-east-1.amazonaws.com/my-bucket/my-key.txt",
"https://my-bucket.s3.us-east-1.amazonaws.com/my-key.txt",
"basic_s3_default_region",
},
S3ToHttpsConversionTestCase{
Expand All @@ -187,12 +235,13 @@ INSTANTIATE_TEST_SUITE_P(
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.eu-west-1.amazonaws.com"},
.path = {"", "prod-cache", "nix", "store", "abc123.nar.xz"},
.authority = ParsedURL::Authority{.host = "prod-cache.s3.eu-west-1.amazonaws.com"},
.path = {"", "nix", "store", "abc123.nar.xz"},
},
"https://s3.eu-west-1.amazonaws.com/prod-cache/nix/store/abc123.nar.xz",
"https://prod-cache.s3.eu-west-1.amazonaws.com/nix/store/abc123.nar.xz",
"with_eu_west_1_region",
},
// Custom endpoint authority: path-style by default
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
Expand All @@ -208,6 +257,7 @@ INSTANTIATE_TEST_SUITE_P(
"http://custom.s3.com/bucket/key",
"custom_endpoint_authority",
},
// Custom endpoint URL: path-style by default
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
Expand Down Expand Up @@ -236,10 +286,10 @@ INSTANTIATE_TEST_SUITE_P(
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.ap-southeast-2.amazonaws.com"},
.path = {"", "bucket", "path", "to", "file.txt"},
.authority = ParsedURL::Authority{.host = "bucket.s3.ap-southeast-2.amazonaws.com"},
.path = {"", "path", "to", "file.txt"},
},
"https://s3.ap-southeast-2.amazonaws.com/bucket/path/to/file.txt",
"https://bucket.s3.ap-southeast-2.amazonaws.com/path/to/file.txt",
"complex_path_and_region",
},
S3ToHttpsConversionTestCase{
Expand All @@ -250,11 +300,11 @@ INSTANTIATE_TEST_SUITE_P(
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.us-east-1.amazonaws.com"},
.path = {"", "my-bucket", "my-key.txt"},
.authority = ParsedURL::Authority{.host = "my-bucket.s3.us-east-1.amazonaws.com"},
.path = {"", "my-key.txt"},
.query = {{"versionId", "abc123xyz"}},
},
"https://s3.us-east-1.amazonaws.com/my-bucket/my-key.txt?versionId=abc123xyz",
"https://my-bucket.s3.us-east-1.amazonaws.com/my-key.txt?versionId=abc123xyz",
"with_versionId",
},
S3ToHttpsConversionTestCase{
Expand All @@ -266,13 +316,185 @@ INSTANTIATE_TEST_SUITE_P(
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.eu-west-1.amazonaws.com"},
.path = {"", "versioned-bucket", "path", "to", "object"},
.authority = ParsedURL::Authority{.host = "versioned-bucket.s3.eu-west-1.amazonaws.com"},
.path = {"", "path", "to", "object"},
.query = {{"versionId", "version456"}},
},
"https://s3.eu-west-1.amazonaws.com/versioned-bucket/path/to/object?versionId=version456",
"https://versioned-bucket.s3.eu-west-1.amazonaws.com/path/to/object?versionId=version456",
"with_region_and_versionId",
},
// Explicit addressing-style=path forces path-style on standard AWS endpoints
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "my-bucket",
.key = {"my-key.txt"},
.region = "us-west-2",
.addressingStyle = S3AddressingStyle::Path,
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.us-west-2.amazonaws.com"},
.path = {"", "my-bucket", "my-key.txt"},
},
"https://s3.us-west-2.amazonaws.com/my-bucket/my-key.txt",
"explicit_path_style",
},
// Explicit addressing-style=virtual forces virtual-hosted-style on custom endpoints
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
.key = {"key"},
.scheme = "http",
.addressingStyle = S3AddressingStyle::Virtual,
.endpoint = ParsedURL::Authority{.host = "custom.s3.com"},
},
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "bucket.custom.s3.com"},
.path = {"", "key"},
},
"http://bucket.custom.s3.com/key",
"explicit_virtual_style_custom_endpoint",
},
// Explicit addressing-style=virtual with full endpoint URL
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
.endpoint =
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "server", .port = 9000},
.path = {""},
},
},
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "bucket.server", .port = 9000},
.path = {"", "key"},
},
"http://bucket.server:9000/key",
"explicit_virtual_style_full_endpoint_url",
},
// Dotted bucket names work normally with explicit path-style
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "my.bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Path,
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.us-east-1.amazonaws.com"},
.path = {"", "my.bucket", "key"},
},
"https://s3.us-east-1.amazonaws.com/my.bucket/key",
"dotted_bucket_with_path_style",
},
// Dotted bucket names fall back to path-style with auto on standard AWS endpoints
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "my.bucket.name",
.key = {"key"},
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.us-east-1.amazonaws.com"},
.path = {"", "my.bucket.name", "key"},
},
"https://s3.us-east-1.amazonaws.com/my.bucket.name/key",
"dotted_bucket_with_auto_style_on_aws",
},
// Dotted bucket names work with auto style on custom endpoints (auto = path-style)
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "my.bucket",
.key = {"key"},
.endpoint = ParsedURL::Authority{.host = "minio.local"},
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "minio.local"},
.path = {"", "my.bucket", "key"},
},
"https://minio.local/my.bucket/key",
"dotted_bucket_with_auto_style_custom_endpoint",
}),
[](const ::testing::TestParamInfo<S3ToHttpsConversionTestCase> & info) { return info.param.description; });

// =============================================================================
// S3 URL to HTTPS Conversion Error Tests
// =============================================================================

struct S3ToHttpsConversionErrorTestCase
{
ParsedS3URL input;
std::string description;
};

class S3ToHttpsConversionErrorTest : public ::testing::WithParamInterface<S3ToHttpsConversionErrorTestCase>,
public ::testing::Test
{};

TEST_P(S3ToHttpsConversionErrorTest, ThrowsOnConversion)
{
auto & [input, description] = GetParam();
ASSERT_THROW(input.toHttpsUrl(), nix::Error);
}

INSTANTIATE_TEST_SUITE_P(
S3ToHttpsConversionErrors,
S3ToHttpsConversionErrorTest,
::testing::Values(
S3ToHttpsConversionErrorTestCase{
ParsedS3URL{
.bucket = "bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
.endpoint = ParsedURL::Authority{.host = ""},
},
"virtual_style_with_empty_host_authority",
},
S3ToHttpsConversionErrorTestCase{
ParsedS3URL{
.bucket = "my.bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
},
"dotted_bucket_with_explicit_virtual_style",
},
S3ToHttpsConversionErrorTestCase{
ParsedS3URL{
.bucket = "my.bucket.name",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
},
"dotted_bucket_with_explicit_virtual_style_multi_dot",
},
S3ToHttpsConversionErrorTestCase{
ParsedS3URL{
.bucket = "my.bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
.endpoint = ParsedURL::Authority{.host = "minio.local"},
},
"dotted_bucket_with_explicit_virtual_style_custom_authority",
},
S3ToHttpsConversionErrorTestCase{
ParsedS3URL{
.bucket = "my.bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
.endpoint =
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "minio.local", .port = 9000},
.path = {""},
},
},
"dotted_bucket_with_explicit_virtual_style_full_endpoint_url",
}),
[](const ::testing::TestParamInfo<S3ToHttpsConversionErrorTestCase> & info) { return info.param.description; });

} // namespace nix
5 changes: 4 additions & 1 deletion src/libstore/daemon.cc
Original file line number Diff line number Diff line change
Expand Up @@ -858,7 +858,10 @@ static void performOp(
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
std::shared_ptr<const ValidPathInfo> info;
logger->startWork();
info = store->queryPathInfo(path);
try {
info = store->queryPathInfo(path);
} catch (InvalidPath &) {
}
logger->stopWork();
if (info) {
conn.to << 1;
Expand Down
6 changes: 6 additions & 0 deletions src/libstore/filetransfer.cc
Original file line number Diff line number Diff line change
Expand Up @@ -502,6 +502,12 @@ struct curlFileTransfer : public FileTransfer

curl_easy_setopt(req, CURLOPT_CONNECTTIMEOUT, fileTransferSettings.connectTimeout.get());

// Enable TCP keep-alive so that idle connections in curl's reuse pool
// are not silently dropped by NATs, firewalls, or load balancers.
curl_easy_setopt(req, CURLOPT_TCP_KEEPALIVE, 1L);
curl_easy_setopt(req, CURLOPT_TCP_KEEPIDLE, 60L);
curl_easy_setopt(req, CURLOPT_TCP_KEEPINTVL, 60L);

curl_easy_setopt(req, CURLOPT_LOW_SPEED_LIMIT, 1L);
curl_easy_setopt(req, CURLOPT_LOW_SPEED_TIME, fileTransferSettings.stalledDownloadTimeout.get());

Expand Down
Loading