Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/features/opensearch/s3-repository.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ Configure in `opensearch.yml`:
| `s3.client.default.endpoint` | Custom S3 endpoint |
| `s3.client.default.region` | AWS region |
| `s3.client.default.protocol` | HTTP or HTTPS |
| `s3.client.default.legacy_md5_checksum_calculation` | Enable legacy MD5 checksum for S3-compatible storage (v3.4.0+) |

### Usage Example

Expand Down Expand Up @@ -210,6 +211,7 @@ graph TB

| Version | PR | Description |
|---------|-----|-------------|
| v3.4.0 | [#19788](https://github.com/opensearch-project/OpenSearch/pull/19788) | Add LEGACY_MD5_CHECKSUM_CALCULATION to repository-s3 settings list |
| v3.3.0 | [#18800](https://github.com/opensearch-project/OpenSearch/pull/18800) | Switch default async HTTP client to AWS CRT for improved throughput |
| v3.3.0 | [#19220](https://github.com/opensearch-project/OpenSearch/pull/19220) | Fix S3-compatible repository checksum trailing headers issue |
| v3.1.0 | [#18312](https://github.com/opensearch-project/OpenSearch/pull/18312) | Add support for SSE-KMS and S3 bucket owner verification |
Expand All @@ -219,6 +221,7 @@ graph TB

## References

- [PR #19788](https://github.com/opensearch-project/OpenSearch/pull/19788): Add LEGACY_MD5_CHECKSUM_CALCULATION to settings list
- [PR #18800](https://github.com/opensearch-project/OpenSearch/pull/18800): AWS CRT async HTTP client implementation
- [Issue #18535](https://github.com/opensearch-project/OpenSearch/issues/18535): Feature request for S3CrtClient support
- [PR #19220](https://github.com/opensearch-project/OpenSearch/pull/19220): S3-compatible repository checksum fix
Expand All @@ -235,6 +238,7 @@ graph TB

## Change History

- **v3.4.0** (2025-10): Added `legacy_md5_checksum_calculation` to opensearch.yml settings for S3-compatible storage configuration
- **v3.3.0** (2025-09): Switched default async HTTP client to AWS CRT for ~5-7% throughput improvement; added `s3_async_client_type` setting
- **v3.3.0** (2025-09): Fixed S3-compatible repository compatibility by making checksum trailing headers conditional
- **v3.1.0** (2025-07): Added SSE-KMS support, bucket owner verification, removed legacy server_side_encryption setting
Expand Down
83 changes: 83 additions & 0 deletions docs/releases/v3.4.0/features/opensearch/s3-repository.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# S3 Repository

## Summary

This release adds the `LEGACY_MD5_CHECKSUM_CALCULATION` setting to the list of repository-s3 plugin settings, enabling it to be configured in `opensearch.yml` in addition to repository creation time. This completes the fix for S3-compatible storage compatibility that was introduced in v3.3.0.

## Details

### What's New in v3.4.0

The `s3.client.<client_name>.legacy_md5_checksum_calculation` setting was added to the list of registered settings in `S3RepositoryPlugin`, allowing users to configure legacy MD5 checksum calculation behavior through `opensearch.yml`.

### Background

In OpenSearch 3.0.0, the AWS SDK was upgraded to version 2.30.31, which introduced mandatory checksum trailing headers. This broke compatibility with S3-compatible storage providers (like Outscale, MinIO, etc.) that don't support the new checksum format.

The fix in v3.3.0 ([PR #19220](https://github.com/opensearch-project/OpenSearch/pull/19220)) introduced the `legacy_md5_checksum_calculation` setting to enable legacy MD5 checksum behavior for compatibility. However, this setting was only configurable at repository creation time, not in `opensearch.yml`.

### Technical Changes

#### Configuration Enhancement

The setting `S3ClientSettings.LEGACY_MD5_CHECKSUM_CALCULATION` is now registered in `S3RepositoryPlugin.getSettings()`:

```java
// S3RepositoryPlugin.java
public List<Setting<?>> getSettings() {
return Arrays.asList(
// ... other settings ...
S3ClientSettings.LEGACY_MD5_CHECKSUM_CALCULATION,
// ... other settings ...
);
}
```

#### New Configuration Option

| Setting | Description | Default |
|---------|-------------|---------|
| `s3.client.<client_name>.legacy_md5_checksum_calculation` | Enable legacy MD5 checksum calculation for S3-compatible storage | `false` |

### Usage Example

Configure in `opensearch.yml` for cluster-wide S3-compatible storage support:

```yaml
# opensearch.yml
s3.client.default.legacy_md5_checksum_calculation: true
s3.client.default.endpoint: "https://s3-compatible-endpoint.example.com"
```

Or configure per-client:

```yaml
# opensearch.yml
s3.client.minio.legacy_md5_checksum_calculation: true
s3.client.minio.endpoint: "http://minio.local:9000"
```

### Migration Notes

If you're using S3-compatible storage and previously had to configure `legacy_md5_checksum_calculation` at repository creation time, you can now move this configuration to `opensearch.yml` for easier management across all repositories using the same client.

## Limitations

- The setting only affects S3-compatible storage providers that don't support AWS SDK v2's checksum trailing headers
- When enabled, uses MD5 checksums instead of the newer CRC-based checksums

## Related PRs

| PR | Description |
|----|-------------|
| [#19788](https://github.com/opensearch-project/OpenSearch/pull/19788) | Add S3Repository.LEGACY_MD5_CHECKSUM_CALCULATION to list of repository-s3 settings |

## References

- [Issue #18240](https://github.com/opensearch-project/OpenSearch/issues/18240): Snapshot repositories don't work with S3 compatible storage in 3.0.0
- [PR #19220](https://github.com/opensearch-project/OpenSearch/pull/19220): Fix issue with s3-compatible repositories due to missing checksum trailing headers (v3.3.0)
- [Register Snapshot Repository](https://docs.opensearch.org/3.0/api-reference/snapshots/create-repository/): OpenSearch documentation

## Related Feature Report

- [Full feature documentation](../../../../features/opensearch/s3-repository.md)
1 change: 1 addition & 0 deletions docs/releases/v3.4.0/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
- [Dependency Updates (OpenSearch Core)](features/opensearch/dependency-updates-opensearch-core.md) - 32 dependency updates including Netty 4.2.4 for HTTP/3 readiness
- [Lucene Upgrade](features/opensearch/lucene-upgrade.md) - Bump Apache Lucene from 10.3.1 to 10.3.2 with MaxScoreBulkScorer bug fix
- [Node Stats Bugfixes](features/opensearch/node-stats-bugfixes.md) - Fix negative CPU usage values in node stats on certain Linux distributions
- [S3 Repository](features/opensearch/s3-repository.md) - Add LEGACY_MD5_CHECKSUM_CALCULATION to opensearch.yml settings for S3-compatible storage
- [Security Kerberos Integration](features/opensearch/security-kerberos-integration.md) - Update Hadoop to 3.4.2 and enable Kerberos integration tests for JDK-24+
- [Settings Bugfixes](features/opensearch/settings-bugfixes.md) - Fix duplicate registration of dynamic settings and patch version build issues
- [Stats Builder Pattern Deprecations](features/opensearch/stats-builder-pattern-deprecations.md) - Deprecated constructors in 30+ Stats classes in favor of Builder pattern
Expand Down