diff --git a/docs/features/opensearch/s3-repository.md b/docs/features/opensearch/s3-repository.md index a5ec419e8..930fc375d 100644 --- a/docs/features/opensearch/s3-repository.md +++ b/docs/features/opensearch/s3-repository.md @@ -83,6 +83,7 @@ Configure in `opensearch.yml`: | `s3.client.default.endpoint` | Custom S3 endpoint | | `s3.client.default.region` | AWS region | | `s3.client.default.protocol` | HTTP or HTTPS | +| `s3.client.default.legacy_md5_checksum_calculation` | Enable legacy MD5 checksum for S3-compatible storage (v3.4.0+) | ### Usage Example @@ -210,6 +211,7 @@ graph TB | Version | PR | Description | |---------|-----|-------------| +| v3.4.0 | [#19788](https://github.com/opensearch-project/OpenSearch/pull/19788) | Add LEGACY_MD5_CHECKSUM_CALCULATION to repository-s3 settings list | | v3.3.0 | [#18800](https://github.com/opensearch-project/OpenSearch/pull/18800) | Switch default async HTTP client to AWS CRT for improved throughput | | v3.3.0 | [#19220](https://github.com/opensearch-project/OpenSearch/pull/19220) | Fix S3-compatible repository checksum trailing headers issue | | v3.1.0 | [#18312](https://github.com/opensearch-project/OpenSearch/pull/18312) | Add support for SSE-KMS and S3 bucket owner verification | @@ -219,6 +221,7 @@ graph TB ## References +- [PR #19788](https://github.com/opensearch-project/OpenSearch/pull/19788): Add LEGACY_MD5_CHECKSUM_CALCULATION to settings list - [PR #18800](https://github.com/opensearch-project/OpenSearch/pull/18800): AWS CRT async HTTP client implementation - [Issue #18535](https://github.com/opensearch-project/OpenSearch/issues/18535): Feature request for S3CrtClient support - [PR #19220](https://github.com/opensearch-project/OpenSearch/pull/19220): S3-compatible repository checksum fix @@ -235,6 +238,7 @@ graph TB ## Change History +- **v3.4.0** (2025-10): Added `legacy_md5_checksum_calculation` to opensearch.yml settings for S3-compatible storage configuration - **v3.3.0** (2025-09): Switched default async HTTP client to AWS CRT for ~5-7% throughput improvement; added `s3_async_client_type` setting - **v3.3.0** (2025-09): Fixed S3-compatible repository compatibility by making checksum trailing headers conditional - **v3.1.0** (2025-07): Added SSE-KMS support, bucket owner verification, removed legacy server_side_encryption setting diff --git a/docs/releases/v3.4.0/features/opensearch/s3-repository.md b/docs/releases/v3.4.0/features/opensearch/s3-repository.md new file mode 100644 index 000000000..ee5479ec1 --- /dev/null +++ b/docs/releases/v3.4.0/features/opensearch/s3-repository.md @@ -0,0 +1,83 @@ +# S3 Repository + +## Summary + +This release adds the `LEGACY_MD5_CHECKSUM_CALCULATION` setting to the list of repository-s3 plugin settings, enabling it to be configured in `opensearch.yml` in addition to repository creation time. This completes the fix for S3-compatible storage compatibility that was introduced in v3.3.0. + +## Details + +### What's New in v3.4.0 + +The `s3.client..legacy_md5_checksum_calculation` setting was added to the list of registered settings in `S3RepositoryPlugin`, allowing users to configure legacy MD5 checksum calculation behavior through `opensearch.yml`. + +### Background + +In OpenSearch 3.0.0, the AWS SDK was upgraded to version 2.30.31, which introduced mandatory checksum trailing headers. This broke compatibility with S3-compatible storage providers (like Outscale, MinIO, etc.) that don't support the new checksum format. + +The fix in v3.3.0 ([PR #19220](https://github.com/opensearch-project/OpenSearch/pull/19220)) introduced the `legacy_md5_checksum_calculation` setting to enable legacy MD5 checksum behavior for compatibility. However, this setting was only configurable at repository creation time, not in `opensearch.yml`. + +### Technical Changes + +#### Configuration Enhancement + +The setting `S3ClientSettings.LEGACY_MD5_CHECKSUM_CALCULATION` is now registered in `S3RepositoryPlugin.getSettings()`: + +```java +// S3RepositoryPlugin.java +public List> getSettings() { + return Arrays.asList( + // ... other settings ... + S3ClientSettings.LEGACY_MD5_CHECKSUM_CALCULATION, + // ... other settings ... + ); +} +``` + +#### New Configuration Option + +| Setting | Description | Default | +|---------|-------------|---------| +| `s3.client..legacy_md5_checksum_calculation` | Enable legacy MD5 checksum calculation for S3-compatible storage | `false` | + +### Usage Example + +Configure in `opensearch.yml` for cluster-wide S3-compatible storage support: + +```yaml +# opensearch.yml +s3.client.default.legacy_md5_checksum_calculation: true +s3.client.default.endpoint: "https://s3-compatible-endpoint.example.com" +``` + +Or configure per-client: + +```yaml +# opensearch.yml +s3.client.minio.legacy_md5_checksum_calculation: true +s3.client.minio.endpoint: "http://minio.local:9000" +``` + +### Migration Notes + +If you're using S3-compatible storage and previously had to configure `legacy_md5_checksum_calculation` at repository creation time, you can now move this configuration to `opensearch.yml` for easier management across all repositories using the same client. + +## Limitations + +- The setting only affects S3-compatible storage providers that don't support AWS SDK v2's checksum trailing headers +- When enabled, uses MD5 checksums instead of the newer CRC-based checksums + +## Related PRs + +| PR | Description | +|----|-------------| +| [#19788](https://github.com/opensearch-project/OpenSearch/pull/19788) | Add S3Repository.LEGACY_MD5_CHECKSUM_CALCULATION to list of repository-s3 settings | + +## References + +- [Issue #18240](https://github.com/opensearch-project/OpenSearch/issues/18240): Snapshot repositories don't work with S3 compatible storage in 3.0.0 +- [PR #19220](https://github.com/opensearch-project/OpenSearch/pull/19220): Fix issue with s3-compatible repositories due to missing checksum trailing headers (v3.3.0) +- [Register Snapshot Repository](https://docs.opensearch.org/3.0/api-reference/snapshots/create-repository/): OpenSearch documentation + +## Related Feature Report + +- [Full feature documentation](../../../../features/opensearch/s3-repository.md) diff --git a/docs/releases/v3.4.0/index.md b/docs/releases/v3.4.0/index.md index 4be81271a..fc25dd522 100644 --- a/docs/releases/v3.4.0/index.md +++ b/docs/releases/v3.4.0/index.md @@ -8,6 +8,7 @@ - [Dependency Updates (OpenSearch Core)](features/opensearch/dependency-updates-opensearch-core.md) - 32 dependency updates including Netty 4.2.4 for HTTP/3 readiness - [Lucene Upgrade](features/opensearch/lucene-upgrade.md) - Bump Apache Lucene from 10.3.1 to 10.3.2 with MaxScoreBulkScorer bug fix - [Node Stats Bugfixes](features/opensearch/node-stats-bugfixes.md) - Fix negative CPU usage values in node stats on certain Linux distributions +- [S3 Repository](features/opensearch/s3-repository.md) - Add LEGACY_MD5_CHECKSUM_CALCULATION to opensearch.yml settings for S3-compatible storage - [Security Kerberos Integration](features/opensearch/security-kerberos-integration.md) - Update Hadoop to 3.4.2 and enable Kerberos integration tests for JDK-24+ - [Settings Bugfixes](features/opensearch/settings-bugfixes.md) - Fix duplicate registration of dynamic settings and patch version build issues - [Stats Builder Pattern Deprecations](features/opensearch/stats-builder-pattern-deprecations.md) - Deprecated constructors in 30+ Stats classes in favor of Builder pattern