HDDS-10979. Support STANDARD_IA S3 storage class to accept EC replication config. #8399

SaketaChalamchala · 2025-05-06T09:12:09Z

What changes were proposed in this pull request?

Right now s3 client can create keys/files with only RATIS replication type using the header x-amz-storage-class=STANDARD(RATIS THREE) or REDUCED_REDUNDANCY(RATIS ONE).

Proposed solution allows creation of EC keys/files by supporting x-amz-storage-class=STANDARD_IA(EC/RS-3-2-1024k by default) along with a custom meta header to configure EC replication strings like x-amz-meta-storage-config=rs-6-3-1024k.
Also, changed the replication config resolution order to be consistent with ozone cli which is

Replication config provided by S3 client.
Replication config provided at client side config.
Bucket replication.
Server side default replication config.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10979

How was this patch tested?

Unit Tests
Acceptance test

…tion config.

jojochuang · 2025-05-08T00:35:15Z

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3StorageType.java

  public static S3StorageType fromReplicationConfig(ReplicationConfig config) {
    if (config instanceof ECReplicationConfig) {
-      return S3StorageType.STANDARD;
+      return STANDARD_IA;


map EC type to STANDARD_IA.

This is done here. STANDARD_IA is mapped to EC/RS-3-2-1024k by default.

jojochuang · 2025-05-08T00:37:21Z

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/EndpointBase.java


  private Set<String> excludeMetadataFields =
-      new HashSet<>(Arrays.asList(OzoneConsts.GDPR_FLAG));
+      new HashSet<>(Arrays.asList(OzoneConsts.GDPR_FLAG, STORAGE_CONFIG_HEADER));


what does this mean

We store any x-amz-meta-key=value for an object in KeyArgs.metadata (or ex., Etag). ^^This will exclude storing EC storage config in the metadata because it will be available in replication config. We only want to use this header to determine EC replication config when client uses STANDARD_IA storage class.

jojochuang · 2025-05-08T00:43:58Z

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java


      copyHeader = headers.getHeaderString(COPY_SOURCE_HEADER);
      storageType = headers.getHeaderString(STORAGE_CLASS_HEADER);
+      storageConfig = headers.getHeaderString(CUSTOM_METADATA_HEADER_PREFIX + STORAGE_CONFIG_HEADER);


curious: how will a client add the header to the request? is it supported by s3 cli or hadoop aws cloud connector?

i wonder where we add this information in the developer doc.

All s3 clients should support it. They would just need to add the "x-amz-metadata-storage-class=" to their requests.
This is demonstrated in the acceptance test with AWS cli which does the above underneath.

jojochuang · 2025-05-08T03:05:51Z

hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/util/TestS3Utils.java

+            "INVALID", null, RATIS3REPLICATIONCONFIG, RATIS1REPLICATIONCONFIG));
+    assertThrows(OS3Exception.class, () -> S3Utils.
+        resolveS3ClientSideReplicationConfig(S3StorageType.STANDARD_IA.name(),
+            "INVALID", RATIS3REPLICATIONCONFIG, RATIS1REPLICATIONCONFIG));


shall we also verify for invalid bucketReplConfig?

also invalid s3StorageTypeHeader

How would we pass an invalid ReplicationConfig here? Wouldn't the creation of an invalid ReplicationConfig itself throw an error?

jojochuang · 2025-05-08T03:08:23Z

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3Utils.java

  }

-  public static S3StorageType toS3StorageType(String storageType)
+  public static ReplicationConfig toReplicationConfig(String storageType, String storageConfig)


All tests for toReplicationConfig() are covered by testing resolveS3ClientSideReplicationConfig in TestS3Utils.java

SaketaChalamchala · 2025-05-15T14:28:10Z

@ivandika3 @kerneltime @adoroszlai could you also take a look?

chungen0126

Thanks @SaketaChalamchala for the patch. Left some comments.

chungen0126 · 2025-05-16T10:00:22Z

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3Utils.java

      throws OS3Exception {
    try {
-      return S3StorageType.valueOf(storageType);
+      if (S3StorageType.STANDARD_IA.name().equals(s3StorageType)) {


Suggest doing valueOf() first, then check the enum makes the logic clearer.

try { S3StorageType storageType = S3StorageType.valueOf(s3StorageType); if (S3StorageType.STANDARD_IA.equals(storageType)) { return (!StringUtils.isEmpty(s3StorageConfig)) ? new ECReplicationConfig(s3StorageConfig) : new ECReplicationConfig(S3StorageType.STANDARD_IA.getEcReplicationString()); } else { return ReplicationConfig.fromProtoTypeAndFactor( ReplicationType.toProto(storageType.getType()), ReplicationFactor.toProto(storageType.getFactor())); }

chungen0126 · 2025-05-16T16:26:31Z

Please take a look at BucketEndpoint#addKey. For objects using EC, listObjects still returns the storage class as STANDARD.

kerneltime · 2025-05-16T16:29:43Z

hadoop-ozone/dist/src/main/smoketest/s3/awss3ecstorage.robot

+${ENDPOINT_URL}       http://s3g:9878
+${BUCKET}             generated
+
+*** Test Cases ***


Maybe add a ozone sh command to validate the replication configuration?

adoroszlai

Thanks @SaketaChalamchala for the patch.

hadoop-ozone/dist/src/main/compose/common/ec-test.sh

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3StorageType.java

hadoop-ozone/dist/src/main/smoketest/s3/awss3ecstorage.robot

…d comments.

SaketaChalamchala · 2025-05-20T00:15:03Z

Thanks @chungen0126 @kerneltime and @adoroszlai. Please also review the updated patch.

adoroszlai

Thanks @SaketaChalamchala for updating the patch.

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3StorageType.java

hadoop-ozone/dist/src/main/smoketest/s3/awss3.robot

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java

hadoop-ozone/s3gateway/pom.xml

adoroszlai · 2025-05-21T16:12:52Z

@chungen0126 please take another look

chungen0126

+1 LGTM

adoroszlai · 2025-05-21T19:56:43Z

Thanks @SaketaChalamchala for the patch, @chungen0126, @jojochuang, @kerneltime for the review.

…tion config (apache#8399)

…239-container-reconciliation Commits: 80 commits 5e273a4 HDDS-12977. Fail build on dependency problems (apache#8574) 5081ba2 HDDS-13034. Refactor DirectoryDeletingService to use ReclaimableDirFilter and ReclaimableKeyFilter (apache#8546) e936e4d HDDS-12134. Implement Snapshot Cache lock for OM Bootstrap (apache#8474) 31d13de HDDS-13165. [Docs] Python client developer guide. (apache#8556) 9e6955e HDDS-13205. Bump common-custom-user-data-maven-extension to 2.0.3 (apache#8581) 750b629 HDDS-13203. Bump Bouncy Castle to 1.81 (apache#8580) ba5177e HDDS-13202. Bump build-helper-maven-plugin to 3.6.1 (apache#8579) 07ee5dd HDDS-13204. Bump awssdk to 2.31.59 (apache#8582) e1964f2 HDDS-13201. Bump jersey2 to 2.47 (apache#8578) 81295a5 HDDS-13013. [Snapshot] Add metrics and tests for snapshot operations. (apache#8436) b3d75ab HDDS-12976. Clean up unused dependencies (apache#8521) e0f08b2 HDDS-13179. rename-generated-config fails on re-compile without clean (apache#8569) f388317 HDDS-12554. Support callback on completed reconfiguration (apache#8391) c13a3fe HDDS-13154 Link more Grafana dashboard json files to the Observability user doc (apache#8533) 2a761f7 HDDS-11967. [Docs]DistCP Integration in Kerberized environment. (apache#8531) 81fc4c4 HDDS-12550. Use DatanodeID instead of UUID in NodeManager CommandQueue. (apache#8560) 2360af4 HDDS-13169. Intermittent failure in testSnapshotOperationsNotBlockedDuringCompaction (apache#8553) f19789d HDDS-13170. Reclaimable filter should always reclaim entries when buckets and volumes have already been deleted (apache#8551) 315ef20 HDDS-13175. Leftover reference to OM-specific trash implementation (apache#8563) 902e715 HDDS-13159. Refactor KeyManagerImpl for getting deleted subdirectories and deleted subFiles (apache#8538) 46a93d0 HDDS-12817. Addendum rename ecIndex to replicaIndex in chunkinfo output (apache#8552) 19b9b9c HDDS-13166. Set pipeline ID in BlockExistenceVerifier to avoid cached pipeline with different node (apache#8549) b3ff67c HDDS-13068. Validate Container Balancer move timeout and replication timeout configs (apache#8490) 7a7b9a8 HDDS-13139. Introduce bucket layout flag in freon rk command (apache#8539) 3c25e7d HDDS-12595. Add verifier for container replica states (apache#8422) 6d59220 HDDS-13104. Move auditparser acceptance test under debug (apache#8527) 8e8c432 HDDS-13071. Documentation for Container Replica Debugger Tool (apache#8485) 0e8c8d4 HDDS-13158. Bump junit to 5.13.0 (apache#8537) 8e552b4 HDDS-13157. Bump exec-maven-plugin to 3.5.1 (apache#8534) 168f690 HDDS-13155. Bump jline to 3.30.4 (apache#8535) cc1e4d1 HDDS-13156. Bump awssdk to 2.31.54 (apache#8536) 3bfb7af HDDS-13136. KeyDeleting Service should not run for already deep cleaned snapshots (apache#8525) 006e691 HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache (apache#8141) 568b228 HDDS-13067. Container Balancer delete commands should not be sent with an expiration time in the past (apache#8491) 53673c5 HDDS-11244. OmPurgeDirectoriesRequest should clean up File and Directory tables of AOS for deleted snapshot directories (apache#8509) 07f4868 HDDS-13099. ozone admin datanode list ignores --json flag when --id filter is used (apache#8500) 08c0ab8 HDDS-13075. Fix default value in description of container placement policy configs (apache#8511) 58c87a8 HDDS-12177. Set runtime scope where missing (apache#8513) 10c470d HDDS-12817. Add EC block index in the ozone debug replicas chunk-info (apache#8515) 7027ab7 HDDS-13124. Respect config hdds.datanode.use.datanode.hostname when reading from datanode (apache#8518) b8b226c HDDS-12928. datanode min free space configuration (apache#8388) fd3d70c HDDS-13026. KeyDeletingService should also delete RenameEntries (apache#8447) 4c1c6cf HDDS-12714. Create acceptance test framework for debug and repair tools (apache#8510) fff80fc HDDS-13118. Remove duplicate mockito-core dependency from hdds-test-utils (apache#8508) 10d5555 HDDS-13115. Bump awssdk to 2.31.50 (apache#8505) 360d139 HDDS-13017. Fix warnings due to non-test scoped test dependencies (apache#8479) 1db1cca HDDS-13116. Bump jline to 3.30.3 (apache#8504) 322ca93 HDDS-13025. Refactor KeyDeletingService to use ReclaimableKeyFilter (apache#8450) 988b447 HDDS-5287. Document S3 ACL classes (apache#8501) 64bb29d HDDS-12777. Use module-specific name for generated config files (apache#8475) 54ed115 HDDS-9210. Update snapshot chain restore test to incorporate snapshot delete. (apache#8484) 87dfa5a HDDS-13014. Improve PrometheusMetricsSink#normalizeName performance (apache#8438) 7cdc865 HDDS-13100. ozone admin datanode list --json should output a newline at the end (apache#8499) 9cc4194 HDDS-13089. [snapshot] Add an integration test to verify snapshotted data can be read by S3 SDK client (apache#8495) cb9867b HDDS-13065. Refactor SnapshotCache to return AutoCloseSupplier instead of ReferenceCounted (apache#8473) a88ff71 HDDS-10979. Support STANDARD_IA S3 storage class to accept EC replication config (apache#8399) 6ec8f85 HDDS-13080. Improve delete metrics to show number of timeout DN command from SCM (apache#8497) 3bb8858 HDDS-12378. Change default hdds.scm.safemode.min.datanode to 3 (apache#8331) 0171bef HDDS-13073. Set pipeline ID in checksums verifier to avoid cached pipeline with different node (apache#8480) 5c7726a HDDS-11539. OzoneClientCache `@PreDestroy` is never called (apache#8493) a8ed19b HDDS-13031. Implement a Flat Lock resource in OzoneManagerLock (apache#8446) e9e8b30 HDDS-12935. Support unsigned chunked upload with STREAMING-UNSIGNED-PAYLOAD-TRAILER (apache#8366) 7590268 HDDS-13079. Improve logging in DN for delete operation. (apache#8489) 435fe7e HDDS-12870. Fix listObjects corner cases (apache#8307) eb5dabd HDDS-12926. Remove *.tmp.* exclusion in DU (apache#8486) eeb98c7 HDDS-13030. Snapshot Purge should unset deep cleaning flag for next 2 snapshots in the chain (apache#8451) 6bf121e HDDS-13032. Support proper S3OwnerId representation (apache#8478) 5d1b43d HDDS-13076. Refactor OzoneManagerLock class to rename Resource class to LeveledResource (apache#8482) bafe6d9 HDDS-13064. [snapshot] Add test coverage for SnapshotUtils.isBlockLocationInfoSame() (apache#8476) 7035846 HDDS-13040. Add user doc highlighting the difference between Ozone ACL and S3 ACL. (apache#8457) 1825cdf HDDS-13049. Deprecate VolumeName & BucketName in OmKeyPurgeRequest and prevent Key version purge on Block Deletion Failure (apache#8463) 211c76c HDDS-13060. Change NodeManager.addDatanodeCommand(..) to use DatanodeID (apache#8471) f410238 HDDS-13061. Add test for key ACL operations without permission (apache#8472) d1a2f48 HDDS-13057. Increment block delete processed transaction counts regardless of log level (apache#8466) 0cc6fcc HDDS-13043. Replace != with assertNotEquals in TestSCMContainerPlacementRackAware (apache#8470) e1c779a HDDS-13051. Use DatanodeID in server-scm. (apache#8465) 35e1126 HDDS-13042. [snapshot] Add future proofing test cases for unsupported file system API (apache#8458) 619c05d HDDS-13008. Exclude same SST files when calculating full snapdiff (apache#8423) 21b49d3 HDDS-12965. Fix warnings about "used undeclared" dependencies (apache#8468) 8136119 HDDS-13048. Create new module for Recon integration tests (apache#8464) Conflicts: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManager.java

HDDS-10979. Support STANDARD_IA S3 storage class to accept EC replica…

58d4603

…tion config.

ivandika3 added the s3 S3 Gateway label May 6, 2025

jojochuang requested a review from chungen0126 May 8, 2025 00:17

jojochuang reviewed May 8, 2025

View reviewed changes

saketa and others added 4 commits May 13, 2025 10:42

HDDS-10979. Added acceptance test.

718bea2

HDDS-10979. Fixed test and checkstyle.

b0f7a6e

Merge branch 'apache:master' into HDDS-10979

9f6a48d

HDDS-10979. Fixed error message in integration test.

57edbeb

SaketaChalamchala marked this pull request as ready for review May 14, 2025 17:19

SaketaChalamchala requested a review from jojochuang May 15, 2025 14:28

ivandika3 requested a review from xichen01 May 15, 2025 14:53

chungen0126 reviewed May 16, 2025

View reviewed changes

kerneltime reviewed May 16, 2025

View reviewed changes

adoroszlai reviewed May 17, 2025

View reviewed changes

saketa added 2 commits May 19, 2025 17:07

HDDS-10979. Optimized S3StorageType. Handled bucket HEAD op. Addresse…

f9187da

…d comments.

HDDS-10979. Changed Execute and check rc to Execute in acceptance test.

b273851

adoroszlai reviewed May 20, 2025

View reviewed changes

HDDS-10979. Fixed awss3 acceptance test. Addressed comments.

5773d58

adoroszlai reviewed May 20, 2025

View reviewed changes

hadoop-ozone/s3gateway/pom.xml Outdated Show resolved Hide resolved

adoroszlai added 2 commits May 21, 2025 15:19

Merge remote-tracking branch 'origin/master' into HDDS-10979

4581468

move OzoneClientUtils to ozone-client

7f66cd6

adoroszlai approved these changes May 21, 2025

View reviewed changes

SaketaChalamchala requested review from chungen0126 and kerneltime May 21, 2025 15:28

chungen0126 approved these changes May 21, 2025

View reviewed changes

adoroszlai merged commit a88ff71 into apache:master May 21, 2025
43 checks passed

Tejaskriya pushed a commit to Tejaskriya/ozone that referenced this pull request May 29, 2025

HDDS-10979. Support STANDARD_IA S3 storage class to accept EC replica…

3c24ac8

…tion config (apache#8399)

HDDS-10979. Support STANDARD_IA S3 storage class to accept EC replication config. #8399

HDDS-10979. Support STANDARD_IA S3 storage class to accept EC replication config. #8399

Uh oh!

Conversation

SaketaChalamchala commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SaketaChalamchala May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SaketaChalamchala commented May 15, 2025

Uh oh!

chungen0126 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chungen0126 commented May 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SaketaChalamchala commented May 20, 2025

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adoroszlai commented May 21, 2025

Uh oh!

chungen0126 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adoroszlai commented May 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

SaketaChalamchala commented May 6, 2025 •

edited

Loading

SaketaChalamchala May 14, 2025 •

edited

Loading