HDDS-13345. Security Token Service (STS) - S3 compatible API for temporary S3 credentials

[len548]

What changes were proposed in this pull request?

This pull request introduces a new S3 compatible API called Security Token Service (STS). STS allows privileged users to generate temporary S3 credentials with:

• Limited duration
• Restricted to specific bucket/prefix paths
• Restricted to specific S3 operations
• Issuing credentials either to self or another identity

With Amazon AWS, there is a central service which has the ability to generate Security Tokens that span resources across services.

As this introduction of this API is too big to be in one ticket and PR, it is divided into sub-tasks. This PR covers only the first sub-task which is creating the API skeleton. So this PR addresses:

• A new STS API endpoint has been added to the S3 Gateway, available on port 9878.
• When a client calls STS API with assume-role action, it will return a mock response with: aws credentials and session token
• Other operations are handled as 'not implemented' and it throws an error in those cases.
• All the rest of incomplete parts of the API are listed as TODOs including actual credential generation, expired token management and other verifications. They will be covered in other subsequent PRs.

What is the link to the Apache JIRA

HDDS-13345

How was this patch tested?

The patch was tested by adding a new unit test for the STS API.

Manual testing was also performed to verify that the STS API works as expected with S3 clients.

[adoroszlai]

Thanks @len548 for working on this.

• Please resolve conflict after merging master into your branch. (You can also rebase before opening PRs.)
• Please open PR after ensuring CI run is clean in your fork. There are various kinds of failures, see: https://github.com/len548/ozone/actions/runs/16252417025.
• I don't think we should add the new service on port 9878, where S3 content is served. If someone has a bucket named sts, it won't be accessible anymore. We could add it on 19878, or may need to introduce a third one.

[len548]

@adoroszlai Thanks for the review.

I don't think we should add the new service on port 9878, where S3 content is served. If someone has a bucket named sts, it won't be accessible anymore. We could add it on 19878, or may need to introduce a third one.

If I introduce it on 19878, there is already s3secret service. Should I introduce it under the s3secret service as sub-service of it? Or completely separate service independent of s3secret endpoint? I guess it would be easier and simpler to do it under s3secret, but do you think there would be any drawback?

[adoroszlai]

If I introduce it on 19878, there is already s3secret service. Should I introduce it under the s3secret service as sub-service of it? Or completely separate service independent of s3secret endpoint?

s3secret is mapped to /secret, not /, so we can add another servlet mapped to /sts here:

ozone/hadoop-ozone/s3gateway/src/main/resources/webapps/s3g-web/WEB-INF/web.xml

Lines 17 to 29 in d76f4ee

	<servlet>
	<servlet-name>secret</servlet-name>
	<servlet-class>org.glassfish.jersey.servlet.ServletContainer</servlet-class>
	<init-param>
	<param-name>javax.ws.rs.Application</param-name>
	<param-value>org.apache.hadoop.ozone.s3secret.Application</param-value>
	</init-param>
	<load-on-startup>1</load-on-startup>
	</servlet>
	<servlet-mapping>
	<servlet-name>secret</servlet-name>
	<url-pattern>/secret/*</url-pattern>
	</servlet-mapping>

[jojochuang]

How do we test the STS implementation for full compatibility?
Looking at the two most popular s3 compatibility test suites, MinIO Mint doesn't seems to support STS. But Ceph S3 test suite seems to support: https://github.com/ceph/s3-tests/blob/f1f55380714a3396fb1fa9935a77525a8311996c/s3tests.conf.SAMPLE#L129

Worth checking later.

[jojochuang]

this would collide with RootEndpoint? I guess you want it at /sts

[jojochuang]

Apart from the endpoint path, the PR looks good to me. A lot is to be added later and this is a new code so it doesn't bother me much.

[peterxcli]

let's use the EndpointBuilder to write the test:

EndpointBuilder<S3STSEndpoint> builder =
    new EndpointBuilder<>(S3STSEndpoint::new);

S3STSEndpoint endpoint = builder
    .setClient(clientStub)
    .setContext(context)
    .build();

[len548]

The EndpointBuilder class is compatible with only those classes that extend EndpointBase. (See EndpointBuilder.java line 31).
Because S3STSEndpoint extends S3EndpointBase, EndpointBuilder cannot be used with S3STSEndpoint.

[peterxcli]

Ah, thanks for the info. Just out of curiosity, is there any specific reason you created a new S3STSEndpointBase instead of using the existing EndpointBase directly?

[len548]

STS service is similar to s3secret. Both services provide AWS credentials. So it was a intuitively natural idea to follow how s3secret is designed. So the idea of using the S3STSEndpointBase comes from S3SecretEndpointBase in s3secret. They are identical classes with different names in this PR but the next PR is supposed to add some utility methods to S3STSEndpointBase. These are necessary for operations of S3STSEndpoint. Also EndpointBase is not suitable for credential management endpoint like STS and S3Secret because it is to provide utility methods for s3 content operation endpoints such as BucketEndpoint. I hope it answers your question. Thank you for asking :)

[adoroszlai]

Is this superseded by the work targeted at feature branch HDDS-13323-sts (#9254 and follow-ups)?