AWS: Add support for enabling access to S3 Requester Pays bucket

[marora90]

Took a stab at resolving #11912

• Adds a new option s3.requester-pays-enabled which defaults to false.
• Modify the default aws client factories to add the requester pays header via override configuration if the option is enabled

Note will also need to document the option, hoping to get some feedback on the PR before that esp. since this is my first time working with iceberg.

[steveloughran]

This isn't the way to do it. Just call withRequesterPays(true) when creating the aws s3 client and let it do all the work

[marora90]

This isn't the way to do it. Just call withRequesterPays(true) when creating the aws s3 client and let it do all the work

Hi,

Thank you for reviewing the PR. I was unable to locate withRequesterPays(true) or any other built-in option to set the RequesterPays flag globally on the S3 client builder.

I do observe this method on individual requests such as UploadPartRequest and ListObjectsV2Request, etc.

However, in my opinion, this approach would be more challenging to maintain as it would require setting the option correctly before each request, including those that may be added in the future.

[marora90]

Hi @steveloughran

I just wanted to follow up on my previous comment and would love to hear any additional thoughts or suggestions you might have.

I looked into the Hadoop S3A implementation, and it seems to use a similar approach by setting the requester-pays header, which I believe you had also reviewed.

Ref:

• HADOOP-14661. Add S3 requester pays bucket support to S3A hadoop#3962
• https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java#L245-L248

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

[github-actions]

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.