Allow pinning to WebIdentityTokenCredentialsProvider in legacy S3 client

[nineinchnick]

Description

Allow users to only use the WebIdentityTokenCredentialsProvider instead of the default credentials provider chain.

This is preferred over #22007. This at least makes the workaround for #15267 easier, I'm not sure if it will allow closing that issue. I haven't updated the docs yet, I'll wait for some initial feedback first.

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
(x) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

# Section
* Fix some things. ({issue}`issuenumber`)

[nineinchnick]

@electrum can you merge this?

[nineinchnick]

I tested this by deploying Trino on EKS, using https://github.com/binayakd/trino-on-eks
and these chart values:

image:
  repository: nineinchnick/test
  tag: 449-SNAPSHOT-amd64
server:
  workers: 1
additionalCatalogs:
  s3_hive: |-
    connector.name=hive
    hive.metastore.uri=thrift://metastore:9083
    fs.native-s3.enabled=false
    hive.s3.region=eu-central-1
    # contains trino.s3.use-web-identity-token-credentials-provider=true
    hive.config.resources=/etc/hadoop/conf/core-site.xml
coordinator:
  additionalJVMConfig:
  - -XX:+UnlockDiagnosticVMOptions
  - -XX:G1NumCollectionsKeepPinned=10000000
worker:
  additionalJVMConfig:
  - -XX:+UnlockDiagnosticVMOptions
  - -XX:G1NumCollectionsKeepPinned=10000000
configMounts:
- name: hadoop
  configMap: hadoop
  path: /etc/hadoop/conf
additionalLogProperties:
- com.amazonaws=DEBUG

Contents of core-site.xml, added in a config map kubectl -n trino create configmap hadoop --from-file=core-site.xml:

<?xml version="1.0"?>
<configuration>
    <property>
        <name>trino.s3.use-web-identity-token-credentials-provider=true</name>
        <value>true</value>
    </property>
</configuration>

I checked the logs and I see it using only one provider:

2024-06-04T08:17:56.748Z        DEBUG   SplitRunner-20240604_081748_00002_5my8g.0.0.0-1-103     com.amazonaws.request   Sending Request: POST https://sts.eu-central-1.amazonaws.com /
...
2024-06-04T08:17:57.333Z        DEBUG   SplitRunner-20240604_081748_00002_5my8g.0.0.0-1-103     com.amazonaws.auth.AWSCredentialsProviderChain  Loading credentials from WebIdentityTokenCredentialsProvider

[nineinchnick]

@ebyhr could you merge this?

[kevinrood]

@ebyhr thanks for merging this! 🙌

It looks like this change will make it in to release 450, is that correct?

If so, when will that release officially land?

[hashhar]

this reads from Hadoop config, not Trino's FS config. How is this exposed to users?

[nineinchnick]

It is exposed through the Hadoop config. See my previous comment on how I tested it.

[hashhar]

there's no config like this in Trino legacy S3 FS.

So the docs (and release notes) are wrong. @nineinchnick can you please update / fix this (both docs and adding a config to expose setting the hadoop config to users).

[nineinchnick]

#23544