Skip to content
Closed
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
package org.apache.hadoop.fs.s3a;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you move to org.apache.hadoop.fs.s3a.auth


import org.apache.commons.lang3.StringUtils;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, we have some layout rules for imports. here's my full intellij settings for this if it helps
https://gist.github.com/steveloughran/817dd90e0f1775ce2b6f24684dfb078c

import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.auth.WebIdentityTokenCredentialsProvider;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.security.ProviderUtils;
import org.slf4j.Logger;

import java.io.IOException;

/**
* WebIdentityTokenCredentialsProvider supports static configuration
* of OIDC token path, role ARN and role session name.
*
*/
//@InterfaceAudience.Public
//@InterfaceStability.Stable
public class OIDCTokenCredentialsProvider implements AWSCredentialsProvider {
public static final String NAME
= "org.apache.hadoop.fs.s3a.OIDCTokenCredentialsProvider";

//these are the parameters to document and to pass along with the class
//usually from import static org.apache.hadoop.fs.s3a.Constants.*;
public static final String JWT_PATH = "fs.s3a.jwt.path";
public static final String ROLE_ARN = "fs.s3a.role.arn";
public static final String SESSION_NAME = "fs.s3a.session.name";

/** Reuse the S3AFileSystem log. */
private static final Logger LOG = S3AFileSystem.LOG;

private String jwtPath;
private String roleARN;
private String sessionName;
private IOException lookupIOE;

public OIDCTokenCredentialsProvider(Configuration conf) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should credential providers be allowed to raise IOEs? we should be able to fix that

try {
Configuration c = ProviderUtils.excludeIncompatibleCredentialProviders(
conf, S3AFileSystem.class);
this.jwtPath = S3AUtils.lookupPassword(c, JWT_PATH, null);
this.roleARN = S3AUtils.lookupPassword(c, ROLE_ARN, null);
this.sessionName = S3AUtils.lookupPassword(c, SESSION_NAME, null);
} catch (IOException e) {
lookupIOE = e;
}
}

public AWSCredentials getCredentials() {
if (lookupIOE != null) {
// propagate any initialization problem
throw new CredentialInitializationException(lookupIOE.toString(),
lookupIOE);
}

LOG.debug("jwtPath {} roleARN {}", jwtPath, roleARN);

if (!StringUtils.isEmpty(jwtPath) && !StringUtils.isEmpty(roleARN)) {
final AWSCredentialsProvider credentialsProvider =
WebIdentityTokenCredentialsProvider.builder()
.webIdentityTokenFile(jwtPath)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will handle local files only, so won't work for jobs across a cluster unless the token is already there.

either cluster fs paths will be needed (download locally and then reference) or we require it on the host of the user launching a job and then include the token data in a delegation token which goes with it. that's a lot more powerful -but a lot more work. best to leave that for a followup patch

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure i get your point. Our service runs accross multiple Kubernetes pods (replicas) using a ServiceAccount, so that any of those pods is automatically attached a volume pointing to a token file created with the same ServiceAcocunt signature. Perhaps i am missing other use-cases within the Hadoop ecosystem?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i was just wondering how the secrets get around. for other credentials we can pick them up from the user launching, say, a distcp job, and they will get passed round. alternatively, they can go into a cluster FS like hdfs.

if it works with your k8s setup, then the docs should say "mount a shared volume in your containers". support for credential propagation can be added by someone else when they needed it

.roleArn(roleARN)
.roleSessionName(sessionName)
.build();
return credentialsProvider.getCredentials();
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: same line as the }

else throw new CredentialInitializationException(
"OIDC token path or role ARN is null");
}

public void refresh() {}

@Override
public String toString() {
return getClass().getSimpleName();
}

}