feat: Add support for using ephemeral credentials for S3 artifacts. F…

…ixes argoproj#5446 (argoproj#5446)

Signed-off-by: Raymond Chow <[email protected]>

docs/configure-artifact-repository.md

@@ -82,6 +82,16 @@ an access key, you will need to create a user with just the permissions you want
 to associate with the access key. Otherwise, you can just create an access key
 using your existing user account.
 
+Although users are not able to have a reduced scope, S3 access grants created
+via a user are. These credentials however, are temporary, and therefore need
+to be periodically generated/refreshed by an external mechanism.
+
+The example below creates a bucket, a user that is able to access the bucket,
+and a set of access keys. The procedure is the same if you wish to use an IAM role,
+except you will need to instead create a role and attach the same policy as below.
+Please note that role credentials are temporary and will need to be periodically
+generated/refreshed by an external mechanism.
+
 ```bash
 $ export mybucket=bucket249
 $ cat > policy.json <<EOF
@@ -130,6 +140,34 @@ specify a bucket region in your workflow configuration.
     ...
 ```
 
+### AWS S3 with IAM credentials or S3 access grants
+
+Below is an example of how to use an credentials from an [S3 access grant](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-grants.html)
+or temporary credentials from an IAM role. If you are using IAM user, omit the `sessionTokenSecret` field.
+
+```yaml
+artifacts:
+  - name: my-output-artifact
+    path: /my-output-artifact
+    s3:
+      endpoint: s3.amazonaws.com
+      bucket: my-s3-bucket
+      # NOTE that, by default, all output artifacts are automatically tarred and
+      # gzipped before saving. So as a best practice, .tgz or .tar.gz
+      # should be incorporated into the key name so the resulting file
+      # has an accurate file extension.
+      key: path/in/bucket/my-output-artifact.tgz
+      accessKeySecret:
+        name: my-s3-credentials
+        key: accessKey
+      secretKeySecret:
+        name: my-s3-credentials
+        key: secretKey
+      sessionTokenSecret:
+        name: my-s3-credentials
+        key: sessionToken
+```
+
 ### AWS S3 IRSA
 
 If you wish to use S3 IRSA instead of passing in an `accessKey` and `secretKey`, you need to annotate the service account of both the running workflow (in order to save logs/artifacts) and the argo-server pod (in order to retrieve the logs/artifacts).

docs/executor_swagger.md

@@ -3795,6 +3795,7 @@ be retried and the retry strategy will be ignored |  |
 | region | string| `string` |  | | Region contains the optional bucket region |  |
 | roleARN | string| `string` |  | | RoleARN is the Amazon Resource Name (ARN) of the role to assume. |  |
 | secretKeySecret | [SecretKeySelector](#secret-key-selector)| `SecretKeySelector` |  | |  |  |
+| sessionTokenSecret | [SecretKeySelector](#secret-key-selector)| `SecretKeySelector` |  | |  |  |
 | useSDKCreds | boolean| `bool` |  | | UseSDKCreds tells the driver to figure out credentials based on sdk defaults. |  |
 
 

docs/fields.md

@@ -3531,6 +3531,7 @@ S3Artifact is the location of an S3 artifact
 |`region`|`string`|Region contains the optional bucket region|
 |`roleARN`|`string`|RoleARN is the Amazon Resource Name (ARN) of the role to assume.|
 |`secretKeySecret`|[`SecretKeySelector`](#secretkeyselector)|SecretKeySecret is the secret selector to the bucket's secret key|
+|`sessionTokenSecret`|[`SecretKeySelector`](#secretkeyselector)|SessionTokenSecret is used for credentials which are ephemeral (e.g. IAM assume role or S3 access grant)|
 |`useSDKCreds`|`boolean`|UseSDKCreds tells the driver to figure out credentials based on sdk defaults.|
 
 ## ValueFrom
@@ -4263,6 +4264,7 @@ S3ArtifactRepository defines the controller configuration for an S3 artifact rep
 |`region`|`string`|Region contains the optional bucket region|
 |`roleARN`|`string`|RoleARN is the Amazon Resource Name (ARN) of the role to assume.|
 |`secretKeySecret`|[`SecretKeySelector`](#secretkeyselector)|SecretKeySecret is the secret selector to the bucket's secret key|
+|`sessionTokenSecret`|[`SecretKeySelector`](#secretkeyselector)|SessionTokenSecret is used for credentials which are ephemeral (e.g. IAM assume role or S3 access grant)|
 |`useSDKCreds`|`boolean`|UseSDKCreds tells the driver to figure out credentials based on sdk defaults.|
 
 ## MutexHolding

go.mod

@@ -2,6 +2,11 @@ module github.com/argoproj/argo-workflows/v3
 
 go 1.21
 
+replace (
+	github.com/argoproj/pkg => github.com/atlassian-forks/argo-pkg v0.0.0-20240104004059-b14f36544f6d
+	github.com/argoproj/pkg/s3 => github.com/atlassian-forks/argo-pkg/s3 v0.0.0-20240104004059-b14f36544f6d
+)
+
 require (
 	cloud.google.com/go/storage v1.36.0
 	github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.4.0
@@ -47,7 +52,7 @@ require (
 	github.com/sirupsen/logrus v1.9.3
 	github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966
 	github.com/soheilhy/cmux v0.1.5
-	github.com/spf13/cobra v1.7.0
+	github.com/spf13/cobra v1.8.0
 	github.com/spf13/pflag v1.0.5
 	github.com/spf13/viper v1.18.2
 	github.com/stretchr/testify v1.8.4
@@ -78,6 +83,8 @@ require (
 
 require (
 	github.com/alibabacloud-go/debug v0.0.0-20190504072949-9472017b5c68 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.10.4 // indirect
+	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.21.5 // indirect
 	github.com/containerd/stargz-snapshotter/estargz v0.14.3 // indirect
 	github.com/evilmonkeyinc/jsonpath v0.8.1 // indirect
 	github.com/go-logr/stdr v1.2.2 // indirect
@@ -140,26 +147,25 @@ require (
 	github.com/ajg/form v1.5.1 // indirect
 	github.com/andybalholm/brotli v1.0.4 // indirect
 	github.com/awalterschulze/gographviz v0.0.0-20200901124122-0eecad45bd71 // indirect
-	github.com/aws/aws-sdk-go v1.45.1 // indirect
-	github.com/aws/aws-sdk-go-v2 v1.16.7 // indirect
-	github.com/aws/aws-sdk-go-v2/config v1.15.14 // indirect
-	github.com/aws/aws-sdk-go-v2/credentials v1.12.9 // indirect
-	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.8 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.14 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.8 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/ini v1.3.15 // indirect
+	github.com/aws/aws-sdk-go-v2 v1.24.0 // indirect
+	github.com/aws/aws-sdk-go-v2/config v1.26.2 // indirect
+	github.com/aws/aws-sdk-go-v2/credentials v1.16.13 // indirect
+	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.14.10 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/configsources v1.2.9 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.5.9 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/ini v1.7.2 // indirect
 	github.com/aws/aws-sdk-go-v2/service/ecr v1.17.8 // indirect
 	github.com/aws/aws-sdk-go-v2/service/ecrpublic v1.13.8 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.8 // indirect
-	github.com/aws/aws-sdk-go-v2/service/sso v1.11.12 // indirect
-	github.com/aws/aws-sdk-go-v2/service/sts v1.16.9 // indirect
-	github.com/aws/smithy-go v1.12.0 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.10.9 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sso v1.18.5 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sts v1.26.6 // indirect
+	github.com/aws/smithy-go v1.19.0 // indirect
 	github.com/awslabs/amazon-ecr-credential-helper/ecr-login v0.0.0-20220706184558-ce46abcd012b // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/cespare/xxhash/v2 v2.2.0 // indirect
 	github.com/chai2010/gettext-go v0.0.0-20160711120539-c6fed771bfd5 // indirect
 	github.com/chrismellard/docker-credential-acr-env v0.0.0-20220327082430-c57b701bfc08 // indirect
-	github.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect
+	github.com/cpuguy83/go-md2man/v2 v2.0.3 // indirect
 	github.com/creack/pty v1.1.21
 	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
 	github.com/daviddengcn/go-colortext v0.0.0-20160507010035-511bcaf42ccd // indirect