Skip to content

[AWS] Use credentials and config from AWS SDK file #1114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
aitorarjona wants to merge 15 commits into from
Closed

[AWS] Use credentials and config from AWS SDK file #1114

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
95437b1
Add support for SSO credentials in AWS Lambda backend
aitorarjona Jun 20, 2023
1fc59df
Add support for SSO credentials in AWS s3
aitorarjona Jun 20, 2023
ce8f499
Remove pin in aws reqs
aitorarjona Jul 6, 2023
bf1ca7e
Add deprectation warning when using aws secrets in lithops config
aitorarjona Jul 11, 2023
30941a0
Merge master
aitorarjona Jul 11, 2023
19831b3
Update lambda backend to support aws config profiles
aitorarjona Jul 12, 2023
118b1e6
Update changelog
aitorarjona Jul 12, 2023
5ba0c46
Update docs
aitorarjona Jul 12, 2023
dc4dce2
Fixes review
aitorarjona Jul 24, 2023
8d4ff11
Add support for unsinged boto3 client
aitorarjona Aug 22, 2023
5344797
Merge upstream master
aitorarjona Aug 22, 2023
06b1dc8
Docs update, removed aws credentials deprecration notice
aitorarjona Sep 13, 2023
e792c2a
Remove secrets from aws lambda payload
aitorarjona Sep 13, 2023
36de64c
Fix automatic storage_bucket name for AWS with config profiles
aitorarjona Sep 13, 2023
506209f
Merge master
aitorarjona Sep 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 62 additions & 13 deletions docs/source/compute_config/aws_lambda.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ python3 -m pip install lithops[aws]
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:*",
Expand All @@ -44,34 +43,72 @@ python3 -m pip install lithops[aws]

7. Choose **Lambda** on the use case list and click **Next: Permissions**. Select the policy created before (`lithops-policy`). Click **Next: Tags** and **Next: Review**. Type a role name, for example `lithops-execution-role`. Click on *Create Role*.

## Configuration
## AWS Credential setup

6. Edit your lithops config and add the following keys:
Lithops loads AWS credentials as specified in the [boto3 configuration guide](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html).

In summary, you can use the following settings:

1. Provide credentials via the `~/.aws/config` file. **This is the preferred option to configure AWS credentials for use with Lithops**:

You can run `aws configure` command if the AWS CLI is installed to setup the credentials.
Comment on lines +52 to +54
Copy link
Member

@JosepSampe JosepSampe Sep 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How the ~/.aws/config looks like in this case? are the keys going into a default profile by defaut? or are the keys set in the file without a profile?

I mean, after calling aws configure, you get this:?

aws_access_key_id=XXXXXXXXX
aws_secret_access_key=XXXXXX

or something like this:?

[profile default]
aws_access_key_id=XXXXXXXXXXXXXXX
aws_secret_access_key=XXXXXXXXXXXXXXXX

Copy link
Member

@JosepSampe JosepSampe Sep 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if in this case it makes sense to force the user to provide a profile_name with aws configure --profile my-unique-profile-name and then configure lithops like in the SSO approach, with:

lithops:
    backend: aws_lambda

aws:
    config_profile: my-unique-profile-name

aws_lambda:
    execution_role: <EXECUTION_ROLE_ARN>
    region: <REGION_NAME>


2. Provide credentials via environment variables:

Lithops needs at least `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_DEFAULT_REGION` environment variables set.
Comment on lines +56 to +58
Copy link
Member

@JosepSampe JosepSampe Sep 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe in this option you can put a config example (and maybe remove AWS_DEFAULT_REGION?):

lithops:
    backend: aws_lambda

aws_lambda:
    execution_role: <EXECUTION_ROLE_ARN>
    region: <REGION_NAME>


3. Provide the credentials in the `aws` section of the Lithops config file:
```yaml
lithops:
backend: aws_lambda

aws:
region: <REGION_NAME>
access_key_id: <AWS_ACCESS_KEY_ID>
secret_access_key: <AWS_SECRET_ACCESS_KEY>
region: <REGION_NAME>

aws_lambda:
execution_role: <EXECUTION_ROLE_ARN>
region: <REGION_NAME>
```

## Summary of configuration keys for AWS
### Setup for SSO-based users

### AWS
Users using SSO-based accounts do not require an IAM user, and have temporal session access tokens instead. To configure access to SSO-based accounts, you can configure a profile in the `~/.aws/config` file for using SSO authentication:

```yaml
[profile my-sso-profile]
sso_start_url = https://XXXXXXXX.awsapps.com/start
sso_region = us-east-1
sso_account_id = XXXXXXXXXXX
sso_role_name = XXXXXXXXXXXXXXXXX
region = us-east-1
```

Then, you can log in or refresh your credentials by using the sso login command:

```
$ aws sso login --profile my-sso-profile
```

To use this profile, you must specify it in the `aws` section of the Lithops config file:

|Group|Key|Default|Mandatory|Additional info|
|---|---|---|---|---|
|aws | region | |yes | AWS Region. For example `us-east-1` |
|aws | access_key_id | |yes | Account access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. |
|aws | secret_access_key | |yes | Account secret access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. |
|aws | session_token | |no | Session token for temporary AWS credentials |
|aws | account_id | |no | *This field will be used if present to retrieve the account ID instead of using AWS STS. The account ID is used to format full image names for container runtimes. |
```yaml
lithops:
backend: aws_lambda

aws:
config_profile: my-sso-profile

aws_lambda:
execution_role: <EXECUTION_ROLE_ARN>
region: <REGION_NAME>
```

More info [here](https://docs.aws.amazon.com/cli/latest/userguide/sso-configure-profile-token.html).


## Summary of configuration keys for AWS Lambda

### AWS Lambda

Expand All @@ -90,6 +127,18 @@ aws_lambda:
| aws_lambda | ephemeral_storage | 512 | no | Ephemeral storage (`/tmp`) size in MB (must be between 512 MB and 10240 MB) |
| aws_lambda | env_vars | {} | no | List of {name: ..., value: ...} pairs for Lambda instance environment variables |

### AWS

|Group| Key | Default | Mandatory | Additional info |
|---|-------------------|----------|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|aws | region | | yes | AWS Region. For example `us-east-1` |
|aws | config_profile | "default" | no | AWS SDK [configuration profile](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#using-a-configuration-file) name. |
|aws | access_key_id | | no | Account access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. |
|aws | secret_access_key | | no | Account secret access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. |
|aws | session_token | | no | Session token for temporary AWS credentials |
|aws | account_id | | no | *This field will be used if present to retrieve the account ID instead of using AWS STS. The account ID is used to format full image names for container runtimes. |


## Additional configuration

### VPC
Expand Down
75 changes: 66 additions & 9 deletions docs/source/storage_config/aws_s3.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,22 +29,79 @@ Lithops with AWS S3 as storage backend.
secret_access_key : <SECRET_ACCESS_KEY>
```

## AWS Credential setup

Lithops loads AWS credentials as specified in the [boto3 configuration guide](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html).

In summary, you can use the following settings:

1. Provide credentials via the `~/.aws/config` file. **This is the preferred option to configure AWS credentials for use with Lithops**:

You can run `aws configure` command if the AWS CLI is installed to setup the credentials.

2. Provide credentials via environment variables:

Lithops needs at least `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_DEFAULT_REGION` environment variables set.

3. Provide the credentials in the `aws` section of the Lithops config file **This option is not ideal and will be removed in future Lithops releases!**:
```yaml
lithops:
storage: aws_s3

aws:
access_key_id: <AWS_ACCESS_KEY_ID>
secret_access_key: <AWS_SECRET_ACCESS_KEY>
region: <REGION_NAME>
```

### Setup for SSO-based users

Users using SSO-based accounts do not require an IAM user, and have temporal session access tokens instead. To configure access to SSO-based accounts, you can configure a profile in the `~/.aws/config` file for using SSO authentication:

```yaml
[profile my-sso-profile]
sso_start_url = https://XXXXXXXX.awsapps.com/start
sso_region = us-east-1
sso_account_id = XXXXXXXXXXX
sso_role_name = XXXXXXXXXXXXXXXXX
region = us-east-1
```

Then, you can log in or refresh your credentials by using the sso login command:

```
$ aws sso login --profile my-sso-profile
```

To use this profile, you must specify it in the `aws` section of the Lithops config file:

```yaml
lithops:
storage: aws_s3

aws:
config_profile: my-sso-profile
```


## Summary of configuration keys for AWS:

### AWS:

|Group|Key|Default|Mandatory|Additional info|
|---|---|---|---|---|
|aws | region | |yes | AWS Region. For example `us-east-1` |
|aws | access_key_id | |yes | Account access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. |
|aws | secret_access_key | |yes | Account secret access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. |
|aws | session_token | |no | Session token for temporary AWS credentials |

### Summary of configuration keys for AWS S3:
### AWS S3

|Group|Key|Default|Mandatory|Additional info|
|---|---|---|---|---|
|aws_s3 | region | |no | Region of your Bcuket. e.g `us-east-1`, `eu-west-1`, etc. Lithops will use the region set under the `aws` section if it is not set here |
|aws_s3 | storage_bucket | | no | The name of a bucket that exists in you account. This will be used by Lithops for intermediate data. Lithops will automatically create a new one if it is not provided |

### AWS

|Group| Key | Default | Mandatory | Additional info |
|---|-------------------|----------|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|aws | region | | yes | AWS Region. For example `us-east-1` |
|aws | config_profile | "default" | no | AWS SDK [configuration profile](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#using-a-configuration-file) name. |
|aws | access_key_id | | no | Account access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. |
|aws | secret_access_key | | no | Account secret access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. |
|aws | session_token | | no | Session token for temporary AWS credentials |
|aws | account_id | | no | *This field will be used if present to retrieve the account ID instead of using AWS STS. The account ID is used to format full image names for container runtimes. |

Loading