Skip to content

[AWS] Use credentials and config from AWS SDK file #1114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
aitorarjona wants to merge 15 commits into from
Closed

[AWS] Use credentials and config from AWS SDK file #1114

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
95437b1
Add support for SSO credentials in AWS Lambda backend
aitorarjona Jun 20, 2023
1fc59df
Add support for SSO credentials in AWS s3
aitorarjona Jun 20, 2023
ce8f499
Remove pin in aws reqs
aitorarjona Jul 6, 2023
bf1ca7e
Add deprectation warning when using aws secrets in lithops config
aitorarjona Jul 11, 2023
30941a0
Merge master
aitorarjona Jul 11, 2023
19831b3
Update lambda backend to support aws config profiles
aitorarjona Jul 12, 2023
118b1e6
Update changelog
aitorarjona Jul 12, 2023
5ba0c46
Update docs
aitorarjona Jul 12, 2023
dc4dce2
Fixes review
aitorarjona Jul 24, 2023
8d4ff11
Add support for unsinged boto3 client
aitorarjona Aug 22, 2023
5344797
Merge upstream master
aitorarjona Aug 22, 2023
06b1dc8
Docs update, removed aws credentials deprecration notice
aitorarjona Sep 13, 2023
e792c2a
Remove secrets from aws lambda payload
aitorarjona Sep 13, 2023
36de64c
Fix automatic storage_bucket name for AWS with config profiles
aitorarjona Sep 13, 2023
506209f
Merge master
aitorarjona Sep 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
- [IBM] Improved token manager
- [Core] Job creation now checks that each element in 'iterdata' is smaller than 8 KB
- [MapReduce] Make 'results' variable name not mandatory in the reduce function signature
- [AWS] Replace 'aws_access_key_id' and 'aws_secret_access_key' in lithops 'aws' config in favor of 'config_profile' key and standard AWS SDK config retrieval procedures.

### Fixed
- [IBM VPC & AWS EC2] Make sure only VMs from the given VPC are removed
Expand Down
80 changes: 67 additions & 13 deletions docs/source/compute_config/aws_lambda.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ python3 -m pip install lithops[aws]
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:*",
Expand Down Expand Up @@ -52,26 +51,69 @@ python3 -m pip install lithops[aws]
lithops:
backend: aws_lambda

aws:
aws_lambda:
execution_role: <EXECUTION_ROLE_ARN>
region: <REGION_NAME>
Copy link
Member

@JosepSampe JosepSampe Sep 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the execution_role mandatory in aws_lambda? if yes I would update this .md file and include it in all the parts where you put some lithops config example, to make it clearer. Is region mandatory in all the cases? I think this lithops config example is confusing here. I would remove it and put the config example in the next section, when necessary, with all the necessary parameters.

Copy link
Contributor Author

@aitorarjona aitorarjona Sep 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, execution_role is mandatory. The user must specify which services can the lambda access to. We could automate this, but the user should have IAM permissions like create role... We can leave like this for now

```

## AWS Credential setup

Lithops loads AWS credentials as specified in the [boto3 configuration guide](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html).

In summary, you can use the following settings:

1. Provide credentials via the `~/.aws/config` file. **This is the preferred option to configure AWS credentials for use with Lithops**:

You can run `aws configure` command if the AWS CLI is installed to setup the credentials.
Comment on lines +52 to +54
Copy link
Member

@JosepSampe JosepSampe Sep 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How the ~/.aws/config looks like in this case? are the keys going into a default profile by defaut? or are the keys set in the file without a profile?

I mean, after calling aws configure, you get this:?

aws_access_key_id=XXXXXXXXX
aws_secret_access_key=XXXXXX

or something like this:?

[profile default]
aws_access_key_id=XXXXXXXXXXXXXXX
aws_secret_access_key=XXXXXXXXXXXXXXXX

Copy link
Member

@JosepSampe JosepSampe Sep 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if in this case it makes sense to force the user to provide a profile_name with aws configure --profile my-unique-profile-name and then configure lithops like in the SSO approach, with:

lithops:
    backend: aws_lambda

aws:
    config_profile: my-unique-profile-name

aws_lambda:
    execution_role: <EXECUTION_ROLE_ARN>
    region: <REGION_NAME>


2. Provide credentials via environment variables:

Lithops needs at least `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_DEFAULT_REGION` environment variables set.
Comment on lines +56 to +58
Copy link
Member

@JosepSampe JosepSampe Sep 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe in this option you can put a config example (and maybe remove AWS_DEFAULT_REGION?):

lithops:
    backend: aws_lambda

aws_lambda:
    execution_role: <EXECUTION_ROLE_ARN>
    region: <REGION_NAME>


3. Provide the credentials in the `aws` section of the Lithops config file **This option is not ideal and will be removed in future Lithops releases!**:
Copy link
Member

@JosepSampe JosepSampe Sep 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for any warning for now. We still have to decide whether we want to deprecate it or not

```yaml
lithops:
backend: aws_lambda

aws:
access_key_id: <AWS_ACCESS_KEY_ID>
secret_access_key: <AWS_SECRET_ACCESS_KEY>
region: <REGION_NAME>
```

aws_lambda:
execution_role: <EXECUTION_ROLE_ARN>
### Setup for SSO-based users

Users using SSO-based accounts do not require an IAM user, and have temporal session access tokens instead. To configure access to SSO-based accounts, you can configure a profile in the `~/.aws/config` file for using SSO authentication:

```yaml
[profile my-sso-profile]
sso_start_url = https://XXXXXXXX.awsapps.com/start
sso_region = us-east-1
sso_account_id = XXXXXXXXXXX
sso_role_name = XXXXXXXXXXXXXXXXX
region = us-east-1
```

## Summary of configuration keys for AWS
Then, you can log in or refresh your credentials by using the sso login command:

### AWS
```
$ aws sso login --profile my-sso-profile
```

To use this profile, you must specify it in the `aws` section of the Lithops config file:

```yaml
lithops:
backend: aws_lambda

aws:
config_profile: my-sso-profile
```

More info [here](https://docs.aws.amazon.com/cli/latest/userguide/sso-configure-profile-token.html).

|Group|Key|Default|Mandatory|Additional info|
|---|---|---|---|---|
|aws | region | |yes | AWS Region. For example `us-east-1` |
|aws | access_key_id | |yes | Account access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. |
|aws | secret_access_key | |yes | Account secret access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. |
|aws | session_token | |no | Session token for temporary AWS credentials |
|aws | account_id | |no | *This field will be used if present to retrieve the account ID instead of using AWS STS. The account ID is used to format full image names for container runtimes. |

## Summary of configuration keys for AWS Lambda

### AWS Lambda

Expand All @@ -90,6 +132,18 @@ aws_lambda:
| aws_lambda | ephemeral_storage | 512 | no | Ephemeral storage (`/tmp`) size in MB (must be between 512 MB and 10240 MB) |
| aws_lambda | env_vars | {} | no | List of {name: ..., value: ...} pairs for Lambda instance environment variables |

### AWS

|Group| Key | Default | Mandatory | Additional info |
|---|-------------------|----------|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|aws | region | | yes | AWS Region. For example `us-east-1` |
|aws | config_profile | "default" | no | AWS SDK [configuration profile](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#using-a-configuration-file) name. |
|aws | access_key_id | | no | Account access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. |
|aws | secret_access_key | | no | Account secret access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. |
|aws | session_token | | no | Session token for temporary AWS credentials |
|aws | account_id | | no | *This field will be used if present to retrieve the account ID instead of using AWS STS. The account ID is used to format full image names for container runtimes. |


## Additional configuration

### VPC
Expand Down
75 changes: 66 additions & 9 deletions docs/source/storage_config/aws_s3.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,22 +29,79 @@ Lithops with AWS S3 as storage backend.
secret_access_key : <SECRET_ACCESS_KEY>
```

## AWS Credential setup

Lithops loads AWS credentials as specified in the [boto3 configuration guide](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html).

In summary, you can use the following settings:

1. Provide credentials via the `~/.aws/config` file. **This is the preferred option to configure AWS credentials for use with Lithops**:

You can run `aws configure` command if the AWS CLI is installed to setup the credentials.

2. Provide credentials via environment variables:

Lithops needs at least `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_DEFAULT_REGION` environment variables set.

3. Provide the credentials in the `aws` section of the Lithops config file **This option is not ideal and will be removed in future Lithops releases!**:
```yaml
lithops:
storage: aws_s3

aws:
access_key_id: <AWS_ACCESS_KEY_ID>
secret_access_key: <AWS_SECRET_ACCESS_KEY>
region: <REGION_NAME>
```

### Setup for SSO-based users

Users using SSO-based accounts do not require an IAM user, and have temporal session access tokens instead. To configure access to SSO-based accounts, you can configure a profile in the `~/.aws/config` file for using SSO authentication:

```yaml
[profile my-sso-profile]
sso_start_url = https://XXXXXXXX.awsapps.com/start
sso_region = us-east-1
sso_account_id = XXXXXXXXXXX
sso_role_name = XXXXXXXXXXXXXXXXX
region = us-east-1
```

Then, you can log in or refresh your credentials by using the sso login command:

```
$ aws sso login --profile my-sso-profile
```

To use this profile, you must specify it in the `aws` section of the Lithops config file:

```yaml
lithops:
storage: aws_s3

aws:
config_profile: my-sso-profile
```


## Summary of configuration keys for AWS:

### AWS:

|Group|Key|Default|Mandatory|Additional info|
|---|---|---|---|---|
|aws | region | |yes | AWS Region. For example `us-east-1` |
|aws | access_key_id | |yes | Account access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. |
|aws | secret_access_key | |yes | Account secret access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. |
|aws | session_token | |no | Session token for temporary AWS credentials |

### Summary of configuration keys for AWS S3:
### AWS S3

|Group|Key|Default|Mandatory|Additional info|
|---|---|---|---|---|
|aws_s3 | region | |no | Region of your Bcuket. e.g `us-east-1`, `eu-west-1`, etc. Lithops will use the region set under the `aws` section if it is not set here |
|aws_s3 | storage_bucket | | no | The name of a bucket that exists in you account. This will be used by Lithops for intermediate data. Lithops will automatically create a new one if it is not provided |

### AWS

|Group| Key | Default | Mandatory | Additional info |
|---|-------------------|----------|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|aws | region | | yes | AWS Region. For example `us-east-1` |
|aws | config_profile | "default" | no | AWS SDK [configuration profile](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#using-a-configuration-file) name. |
|aws | access_key_id | | no | Account access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. |
|aws | secret_access_key | | no | Account secret access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. |
|aws | session_token | | no | Session token for temporary AWS credentials |
|aws | account_id | | no | *This field will be used if present to retrieve the account ID instead of using AWS STS. The account ID is used to format full image names for container runtimes. |

Loading