[AWS] Use credentials and config from AWS SDK file #1114

aitorarjona · 2023-06-20T14:19:42Z

Fix for #1107

This pull request adds functionality to retrieve AWS SDK config and credentials from the standard config file (~/.aws/config and ~/.aws/credentials) or env vars (more info).

Consequently, it deprecates using aws_access_key_id and aws_secret_access_key in aws Lithops config section.

This approach is not only more secure, as we avoid sending secrets to the runtime via payload, but also we support users with SSO-based accounts, which will need configure a profile in their ~/.aws/config file and retrieve their session credentials dynamically. E.g.:

[profile my-sso-profile]
sso_start_url = https://XXXXXXXX.awsapps.com/start
sso_region = us-east-1
sso_account_id = XXXXXXXXXXX
sso_role_name = XXXXXXXXXXXXXXXXX
region = us-east-1

Summary:

Added new parameter in AWS config: config_profile.

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

aitorarjona · 2023-06-20T14:20:12Z

@JosepSampe please don't merge yet

aitorarjona · 2023-07-12T14:04:27Z

@JosepSampe ready for review and merge

JosepSampe · 2023-07-13T12:18:56Z

lithops/serverless/backends/aws_lambda/config.py

-    temp = copy.deepcopy(config_data['aws_lambda'])
-    config_data['aws_lambda'].update(config_data['aws'])
-    config_data['aws_lambda'].update(temp)


Is there any particular reason to, instead of copying the aws keys inside aws_lambda, create another level of configuration inside aws_lambda and put aws section inside aws_lambda section? I think this new approach will breake a functionality explained below.

Yes, the idea was to remove the "aws" section altogether when clients are created. Reverted

lithops/serverless/backends/aws_lambda/config.py

JosepSampe · 2023-07-13T12:19:55Z

lithops/serverless/backends/aws_lambda/config.py

+    if "secret_access_key" in config_data["aws_lambda"]["aws"] or "access_key_id" in config_data["aws_lambda"]["aws"]:
+        logger.warning('Using "secret_access_key" and "access_key_id" in lithops configuration is deprecated and '
+                       'it will be removed in future releases '
+                       '- Use boto3 configuration with environment variables or config file in ~/.aws instead '
+                       '(https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html)')


Is there any particular reason to deprecate access_key_id secret_access_key? IMO we should keep it as an option

Removed the deprecation notice but kept the warning

JosepSampe · 2023-07-13T12:21:24Z

lithops/storage/backends/aws_s3/aws_s3.py

        else:
+            logger.debug("Creating default boto3 client")
            client_config = Config(
-                signature_version=UNSIGNED,


If I remember correctly, this line is necessary for accessing public buckets when no s3 config is provided in the lithops config

For now I will keep the default client as is, so that it can get credentials from the AWS role in the lambda runtime. If any user needs unsigned requests, it should explicitly create an s3 client or specify it in the s3 config

We have an app that is analyzing an AWS S3 public bucket, where the client is automatically created by lithops because we put in iterdata a bucket address, like in this example but for AWS S3, so we should find a way to keep this unsigned flag in the case that a user does not have credentials at all in his machine. If the user has credentials in his machine, the unsigned flag is not necessary.

JosepSampe · 2023-07-13T12:28:01Z

lithops/serverless/backends/aws_lambda/config.py

+    elif 'region' not in config_data['aws_lambda']['aws']:
+        config_data['aws_lambda']['aws']['region'] = config_data['aws_lambda']['region']


Previously, these lines where used to propagate the region from the aws_lambda section to the aws section, and later to aws_s3, in that case where someone puts a region in aws_lambda, but not inaws and aws_s3, this way the s3 backend would use the same region as the lambda backend. I think this new approach will break this propagation.

Hi Josep, now I understand... I'll add some comments to the code to explain this. What I was thinking is that all AWS configuration should be inherited from the standard AWS SDK ways, but I understand that is also important to enforce users deploy lambdas and buckets in the same region to avoid data transfer costs. In this case, the safest option is require to specifiy the "region" parameter the "aws" section, regardless of the AWS SDK config, and not have it repeated in "aws_s3" or "aws_lambda"?

JosepSampe · 2023-07-13T12:34:37Z

lithops/serverless/backends/aws_lambda/aws_lambda.py

+        sts_client = self.aws_session.client('sts', region_name=self.region_name)
+        caller_id = sts_client.get_caller_identity()
+
+        self.user_key = caller_id["UserId"].split(":")[1]


I tested these particular lines, and in my case the caller_id["UserId"] does not contain :, so it fails. My UserID looks like: {'UserId': 'XIPOZEKLFKWLQSXXX7587', ...}

JosepSampe · 2023-07-13T12:38:59Z

lithops/storage/backends/aws_s3/config.py

+        if "secret_access_key" in config_data["aws"] or "access_key_id" in config_data["aws"]:
+            logger.warning(
+                'using "secret_access_key" and "access_key_id" in the lithops configuration is deprecated and '
+                'they will be removed in future releases '
+                '- Use boto3 configuration with environment variables or config file in ~/.aws instead '
+                '(https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html)')
+
+        # Put "aws" section inside AWS backends, so we can access credentials at the backend class
+        # Remove from config_data to avoid storing secrets
+        if "aws" in config_data:
+            if "aws_lambda" in config_data:
+                config_data["aws_lambda"]["aws"] = config_data["aws"]
+            if "aws_s3" in config_data:
+                config_data["aws_s3"]["aws"] = config_data["aws"]
+            if "aws_batch" in config_data:
+                config_data["aws_batch"]["aws"] = config_data["aws"]
+            if "aws_ec2" in config_data:
+                config_data["aws_ec2"]["aws"] = config_data["aws"]
+            del config_data["aws"]


Same comments as in the lambda backend

JosepSampe · 2023-07-13T12:39:13Z

lithops/storage/backends/aws_s3/config.py

-        temp = copy.deepcopy(config_data['aws_s3'])
-        config_data['aws_s3'].update(config_data['aws'])
-        config_data['aws_s3'].update(temp)


Same comments as in the lambda backend

JosepSampe · 2023-07-26T14:11:01Z

lithops/serverless/backends/aws_lambda/config.py

+    # temp = copy.deepcopy(config_data["aws_lambda"])
+    config_data["aws_lambda"].update(config_data["aws"])
+    # config_data["aws_lambda"].update(temp)


The reason of this roundtrip is to make sure that the most specific config is always applied. For example, if you set region both in aws and aws_lamda, the aws_lamda region must be applied. I don't know if there is a better way to make sure the most specific config is applied and not overwritten by the .update(), but by commenting those 2 lines, if you have a region set in your aws section of the config, for example us-east1, and then you do this in your code: fexec = lithops.FunctionExecutor(region="eu-west2"), the us-east1 region will always overwrite the region you explicitly set in the function executor.

JosepSampe · 2023-07-26T14:15:11Z

lithops/storage/backends/aws_s3/aws_s3.py

        else:
+            logger.debug("Creating default boto3 client")
            client_config = Config(
-                signature_version=UNSIGNED,


We have an app that is analyzing an AWS S3 public bucket, where the client is automatically created by lithops because we put in iterdata a bucket address, like in this example but for AWS S3, so we should find a way to keep this unsigned flag in the case that a user does not have credentials at all in his machine. If the user has credentials in his machine, the unsigned flag is not necessary.

aitorarjona · 2023-09-04T09:14:03Z

@JosepSampe Hi Josep, all requests have been implemented. Please we should need this merged ASAP, we switched to an SSO-based account and the current implementation in main does not work well (and also to be ready for the next release #1137 ). Thanks!

JosepSampe · 2023-09-04T15:06:59Z

lithops/serverless/backends/aws_lambda/config.py

+    if "region" not in config_data["aws_lambda"]:
+        raise Exception("\"region\" is mandatory under the \"aws_lambda\" or \"aws\" section of the configuration")
+    elif "region" not in config_data["aws"]:
+        config_data["aws"]["region"] = config_data["aws_lambda"]["region"]


What if region is set using options 1 (~/.aws/config) or 2 (env var)? Or is region mandatory in any case in the lithops config like in the documentation?

lithops: backend: aws_lambda aws_lambda: execution_role: <EXECUTION_ROLE_ARN> region: <REGION_NAME>

Yes, I think we still need the region name for this or this. In any case, the region from the Lithops config would override the region stated in the aws config.

JosepSampe · 2023-09-04T15:08:56Z

lithops/storage/backends/aws_s3/config.py

+        if "secret_access_key" in config_data["aws"] or "access_key_id" in config_data["aws"]:
+            logger.warning("Using 'secret_access_key' and 'access_key_id' in lithops configuration is not recommended "
+                           "- Use boto3 configuration file in ~/.aws or environment variables instead "
+                           "(https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html)")



No need for any warning for now. We still have to decide whether we want to deprecate it or not

I can't add comments in unmodifed code, but:
In lines 42-43: if in options 1 and 2 region is not mandatory, this should be fixed.
Now lithops supports the automatic creation of the storage bucket if it is not provided in the config. So lines 45-48 should be fixed.

Now lithops supports the automatic creation of the storage bucket if it is not provided in the config. So lines 45-48 should be fixed.

Not sure how to proceed with this... With the SSO approach, we don't have a "fixed" key or ID to read from, contrary to the key pair approach.

Is the S3 instance shared between all the users in the SSO approach?
Or each SSO user has its own instance ?
Or the same insytance but s/he can only see his own buckets?

I wonder if we can simply use the config_profile name (or a hash of it) for the bucket name

Multiple SSO users can access to a same account and share buckets, but each one will have a different profile name. Yes, using config profile is viable 👍

JosepSampe · 2023-09-04T15:09:58Z

docs/source/compute_config/aws_lambda.md

+
+    Lithops needs at least `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_DEFAULT_REGION` environment variables set.
+
+3. Provide the credentials in the `aws` section of the Lithops config file **This option is not ideal and will be removed in future Lithops releases!**:


No need for any warning for now. We still have to decide whether we want to deprecate it or not

JosepSampe · 2023-09-04T15:18:43Z

lithops/serverless/backends/aws_lambda/config.py

+    if "secret_access_key" in config_data["aws"] or "access_key_id" in config_data["aws"]:
+        logger.warning("Using 'secret_access_key' and 'access_key_id' in lithops configuration is not recommended "
+                       "- Use boto3 configuration file in ~/.aws or environment variables instead "
+                       "(https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html)")


No need for any warning for now. We still have to decide whether we want to deprecate it or not

JosepSampe · 2023-09-04T15:23:59Z

docs/source/compute_config/aws_lambda.md

+aws_lambda:
+    execution_role: <EXECUTION_ROLE_ARN>
    region: <REGION_NAME>


Is the execution_role mandatory in aws_lambda? if yes I would update this .md file and include it in all the parts where you put some lithops config example, to make it clearer. Is region mandatory in all the cases? I think this lithops config example is confusing here. I would remove it and put the config example in the next section, when necessary, with all the necessary parameters.

Yes, execution_role is mandatory. The user must specify which services can the lambda access to. We could automate this, but the user should have IAM permissions like create role... We can leave like this for now

JosepSampe · 2023-09-13T13:24:29Z

lithops/serverless/backends/aws_lambda/aws_lambda.py

+        if "access_key_id" in payload["config"]["aws"] and "secret_access_key" in payload["config"]["aws"]:
+            del payload["config"]["aws"]["access_key_id"]
+            del payload["config"]["aws"]["secret_access_key"]


Is it necessary/convenient here to remove the session_token too?
I think here you can simply pop the keys instead of checking one by one if they exists

payload["config"]["aws"].pop("access_key_id", None) payload["config"]["aws"].pop("secret_access_key", None) payload["config"]["aws"].pop("session_token", None)

JosepSampe · 2023-09-13T13:44:02Z

My last comments are about the 2 other AWS backend (Batch & EC2).

Does the changes made here in the way to configure aws affect to those backends?
Is it convenient to copy the changes made in the Lambda docs to the docs of Batch & EC2?
In order to adapt the Batch & EC2 backends, is it as simple as copy the relevant code in the __init__ of the Lambda backend to the other 2 backends?

JosepSampe · 2023-09-13T13:46:47Z

lithops/storage/backends/aws_s3/config.py

+            if 'access_key_id' in config_data['aws']:
+                key = config_data['aws_s3']['access_key_id']
+            elif 'config_profile' in config_data['aws']:
+                key = hashlib.md5(config_data['aws']['config_profile'].encode("utf-8"), usedforsecurity=False).hexdigest()
+            else:
+                raise Exception("'access_key_id' or 'config_profile' is mandatory in 'aws' section of the configuration")


Is the option of AWS_ACCESS_KEY_ID in the env missing here?

JosepSampe · 2023-09-13T13:52:14Z

docs/source/compute_config/aws_lambda.md

+1. Provide credentials via the `~/.aws/config` file. **This is the preferred option to configure AWS credentials for use with Lithops**:
+
+    You can run `aws configure` command if the AWS CLI is installed to setup the credentials.


How the ~/.aws/config looks like in this case? are the keys going into a default profile by defaut? or are the keys set in the file without a profile?

I mean, after calling aws configure, you get this:?

aws_access_key_id=XXXXXXXXX aws_secret_access_key=XXXXXX

or something like this:?

[profile default] aws_access_key_id=XXXXXXXXXXXXXXX aws_secret_access_key=XXXXXXXXXXXXXXXX

I wonder if in this case it makes sense to force the user to provide a profile_name with aws configure --profile my-unique-profile-name and then configure lithops like in the SSO approach, with:

lithops: backend: aws_lambda aws: config_profile: my-unique-profile-name aws_lambda: execution_role: <EXECUTION_ROLE_ARN> region: <REGION_NAME>

JosepSampe · 2023-09-13T14:02:57Z

docs/source/compute_config/aws_lambda.md

+2. Provide credentials via environment variables:
+
+    Lithops needs at least `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_DEFAULT_REGION` environment variables set.


Maybe in this option you can put a config example (and maybe remove AWS_DEFAULT_REGION?):

lithops: backend: aws_lambda aws_lambda: execution_role: <EXECUTION_ROLE_ARN> region: <REGION_NAME>

aitorarjona · 2024-03-15T14:57:17Z

Closing for now, #1164 partially solves the issue described

aitorarjona added 2 commits June 20, 2023 16:08

Add support for SSO credentials in AWS Lambda backend

95437b1

Add support for SSO credentials in AWS s3

1fc59df

aitorarjona marked this pull request as draft June 21, 2023 09:31

aitorarjona mentioned this pull request Jul 3, 2023

[Request] New lithops release #1099

Closed

aitorarjona added 4 commits July 6, 2023 12:23

Remove pin in aws reqs

ce8f499

Add deprectation warning when using aws secrets in lithops config

bf1ca7e

Merge master

30941a0

Update lambda backend to support aws config profiles

19831b3

aitorarjona marked this pull request as ready for review July 12, 2023 12:22

aitorarjona added 2 commits July 12, 2023 14:31

Update changelog

118b1e6

Update docs

5ba0c46

aitorarjona changed the title ~~[AWS] Support for SSO credentials~~ [AWS] Use credentials and config from AWS SDK file Jul 13, 2023

JosepSampe requested changes Jul 14, 2023

View reviewed changes

Fixes review

dc4dce2

aitorarjona requested a review from JosepSampe July 24, 2023 14:08

JosepSampe reviewed Jul 26, 2023

View reviewed changes

aitorarjona added 2 commits August 22, 2023 17:09

Add support for unsinged boto3 client

8d4ff11

Merge upstream master

5344797

aitorarjona requested a review from JosepSampe August 22, 2023 15:12

JosepSampe requested changes Sep 4, 2023

View reviewed changes

Docs update, removed aws credentials deprecration notice

06b1dc8

JosepSampe reviewed Sep 13, 2023

View reviewed changes

aitorarjona added 3 commits September 13, 2023 15:27

Remove secrets from aws lambda payload

e792c2a

Fix automatic storage_bucket name for AWS with config profiles

36de64c

Merge master

506209f

JosepSampe reviewed Sep 13, 2023

View reviewed changes

JosepSampe mentioned this pull request Sep 28, 2023

Rebuilding and deploying runtime after temporal credentials expire #1107

Closed

aitorarjona closed this Mar 15, 2024

JosepSampe mentioned this pull request Mar 29, 2024

Why do AWS credentials have to be hard coded into config files? #1293

Closed

rabernat mentioned this pull request Mar 29, 2024

[PoC] use aws default credentials #1295

Closed

		elif 'region' not in config_data['aws_lambda']['aws']:
		config_data['aws_lambda']['aws']['region'] = config_data['aws_lambda']['region']


		Lithops needs at least `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_DEFAULT_REGION` environment variables set.

		3. Provide the credentials in the `aws` section of the Lithops config file This option is not ideal and will be removed in future Lithops releases!:

		1. Provide credentials via the `~/.aws/config` file. This is the preferred option to configure AWS credentials for use with Lithops:

		You can run `aws configure` command if the AWS CLI is installed to setup the credentials.

		2. Provide credentials via environment variables:

		Lithops needs at least `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_DEFAULT_REGION` environment variables set.

[AWS] Use credentials and config from AWS SDK file #1114

[AWS] Use credentials and config from AWS SDK file #1114

Uh oh!

Conversation

aitorarjona commented Jun 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aitorarjona commented Jun 20, 2023

Uh oh!

aitorarjona commented Jul 12, 2023

Uh oh!

JosepSampe Jul 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aitorarjona Jul 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JosepSampe Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aitorarjona Jul 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JosepSampe Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aitorarjona commented Sep 4, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JosepSampe Sep 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aitorarjona commented Jun 20, 2023 •

edited

Loading

JosepSampe Jul 13, 2023 •

edited

Loading

aitorarjona Jul 24, 2023 •

edited

Loading

JosepSampe Jul 26, 2023 •

edited

Loading

aitorarjona Jul 24, 2023 •

edited

Loading

JosepSampe Jul 26, 2023 •

edited

Loading

JosepSampe Sep 13, 2023 •

edited

Loading

JosepSampe Sep 13, 2023 •

edited

Loading

JosepSampe Sep 13, 2023 •

edited

Loading

JosepSampe Sep 13, 2023 •

edited

Loading