diff --git a/docs/source/compute_config/aws_lambda.md b/docs/source/compute_config/aws_lambda.md index 57696f287..4f7d00ff1 100644 --- a/docs/source/compute_config/aws_lambda.md +++ b/docs/source/compute_config/aws_lambda.md @@ -20,7 +20,6 @@ python3 -m pip install lithops[aws] "Version": "2012-10-17", "Statement": [ { - "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "s3:*", @@ -44,34 +43,72 @@ python3 -m pip install lithops[aws] 7. Choose **Lambda** on the use case list and click **Next: Permissions**. Select the policy created before (`lithops-policy`). Click **Next: Tags** and **Next: Review**. Type a role name, for example `lithops-execution-role`. Click on *Create Role*. -## Configuration +## AWS Credential setup -6. Edit your lithops config and add the following keys: +Lithops loads AWS credentials as specified in the [boto3 configuration guide](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html). +In summary, you can use the following settings: + +1. Provide credentials via the `~/.aws/config` file. **This is the preferred option to configure AWS credentials for use with Lithops**: + + You can run `aws configure` command if the AWS CLI is installed to setup the credentials. + +2. Provide credentials via environment variables: + + Lithops needs at least `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_DEFAULT_REGION` environment variables set. + +3. Provide the credentials in the `aws` section of the Lithops config file: ```yaml lithops: backend: aws_lambda aws: - region: access_key_id: secret_access_key: + region: aws_lambda: execution_role: + region: ``` -## Summary of configuration keys for AWS +### Setup for SSO-based users -### AWS +Users using SSO-based accounts do not require an IAM user, and have temporal session access tokens instead. To configure access to SSO-based accounts, you can configure a profile in the `~/.aws/config` file for using SSO authentication: + +```yaml +[profile my-sso-profile] +sso_start_url = https://XXXXXXXX.awsapps.com/start +sso_region = us-east-1 +sso_account_id = XXXXXXXXXXX +sso_role_name = XXXXXXXXXXXXXXXXX +region = us-east-1 +``` + +Then, you can log in or refresh your credentials by using the sso login command: + +``` +$ aws sso login --profile my-sso-profile +``` + +To use this profile, you must specify it in the `aws` section of the Lithops config file: -|Group|Key|Default|Mandatory|Additional info| -|---|---|---|---|---| -|aws | region | |yes | AWS Region. For example `us-east-1` | -|aws | access_key_id | |yes | Account access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. | -|aws | secret_access_key | |yes | Account secret access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. | -|aws | session_token | |no | Session token for temporary AWS credentials | -|aws | account_id | |no | *This field will be used if present to retrieve the account ID instead of using AWS STS. The account ID is used to format full image names for container runtimes. | +```yaml +lithops: + backend: aws_lambda + +aws: + config_profile: my-sso-profile + +aws_lambda: + execution_role: + region: +``` + +More info [here](https://docs.aws.amazon.com/cli/latest/userguide/sso-configure-profile-token.html). + + +## Summary of configuration keys for AWS Lambda ### AWS Lambda @@ -90,6 +127,18 @@ aws_lambda: | aws_lambda | ephemeral_storage | 512 | no | Ephemeral storage (`/tmp`) size in MB (must be between 512 MB and 10240 MB) | | aws_lambda | env_vars | {} | no | List of {name: ..., value: ...} pairs for Lambda instance environment variables | +### AWS + +|Group| Key | Default | Mandatory | Additional info | +|---|-------------------|----------|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------| +|aws | region | | yes | AWS Region. For example `us-east-1` | +|aws | config_profile | "default" | no | AWS SDK [configuration profile](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#using-a-configuration-file) name. | +|aws | access_key_id | | no | Account access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. | +|aws | secret_access_key | | no | Account secret access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. | +|aws | session_token | | no | Session token for temporary AWS credentials | +|aws | account_id | | no | *This field will be used if present to retrieve the account ID instead of using AWS STS. The account ID is used to format full image names for container runtimes. | + + ## Additional configuration ### VPC diff --git a/docs/source/storage_config/aws_s3.md b/docs/source/storage_config/aws_s3.md index 21f89f609..de7389866 100644 --- a/docs/source/storage_config/aws_s3.md +++ b/docs/source/storage_config/aws_s3.md @@ -29,22 +29,79 @@ Lithops with AWS S3 as storage backend. secret_access_key : ``` +## AWS Credential setup + +Lithops loads AWS credentials as specified in the [boto3 configuration guide](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html). + +In summary, you can use the following settings: + +1. Provide credentials via the `~/.aws/config` file. **This is the preferred option to configure AWS credentials for use with Lithops**: + + You can run `aws configure` command if the AWS CLI is installed to setup the credentials. + +2. Provide credentials via environment variables: + + Lithops needs at least `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_DEFAULT_REGION` environment variables set. + +3. Provide the credentials in the `aws` section of the Lithops config file **This option is not ideal and will be removed in future Lithops releases!**: +```yaml +lithops: + storage: aws_s3 + +aws: + access_key_id: + secret_access_key: + region: +``` + +### Setup for SSO-based users + +Users using SSO-based accounts do not require an IAM user, and have temporal session access tokens instead. To configure access to SSO-based accounts, you can configure a profile in the `~/.aws/config` file for using SSO authentication: + +```yaml +[profile my-sso-profile] +sso_start_url = https://XXXXXXXX.awsapps.com/start +sso_region = us-east-1 +sso_account_id = XXXXXXXXXXX +sso_role_name = XXXXXXXXXXXXXXXXX +region = us-east-1 +``` + +Then, you can log in or refresh your credentials by using the sso login command: + +``` +$ aws sso login --profile my-sso-profile +``` + +To use this profile, you must specify it in the `aws` section of the Lithops config file: + +```yaml +lithops: + storage: aws_s3 + +aws: + config_profile: my-sso-profile +``` + ## Summary of configuration keys for AWS: -### AWS: -|Group|Key|Default|Mandatory|Additional info| -|---|---|---|---|---| -|aws | region | |yes | AWS Region. For example `us-east-1` | -|aws | access_key_id | |yes | Account access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. | -|aws | secret_access_key | |yes | Account secret access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. | -|aws | session_token | |no | Session token for temporary AWS credentials | - -### Summary of configuration keys for AWS S3: +### AWS S3 |Group|Key|Default|Mandatory|Additional info| |---|---|---|---|---| |aws_s3 | region | |no | Region of your Bcuket. e.g `us-east-1`, `eu-west-1`, etc. Lithops will use the region set under the `aws` section if it is not set here | |aws_s3 | storage_bucket | | no | The name of a bucket that exists in you account. This will be used by Lithops for intermediate data. Lithops will automatically create a new one if it is not provided | +### AWS + +|Group| Key | Default | Mandatory | Additional info | +|---|-------------------|----------|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------| +|aws | region | | yes | AWS Region. For example `us-east-1` | +|aws | config_profile | "default" | no | AWS SDK [configuration profile](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#using-a-configuration-file) name. | +|aws | access_key_id | | no | Account access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. | +|aws | secret_access_key | | no | Account secret access key to AWS services. To find them, navigate to *My Security Credentials* and click *Create Access Key* if you don't already have one. | +|aws | session_token | | no | Session token for temporary AWS credentials | +|aws | account_id | | no | *This field will be used if present to retrieve the account ID instead of using AWS STS. The account ID is used to format full image names for container runtimes. | + diff --git a/lithops/serverless/backends/aws_lambda/aws_lambda.py b/lithops/serverless/backends/aws_lambda/aws_lambda.py index 3734778bf..5c80eecc9 100644 --- a/lithops/serverless/backends/aws_lambda/aws_lambda.py +++ b/lithops/serverless/backends/aws_lambda/aws_lambda.py @@ -31,6 +31,7 @@ from lithops import utils from lithops.version import __version__ from lithops.constants import COMPUTE_CLI_MSG +from lithops.utils import is_lithops_worker from . import config @@ -50,26 +51,34 @@ def __init__(self, lambda_config, internal_storage): Initialize AWS Lambda Backend """ logger.debug('Creating AWS Lambda client') - self.name = 'aws_lambda' self.type = 'faas' - self.lambda_config = lambda_config self.internal_storage = internal_storage self.user_agent = lambda_config['user_agent'] - - self.user_key = lambda_config['access_key_id'][-4:].lower() - self.package = f'lithops_v{__version__.replace(".", "-")}_{self.user_key}' self.region_name = lambda_config['region'] - self.role_arn = lambda_config['execution_role'] + self.lambda_config = lambda_config + + if "config_profile" in lambda_config and not is_lithops_worker(): + logger.debug("Creating lambda session using profile %s", lambda_config["config_profile"]) + self.aws_session = boto3.Session( + profile_name=lambda_config["config_profile"], + region_name=self.region_name + ) + else: # If it's a lithops worker (lambda), get credentials from execution role + self.aws_session = boto3.Session( + aws_access_key_id=lambda_config.get('access_key_id'), + aws_secret_access_key=lambda_config.get('secret_access_key'), + aws_session_token=lambda_config.get('session_token'), + region_name=self.region_name + ) - logger.debug('Creating Boto3 AWS Session and Lambda Client') + sts_client = self.aws_session.client('sts', region_name=self.region_name) + caller_id = sts_client.get_caller_identity() - self.aws_session = boto3.Session( - aws_access_key_id=lambda_config['access_key_id'], - aws_secret_access_key=lambda_config['secret_access_key'], - aws_session_token=lambda_config.get('session_token'), - region_name=self.region_name - ) + if ":" in caller_id["UserId"]: # SSO user + self.user_key = caller_id["UserId"].split(":")[1] + else: # IAM user + self.user_key = caller_id["UserId"][-4:].lower() self.lambda_client = self.aws_session.client( 'lambda', region_name=self.region_name, @@ -81,17 +90,17 @@ def __init__(self, lambda_config, internal_storage): self.credentials = self.aws_session.get_credentials() self.session = URLLib3Session() self.host = f'lambda.{self.region_name}.amazonaws.com' + self.package = f'lithops_v{__version__.replace(".", "-")}_{self.user_key}' - if 'account_id' in self.lambda_config: - self.account_id = self.lambda_config['account_id'] + if 'account_id' in lambda_config: + self.account_id = lambda_config['account_id'] else: - sts_client = self.aws_session.client('sts', region_name=self.region_name) - self.account_id = sts_client.get_caller_identity()["Account"] + self.account_id = caller_id["Account"] self.ecr_client = self.aws_session.client('ecr', region_name=self.region_name) msg = COMPUTE_CLI_MSG.format('AWS Lambda') - logger.info(f"{msg} - Region: {self.region_name}") + logger.info("%s - Region: %s", msg, self.region_name) def _format_function_name(self, runtime_name, runtime_memory, version=__version__): runtime_name = runtime_name.replace('/', '__').replace('.', '').replace(':', '--') @@ -115,11 +124,11 @@ def _format_layer_name(self, runtime_name, version=__version__): def _get_default_runtime_name(self): py_version = utils.CURRENT_PY_VERSION.replace('.', '') - return f'lithops-default-runtime-v{py_version}' + return f'default-v{py_version}' def _is_container_runtime(self, runtime_name): name = runtime_name.split('/', 1)[-1] - return 'lithops-default-runtime-v' not in name + return 'default-v' not in name def _format_repo_name(self, runtime_name): if ':' in runtime_name: @@ -159,17 +168,16 @@ def _wait_for_function_deployed(self, func_name): state = res['Configuration']['State'] if state == 'Pending': time.sleep(sleep_seconds) - logger.debug('"{}" function is being deployed... ' - '(status: {})'.format(func_name, res['Configuration']['State'])) + logger.debug('"%s" function is being deployed (status: %s)', func_name, res['Configuration']['State']) retries -= 1 if retries == 0: - raise Exception('"{}" function not deployed (timed out): {}'.format(func_name, res)) + raise Exception(f'"{func_name}" function not deployed (timed out): {res}') elif state == 'Failed' or state == 'Inactive': - raise Exception('"{}" function not deployed (state is "{}"): {}'.format(func_name, state, res)) + raise Exception(f'"{func_name}" function not deployed (state is "{state}"): {res}') elif state == 'Active': break - logger.debug('Ok --> function "{}" is active'.format(func_name)) + logger.debug('Function "%s" is active', func_name) def _get_layer(self, runtime_name): """ @@ -191,7 +199,7 @@ def _create_layer(self, runtime_name): @param runtime_name: runtime name from which to create the layer @return: ARN of the created layer """ - logger.info('Creating default lambda layer for runtime {}'.format(runtime_name)) + logger.info('Creating default lambda layer for runtime %s', runtime_name) with zipfile.ZipFile(BUILD_LAYER_FUNCTION_ZIP, 'w') as build_layer_zip: current_location = os.path.dirname(os.path.abspath(__file__)) @@ -206,10 +214,10 @@ def _create_layer(self, runtime_name): logger.debug('Creating "layer builder" function') try: - resp = self.lambda_client.create_function( + res = self.lambda_client.create_function( FunctionName=func_name, Runtime=config.AVAILABLE_PY_RUNTIMES[utils.CURRENT_PY_VERSION], - Role=self.role_arn, + Role=self.lambda_config["execution_role"], Handler='build_layer.lambda_handler', Code={ 'ZipFile': build_layer_zip_bin @@ -219,12 +227,12 @@ def _create_layer(self, runtime_name): ) # wait until the function is created - if resp['ResponseMetadata']['HTTPStatusCode'] not in (200, 201): - msg = 'An error occurred creating/updating action {}: {}'.format(runtime_name, resp) + if res['ResponseMetadata']['HTTPStatusCode'] not in (200, 201): + msg = f'An error occurred creating/updating action {runtime_name}: {res}' raise Exception(msg) self._wait_for_function_deployed(func_name) - logger.debug('OK --> Created "layer builder" function {}'.format(runtime_name)) + logger.debug('Created "layer builder" function %s', runtime_name) dependencies = [dependency.strip().replace(' ', '') for dependency in config.DEFAULT_REQUIREMENTS] layer_name = self._format_layer_name(runtime_name) @@ -237,9 +245,9 @@ def _create_layer(self, runtime_name): logger.debug('Invoking "layer builder" function') response = self.lambda_client.invoke(FunctionName=func_name, Payload=json.dumps(payload)) if response['ResponseMetadata']['HTTPStatusCode'] == 200: - logger.debug('OK --> Layer {} built'.format(layer_name)) + logger.debug('Layer %s built', layer_name) else: - msg = 'An error occurred creating layer {}: {}'.format(layer_name, response) + msg = f'An error occurred creating layer {layer_name}: {response}' raise Exception(msg) finally: os.remove(BUILD_LAYER_FUNCTION_ZIP) @@ -251,7 +259,7 @@ def _create_layer(self, runtime_name): raise # Publish layer from S3 - logger.debug('Creating layer {} ...'.format(layer_name)) + logger.debug('Creating layer %s', layer_name) response = self.lambda_client.publish_layer_version( LayerName=layer_name, Description='Lithops Function for ' + self.package, @@ -268,7 +276,7 @@ def _create_layer(self, runtime_name): logger.warning(e) if response['ResponseMetadata']['HTTPStatusCode'] == 201: - logger.debug('OK --> Layer {} created'.format(layer_name)) + logger.debug('Layer %s created', layer_name) return response['LayerVersionArn'] else: raise Exception(f'An error occurred creating layer {layer_name}: {response}') @@ -278,7 +286,7 @@ def _delete_layer(self, layer_name): Delete a layer @param layer_name: Formatted layer name """ - logger.debug('Deleting lambda layer: {}'.format(layer_name)) + logger.debug('Deleting lambda layer %s', layer_name) versions = [] response = self.lambda_client.list_layer_versions(LayerName=layer_name) @@ -294,7 +302,7 @@ def _delete_layer(self, layer_name): VersionNumber=version ) if response['ResponseMetadata']['HTTPStatusCode'] == 204: - logger.debug('OK --> Layer {} version {} deleted'.format(layer_name, version)) + logger.debug('Layer %s version %s deleted', layer_name, version) def _list_layers(self): """ @@ -305,7 +313,7 @@ def _list_layers(self): response = self.lambda_client.list_layers() layers = response['Layers'] if 'Layers' in response else [] - logger.debug('Listed {} layers'.format(len(layers))) + logger.debug('Listed %d layers', len(layers)) lithops_layers = [] for layer in layers: if 'lithops' in layer['LayerName'] and self.user_key in layer['LayerName']: @@ -317,7 +325,7 @@ def _delete_function(self, function_name): Deletes a function by its formatted name @param function_name: function name to delete """ - logger.info(f'Deleting function: {function_name}') + logger.info('Deleting function "%s"', function_name) try: response = self.lambda_client.delete_function( FunctionName=function_name @@ -326,11 +334,11 @@ def _delete_function(self, function_name): raise err if response['ResponseMetadata']['HTTPStatusCode'] == 204: - logger.debug('OK --> Deleted function {}'.format(function_name)) + logger.debug('Deleted function "%s"', function_name) elif response['ResponseMetadata']['HTTPStatusCode'] == 404: - logger.debug('OK --> Function {} does not exist'.format(function_name)) + logger.debug('Function "%s" does not exist', function_name) else: - msg = 'An error occurred creating/updating action {}: {}'.format(function_name, response) + msg = f'An error occurred creating/updating action {function_name}: {response}' raise Exception(msg) def build_runtime(self, runtime_name, runtime_file, extra_args=[]): @@ -339,7 +347,7 @@ def build_runtime(self, runtime_name, runtime_file, extra_args=[]): @param runtime_name: name of the runtime to be built @param runtime_file: path of a Dockerfile for a container runtime """ - logger.info(f'Building runtime {runtime_name} from {runtime_file}') + logger.info('Building runtime %s from %s', runtime_name, runtime_file) docker_path = utils.get_docker_path() if runtime_file: @@ -374,10 +382,10 @@ def build_runtime(self, runtime_name, runtime_file, extra_args=[]): try: self.ecr_client.create_repository(repositoryName=repo_name) except self.ecr_client.exceptions.RepositoryAlreadyExistsException: - logger.debug(f'Repository {repo_name} already exists') + logger.debug('Repository "%s" already exists', repo_name) image_name = f'{registry}/{repo_name}:{tag}' - logger.debug(f'Pushing runtime {image_name} to AWS container registry') + logger.debug('Pushing runtime "%s" to AWS container registry', image_name) cmd = f'{docker_path} tag {runtime_name} {image_name}' utils.run_command(cmd) if utils.is_podman(docker_path): @@ -392,7 +400,7 @@ def _deploy_default_runtime(self, runtime_name, memory, timeout): """ Deploy the default runtime based on layers """ - logger.info(f"Deploying runtime: {runtime_name} - Memory: {memory} - Timeout: {timeout}") + logger.info("Deploying runtime: %s - Memory: %d - Timeout: %d", runtime_name, memory, timeout) function_name = self._format_function_name(runtime_name, memory) layer_arn = self._get_layer(runtime_name) @@ -406,7 +414,7 @@ def _deploy_default_runtime(self, runtime_name, memory, timeout): response = self.lambda_client.create_function( FunctionName=function_name, Runtime=config.AVAILABLE_PY_RUNTIMES[utils.CURRENT_PY_VERSION], - Role=self.role_arn, + Role=self.lambda_config["execution_role"], Handler='entry_point.lambda_handler', Code={ 'ZipFile': code @@ -421,7 +429,7 @@ def _deploy_default_runtime(self, runtime_name, memory, timeout): }, FileSystemConfigs=[ {'Arn': efs_conf['access_point'], - 'LocalMountPath': efs_conf['mount_path']} + 'LocalMountPath': efs_conf['mount_path']} for efs_conf in self.lambda_config['efs'] ], Tags={ @@ -446,13 +454,13 @@ def _deploy_default_runtime(self, runtime_name, memory, timeout): raise e self._wait_for_function_deployed(function_name) - logger.debug('OK --> Created lambda function {}'.format(function_name)) + logger.debug('Created lambda function "%s"', function_name) def _deploy_container_runtime(self, runtime_name, memory, timeout): """ Deploy a runtime based on a container """ - logger.info(f"Deploying runtime: {runtime_name} - Memory: {memory} Timeout: {timeout}") + logger.info("Deploying runtime: %s - Memory: %d Timeout: %d", runtime_name, memory, timeout) function_name = self._format_function_name(runtime_name, memory) if ':' in runtime_name: @@ -480,7 +488,7 @@ def _deploy_container_runtime(self, runtime_name, memory, timeout): try: response = self.lambda_client.create_function( FunctionName=function_name, - Role=self.role_arn, + Role=self.lambda_config["execution_role"], Code={ 'ImageUri': image_uri }, @@ -494,7 +502,7 @@ def _deploy_container_runtime(self, runtime_name, memory, timeout): }, FileSystemConfigs=[ {'Arn': efs_conf['access_point'], - 'LocalMountPath': efs_conf['mount_path']} + 'LocalMountPath': efs_conf['mount_path']} for efs_conf in self.lambda_config['efs'] ], Tags={ @@ -520,7 +528,7 @@ def _deploy_container_runtime(self, runtime_name, memory, timeout): raise e self._wait_for_function_deployed(function_name) - logger.debug(f'OK --> Created lambda function {function_name}') + logger.debug('Created lambda function "%s"', function_name) def deploy_runtime(self, runtime_name, memory, timeout): """ @@ -545,7 +553,7 @@ def delete_runtime(self, runtime_name, runtime_memory, version=__version__): @param runtime_name: name of the runtime to be deleted @param runtime_memory: memory of the runtime to be deleted in MB """ - logger.info(f'Deleting lambda runtime: {runtime_name} - {runtime_memory}MB') + logger.info('Deleting lambda runtime: %s - %d MB', runtime_name, runtime_memory) func_name = self._format_function_name(runtime_name, runtime_memory, version) self._delete_function(func_name) @@ -560,14 +568,14 @@ def delete_runtime(self, runtime_name, runtime_memory, version=__version__): image, tag = runtime_name, 'latest' package = '_'.join(func_name.split('_')[:3]) repo_name = f"{package}/{image}" - logger.debug(f'Going to delete ECR repository {repo_name} tag {tag}') + logger.debug('Going to delete ECR repository "%s" with tag "%s"', repo_name, tag) try: self.ecr_client.batch_delete_image(repositoryName=repo_name, imageIds=[{'imageTag': tag}]) images = self.ecr_client.list_images(repositoryName=repo_name, filter={'tagStatus': 'TAGGED'}) if not images['imageIds']: - logger.debug(f'Going to delete ECR repository {repo_name}') + logger.debug('Going to delete ECR repository %s', repo_name) self.ecr_client.delete_repository(repositoryName=repo_name, force=True) - except Exception: + except: pass else: layer = self._format_layer_name(runtime_name, version) @@ -629,9 +637,12 @@ def invoke(self, runtime_name, runtime_memory, payload): @param payload: invoke dict payload @return: invocation ID """ - function_name = self._format_function_name(runtime_name, runtime_memory) + payload["config"]["aws"].pop("access_key_id", None) + payload["config"]["aws"].pop("secret_access_key", None) + payload["config"]["aws"].pop("session_token", None) + headers = {'Host': self.host, 'X-Amz-Invocation-Type': 'Event', 'User-Agent': self.user_agent} url = f'https://{self.host}/2015-03-31/functions/{function_name}/invocations' request = AWSRequest(method="POST", url=url, data=json.dumps(payload, default=str), headers=headers) @@ -642,7 +653,7 @@ def invoke(self, runtime_name, runtime_memory, payload): try: r = self.session.send(request.prepare()) invoked = True - except Exception: + except: pass if r.status_code == 202: @@ -713,7 +724,7 @@ def _generate_runtime_meta(self, runtime_name, runtime_memory): Extract preinstalled Python modules from lambda function execution environment return : runtime meta dictionary """ - logger.debug(f'Extracting runtime metadata from: {runtime_name}') + logger.debug('Extracting runtime metadata from runtime "%s"', runtime_name) function_name = self._format_function_name(runtime_name, runtime_memory) diff --git a/lithops/serverless/backends/aws_lambda/config.py b/lithops/serverless/backends/aws_lambda/config.py index 7178b4a9f..c4d28cef2 100644 --- a/lithops/serverless/backends/aws_lambda/config.py +++ b/lithops/serverless/backends/aws_lambda/config.py @@ -20,14 +20,14 @@ logger = logging.getLogger(__name__) DEFAULT_REQUIREMENTS = [ - 'numpy', - 'requests', - 'redis', - 'pika', - 'cloudpickle', - 'ps-mem', - 'tblib', - 'urllib3<2' + "numpy", + "requests", + "redis", + "pika", + "cloudpickle", + "ps-mem", + "tblib", + "urllib3<2" ] AVAILABLE_PY_RUNTIMES = { @@ -39,22 +39,22 @@ '3.11': 'python3.11' } -USER_RUNTIME_PREFIX = 'lithops.user_runtimes' +USER_RUNTIME_PREFIX = "lithops.user_runtimes" DEFAULT_CONFIG_KEYS = { - 'runtime_timeout': 180, # Default: 180 seconds => 3 minutes - 'runtime_memory': 256, # Default memory: 256 MB - 'max_workers': 1000, - 'worker_processes': 1, - 'invoke_pool_threads': 64, - 'architecture': 'x86_64', - 'ephemeral_storage': 512, - 'env_vars': {}, - 'vpc': {'subnets': [], 'security_groups': []}, - 'efs': [] + "runtime_timeout": 180, # Default: 180 seconds => 3 minutes + "runtime_memory": 256, # Default memory: 256 MB + "max_workers": 1000, + "worker_processes": 1, + "invoke_pool_threads": 64, + "architecture": "x86_64", + "ephemeral_storage": 512, + "env_vars": {}, + "vpc": {"subnets": [], "security_groups": []}, + "efs": [] } -REQ_PARAMS = ('execution_role',) +REQ_PARAMS = ("execution_role",) RUNTIME_TIMEOUT_MAX = 900 # Max. timeout: 900 s == 15 min RUNTIME_MEMORY_MIN = 128 # Max. memory: 128 MB @@ -65,74 +65,71 @@ def load_config(config_data): - if 'aws' not in config_data: - raise Exception("'aws' section is mandatory in the configuration") + if "aws" not in config_data: + config_data["aws"] = {} - if not {'access_key_id', 'secret_access_key'}.issubset(set(config_data['aws'])): - raise Exception("'access_key_id' and 'secret_access_key' are mandatory under the 'aws' section of the configuration") - - if not config_data['aws_lambda']: - raise Exception("'aws_lambda' section is mandatory in the configuration") + if not config_data["aws_lambda"]: + raise Exception("\"aws_lambda\" section is mandatory in the configuration") temp = copy.deepcopy(config_data['aws_lambda']) config_data['aws_lambda'].update(config_data['aws']) config_data['aws_lambda'].update(temp) for param in REQ_PARAMS: - if param not in config_data['aws_lambda']: - msg = f'"{param}" is mandatory in the "aws_lambda" section of the configuration' + if param not in config_data["aws_lambda"]: + msg = f"\"{param}\" is mandatory in the \"aws_lambda\" section of the configuration" raise Exception(msg) for key in DEFAULT_CONFIG_KEYS: - if key not in config_data['aws_lambda']: - config_data['aws_lambda'][key] = DEFAULT_CONFIG_KEYS[key] + if key not in config_data["aws_lambda"]: + config_data["aws_lambda"][key] = DEFAULT_CONFIG_KEYS[key] - if config_data['aws_lambda']['runtime_memory'] > RUNTIME_MEMORY_MAX: - logger.warning("Memory set to {} - {} exceeds " - "the maximum amount".format(RUNTIME_MEMORY_MAX, config_data['aws_lambda']['runtime_memory'])) - config_data['aws_lambda']['runtime_memory'] = RUNTIME_MEMORY_MAX + if config_data["aws_lambda"]["runtime_memory"] > RUNTIME_MEMORY_MAX: + logger.warning("Memory set to %d - %d exceeds the maximum amount", RUNTIME_MEMORY_MAX, + config_data["aws_lambda"]["runtime_memory"]) + config_data["aws_lambda"]["runtime_memory"] = RUNTIME_MEMORY_MAX - if config_data['aws_lambda']['runtime_memory'] < RUNTIME_MEMORY_MIN: - logger.warning("Memory set to {} - {} is lower than " - "the minimum amount".format(RUNTIME_MEMORY_MIN, config_data['aws_lambda']['runtime_memory'])) - config_data['aws_lambda']['runtime_memory'] = RUNTIME_MEMORY_MIN + if config_data["aws_lambda"]["runtime_memory"] < RUNTIME_MEMORY_MIN: + logger.warning("Memory set to %d - %d is lower than " + "the minimum amount", RUNTIME_MEMORY_MIN, config_data["aws_lambda"]["runtime_memory"]) + config_data["aws_lambda"]["runtime_memory"] = RUNTIME_MEMORY_MIN - if config_data['aws_lambda']['runtime_timeout'] > RUNTIME_TIMEOUT_MAX: - logger.warning("Timeout set to {} - {} exceeds the " - "maximum amount".format(RUNTIME_TIMEOUT_MAX, config_data['aws_lambda']['runtime_timeout'])) - config_data['aws_lambda']['runtime_timeout'] = RUNTIME_TIMEOUT_MAX + if config_data["aws_lambda"]["runtime_timeout"] > RUNTIME_TIMEOUT_MAX: + logger.warning("Timeout set to %d - %d exceeds the maximum amount", RUNTIME_TIMEOUT_MAX, + config_data["aws_lambda"]["runtime_timeout"]) + config_data["aws_lambda"]["runtime_timeout"] = RUNTIME_TIMEOUT_MAX - if not {'subnets', 'security_groups'}.issubset(set(config_data['aws_lambda']['vpc'])): - raise Exception("'subnets' and 'security_groups' are mandatory sections under 'aws_lambda/vpc'") + if not {"subnets", "security_groups"}.issubset(set(config_data["aws_lambda"]["vpc"])): + raise Exception("\"subnets\" and \"security_groups\" are mandatory sections under \"aws_lambda/vpc\"") - if not isinstance(config_data['aws_lambda']['vpc']['subnets'], list): - raise Exception("Unknown type {} for 'aws_lambda/" - "vpc/subnet' section".format(type(config_data['aws_lambda']['vpc']['subnets']))) + if not isinstance(config_data["aws_lambda"]["vpc"]["subnets"], list): + raise Exception("Invalid type {} for \"aws_lambda/vpc/subnet\" section" + .format(type(config_data["aws_lambda"]["vpc"]["subnets"]))) - if not isinstance(config_data['aws_lambda']['vpc']['security_groups'], list): - raise Exception("Unknown type {} for 'aws_lambda/" - "vpc/security_groups' section".format(type(config_data['aws_lambda']['vpc']['security_groups']))) + if not isinstance(config_data["aws_lambda"]["vpc"]["security_groups"], list): + raise Exception("Invalid type {} for \"aws_lambda/vpc/security_groups\" section" + .format(type(config_data["aws_lambda"]["vpc"]["security_groups"]))) - if not isinstance(config_data['aws_lambda']['efs'], list): - raise Exception("Unknown type {} for " - "'aws_lambda/efs' section".format(type(config_data['aws_lambda']['vpc']['security_groups']))) + if not isinstance(config_data["aws_lambda"]["efs"], list): + raise Exception("Unknown type {} for \"aws_lambda/efs\" section" + .format(type(config_data["aws_lambda"]["vpc"]["security_groups"]))) if not all( - ['access_point' in efs_conf and 'mount_path' in efs_conf for efs_conf in config_data['aws_lambda']['efs']]): - raise Exception("List of 'access_point' and 'mount_path' mandatory in 'aws_lambda/efs section'") + ["access_point" in efs_conf and "mount_path" in efs_conf for efs_conf in config_data["aws_lambda"]["efs"]]): + raise Exception("List of \"access_point\" and \"mount_path\" mandatory in \"aws_lambda/efs section\"") - if not all([efs_conf['mount_path'].startswith('/mnt') for efs_conf in config_data['aws_lambda']['efs']]): - raise Exception("All mount paths must start with '/mnt' on 'aws_lambda/efs/*/mount_path' section") + if not all([efs_conf["mount_path"].startswith("/mnt") for efs_conf in config_data["aws_lambda"]["efs"]]): + raise Exception("All mount paths must start with \"/mnt\" on \"aws_lambda/efs/*/mount_path\" section") # Lambda runtime config - if config_data['aws_lambda']['ephemeral_storage'] < RUNTIME_TMP_SZ_MIN \ - or config_data['aws_lambda']['ephemeral_storage'] > RUNTIME_TMP_SZ_MAX: - raise Exception(f'Ephemeral storage value must be between {RUNTIME_TMP_SZ_MIN} and {RUNTIME_TMP_SZ_MAX}') + if config_data["aws_lambda"]["ephemeral_storage"] < RUNTIME_TMP_SZ_MIN \ + or config_data["aws_lambda"]["ephemeral_storage"] > RUNTIME_TMP_SZ_MAX: + raise Exception(f"Ephemeral storage value must be between {RUNTIME_TMP_SZ_MIN} and {RUNTIME_TMP_SZ_MAX}") - if 'region_name' in config_data['aws_lambda']: - config_data['aws_lambda']['region'] = config_data['aws_lambda'].pop('region_name') + if "region_name" in config_data["aws_lambda"]: + config_data["aws_lambda"]["region"] = config_data["aws_lambda"].pop("region_name") - if 'region' not in config_data['aws_lambda']: - raise Exception('"region" is mandatory under the "aws_lambda" or "aws" section of the configuration') - elif 'region' not in config_data['aws']: - config_data['aws']['region'] = config_data['aws_lambda']['region'] + if "region" not in config_data["aws_lambda"]: + raise Exception("\"region\" is mandatory under the \"aws_lambda\" or \"aws\" section of the configuration") + elif "region" not in config_data["aws"]: + config_data["aws"]["region"] = config_data["aws_lambda"]["region"] diff --git a/lithops/storage/backends/aws_s3/aws_s3.py b/lithops/storage/backends/aws_s3/aws_s3.py index 2fcadbbe6..16f6426fc 100644 --- a/lithops/storage/backends/aws_s3/aws_s3.py +++ b/lithops/storage/backends/aws_s3/aws_s3.py @@ -14,17 +14,18 @@ # limitations under the License. # -import os import logging +import os + import boto3 +import botocore from botocore import UNSIGNED from botocore.config import Config -import botocore -from lithops.storage.utils import StorageNoSuchKeyError -from lithops.utils import sizeof_fmt from lithops.constants import STORAGE_CLI_MSG from lithops.libs.globber import match +from lithops.storage.utils import StorageNoSuchKeyError +from lithops.utils import sizeof_fmt, is_lithops_worker logger = logging.getLogger(__name__) @@ -38,11 +39,9 @@ def __init__(self, s3_config): self.config = s3_config self.user_agent = s3_config['user_agent'] self.region_name = s3_config.get('region') - self.access_key_id = s3_config.get('access_key_id') - self.secret_access_key = s3_config.get('secret_access_key') - self.session_token = s3_config.get('session_token') - if self.access_key_id and self.secret_access_key: + if "config_profile" in s3_config and not is_lithops_worker(): + logger.debug("Creating s3 client using profile %s", s3_config["config_profile"]) client_config = Config( max_pool_connections=128, user_agent_extra=self.user_agent, @@ -50,28 +49,55 @@ def __init__(self, s3_config): read_timeout=CONN_READ_TIMEOUT, retries={'max_attempts': OBJ_REQ_RETRIES} ) - self.s3_client = boto3.client( - 's3', aws_access_key_id=self.access_key_id, - aws_secret_access_key=self.secret_access_key, - aws_session_token=self.session_token, + session = boto3.Session(profile_name=s3_config["config_profile"], region_name=self.region_name) + self.s3_client = session.client( + 's3', config=client_config, region_name=self.region_name ) - else: + elif "access_key_id" in s3_config and "secret_access_key" in s3_config: + logger.debug("Creating s3 client using IAM key pair") client_config = Config( - signature_version=UNSIGNED, - user_agent_extra=self.user_agent + max_pool_connections=128, + user_agent_extra=self.user_agent, + connect_timeout=CONN_READ_TIMEOUT, + read_timeout=CONN_READ_TIMEOUT, + retries={'max_attempts': OBJ_REQ_RETRIES} ) - self.s3_client = boto3.client('s3', config=client_config) + self.s3_client = boto3.client( + 's3', aws_access_key_id=s3_config["access_key_id"], + aws_secret_access_key=s3_config["secret_access_key"], + aws_session_token=s3_config.get("session_token"), + config=client_config, + region_name=self.region_name + ) + else: + logger.debug("Creating default s3 client") + session = boto3.Session() + credentials = session.get_credentials() # Returns not None if credentials are configured for this host + if credentials is None: + # Create unsigned client if no credentials are found + logger.debug("No AWS credentials found, creating unsigned boto3 client") + client_config = Config( + signature_version=UNSIGNED, + user_agent_extra=self.user_agent + ) + session = boto3.Session() + session.get_credentials() + self.s3_client = boto3.client('s3', config=client_config) + else: + # Let botocore load credentials from file, env, IAM execution role... + client_config = Config(user_agent_extra=self.user_agent) + self.s3_client = session.client("s3", config=client_config) msg = STORAGE_CLI_MSG.format('S3') logger.info(f"{msg} - Region: {self.region_name}") def get_client(self): - ''' + """ Get boto3 client. :return: boto3 client - ''' + """ return self.s3_client def create_bucket(self, bucket_name): @@ -90,13 +116,13 @@ def create_bucket(self, bucket_name): raise e def put_object(self, bucket_name, key, data): - ''' + """ Put an object in COS. Override the object if the key already exists. :param key: key of the object. :param data: data of the object :type data: str/bytes :return: None - ''' + """ try: res = self.s3_client.put_object(Bucket=bucket_name, Key=key, Body=data) status = 'OK' if res['ResponseMetadata']['HTTPStatusCode'] == 200 else 'Error' @@ -111,12 +137,12 @@ def put_object(self, bucket_name, key, data): raise e def get_object(self, bucket_name, key, stream=False, extra_get_args={}): - ''' + """ Get object from COS with a key. Throws StorageNoSuchKeyError if the given key does not exist. :param key: key of the object :return: Data of the object :rtype: str/bytes - ''' + """ try: r = self.s3_client.get_object(Bucket=bucket_name, Key=key, **extra_get_args) if stream: @@ -171,12 +197,12 @@ def download_file(self, bucket, key, file_name=None, extra_args={}): return True def head_object(self, bucket_name, key): - ''' + """ Head object from COS with a key. Throws StorageNoSuchKeyError if the given key does not exist. :param key: key of the object :return: Data of the object :rtype: str/bytes - ''' + """ try: metadata = self.s3_client.head_object(Bucket=bucket_name, Key=key) return metadata['ResponseMetadata']['HTTPHeaders'] @@ -187,19 +213,19 @@ def head_object(self, bucket_name, key): raise e def delete_object(self, bucket_name, key): - ''' + """ Delete an object from storage. :param bucket: bucket name :param key: data key - ''' + """ return self.s3_client.delete_object(Bucket=bucket_name, Key=key) def delete_objects(self, bucket_name, key_list): - ''' + """ Delete a list of objects from storage. :param bucket: bucket name :param key_list: list of keys - ''' + """ result = [] max_keys_num = 1000 for i in range(0, len(key_list), max_keys_num): @@ -209,12 +235,12 @@ def delete_objects(self, bucket_name, key_list): return result def head_bucket(self, bucket_name): - ''' + """ Head bucket from COS with a name. Throws StorageNoSuchKeyError if the given bucket does not exist. :param bucket_name: name of the bucket :return: Metadata of the bucket :rtype: str/bytes - ''' + """ try: return self.s3_client.head_bucket(Bucket=bucket_name) except botocore.exceptions.ClientError as e: @@ -224,13 +250,13 @@ def head_bucket(self, bucket_name): raise e def list_objects(self, bucket_name, prefix=None, match_pattern=None): - ''' + """ Return a list of objects for the given bucket and prefix. :param bucket_name: Name of the bucket. :param prefix: Prefix to filter object names. :return: List of objects in bucket that match the given prefix. :rtype: list of str - ''' + """ try: prefix = '' if prefix is None else prefix paginator = self.s3_client.get_paginator('list_objects_v2') @@ -251,13 +277,13 @@ def list_objects(self, bucket_name, prefix=None, match_pattern=None): raise e def list_keys(self, bucket_name, prefix=None): - ''' + """ Return a list of keys for the given prefix. :param bucket_name: Name of the bucket. :param prefix: Prefix to filter object names. :return: List of keys in bucket that match the given prefix. :rtype: list of str - ''' + """ try: prefix = '' if prefix is None else prefix paginator = self.s3_client.get_paginator('list_objects_v2') diff --git a/lithops/storage/backends/aws_s3/config.py b/lithops/storage/backends/aws_s3/config.py index 60585a827..3f74dcbcf 100644 --- a/lithops/storage/backends/aws_s3/config.py +++ b/lithops/storage/backends/aws_s3/config.py @@ -16,15 +16,14 @@ import copy +import logging +import hashlib +logger = logging.getLogger(__name__) -def load_config(config_data): +def load_config(config_data): if 'aws' in config_data: - - if not {'access_key_id', 'secret_access_key'}.issubset(set(config_data['aws'])): - raise Exception("'access_key_id' and 'secret_access_key' are mandatory under the 'aws' section of the configuration") - if 'aws_s3' not in config_data: config_data['aws_s3'] = {} @@ -39,6 +38,11 @@ def load_config(config_data): raise Exception("'region' is mandatory under 'aws_s3' or 'aws' section of the configuration") if 'storage_bucket' not in config_data['aws_s3']: - key = config_data['aws_s3']['access_key_id'] + if 'access_key_id' in config_data['aws']: + key = config_data['aws_s3']['access_key_id'] + elif 'config_profile' in config_data['aws']: + key = hashlib.md5(config_data['aws']['config_profile'].encode("utf-8"), usedforsecurity=False).hexdigest() + else: + raise Exception("'access_key_id' or 'config_profile' is mandatory in 'aws' section of the configuration") region = config_data['aws_s3']['region'] config_data['aws_s3']['storage_bucket'] = f'lithops-{region}-{key[:6].lower()}'