diff --git a/docs/upgrade.md b/docs/upgrade.md index 11d8cdbe83a..9d29be758e6 100644 --- a/docs/upgrade.md +++ b/docs/upgrade.md @@ -5,222 +5,289 @@ description: Guide to update between major Powertools for AWS Lambda (Python) ve -## End of support v1 +## Migrate to v3 from v2 -!!! warning "On March 31st, 2023, Powertools for AWS Lambda (Python) v1 reached end of support and will no longer receive updates or releases. If you are still using v1, we strongly recommend you to read our upgrade guide and update to the latest version." +!!! info "We strongly encourage you to migrate to v3. However, if you still need to upgrade from v1 to v2, you can find the [upgrade guide](/lambda/python/2.43.1/upgrade/)." -Given our commitment to all of our customers using Powertools for AWS Lambda (Python), we will keep [Pypi](https://pypi.org/project/aws-lambda-powertools/){target="_blank"} v1 releases and documentation 1.x versions to prevent any disruption. - -## Migrate to v2 from v1 - -We've made minimal breaking changes to make your transition to v2 as smooth as possible. +We've made minimal breaking changes to make your transition to v3 as smooth as possible. ### Quick summary -| Area | Change | Code change required | IAM Permissions change required | -| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | ------------------------------- | -| **Batch** | Removed legacy [SQS batch processor](#legacy-sqs-batch-processor) in favour of **`BatchProcessor`**. | Yes | - | -| **Environment variables** | Removed legacy **`POWERTOOLS_EVENT_HANDLER_DEBUG`** in favour of [`POWERTOOLS_DEV`](index.md#optimizing-for-non-production-environments){target="_blank"}. | - | - | -| **Event Handler** | Updated [headers response format](#event-handler-headers-response-format) due to [multi-value headers and cookie support](./core/event_handler/api_gateway.md#fine-grained-responses){target="_blank"}. | Tests only | - | -| **Event Source Data Classes** | Replaced [DynamoDBStreamEvent](#dynamodbstreamevent-in-event-source-data-classes) `AttributeValue` with native Python types. | Yes | - | -| **Feature Flags** / **Parameters** | Updated [AppConfig API calls](#feature-flags-and-appconfig-parameter-utility) due to **`GetConfiguration`** API deprecation. | - | Yes | -| **Idempotency** | Updated [partition key](#idempotency-partition-key-format) to include fully qualified function/method names. | - | - | +| Area | Change | Code change required | +| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | -------------------- | +| **Pydantic** | We have removed support for [Pydantic v1](#drop-support-for-pydantic-v1) | No | +| **Parser** | We have replaced [DynamoDBStreamModel](#dynamodbstreammodel-in-parser) `AttributeValue` with native Python types | Yes | +| **Lambda layer** | [Lambda layers](#new-aws-lambda-layer-arns) are now compiled according to the specific Python version and architecture | No | +| **Event Handler** | We [have deprecated](#event-handler-headers-are-case-insensitive) the `get_header_value` function. | Yes | +| **Batch Processor** | `@batch_processor` and `@async_batch_processor` decorators [are now deprecated](#deprecated-batch-processing-decorators) | Yes | +| **Event Source Data Classes** | We have updated [default values](#event-source-default-values) for optional fields. | Yes | +| **Parameters** | The [default cache TTL](#parameters-default-cache-ttl-updated-to-5-minutes) is now set to **5 minutes** | No | +| **Parameters** | The `config` parameter [is deprecated](#parameters-using-the-new-boto_config-parameter) in favor of `boto_config` | Yes | +| **JMESPath Functions** | The `extract_data_from_envelope` function is [deprecated in favor](#utilizing-the-new-query-function-in-jmespath-functions) of `query` | Yes | +| **Types file** | We have removed the [type imports](#importing-types-from-typing-and-typing_annotations) from the `shared/types.py` file | Yes | ### First Steps -!!! note "All dependencies are optional now. [Tracer](core/tracer.md#install){target="_blank"}, [Validation](./utilities/validation.md#install){target="_blank"}, and [Parser](./utilities/parser.md){target="_blank"} now require additional dependencies." - Before you start, we suggest making a copy of your current working project or create a new branch with git. -1. **Upgrade** Python to at least v3.8 +1. **Upgrade** Python to at least v3.8. 2. **Ensure** you have the latest version via [Lambda Layer or PyPi](index.md#install){target="_blank"}. -3. **Review** the following sections to confirm whether they affect your code +3. **Review** the following sections to confirm if you need to make changes to your code. + +## Drop support for Pydantic v1 + +!!! note "No code changes required" -## Legacy SQS Batch Processor +As of June 30, 2024, Pydantic v1 has reached its end-of-life, and we have discontinued support for this version. We now exclusively support Pydantic v2. -We removed the deprecated `PartialSQSProcessor` class and `sqs_batch_processor` decorator. +Use [Pydantic v2 Migration Guide](https://docs.pydantic.dev/latest/migration/){target="_blank"} to migrate your custom Pydantic models to v2. -You can migrate to `BatchProcessor` with the following changes: +## DynamoDBStreamModel in parser -1. If you use **`sqs_batch_decorator`**, change to **`batch_processor`** decorator -2. If you use **`PartialSQSProcessor`**, change to **`BatchProcessor`** -3. [Enable **`ReportBatchItemFailures`** in your Lambda Event Source](./utilities/batch.md#required-resources){target="_blank"} -4. Change your Lambda Handler to return the new response format +!!! info "This also applies if you're using [**DynamoDB BatchProcessor**](https://docs.powertools.aws.dev/lambda/python/latest/utilities/batch/#processing-messages-from-dynamodb){target="_blank"}." + +`DynamoDBStreamModel` now returns native Python types when you access DynamoDB records through `Keys`, `NewImage`, and `OldImage` attributes. -=== "[Before] Decorator" +Previously, you'd receive a raw JSON and need to deserialize each item to the type you'd want for convenience, or to the type DynamoDB stored via `get` method. - ```python hl_lines="1 6" - from aws_lambda_powertools.utilities.batch import sqs_batch_processor +With this change, you can access data deserialized as stored in DynamoDB, and no longer need to recursively deserialize nested objects (Maps) if you had them. - def record_handler(record): - return do_something_with(record["body"]) +???+ note + For a lossless conversion of DynamoDB `Number` type, we follow AWS Python SDK (boto3) approach and convert to `Decimal`. - @sqs_batch_processor(record_handler=record_handler) - def lambda_handler(event, context): - return {"statusCode": 200} - ``` +```diff +from __future__ import annotations -=== "[After] Decorator" +import json +from typing import Any - ```python hl_lines="3 5 11 13" - import json +from aws_lambda_powertools.utilities.parser import event_parser +from aws_lambda_powertools.utilities.parser.models import DynamoDBStreamModel +from aws_lambda_powertools.utilities.typing import LambdaContext - from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType, batch_processor - processor = BatchProcessor(event_type=EventType.SQS) +def send_to_sqs(data: dict): + body = json.dumps(data) + ... +@event_parser +def lambda_handler(event: DynamoDBStreamModel, context: LambdaContext): - def record_handler(record): - return do_something_with(record["body"]) + for record in event.Records: - @batch_processor(record_handler=record_handler, processor=processor) - def lambda_handler(event, context): - return processor.response() - ``` +- # BEFORE - v2 +- new_image: dict[str, Any] = record.dynamodb.NewImage +- event_type = new_image["eventType"]["S"] +- if event_type == "PENDING": +- # deserialize attribute value into Python native type +- # NOTE: nested objects would need additional logic +- data = dict(new_image) +- send_to_sqs(data) -=== "[Before] Context manager" ++ # NOW - v3 ++ new_image: dict[str, Any] = record.dynamodb.NewImage ++ if new_image.get("eventType") == "PENDING": ++ send_to_sqs(new_image) # Here new_image is just a Python Dict type - ```python hl_lines="1-2 4 14 19" - from aws_lambda_powertools.utilities.batch import PartialSQSProcessor - from botocore.config import Config +``` - config = Config(region_name="us-east-1") +## New AWS Lambda Layer ARNs - def record_handler(record): - return_value = do_something_with(record["body"]) - return return_value +!!! note "No code changes required" +To give you better a better experience, we're now building Powertools for AWS Lambda (Python)'s Lambda layers for specific Python versions (`3.8-3.12`) and architectures (`x86_64` & `arm64`). - def lambda_handler(event, context): - records = event["Records"] +This also allows us to include architecture-specific versions of both Pydantic v2 and AWS Encryption SDK and give you a more streamlined setup. - processor = PartialSQSProcessor(config=config) +To take advantage of the new layers, you need to update your functions or deployment setup to include one of the new Lambda layer ARN from the table below: - with processor(records, record_handler): - result = processor.process() +| Architecture | Python version | Layer ARN | +| ------------ | -------------- | ------------------------------------------------------------------------------------------------ | +| x86_64 | 3.8 | arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python38-x86:{version} | +| x86_64 | 3.9 | arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python39-x86:{version} | +| x86_64 | 3.10 | arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python310-x86:{version} | +| x86_64 | 3.11 | arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python311-x86:{version} | +| x86_64 | 3.12 | arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python312-x86:{version} | +| arm64 | 3.8 | arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python38-arm64:{version} | +| arm64 | 3.9 | arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python39-arm64:{version} | +| arm64 | 3.10 | arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python310-arm64:{version} | +| arm64 | 3.11 | arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python311-arm64:{version} | +| arm64 | 3.12 | arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python312-arm64:{version} | - return result - ``` +## Event Handler: headers are case-insensitive -=== "[After] Context manager" +According to the [HTTP RFC](https://datatracker.ietf.org/doc/html/rfc9110#section-5.1){target="_blank"}, HTTP headers are case-insensitive. As a result, we have deprecated the `get_header_value` function to align with this standard. Instead, we recommend using `app.current_event.headers.get` to access header values directly - ```python hl_lines="1 11 16" - from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType, batch_processor +Consequently, the `case_sensitive` parameter in this function no longer has any effect, as we now ensure consistent casing by normalizing headers for you. This function will be removed in a future release, and we encourage users to adopt the new method to access header values. +```diff +import requests +from requests import Response - def record_handler(record): - return_value = do_something_with(record["body"]) - return return_value +from aws_lambda_powertools import Logger, Tracer +from aws_lambda_powertools.event_handler import APIGatewayRestResolver +from aws_lambda_powertools.logging import correlation_paths +from aws_lambda_powertools.utilities.typing import LambdaContext - def lambda_handler(event, context): - records = event["Records"] +tracer = Tracer() +logger = Logger() +app = APIGatewayRestResolver() - processor = BatchProcessor(event_type=EventType.SQS) - with processor(records, record_handler): - result = processor.process() +@app.get("/todos") +@tracer.capture_method +def get_todos(): + endpoint = "https://jsonplaceholder.typicode.com/todos" - return processor.response() - ``` + # BEFORE - v2 +- api_key: str = app.current_event.get_header_value(name="X-Api-Key", case_sensitive=True, default_value="") -## Event Handler headers response format + # NOW - v3 ++ api_key: str = app.current_event.headers.get("X-Api-Key", "") -!!! note "No code changes required" + todos: Response = requests.get(endpoint, headers={"X-Api-Key": api_key}) + todos.raise_for_status() -This only applies if you're using `APIGatewayRestResolver` and asserting custom header values in your tests. + return {"todos": todos.json()} -Previously, custom headers were available under `headers` key in the Event Handler response. -```python title="V1 response headers" hl_lines="2" -{ - "headers": { - "Content-Type": "application/json" - } -} +# You can continue to use other utilities just as before +@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST) +@tracer.capture_lambda_handler +def lambda_handler(event: dict, context: LambdaContext) -> dict: + return app.resolve(event, context) ``` -In V2, we add all headers under `multiValueHeaders` key. This enables seamless support for multi-value headers and cookies in [fine grained responses](./core/event_handler/api_gateway.md#fine-grained-responses){target="_blank"}. +## Deprecated Batch Processing decorators -```python title="V2 response headers" hl_lines="2" -{ - "multiValueHeaders": { - "Content-Type": "application/json" - } -} -``` +In v2, we designated `@batch_processor` and `@async_batch_processor` as legacy modes for using the Batch Processing utility. -## DynamoDBStreamEvent in Event Source Data Classes +In v3, these have been marked as deprecated. Continuing to use them will result in warnings in your IDE and during Lambda execution. -!!! info "This also applies if you're using [**DynamoDB BatchProcessor**](https://docs.powertools.aws.dev/lambda/python/latest/utilities/batch/#processing-messages-from-dynamodb){target="_blank"}." +```diff +import json -You will now receive native Python types when accessing DynamoDB records via `keys`, `new_image`, and `old_image` attributes in `DynamoDBStreamEvent`. +from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType, batch_processor, process_partial_response +from aws_lambda_powertools.utilities.data_classes.sqs_event import SQSRecord +from aws_lambda_powertools.utilities.typing import LambdaContext -Previously, you'd receive a `AttributeValue` instance and need to deserialize each item to the type you'd want for convenience, or to the type DynamoDB stored via `get_value` method. +processor = BatchProcessor(event_type=EventType.SQS) -With this change, you can access data deserialized as stored in DynamoDB, and no longer need to recursively deserialize nested objects (Maps) if you had them. +@tracer.capture_method +def record_handler(record: SQSRecord): + payload: str = record.body + if payload: + item: dict = json.loads(payload) + logger.info(item) -???+ note - For a lossless conversion of DynamoDB `Number` type, we follow AWS Python SDK (boto3) approach and convert to `Decimal`. +-# BEFORE - v2 +-@batch_processor(record_handler=record_handler, processor=processor) +-def lambda_handler(event, context: LambdaContext): +- return processor.response() -```python hl_lines="15-20 24-25" -from aws_lambda_powertools.utilities.data_classes.dynamo_db_stream_event import ( - DynamoDBStreamEvent, - DynamoDBRecordEventName -) ++ # NOW - v3 ++def lambda_handler(event, context: LambdaContext): ++ return process_partial_response( ++ event=event, ++ record_handler=record_handler, ++ processor=processor, ++ context=context, ++ ) +``` + +## Event source default values + +We've modified the **Event Source Data classes** so that optional dictionaries and lists now return empty strings, dictionaries or lists instead of `None`. This improvement simplifies your code by eliminating the need for conditional checks when accessing these fields, while maintaining backward compatibility with previous implementations. + +We've applied this change broadly across various event source data classes, ensuring a more consistent and streamlined coding experience for you. + +```diff +from aws_lambda_powertools.utilities.data_classes import DynamoDBStreamEvent, event_source +from aws_lambda_powertools.utilities.typing import LambdaContext -def send_to_sqs(data: Dict): - body = json.dumps(data) - ... @event_source(data_class=DynamoDBStreamEvent) -def lambda_handler(event: DynamoDBStreamEvent, context): +def lambda_handler(event: DynamoDBStreamEvent, context: LambdaContext): for record in event.records: - # BEFORE - new_image: Dict[str, AttributeValue] = record.dynamodb.new_image - event_type: AttributeValue = new_image["eventType"].get_value - if event_type == "PENDING": - # deserialize attribute value into Python native type - # NOTE: nested objects would need additional logic - data = {k: v.get_value for k, v in image.items()} - send_to_sqs(data) - - # AFTER - new_image: Dict[str, Any] = record.dynamodb.new_image - if new_image.get("eventType") == "PENDING": - send_to_sqs(new_image) # Here new_image is just a Python Dict type +- # BEFORE - v2 +- old_image_type_return_v2 = type(record.dynamodb.old_image) +- # Output is ++ # NOW - v3 ++ old_image_type_return_v3 = type(record.dynamodb.old_image) ++ # Output is ``` -## Feature Flags and AppConfig Parameter utility +## Parameters: default cache TTL updated to 5 minutes !!! note "No code changes required" -We replaced `GetConfiguration` API ([now deprecated](https://github.com/aws-powertools/powertools-lambda-python/issues/1506#issuecomment-1266645884){target="_blank"}) with `GetLatestConfiguration` and `StartConfigurationSession`. +We have updated the cache TTL from 5 seconds to 5 minutes to reduce the number of API calls to AWS, leading to improved performance and lower costs. -As such, you must update your IAM Role permissions to allow the following IAM actions: +No code changes are necessary for this update; however, if you prefer the previous behavior, you can set the `max_age` parameter back to 5 seconds. -* `appconfig:GetLatestConfiguration` -* `appconfig:StartConfigurationSession` +## Parameters: using the new boto_config parameter -## Idempotency partition key format +In v2, you could use the `config` parameter to modify the **botocore Config** session settings. + +In v3, we renamed this parameter to `boto_config` to standardize the name with other features, such as Idempotency, and introduced deprecation warnings for users still using `config`. + +```diff +from botocore.config import Config + +from aws_lambda_powertools.utilities import parameters + +-# BEFORE - v2 +-ssm_provider = parameters.SSMProvider(config=Config(region_name="us-west-1")) + ++# NOW - v3 ++ssm_provider = parameters.SSMProvider(boto_config=Config(region_name="us-west-1")) + +def handler(event, context): + value = ssm_provider.get("/my/parameter") + return {"message": value} + +``` + +## Utilizing the new query function in JMESPath Functions + +In v2, you could use the `extract_data_from_envelope` function to search and extract data from dictionaries with JMESPath. This name was too generic and customers told us it was confusing. + +In v3, we renamed this function to `query` to align with similar frameworks in the ecosystem, and introduced deprecation warnings for users still using `extract_data_from_envelope`. + +```diff +from aws_lambda_powertools.utilities.jmespath_utils import extract_data_from_envelope, query +from aws_lambda_powertools.utilities.typing import LambdaContext -!!! note "No code changes required" -We replaced the DynamoDB partition key format to include fully qualified function/method names. This means that recent non-expired idempotent transactions will be ignored. +def handler(event: dict, context: LambdaContext) -> dict: +- # BEFORE - v2 +- some_data = extract_data_from_envelope(data=event, envelope="powertools_json(body)") -Previously, we used the function/method name to generate the partition key value. ++ # NOW - v3 ++ some_data = query(data=event, envelope="powertools_json(body)") -> e.g. `HelloWorldFunction.lambda_handler#99914b932bd37a50b983c5e7c90ae93b` + return {"data": some_data} +``` + +## Importing types from typing and typing_annotations + +We refactored our codebase to align with Python guidelines and eliminated the use of `aws_lambda_powertools.shared.types` imports. -![Idempotency Before](./media/upgrade_idempotency_before.png) +Instead, we now utilize types from the standard `typing` library, which are compatible with Python versions 3.8 and above, or from `typing_extensions` (included as a required dependency) for additional type support. -In V2, we now distinguish between distinct classes or modules that may have the same function/method name. +```diff +-# BEFORE - v2 +-from aws_lambda_powertools.shared.types import Annotated -[For example](https://github.com/aws-powertools/powertools-lambda-python/issues/1330){target="_blank"}, an ABC or Protocol class may have multiple implementations of `process_payment` method and may have different results. ++# NOW - v3 ++from typing_extensions import Annotated - +from aws_lambda_powertools.utilities.typing import LambdaContext -> e.g. `HelloWorldFunction.app.lambda_handler#99914b932bd37a50b983c5e7c90ae93b` -![Idempotency Before](./media/upgrade_idempotency_after.png) +def handler(event: dict, context: LambdaContext) -> dict: + ... + +```