feat(Idempotency): add feature for manipulating idempotent responses

[walmsles]

Issue number: #2164

Summary

I have added the capability for developers to provide a response hook, which is called when the idempotency handler finds a valid idempotent response.

Changes

Addition of "response_hook" in IdempotentConfig class to enable developers to register a repsonse_hook

User experience

import datetime
import uuid
from typing import Dict

from aws_lambda_powertools import Logger
from aws_lambda_powertools.utilities.idempotency import (
    DynamoDBPersistenceLayer,
    IdempotencyConfig,
    idempotent_function,
)
from aws_lambda_powertools.utilities.idempotency.persistence.base import (
    DataRecord,
)
from aws_lambda_powertools.utilities.typing import LambdaContext

logger = Logger()

def my_response_hook(response: Dict, idempotent_data: DataRecord) -> Dict:
    # Return inserted Header data into the Idempotent Response
    response["x-idempotent-key"] = idempotent_data.idempotency_key

    # expiry_timestamp could be None so include if set
    expiry_timestamp = idempotent_data.expiry_timestamp
    if expiry_timestamp:
        expiry_time = datetime.datetime.fromtimestamp(int(expiry_timestamp))
        response["x-idempotent-expiration"] = expiry_time.isoformat()

    # Must return the response here
    return response

dynamodb = DynamoDBPersistenceLayer(table_name="IdempotencyTable")
config = IdempotencyConfig(response_hook=my_response_hook)

@idempotent_function(data_keyword_argument="order", config=config, persistence_store=dynamodb)
def process_order(order: dict) -> dict:
    # create the order_id
    order_id = str(uuid.uuid4())

    # create your logic to save the order
    # append the order_id created
    order["order_id"] = order_id

    # return the order
    return {"order": order}


def lambda_handler(event: dict, context: LambdaContext):
    config.register_lambda_context(context)  # see Lambda timeouts section
    try:
        logger.info(f"Processing order id {event.get('order_id')}")
        return process_order(order=event.get("order"))
    except Exception as err:
        return {"status_code": 400, "error": f"Erro processing {str(err)}"}

Developer experience and implementation

Response hooks

We are introducing support for hooks in the Idempotency utility. Customers can now create a function and pass it as the hook in the IdempotencyConfig. This hook will be invoked every time a request is idempotent. If the request is not idempotent, the hook will not execute.

from aws_lambda_powertools.utilities.idempotency import IdempotencyConfig
def my_response_hook(response: Dict, idempotent_data: DataRecord): ...
config = IdempotencyConfig(response_hook=my_response_hook)

Manipulating response

This hook will be able to access both: the response from the invocation and the DataRecord from the Idempotency Store. This means customers can inject new keys into the response or utilize attributes from the DataRecord to log information or create metrics, for example.

Persistent Store

This hook is compatible with any Persistent Store. Currently, we support DynamoDB and Redis for this implementation.

Handling exceptions

Customers are responsible for handling exceptions when using Response hooks. During the implementation of this new feature, we considered the possibility of implementing soft failure or raising warnings to ignore hook execution. For instance, this approach could prevent customers from having exceptions in the hook, thereby avoiding potential malfunctions in the Lambda function execution. However, we opted against this approach because it could result in unexpected behaviors and potentially breaking the contract that customers expect for Response hooks. Let me provide some context with an example.
Consider a scenario where a customer creates a hook to inject a specific key into the response when the operation is idempotent and wants to use it to trigger an alert for third-party integration. The code might appear like this:

def my_response_hook(response: Dict, idempotent_data: DataRecord) -> Dict:
    # Return inserted Header data into the Idempotent Response
    response["x-idempotent-key"] = idempotent_data.idempotency_key

    imagine_an_exception_here()

    # expiry_timestamp could be None so include if set
    expiry_timestamp = idempotent_data.expiry_timestamp
    if expiry_timestamp:
        expiry_time = datetime.datetime.fromtimestamp(int(expiry_timestamp))
        response["x-idempotent-expiration"] = expiry_time.isoformat()

    # Must return the response here
    return response

dynamodb = DynamoDBPersistenceLayer(table_name="IdempotencyTable")
config = IdempotencyConfig(response_hook=my_response_hook)

@idempotent_function(data_keyword_argument="order", config=config, persistence_store=dynamodb)
def process_order(order: dict) -> dict: ...

def lambda_handler(event: dict, context: LambdaContext):
    config.register_lambda_context(context)  # see Lambda timeouts section
    process_order = process_order(event.get("order"))

    if process_order["x-idempotent-key"] is not None:
        trigger_some_sns_alert()

If we choose to ignore the hook execution exception and only raise a warning instead of propagating the exception, customers may encounter difficulties in debugging. They may struggle to understand why the if process_order["x-idempotent-key"] is not None: condition is failing and why the x-idempotent-key doesn't exist in the response.

Documentation

The documentation will be updated to inform customers on the best practices for using Response hooks. Additionally, it will feature an example demonstrating how to utilize Response hooks and effectively handle exceptions.

Checklist

If your change doesn't seem to apply, please leave them unchecked.

• Meet tenets criteria
• I have performed a self-review of this change
• Changes have been tested
• Changes are documented
• PR title follows conventional commit semantics

Is this a breaking change?

No, not a breaking change. Configuration is optional and implementation checked for existence of the hook

RFC issue number: 2164

Checklist:

• Migration process documented
• Implement warnings (if it can live side by side)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

[walmsles]

Still need to work on documentation - will do this next.

[leandrodamascena]

Hi @walmsles! Your contributions continue to shine brightly. Access to the Idempotency operation results has long been on our backlog and I'm so happy that we will add this new feature. If you need any assistance with documentation, testing, or anything else, please don't hesitate to let me know.

Thanks

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 92.85714% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 96.25%. Comparing base (e14e768) to head (5f50be5).
Report is 265 commits behind head on develop.

Files	Patch %	Lines
...ws_lambda_powertools/utilities/idempotency/hook.py	80.00%	0 Missing and 1 partial ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #4037      +/-   ##
===========================================
- Coverage    96.38%   96.25%   -0.13%     
===========================================
  Files          214      216       +2     
  Lines        10030    10387     +357     
  Branches      1846     1927      +81     
===========================================
+ Hits          9667     9998     +331     
- Misses         259      275      +16     
- Partials       104      114      +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[walmsles]

I added documentation for review. I also changed the order so that response_hook processes after the custom de-serializer if it is used. This makes the most sense to developers so the response_hook is passed an expected structure rather than a raw event format.

[leandrodamascena]

Hey, @walmsles! I looked at the code and ran some tests, and everything looked good to me at first glance! I left some comments for us to discuss before a final review. Thank you so much!

[leandrodamascena]

Hey @walmsles! I've updated the documentation and example. I believe it's ready for the final review now.

I'll also be updating the issue with the decisions regarding exception/error handling, including why we allow customers to handle their own exceptions and why we fail in case of an exception.

[walmsles]

I love the being a Good Citizen sections! That makes so much more sense - I knew it was missing something.

[rubenfonseca]

Reviewing now

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

[leandrodamascena]

@rubenfonseca ready to one more review.

[rubenfonseca]

Great job @walmsles @leandrodamascena pulling this off!