Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DynamoDB TypeDeserializer support encoding by aws step functions operations #4253

Open
1 of 2 tasks
johnclarke94 opened this issue Sep 2, 2024 · 2 comments
Open
1 of 2 tasks
Labels
dynamodb feature-request This issue requests a feature. p2 This is a standard priority issue service-api This issue is caused by the service API, not the SDK implementation. stepfunctions

Comments

@johnclarke94
Copy link

Describe the feature

The TypeDeserializer can't currently decode all items returned from arn:aws:states:::aws-sdk:dynamodb:scan by AWS step functions (and presumably other database operations).

This is caused by non-standard encoding where NULL is Nul and BOOL is Bool within a nested map. Top level attributes do however use the more standard NULL and BOOL.

Use Case

To be able to handle this form of encoding provided by AWS without a custom wrapper which will need to be added by many users. It can easily lead to unexpected errors when this special encoding is not known by a developer. It would be great to have this abstracted by boto3.

The current implementation returns this exception or similar for Bool

[ERROR] TypeError: Dynamodb type Nul is not supported
Traceback (most recent call last):
  File "/var/lang/lib/python3.12/site-packages/boto3/dynamodb/types.py", line 280, in deserialize
    return deserializer(value[dynamodb_type])
  File "/var/lang/lib/python3.12/site-packages/boto3/dynamodb/types.py", line 310, in _deserialize_m
    return {k: self.deserialize(v) for k, v in value.items()}
  File "/var/lang/lib/python3.12/site-packages/boto3/dynamodb/types.py", line 279, in deserialize
    raise TypeError(f'Dynamodb type {dynamodb_type} is not supported')

Proposed Solution

I believe a minor update to the TypeDeserializer to understand these two additional cases.

Other Information

No response

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change

SDK version used

1.34.107

Environment details (OS name and version, etc.)

Python 3.12 on AWS Lambda

@johnclarke94 johnclarke94 added feature-request This issue requests a feature. needs-triage This issue or PR still needs to be triaged. labels Sep 2, 2024
@johnclarke94 johnclarke94 changed the title DynamoDB TypeSerializer support encoding by aws step functions operations DynamoDB TypeDeserializer support encoding by aws step functions operations Sep 2, 2024
@tim-finnigan
Copy link
Contributor

Thanks for reaching out. It looks like similar issues have been reported in other AWS SDK repositories, for example: aws/aws-sdk-js#4294, aws/aws-sdk-js-v3#5423, aws/aws-sdk-js-v3#4948

Based on this comment it sounds like the Step Functions behavior is expected, but has forced workarounds like the one mentioned here in another comment. Since this issue is upstream with Step Functions API behavior, it probably needs to either be addressed by them or further clarified in their documentation. I'll try reaching out internally to get more feedback on this.

@tim-finnigan tim-finnigan added dynamodb p2 This is a standard priority issue stepfunctions service-api This issue is caused by the service API, not the SDK implementation. and removed needs-triage This issue or PR still needs to be triaged. labels Sep 3, 2024
@tim-finnigan
Copy link
Contributor

I heard back from the Step Functions team, who wanted to highlight the updated wording on parameters in their documentation:

Parameters in Step Functions are expressed in PascalCase, even if the native service API is in camelCase.

They also created a backlog item for tracking potential improvements to this workflow. But any future changes would need to be backwards compatible in order to not break customers who rely on the current behavior.

Ideally the Step Functions team can provide a solution to improve integration with DynamoDB connectors (along with any other services that might be affected.) If this were to be addressed somehow on the SDK side, then that would require coordination with other AWS SDKs, along with the Step Functions team, in order to provide consistent behavior. For now it should be a straightforward workaround to convert the types manually. In the long term, we probably need to wait for the Step Functions team to research this further and propose alternative approaches.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dynamodb feature-request This issue requests a feature. p2 This is a standard priority issue service-api This issue is caused by the service API, not the SDK implementation. stepfunctions
Projects
None yet
Development

No branches or pull requests

2 participants