Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(dynamodb): import data from the bucket #28610

Merged
merged 7 commits into from
Jan 19, 2024

Conversation

sakurai-ryo
Copy link
Contributor

@sakurai-ryo sakurai-ryo commented Jan 8, 2024

This PR adds support for importing data from a bucket.
This feature is not supported by AWS::DynamoDB::GlobalTable, so it is implemented only in the Table construct.

Details of this feature

Users can create DynamoDB table using data from the bucket.
The data to be imported can be a single object or multiple objects.
This can be selected by specifying the prefix of the key.

Data format must be CSV, DynamoDB JSON, or Amazon Ion.
ZSTD or GZIP compressed data can also be imported.

For CSV files, delimiter and header can be specified as options.
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/S3forDynamoDB.html

In CloudFormation, use this feat as follows.

AWSTemplateFormatVersion: "2010-09-09"
Resources:
  MyTable:
    Type: AWS::DynamoDB::Table
    Properties:
      AttributeDefinitions:
        -
          AttributeName: "id"
          AttributeType: "S"
      KeySchema:
        -
          AttributeName: "id"
          KeyType: "HASH"
      TableName: "myTableName"
      ImportSourceSpecification:
        InputCompressionType: NONE # GZIP | ZSTD 
        InputFormat: DYNAMODB_JSON # ION | CSV
        S3BucketSource:
          S3Bucket: cloudfs-s3
          S3KeyPrefix: db

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-dynamodb-table-importsourcespecification.html

Closes #21825


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@github-actions github-actions bot added effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p1 admired-contributor [Pilot] contributed between 13-24 PRs to the CDK labels Jan 8, 2024
@aws-cdk-automation aws-cdk-automation requested a review from a team January 8, 2024 10:53
@sakurai-ryo sakurai-ryo changed the title feat(dynamodb): support for importing data from the bucket feat(dynamodb): import data from the bucket Jan 8, 2024
@aws-cdk-automation aws-cdk-automation added the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Jan 8, 2024
Copy link
Contributor

@paulhcsun paulhcsun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall these changes look good to me. Only a single comment.

*/
private static validCsvDelimiters = [',', '\t', ':', ';', '|', ' '];

private static readableValidCsvDelimiters = ['comma (,)', 'tab (\\t)', 'colon (:)', 'semicolon (;)', 'pipe (|)', 'space ( )'];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this an extra backslash or is it supposed to be there?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @paulhcsun.
This backslash is intended.
Because \t is an escape sequence in js, it is necessary to escape \t with a backslash to display it as a string.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Lexical_grammar#white_space

Copy link
Contributor

@paulhcsun paulhcsun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work @sakurai-ryo!

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: 79cadce
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Copy link
Contributor

mergify bot commented Jan 19, 2024

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@mergify mergify bot merged commit 45b8398 into aws:main Jan 19, 2024
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
admired-contributor [Pilot] contributed between 13-24 PRs to the CDK effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p1 pr/needs-maintainer-review This PR needs a review from a Core Team Member
Projects
None yet
Development

Successfully merging this pull request may close these issues.

DynamoDB: Import table from S3
3 participants