Skip to content

Conversation

@dpilch
Copy link
Contributor

@dpilch dpilch commented Jun 18, 2024

Description of changes

Add a feature to import existing DynamoDB tables to an Amplify managed table. Import validation will come with a later PR.

CDK / CloudFormation Parameters Changed
  • New data source strategy: IMPORTED_AMPLIFY_TABLE.
  • New construct prop: importedAmplifyDynamoDBTableMap.
  • Final naming on these can be updated before merging to main.

importedAmplifyDynamoDBTableMap will define a mapping of model name to imported table name. All models defined with the IMPORTED_AMPLIFY_TABLE should have a matching key in the importedAmplifyDynamoDBTableMap prop.

import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import {
  AmplifyGraphqlApi,
  AmplifyGraphqlDefinition,
} from "@aws-amplify/graphql-api-construct";
import * as path from "path";

export class BackendStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const amplifyApi = new AmplifyGraphqlApi(this, "MyNewApi", {
      definition: AmplifyGraphqlDefinition.combine([
        AmplifyGraphqlDefinition.fromString(
          `
            type Todo @model @auth(rules: [{ allow: public }]) {
              title: String
            }
          `,
          {
            dbType: "DYNAMODB",
            provisionStrategy: "DEFAULT",
          },
        ),
        AmplifyGraphqlDefinition.fromString(
          `
            type Blog @model @auth(rules: [{ allow: public }]) {
              title: String
              content: String
              authors: [String]
            }
          `,
          {
            dbType: "DYNAMODB",
            provisionStrategy: "IMPORTED_AMPLIFY_TABLE",
          },
        ),
      ]),
      importedAmplifyDynamoDBTableMap: {
        // existing table
        Blog: 'Blog-xxxx-NONE',
      },
      authorizationModes: {
        defaultAuthorizationMode: "API_KEY",
        apiKeyConfig: {
          expires: cdk.Duration.days(30),
        },
      },
    });
  }
}

Issue #, if available

N/A

Description of how you validated changes

  • Manual testing
  • Unit tests, more to come in future PRs
  • This is merging to a feature branch and E2E tests will be added with a separate PR

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@dpilch dpilch changed the base branch from main to feature/migration June 18, 2024 19:13
@dpilch dpilch changed the title Import table feat: import existing table to amplify managed table Jul 3, 2024
@dpilch dpilch changed the base branch from feature/migration to feature/gen2-migration July 3, 2024 14:36
@dpilch dpilch marked this pull request as ready for review July 5, 2024 15:18
@dpilch dpilch requested review from a team as code owners July 5, 2024 15:18
expect(() => validateImportedTableMap(definition)).toThrow('Table mapping is missing for imported Amplify DynamoDB table strategy.');
});
});
});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also have multi-strategy tests. One case should ensure that a model that uses a different strategy throws an error if it is included in the importedAmplifyDynamoDBTableMap. Ideally, the error would note that the strategy is invalid, rather than a generic "missing" or "invalid" error. E.g., from your generator test below:

dataSourceStrategies: {
        Comment: DDB_DEFAULT_DATASOURCE_STRATEGY,
        Post: DDB_AMPLIFY_MANAGED_DATASOURCE_STRATEGY,
        Author: IMPORTED_DDB_AMPLIFY_MANAGED_DATASOURCE_STRATEGY,
      },
      importedAmplifyDynamoDBTableMap: {
        Author: 'Author-myApiId-myEnv',
      },

If you add Post to the map, we'd hope to see an error like "Table mapping includes models that do not use an imported strategy (Post)." or similar

};

/**
* Type predicate that returns true if `obj` is a AmplifyDynamoDbModelDataSourceStrategy
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Type predicate that returns true if `obj` is a AmplifyDynamoDbModelDataSourceStrategy
* Type predicate that returns true if `obj` is an ImportedAmplifyDynamoDbModelDataSourceStrategy

const tableLogicalName = ModelResourceIDs.ModelTableResourceID(modelName);
const tableName = context.resourceHelper.generateTableName(modelName);
const importedTableMap = context.importedAmplifyDynamoDBTableMap ?? {};
const isTableImported = importedTableMap[modelName] !== undefined;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Piling onto my above comments -- here you check for the existence of the importedTableMap, but don't validate the strategy. If we have conditions that can result in inconsistency b/t strategy & mapping, this is a source of potential future bugs.

let result;
switch (event.RequestType) {
case 'Create':
if (tableDef.isImported) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. rather than inlining this big if block, can we refactor to a standalone function to make the "Create" case a bit easier to parse?
  2. I don't see anything to suggest otherwise, but please confirm that this won't cause any issues with the Tagging support for Amplify-managed tables that we're introducing in fix: add tags to dynamodb tables #2694

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Will do.
  2. I don't see any issues here, but I'll add a test to cover this case before merging to main.

switch (event.RequestType) {
case 'Create':
if (tableDef.isImported) {
// TODO: Add import validation
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What validation needs to be added?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The validation is to ensure the DDB properties align (Example: KeySchema, SSESpecification, etc.). WIP PR: #2696

palpatim
palpatim previously approved these changes Jul 11, 2024
},
},
};
// @ts-expect-error
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume the @ts-expect-error is to suppress the compile warning about the missing tableName attribute?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, updated to clarify

typeof (strategy as any)['provisionStrategy'] === 'string' &&
(strategy as any)['provisionStrategy'] === 'IMPORTED_AMPLIFY_TABLE' &&
typeof (strategy as any)['tableName'] === 'string' &&
(strategy as any)['tableName'] !== ''
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we turn this "empty imported table name" scenario handling into a validation check and fail instead of treating this as a greenfield scenario?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would actually fail with No resource generator assigned for Post with dbType DYNAMODB. The error message could be improved though. I can tackle this in the next PR.

// TODO: Add import validation
console.log('Initiating table import process');
console.log('Fetching current table state');
console.log(`Table name: ${tableDef.tableName}`);
Copy link
Contributor

@phani-srikar phani-srikar Jul 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor nit for next PR: (since you already have a TODO up there):
Can combine above two lines to console.log('Fetching current state of table ${tableDef.tableName}'); to make it more readable in the logs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll update this in the next PR.

@dpilch dpilch merged commit b3fb28f into feature/gen2-migration Jul 12, 2024
@dpilch dpilch deleted the import-table branch July 12, 2024 22:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants