Skip to content

Commit

Permalink
Merge branch 'master' of github.com:trustpilot/kafka-connect-dynamodb
Browse files Browse the repository at this point in the history
  • Loading branch information
ebartkus committed Jul 16, 2019
2 parents b6b4952 + 1518dbf commit af4b433
Show file tree
Hide file tree
Showing 4 changed files with 7 additions and 7 deletions.
4 changes: 2 additions & 2 deletions docs/details.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

This connector can sync multiple DynamoDB tables at the same time and it does so without requiring explicit configuration for each one. On start and at regular time intervals (by default 60s) after, it queries AWS api for DynamoDB tables which match following criteria and starts Kafka Connect task for each of them:
* ingestion TAG key set
* stack(environment) TAG key and value set
* environment TAG key and value set
* DynamoDB streams enabled (in `new_image` or `new_and_old_image` mode)


Expand Down Expand Up @@ -40,7 +40,7 @@ Since we are using two different frameworks/libraries together there are two dif
### `DISCOVERY` state and task configuration

Connector uses AWS resource group API to receive a list of DynamoDB tables which have ingestion TAG defined. Then it iterates over this list and checks if stack TAG is matched and streams are actually enabled. Connect task is started for each table which meats all requirements.
Connector uses AWS resource group API to receive a list of DynamoDB tables which have ingestion TAG defined. Then it iterates over this list and checks if environment TAG is matched and streams are actually enabled. Connect task is started for each table which meats all requirements.

`discovery` phase is executed on start and every 60 seconds(default config value) after initial start.

Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ These instructions will get you up and running this connector on your own machin
* Create `test-dynamodb-connector` DynamoDB table (or use any other name you choose)
* Enable DynamoDB streams with mode `new image` or `new and old image`
* Set TAG's:
* `stack=dev`
* `environment=dev`
* `datalake-ingest=`
* Put some random test data into it

Expand Down
6 changes: 3 additions & 3 deletions docs/options.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
"aws.access.key.id": "",
"aws.secret.key": "",

"dynamodb.table.env.tag.key": "stack",
"dynamodb.table.env.tag.key": "environment",
"dynamodb.table.env.tag.value": "dev",
"dynamodb.table.ingestion.tag.key": "datalake-ingest",

Expand All @@ -32,9 +32,9 @@
"connect.dynamodb.rediscovery.period": "60000"
}
```
`dynamodb.table.env.tag.key` - tag key used to define environment(stack). Useful if you have `staging` and `production` under same AWS account. Or if you want to use different Kafka Connect clusters to sync different tables.
`dynamodb.table.env.tag.key` - tag key used to define environment. Useful if you have `staging` and `production` under same AWS account. Or if you want to use different Kafka Connect clusters to sync different tables.

`dynamodb.table.env.tag.value` - defines from which environment or stack to ingest tables. For e.g. 'staging' or 'production'...
`dynamodb.table.env.tag.value` - defines from which environment to ingest tables. For e.g. 'staging' or 'production'...

`dynamodb.table.ingestion.tag.key` - only tables marked with this tag key will be ingested.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ public class DynamoDBSourceConnectorConfig extends AbstractConfig {
public static final String SRC_DYNAMODB_TABLE_ENV_TAG_KEY_CONFIG = "dynamodb.table.env.tag.key";
public static final String SRC_DYNAMODB_TABLE_ENV_TAG_KEY_DOC = "Define DynamoDB tables environment tag name. Only tables with dynamodb.table.env.tag.value value in this key will be ingested.";
public static final String SRC_DYNAMODB_TABLE_ENV_TAG_KEY_DISPLAY = "Environment tag key";
public static final String SRC_DYNAMODB_TABLE_ENV_TAG_KEY_DEFAULT = "stack";
public static final String SRC_DYNAMODB_TABLE_ENV_TAG_KEY_DEFAULT = "environment";

public static final String SRC_DYNAMODB_TABLE_ENV_TAG_VALUE_CONFIG = "dynamodb.table.env.tag.value";
public static final String SRC_DYNAMODB_TABLE_ENV_TAG_VALUE_DOC = "Define environment name which must be present in dynamodb.table.env.tag.key.";
Expand Down

0 comments on commit af4b433

Please sign in to comment.