Feature/init sync #11
Conversation
* update kcl and dynamo packages * add dist property to travis file * override abstract CW methods Co-authored-by: fal-trustpilot <[email protected]>
| .define(SRC_INIT_SYNC_DELAY_CONFIG, | ||
| ConfigDef.Type.INT, | ||
| SRC_INIT_SYNC_DELAY_DEFAULT, | ||
| ConfigDef.Importance.LOW, | ||
| SRC_INIT_SYNC_DELAY_DOC, | ||
| CONNECTOR_GROUP, 2, | ||
| CONNECTOR_GROUP, 3, |
There was a problem hiding this comment.
What is 'CONNECTOR_GROUP' ?
There was a problem hiding this comment.
This class is used to display configs in the UI on something like Confluent Hub (width, grouping, importance). No real use outside of that.
| HashMap<String, Object> offset = new HashMap<>(); | ||
| offset.put("table_name", tableName); | ||
| offset.put("init_sync_state", "SKIPPED"); | ||
| offset.put("init_sync_start", Instant.parse("2001-01-02T00:00:00.00Z").toEpochMilli()); |
There was a problem hiding this comment.
does this require putting a sync_start time even if the init sync is skipped?
There was a problem hiding this comment.
That function does require it- which shouldn't be a problem because init sync skip logic puts that field to 0. If it's coming from an offset, it means it will have that field already.
I'll change the test to be epoch 0.
|
Can you add a comment to the README documenting this setting? |
|
for larger stream - do we want to increase Stream read limit - default is 1000 records - configured under com.trustpilot.connector.dynamodb.Constants.java |
That's a great find. Looks like that class' default is 10,000- not 1,000. I will address throughput in the next PR.
There's no
Take a look at the README and documentation. |
Whoops- missed that. Thanks for updating |
amazon-kinesis-clientanddynamodb-streams-kinesis-adapterand adds some CloudWatch metrics.init.sync.skipflag. This optional flag will skip the initial load of dynamodb and read fromLATESTon the DynamoDBStream.