DBZ-9696 Add config for Vitess tables to copy flag

[twthorn]

Support for table copy list added here vitessio/vitess#18184

Implement this support for Debezium as well with a config.

Add unit test & itests.

Also fix one other itest now that the bug with open lineage was fixed in debezium main repo (we need to enable open lineage explicitly now to get this test to assert correctly) debezium/debezium#6873

[github-actions]

Hi @twthorn, thanks for your contribution. Please prefix the commit message(s) with the DBZ-xxx JIRA issue key.

[twthorn]

@jpechane can you take a look at this when you get the chance? Thanks!

[jpechane]

@twthorn I wonder if there is any use case for this to be different from table.include.list?

[github-actions]

Hi @twthorn, thanks for your contribution. Please prefix the commit message(s) with the DBZ-xxx JIRA issue key.

[twthorn]

@jpechane Thanks for the review

@twthorn I wonder if there is any use case for this to be different from table.include.list?

Yes, there is. For example, assume we are streaming data for N tables for a single connector, so table include list is size N. However a downstream user only needs a copy or snapshot of just table t1 due to some issue (e.g., maybe they dropped data or data was corrupted). In this case, we use the separate tables to copy config to perform the copy just for that table t1 (and not all N tables).

[twthorn]

Note: Two of the tests will trivially fail until we get the latest vitess/lite docker build in quay.io (or wait and add a new released version) in quay.io because we need this bug fix vitessio/vitess#18938

[jpechane]

@jpechane Thanks for the review

@twthorn I wonder if there is any use case for this to be different from table.include.list?

Yes, there is. For example, assume we are streaming data for N tables for a single connector, so table include list is size N. However a downstream user only needs a copy or snapshot of just table t1 due to some issue (e.g., maybe they dropped data or data was corrupted). In this case, we use the separate tables to copy config to perform the copy just for that table t1 (and not all N tables).

Can't we use io.debezium.config.CommonConnectorConfig.SNAPSHOT_MODE_TABLES for that purpose then?

[jpechane]

@twthorn Applied, thanks! Could you please send a docs PR for the feature?

[twthorn]

@jpechane thanks for the review! I opened the docs PR here debezium/debezium#6922