cdc: Support Kafka sink #381

pingyu · 2023-12-31T15:56:55Z

Feature Request

Is your feature request related to a problem? Please describe:

Describe the feature you'd like:

Support sink to Kafka.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

* wip Signed-off-by: Ping Yu <[email protected]> * add go:generate Signed-off-by: Ping Yu <[email protected]> * add tools/Makefile Signed-off-by: Ping Yu <[email protected]> * fix build error Signed-off-by: Ping Yu <[email protected]> * built Signed-off-by: Ping Yu <[email protected]> * fix Signed-off-by: Ping Yu <[email protected]> * fix ut Signed-off-by: Ping Yu <[email protected]> * clean up Signed-off-by: Ping Yu <[email protected]> * fix ut Signed-off-by: Ping Yu <[email protected]> * add ut Signed-off-by: Ping Yu <[email protected]> * fix check Signed-off-by: Ping Yu <[email protected]> * polish Signed-off-by: Ping Yu <[email protected]> --------- Signed-off-by: Ping Yu <[email protected]>

* wip Signed-off-by: Ping Yu <[email protected]> * add go:generate Signed-off-by: Ping Yu <[email protected]> * add tools/Makefile Signed-off-by: Ping Yu <[email protected]> * fix build error Signed-off-by: Ping Yu <[email protected]> * built Signed-off-by: Ping Yu <[email protected]> * fix Signed-off-by: Ping Yu <[email protected]> * fix ut Signed-off-by: Ping Yu <[email protected]> * clean up Signed-off-by: Ping Yu <[email protected]> * fix ut Signed-off-by: Ping Yu <[email protected]> * add ut Signed-off-by: Ping Yu <[email protected]> * fix check Signed-off-by: Ping Yu <[email protected]> * first works Signed-off-by: Ping Yu <[email protected]> * kafka sink in run.sh Signed-off-by: Ping Yu <[email protected]> * fix failpoint Signed-off-by: Ping Yu <[email protected]> * fix fmt Signed-off-by: Ping Yu <[email protected]> * fix sigstop Signed-off-by: Ping Yu <[email protected]> * fix check Signed-off-by: Ping Yu <[email protected]> * skip not support case Signed-off-by: Ping Yu <[email protected]> * default to kafka Signed-off-by: Ping Yu <[email protected]> * update readme.md in tests Signed-off-by: Ping Yu <[email protected]> * polish Signed-off-by: Ping Yu <[email protected]> --------- Signed-off-by: Ping Yu <[email protected]>

* kafka consumer tolerate TiKV errors Signed-off-by: Ping Yu <[email protected]> * polish Signed-off-by: Ping Yu <[email protected]> * improve download Signed-off-by: Ping Yu <[email protected]> * fix no wget Signed-off-by: Ping Yu <[email protected]> * trap kafka consumer log on exit Signed-off-by: Ping Yu <[email protected]> * more logs Signed-off-by: Ping Yu <[email protected]> * more memory usage for kafka sink Signed-off-by: Ping Yu <[email protected]> * trigger CI Signed-off-by: Ping Yu <[email protected]> * skip error on memory oversize for Kafka sink Signed-off-by: Ping Yu <[email protected]> * longer check_sync_diff timeout Signed-off-by: Ping Yu <[email protected]> * check event nil Signed-off-by: Ping Yu <[email protected]> --------- Signed-off-by: Ping Yu <[email protected]>

This was referenced Dec 31, 2023

[close #381] Support Kafka sink #380

Merged

[close #381] Add integration tests for Kafka sink #382

Merged

pingyu closed this as completed in #380 Jan 5, 2024

pingyu mentioned this issue Jan 20, 2024

[to #381] Add more integration tests for Kafka sink #387

Merged