feat(cdc): support sql server cdc #17429

KeXiangWang · 2024-06-24T19:50:40Z

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Support sql server cdc.

There're several potential issues in this implementation:

The time precision is not sufficient (only 3 digit numbers after decimal point). It's Debezium's bug, and will be fixed in their 2.7.0+ release. We can wait for their fix.
Have not check the timezone related issues.
Currently, the conversion between sql server types and rw types are anonymous without a type specification. As we have multiple sql server types mapped into one rw type, it may have problems when trying to map it back to sql server types. This may need a refactorization of the tiberius library.
tiberius, which is library that provides the client and type conversion functions, is now out of maintenance. The good news is that the library is not complicated.
Both mysql and postgres support transactional CDC. Due to the design of debezium, for Sql Server CDC, it would come at the expense of the freshness of cdc sourcec. So in this implementation, Sql Server CDC is not transactional.
There're still some types not supported.
Sql server allows user to create multiple capture instances for a single table (similar to slot in postgres), but debezium does not allow user to specify which capture instance to track. We may need to add some constraints for this.
More tests are needed:

Create source xxx (*)
Tests with larger scale data.

Checklist

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
I have added test labels as necessary. See details.
I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
All checks passed in ./risedev check (or alias, ./risedev c)
My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

ci/scripts/e2e-paid-source-test.sh

BugenZhao · 2024-07-03T03:07:12Z

ci/scripts/e2e-paid-source-test.sh

+echo "--- Install sql server client"
+curl https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add -
+curl https://packages.microsoft.com/config/ubuntu/20.04/prod.list | sudo tee /etc/apt/sources.list.d/msprod.list
+apt-get update -y
+ACCEPT_EULA=Y DEBIAN_FRONTEND=noninteractive apt-get install -y mssql-tools unixodbc-dev


We should consider doing this in ci/Dockerfile.

ci/scripts/e2e-paid-source-test.sh

e2e_test/source/cdc_paid/cdc.load.slt

e2e_test/source/cdc_paid/cdc.share_stream.slt

...nnector-service/src/main/java/com/risingwave/connector/source/common/DbzConnectorConfig.java

src/connector/src/source/cdc/external/sql_server.rs

KeXiangWang · 2024-07-25T19:42:06Z

A short summary of required following up:

Refactoring the test cases, following the inline test cases.
Fulfill task id in connector metrics.
Make sure the impl works well with uppercase table/schema/dbname

StrikeW · 2024-07-26T02:10:16Z

src/connector/src/source/cdc/external/sql_server.rs

+                match item {
+                    QueryItem::Metadata(meta) => {
+                        for col in meta.columns() {
+                            column_descs.push(ColumnDesc::named(


can we get these column meta from the INFORMATION_SCHEMA (in the following query)? Because our pg and mysql source all rely on the information in INFORMATION_SCHEMA to support the auto schema mapping.

Will do it later.

xxchan

Wait, we are going to made this paid feature, right? But it seems this is not gated yet (like #17683, #17795). This should be a blocker before this feature goes into 1.11.

RickLeite · 2024-07-29T14:52:59Z

Awesome 🚀

KeXiangWang · 2024-08-06T00:36:03Z

Wait, we are going to made this paid feature, right? But it seems this is not gated yet (like #17683, #17795). This should be a blocker before this feature goes into 1.11.

Added

xxchan

Others generally LGTM. Thank you for the hard work! (CDC business logic not reviewed)

ci/scripts/e2e-source-test.sh

xxchan · 2024-08-06T02:54:29Z

e2e_test/source/cdc_inline/sql_server_cdc/sql_server_cdc.slt

+# ------------ kill cluster ------------
+# system ok
+# risedev kill
+
+sleep 30s


What's the intention here? 👀

Ah, I forget to uncomment them. It's for testing sql server cdc with recovery.

Yeah, I think we do lack recovery tests, but perhaps need to come up with a design.
related issue: #16356

It seems we cannot handle recovery with current cdc_inline design. The sqllogictest gets query failed: connection closed after the RW cluster's recovery. I'll comment them for now.

e2e_test/source/cdc_inline/sql_server_cdc/sql_server_cdc.slt

src/frontend/src/handler/create_table.rs

src/common/src/types/datetime.rs

src/connector/Cargo.toml

StrikeW

Rest LGTM!

e2e_test/source/cdc_inline/sql_server_cdc/sql_server_cdc.slt

src/connector/src/sink/sqlserver.rs

src/connector/src/source/cdc/external/sql_server.rs

src/connector/src/source/reader/reader.rs

already done

Cargo.lock

KeXiangWang requested a review from a team as a code owner June 24, 2024 19:50

graphite-app bot requested a review from a team June 24, 2024 19:50

github-actions bot added the type/feature label Jun 24, 2024

KeXiangWang marked this pull request as draft June 25, 2024 18:46

KeXiangWang force-pushed the wkx/sql-server-cdc branch from c30b598 to 3e1171b Compare June 25, 2024 19:02

KeXiangWang marked this pull request as ready for review June 25, 2024 19:39

xxchan mentioned this pull request Jun 26, 2024

update developer guide #17436

Open

10 tasks

KeXiangWang mentioned this pull request Jul 1, 2024

feat(debeizum): upgrade version of debeizum from 2.4.2 to 2.6.2 #17504

Merged

9 tasks

KeXiangWang force-pushed the wkx/sql-server-cdc branch 2 times, most recently from 7567d83 to 0c371b7 Compare July 2, 2024 22:32

KeXiangWang requested review from StrikeW and xiangjinwu July 3, 2024 01:42

xxchan previously requested changes Jul 3, 2024

View reviewed changes

ci/scripts/e2e-paid-source-test.sh Outdated Show resolved Hide resolved

ci/scripts/e2e-paid-source-test.sh Outdated Show resolved Hide resolved

BugenZhao reviewed Jul 3, 2024

View reviewed changes

xxchan mentioned this pull request Jul 3, 2024

doc: add connector dev guide #17555

Merged

9 tasks

StrikeW reviewed Jul 19, 2024

View reviewed changes

KeXiangWang force-pushed the wkx/sql-server-cdc branch from 81d203b to 54e24f0 Compare July 25, 2024 19:43

StrikeW reviewed Jul 26, 2024

View reviewed changes

graphite-app bot requested a review from xxchan July 26, 2024 02:51

xxchan reviewed Jul 26, 2024

View reviewed changes

chrisolszewski mentioned this pull request Jul 29, 2024

feat(connector): SQL Server sink connection redirect support #17848

Closed

9 tasks

KeXiangWang added 2 commits August 5, 2024 18:27

feat(cdc): support sql server cdc

7f450b5

small refactor

33a61e0

KeXiangWang force-pushed the wkx/sql-server-cdc branch from 9b28189 to 33a61e0 Compare August 5, 2024 22:34

KeXiangWang added 4 commits August 5, 2024 18:39

tmp

0f92d8a

tmp

8d61f7d

tmp

0bfafbe

try

f377461

tmp

827fade

xxchan reviewed Aug 6, 2024

View reviewed changes

KeXiangWang added 4 commits August 5, 2024 23:33

fix

fb085ab

small refactor

311ae44

fix test

0c22d78

fix test

8536f5d

StrikeW approved these changes Aug 7, 2024

View reviewed changes

KeXiangWang added 2 commits August 6, 2024 22:05

fix comments

6f318f3

tmp

63801dd

KeXiangWang requested a review from a team August 7, 2024 02:24

xxchan reviewed Aug 7, 2024

View reviewed changes

Cargo.lock Show resolved Hide resolved

xxchan approved these changes Aug 7, 2024

View reviewed changes

xxchan enabled auto-merge August 7, 2024 02:43

xxchan added this pull request to the merge queue Aug 7, 2024

KeXiangWang removed this pull request from the merge queue due to a manual request Aug 7, 2024

tmp

1bca226

KeXiangWang enabled auto-merge August 7, 2024 03:20

xxchan disabled auto-merge August 7, 2024 03:24

KeXiangWang added this pull request to the merge queue Aug 7, 2024

Merged via the queue into main with commit accfb49 Aug 7, 2024
31 of 32 checks passed

KeXiangWang deleted the wkx/sql-server-cdc branch August 7, 2024 04:33

This was referenced Aug 13, 2024

fix(mssql-cdc): remove vendored-openssl of tiberus #18013

Merged

doc(source): add sqlserver-cdc risingwavelabs/risingwave-docs#2482

Merged

lmatz mentioned this pull request Sep 3, 2024

SQL Server CDC Source #16178

Closed

This was referenced Oct 23, 2024

RUSTSEC-2021-0153: encoding is unmaintained #17285

Closed

RUSTSEC-2024-0336: rustls::ConnectionCommon::complete_io could fall into an infinite loop based on network input #17284

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(cdc): support sql server cdc #17429

feat(cdc): support sql server cdc #17429

KeXiangWang commented Jun 24, 2024 •

edited

Loading

BugenZhao Jul 3, 2024

KeXiangWang commented Jul 25, 2024

StrikeW Jul 26, 2024

KeXiangWang Aug 7, 2024

xxchan left a comment

RickLeite commented Jul 29, 2024

KeXiangWang commented Aug 6, 2024

xxchan left a comment

xxchan Aug 6, 2024

KeXiangWang Aug 6, 2024

xxchan Aug 6, 2024 •

edited

Loading

KeXiangWang Aug 7, 2024

StrikeW left a comment

feat(cdc): support sql server cdc #17429

feat(cdc): support sql server cdc #17429

Conversation

KeXiangWang commented Jun 24, 2024 • edited Loading

What's changed and what's your intention?

Checklist

Documentation

Release note

BugenZhao Jul 3, 2024

Choose a reason for hiding this comment

KeXiangWang commented Jul 25, 2024

StrikeW Jul 26, 2024

Choose a reason for hiding this comment

KeXiangWang Aug 7, 2024

Choose a reason for hiding this comment

xxchan left a comment

Choose a reason for hiding this comment

RickLeite commented Jul 29, 2024

KeXiangWang commented Aug 6, 2024

xxchan left a comment

Choose a reason for hiding this comment

xxchan Aug 6, 2024

Choose a reason for hiding this comment

KeXiangWang Aug 6, 2024

Choose a reason for hiding this comment

xxchan Aug 6, 2024 • edited Loading

Choose a reason for hiding this comment

KeXiangWang Aug 7, 2024

Choose a reason for hiding this comment

StrikeW left a comment

Choose a reason for hiding this comment

KeXiangWang commented Jun 24, 2024 •

edited

Loading

xxchan Aug 6, 2024 •

edited

Loading