Skip to content

[feature][Connector-V2][cdc] Support flink running cdc job#4918

Merged
TyrantLucifer merged 4 commits into
apache:devfrom
Carl-Zhou-CN:flink-cdc
Sep 12, 2023
Merged

[feature][Connector-V2][cdc] Support flink running cdc job#4918
TyrantLucifer merged 4 commits into
apache:devfrom
Carl-Zhou-CN:flink-cdc

Conversation

@Carl-Zhou-CN
Copy link
Copy Markdown
Member

@Carl-Zhou-CN Carl-Zhou-CN commented Jun 14, 2023 •
edited by hailin0
Loading

Purpose of this pull request

close #4605

Check list

@hailin0
Copy link
Copy Markdown
Member

hailin0 commented Jun 14, 2023

good

@TyrantLucifer
Copy link
Copy Markdown
Member

TyrantLucifer commented Jun 15, 2023

good pr

@EricJoy2048
Copy link
Copy Markdown
Member

EricJoy2048 commented Jun 15, 2023

niubility!

@hailin0
Copy link
Copy Markdown
Member

hailin0 commented Jul 9, 2023

waitting for ci check

@Carl-Zhou-CN
Copy link
Copy Markdown
Member Author

Carl-Zhou-CN commented Jul 9, 2023

@hailin0 Please help to start the client,the mysql cdc driver is abnormal

@Carl-Zhou-CN
Copy link
Copy Markdown
Member Author

Carl-Zhou-CN commented Jul 9, 2023

image
Appears to be a zeta startup exception

@hailin0
Copy link
Copy Markdown
Member

hailin0 commented Jul 10, 2023

@EricJoy2048

@Carl-Zhou-CN
Copy link
Copy Markdown
Member Author

Carl-Zhou-CN commented Jul 12, 2023

@evan766 Can you share your json configuration?

@Carl-Zhou-CN
Copy link
Copy Markdown
Member Author

Carl-Zhou-CN commented Jul 12, 2023 •
edited
Loading

image
Whether this was removed when merged？@evan766

@Carl-Zhou-CN
Copy link
Copy Markdown
Member Author

Carl-Zhou-CN commented Jul 12, 2023

@evan766 Can I add my wechat zy330600

@liugddx liugddx closed this Jul 18, 2023
@liugddx liugddx reopened this Jul 18, 2023
@hailin0
Copy link
Copy Markdown
Member

hailin0 commented Jul 25, 2023 •
edited
Loading

Please merge dev branch into this pr

@Carl-Zhou-CN

@Carl-Zhou-CN
Copy link
Copy Markdown
Member Author

Carl-Zhou-CN commented Jul 26, 2023

@hailin0 ok

@hailin0 hailin0 closed this Jul 27, 2023
@hailin0 hailin0 reopened this Jul 27, 2023
hailin0
hailin0 reviewed Jul 27, 2023
View reviewed changes
Comment thread ...erver-e2e/src/test/java/org/apache/seatunnel/e2e/connector/cdc/sqlserver/SqlServerCDCIT.java
@DisabledOnContainer(
value = {},
type = {EngineType.SPARK, EngineType.FLINK},
type = {EngineType.SPARK},
Copy link
Copy Markdown
Member

@hailin0 hailin0 Jul 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update testcase

https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mongodb-e2e/src/test/java/mongodb/MongodbCDCIT.java#L68
https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mysql-e2e/src/test/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/MysqlCDCIT.java#L58

@hailin0
Copy link
Copy Markdown
Member

hailin0 commented Jul 28, 2023

check ci error

@EricJoy2048 EricJoy2048 added this to the 2.3.3 milestone Aug 4, 2023
@ic4y ic4y removed this from the 2.3.3 milestone Aug 6, 2023
@Carl-Zhou-CN
Copy link
Copy Markdown
Member Author

Carl-Zhou-CN commented Aug 9, 2023

Temporarily blocks mongo cdc effects #5251

@Carl-Zhou-CN
Copy link
Copy Markdown
Member Author

Carl-Zhou-CN commented Aug 14, 2023

image
In his own warehouse he was successful

@hailin0 hailin0 added this to the 2.3.4 milestone Aug 23, 2023
@Carl-Zhou-CN
Copy link
Copy Markdown
Member Author

Carl-Zhou-CN commented Aug 25, 2023

@ic4y @hailin0 @EricJoy2048 @Hisoka-X @TyrantLucifer Please help review, thank you

@hailin0
Copy link
Copy Markdown
Member

hailin0 commented Sep 12, 2023

LGTM

TyrantLucifer
TyrantLucifer approved these changes Sep 12, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@TyrantLucifer TyrantLucifer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you for your contribution!

@TyrantLucifer TyrantLucifer merged commit 5e37883 into apache:dev Sep 12, 2023
@hailin0 hailin0 mentioned this pull request Sep 12, 2023
Closed
3 tasks
Hisoka-X
Hisoka-X reviewed Sep 27, 2023
View reviewed changes
Comment thread ...a/org/apache/seatunnel/core/starter/flink/execution/FlinkAbstractPluginExecuteProcessor.java
Comment on lines +95 to +109
if (pluginConfig.hasPath(SOURCE_TABLE_NAME)) {
String sourceTable = pluginConfig.getString(SOURCE_TABLE_NAME);
flinkRuntimeEnvironment.registerResultTable(
pluginConfig,
dataStream,
resultTable,
isAppendMap.getOrDefault(sourceTable, true));
registerAppendStream(pluginConfig);
return;
}
flinkRuntimeEnvironment.registerResultTable(
pluginConfig,
dataStream,
resultTable,
isAppendMap.getOrDefault(resultTable, true));
Copy link
Copy Markdown
Member

@Hisoka-X Hisoka-X Sep 27, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Carl-Zhou-CN , could you explain for me why we should register table as source table name when both have RESULT_TABLE_NAME and SOURCE_TABLE_NAME? Thanks.

I got it. Please ignore it.

gnehil pushed a commit to gnehil/seatunnel that referenced this pull request Oct 12, 2023
@Carl-Zhou-CN
[Feature][Connector-V2][CDC] Support flink running cdc job (apache#4918)
aad623f 
* [feature][Connector-V2][cdc] Support flink running cdc job

* [feature][Connector-V2][cdc] Support flink running cdc job

* [feature][Connector-V2][cdc] mongo e2e delete

---------

Co-authored-by: zhouyao <yao.zhou@marketingforce.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Hisoka-X Hisoka-X Hisoka-X left review comments

@hailin0 hailin0 hailin0 approved these changes

@TyrantLucifer TyrantLucifer TyrantLucifer approved these changes

Assignees

No one assigned

Labels

approved cdc flink

Projects

None yet

Milestone

2.3.4

Development

Successfully merging this pull request may close these issues.

[Feature][CDC] Support flink running cdc job

7 participants

@Carl-Zhou-CN @hailin0 @TyrantLucifer @EricJoy2048 @Hisoka-X @liugddx @ic4y