Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improve][Doris Connector] Unified serialization method,Use RowToJsonConverter and TextSerializationSchema #7229

Merged
merged 4 commits into from
Jul 22, 2024
Merged

[Improve][Doris Connector] Unified serialization method,Use RowToJsonConverter and TextSerializationSchema #7229

merged 4 commits into from
Jul 22, 2024

Conversation

liugddx
Copy link
Member

@liugddx liugddx commented Jul 17, 2024

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Hisoka-X
Hisoka-X reviewed Jul 17, 2024
View reviewed changes
Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall LGTM. Only one minior suggestion.

...is/src/main/java/org/apache/seatunnel/connectors/doris/serialize/SeaTunnelRowSerializer.java Show resolved Hide resolved
@liugddx liugddx requested a review from Hisoka-X July 21, 2024 09:03
Hisoka-X
Hisoka-X reviewed Jul 21, 2024
View reviewed changes
...src/test/java/org/apache/seatunnel/connectors/doris/serialize/SeaTunnelRowConverterTest.java Show resolved Hide resolved
...doris/src/main/java/org/apache/seatunnel/connectors/doris/source/serialization/RowBatch.java
@@ -71,21 +75,22 @@

@Slf4j
public class RowBatch {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why change RowBatch? Any test case?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why change RowBatch? Any test case?

I adapted the latest version of doris, and I found that many types were incompatible, so I added some type conversions.It is backwards compatible

...doris/src/main/java/org/apache/seatunnel/connectors/doris/source/serialization/RowBatch.java Outdated Show resolved Hide resolved
@liugddx liugddx requested a review from Hisoka-X July 22, 2024 01:13
@liugddx
Copy link
Member Author

liugddx commented Jul 22, 2024

After apache/doris#38174 is resolved, I will upgrade to the latest doris version as a image for test cases.

@Hisoka-X
Copy link
Member

After apache/doris#38174 is resolved, I will upgrade to the latest doris version as a image for test cases.

Please create an issue for this.

Hisoka-X
Hisoka-X reviewed Jul 22, 2024
View reviewed changes
Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM except one minor problem.

...doris/src/main/java/org/apache/seatunnel/connectors/doris/source/serialization/RowBatch.java Outdated Show resolved Hide resolved
gdliu3 and others added 2 commits July 22, 2024 11:08
1
8e21dd9
@liugddx @Hisoka-X
Update seatunnel-connectors-v2/connector-doris/src/main/java/org/apac…
d291a51 
…he/seatunnel/connectors/doris/source/serialization/RowBatch.java

Co-authored-by: Jia Fan <[email protected]>
@liugddx liugddx merged commit 4b3af9b into apache:dev Jul 22, 2024
4 of 5 checks passed
hawk9821 pushed a commit to hawk9821/seatunnel that referenced this pull request Jul 27, 2024
@liugddx @Hisoka-X @hawk9821
[Improve][Doris Connector] Unified serialization method,Use RowToJson…
de1c1dc 
…Converter and TextSerializationSchema (apache#7229)

* 1

* 1

* 1

* Update seatunnel-connectors-v2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/source/serialization/RowBatch.java

Co-authored-by: Jia Fan <[email protected]>

---------

Co-authored-by: gdliu3 <[email protected]>
Co-authored-by: Jia Fan <[email protected]>
jovezhong added a commit to jovezhong/seatunnel that referenced this pull request Aug 12, 2024
@jovezhong
Squashed commit of the following:
e8e887b 
commit 9d161d2
Author: lizhenglei <[email protected]>
Date:   Mon Aug 12 14:03:21 2024 +0800

    [Fix] Fix http e2e case (apache#7356)

    * 1

    * fix

    * fix

    * fix

    * fix

    * fix

    ---------

    Co-authored-by: lizhenglei <[email protected]>

commit 2489f64
Author: hailin0 <[email protected]>
Date:   Mon Aug 12 13:36:15 2024 +0800

    [Improve][Connector] Add multi-table sink option check (apache#7360)

    * [Improve][Connector] Add multi-table sink option check

    * fix

commit fa34ac9
Author: Jia Fan <[email protected]>
Date:   Mon Aug 12 13:35:43 2024 +0800

    [Improve][API] Check catalog table fields name legal before send to downstream (apache#7358)

    * [Improve][API] Check catalog table fields name legal before send to downstream

    * update

commit 068c5e3
Author: Tyrantlucifer <[email protected]>
Date:   Mon Aug 12 12:55:41 2024 +0800

    [Core][Flink] refactor flink proxy source/sink (apache#7355)

commit e23e3ac
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Sun Aug 11 17:40:02 2024 +0800

    Bump org.apache.activemq:activemq-client (apache#7323)

    Bumps org.apache.activemq:activemq-client from 5.14.5 to 5.15.16.

commit d46cf16
Author: Jarvis <[email protected]>
Date:   Sat Aug 10 23:47:12 2024 +0800

    [Feature] Split transform and move jar into connectors directory (apache#7218)

commit 862e205
Author: Jia Fan <[email protected]>
Date:   Sat Aug 10 15:21:36 2024 +0800

    [Improve][Document] Change deprecated connector name in setup.md (apache#7366)

commit 2769ed5
Author: hailin0 <[email protected]>
Date:   Sat Aug 10 09:58:03 2024 +0800

    [Hotfix][Connector] Fix jdbc compile error (apache#7359)

commit 3130ae0
Author: xxsc0529 <[email protected]>
Date:   Fri Aug 9 20:02:06 2024 +0800

    [Fix][Connector-V2][OceanBase] Remove OceanBase catalog's dependency on mysql driver (apache#7311)

commit 9216627
Author: chaos <[email protected]>
Date:   Fri Aug 9 13:53:28 2024 +0800

    Update ConsoleSinkFactory.java (apache#7350)

    Add existing rules to the optionRule

commit dc3c239
Author: dailai <[email protected]>
Date:   Fri Aug 9 13:11:30 2024 +0800

    [Improve][Jdbc] Skip all index when auto create table to improve performance of write (apache#7288)

commit 16950a6
Author: Jia Fan <[email protected]>
Date:   Thu Aug 8 17:27:47 2024 +0800

    [Improve][Connector-V2] Improve doris error msg (apache#7343)

commit a18fca8
Author: Alex Ting <[email protected]>
Date:   Thu Aug 8 11:22:41 2024 +0800

    [Fix][Connector-tdengine] Fix sql exception and concurrentmodifyexception when connect to taos and read data

commit 064fcad
Author: lizhenglei <[email protected]>
Date:   Wed Aug 7 21:57:57 2024 +0800

    [Improve][Transform] Improve DynamicCompile transform (apache#7319)

commit 16eeb1c
Author: He Wang <[email protected]>
Date:   Wed Aug 7 14:54:46 2024 +0800

    [Improve][Jdbc] Remove MysqlType references in JdbcDialect (apache#7333)

commit 191d9e1
Author: ChunFuWu <[email protected]>
Date:   Wed Aug 7 14:53:51 2024 +0800

    [FIX][SFTP] Fix username parameter error in sftp sink document (apache#7334)

commit 764d8b0
Author: Jia Fan <[email protected]>
Date:   Wed Aug 7 12:46:30 2024 +0800

    [Improve][API] Make sure the table name in TablePath not be null (apache#7252)

commit 8916523
Author: hailin0 <[email protected]>
Date:   Wed Aug 7 12:46:12 2024 +0800

    [Hotfix][Connector] Fix kafka consumer log next startup offset (apache#7312)

commit 819c685
Author: hailin0 <[email protected]>
Date:   Wed Aug 7 12:45:54 2024 +0800

    [Improve][Jdbc] Merge user config primary key when create table (apache#7313)

commit 7c3cd99
Author: hailin0 <[email protected]>
Date:   Wed Aug 7 12:44:41 2024 +0800

    [Improve] Remove unused code (apache#7324)

commit 460e73e
Author: hailin0 <[email protected]>
Date:   Wed Aug 7 12:44:24 2024 +0800

    [E2E] Enable fakesource e2e of spark/flink (apache#7325)

commit f6a1e51
Author: hailin0 <[email protected]>
Date:   Wed Aug 7 12:44:13 2024 +0800

    [E2E] Enable JdbcPostgresIdentifierIT (apache#7326)

commit a12786b
Author: Carl-Zhou-CN <[email protected]>
Date:   Wed Aug 7 12:27:50 2024 +0800

    [DOC][Oss] fix document configuration is rectified when the oss is selected as the checkpoint base (apache#7332)

commit a4db64d
Author: zhangdonghao <[email protected]>
Date:   Wed Aug 7 11:52:19 2024 +0800

    [Improve][E2E] Support windows for the e2e of paimon (apache#7329)

commit 855254e
Author: Jia Fan <[email protected]>
Date:   Wed Aug 7 11:03:06 2024 +0800

    [Feature][Transform] Add LLM transform (apache#7303)

    * [Feature][Transform] Add LLM transform

    * update

    * update

    * retrigger

commit f0cefbe
Author: Arin <[email protected]>
Date:   Tue Aug 6 19:23:38 2024 +0530

    [Feature] [Activemq] Added activemq sink  (apache#7251)

commit bb2c912
Author: Guangdong Liu <[email protected]>
Date:   Tue Aug 6 21:46:12 2024 +0800

    [Fix][Doris] Fix the abnormality of deleting data in CDC scenario. (apache#7315)

commit c94ea32
Author: Jia Fan <[email protected]>
Date:   Tue Aug 6 11:23:52 2024 +0800

    [Fix][Doc] Fix miss sink-options-placeholders.md in sidebars (apache#7310)

commit 73632ba
Author: hailin0 <[email protected]>
Date:   Tue Aug 6 10:08:45 2024 +0800

    [Hotfix][Zeta] Fix task cannot be stopped when system is busy (apache#7292)

commit 9d56cc3
Author: virvle <[email protected]>
Date:   Mon Aug 5 17:51:26 2024 +0800

    Update Oracle-CDC.md (apache#7285)

commit 4f120ff
Author: Jia Fan <[email protected]>
Date:   Mon Aug 5 14:34:01 2024 +0800

    [Improve] Update snapshot version to 2.3.7 (apache#7305)

commit 2fd4eec
Author: Jia Fan <[email protected]>
Date:   Mon Aug 5 11:35:02 2024 +0800

    [Fix][Doc] Fix hybrid cluster deployment document display error (apache#7306)

commit 82f5d8c
Author: hailin0 <[email protected]>
Date:   Sat Aug 3 12:02:19 2024 +0800

    [Improve][SQL-Transform] Remove escape identifier from output fields (apache#7297)

commit 34a6b8e
Author: Jast <[email protected]>
Date:   Wed Jul 31 14:43:11 2024 +0800

    [hotfix][connector-v2-hbase]fix and  optimize hbase source problem (apache#7148)

    * [hotfix][improve][doc]optimize connector hbase source

    * [doc]add dependent document

    * [doc]update dependent document

    * [improve]improve static use

    * [hotfix]add test case

    * [hotfix]add test case

    ---------

    Co-authored-by: Jia Fan <[email protected]>

commit 9df557c
Author: lizhenglei <[email protected]>
Date:   Tue Jul 30 21:37:04 2024 +0800

    [Improve][Transform] Improve DynamicCompile transform  (apache#7264)

commit b9acb57
Author: hailin0 <[email protected]>
Date:   Tue Jul 30 16:12:57 2024 +0800

    [Feature][Transforms] Support cast to bytes function of sql (apache#7284)

commit ec1c319
Author: Nothing. <[email protected]>
Date:   Mon Jul 29 21:53:01 2024 +0800

    [Docs]translate event-listener doc into chinese (apache#7274)

commit f012b2a
Author: dailai <[email protected]>
Date:   Mon Jul 29 21:51:59 2024 +0800

    [Improve][Connector-v2] Optimize the way of databases and tables are checked for existence (apache#7261)

commit 4897491
Author: 卢宗柱 <[email protected]>
Date:   Mon Jul 29 17:29:23 2024 +0800

    fix [Bug] Unable to create a source for identifier 'Iceberg'. apache#7182 (apache#7279)

    Co-authored-by: luzongzhu <[email protected]>

commit 21c4f52
Author: corgy-w <[email protected]>
Date:   Mon Jul 29 17:28:10 2024 +0800

    [Improve][Core] Improve base on plugin name of lookup strategy (apache#7278)

    * [bug][plugin-discovery] fix multi plugin discovery

    * [bug][plugin-discovery] optimize code

    ---------

    Co-authored-by: wangchao <[email protected]>

commit 3ccc6a8
Author: hailin0 <[email protected]>
Date:   Mon Jul 29 10:41:09 2024 +0800

    [Hotfix][Zeta] Fix task cannot be stopped when system is busy (apache#7280)

commit d886495
Author: Guangdong Liu <[email protected]>
Date:   Fri Jul 26 10:45:32 2024 +0800

    [Fix][Connector-V2] Fix doris TRANSFER_ENCODING header error (apache#7267)

commit 31e59cd
Author: Xuzz <[email protected]>
Date:   Fri Jul 26 10:44:50 2024 +0800

    [Feature][Jdbc] Support hive compatibleMode add inceptor dialect (apache#7262)

commit d003bd8
Author: zhangdonghao <[email protected]>
Date:   Fri Jul 26 10:44:09 2024 +0800

    [Feature][Zeta] Added the metrics information of table statistics in multi-table mode (apache#7212)

commit 876d2f0
Author: Jia Fan <[email protected]>
Date:   Thu Jul 25 10:24:55 2024 +0800

    [Fix][Dist] Remove aws jar dependency in release package (apache#7257)

    * [Fix][Dist] Remove aws jar dependency in release package

    * [Fix][Dist] Remove aws jar dependency in release package

commit 0d08b20
Author: dailai <[email protected]>
Date:   Thu Jul 25 10:00:13 2024 +0800

    [Improve][Connector-v2] Optimize the count table rows for jdbc-oracle and oracle-cdc (apache#7248)

commit cc59499
Author: Jia Fan <[email protected]>
Date:   Tue Jul 23 14:27:07 2024 +0800

    [Improve][Core] Move MultiTableSink to seatunnel-api module (apache#7243)

    * [Improve][Core] Move MultiTableSink to seatunnel-api module

    * [Improve][Core] Move MultiTableSink to seatunnel-api module

commit fe0c477
Author: Eric <[email protected]>
Date:   Tue Jul 23 08:36:58 2024 +0800

    [Hotfix][Zeta] Fix taskgroup failed log lost (apache#7241)

commit b0fe432
Author: hawk9821 <[email protected]>
Date:   Mon Jul 22 16:12:12 2024 +0800

    The isNullable attribute is true when the primary key field in the Paimon table converts the Column object. apache#7231 (apache#7242)

commit 4b3af9b
Author: Guangdong Liu <[email protected]>
Date:   Mon Jul 22 13:04:19 2024 +0800

    [Improve][Doris Connector] Unified serialization method,Use RowToJsonConverter and TextSerializationSchema (apache#7229)

    * 1

    * 1

    * 1

    * Update seatunnel-connectors-v2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/source/serialization/RowBatch.java

    Co-authored-by: Jia Fan <[email protected]>

    ---------

    Co-authored-by: gdliu3 <[email protected]>
    Co-authored-by: Jia Fan <[email protected]>

commit 4ec25f3
Author: Jia Fan <[email protected]>
Date:   Mon Jul 22 12:01:36 2024 +0800

    [Fix][Zeta] Fix release slot resource twice (apache#7236)

commit 7dc3fa8
Author: litiliu <[email protected]>
Date:   Mon Jul 22 11:41:33 2024 +0800

    [Fix][Connector kafka]Fix Kafka consumer stop fetching after TM node restarted (apache#7233)

commit 063f83c
Author: Guangdong Liu <[email protected]>
Date:   Mon Jul 22 10:40:58 2024 +0800

    [Improve][Restapi] Add ip and port to monitoring information (apache#7203)

commit c4ca741
Author: hailin0 <[email protected]>
Date:   Sat Jul 20 21:00:06 2024 +0800

    [Feature][Core] Support using upstream table placeholders in sink options and auto replacement (apache#7131)

    、

commit 821dfc8
Author: lizhenglei <[email protected]>
Date:   Sat Jul 20 20:25:50 2024 +0800

    [Feature][Transform] DynamicCompile add transform  (apache#7170)

commit 1e7c78d
Author: hilo <[email protected]>
Date:   Fri Jul 19 19:44:18 2024 +0800

    [Improve][Doc] Add IMap OSS config jars into document (apache#7029)

commit d39ad93
Author: Jia Fan <[email protected]>
Date:   Fri Jul 19 19:42:03 2024 +0800

    [Fix][Zeta] Fix hybrid deployment can not get worker when init (apache#7235)

commit 2058f5d
Author: ChunFuWu <[email protected]>
Date:   Fri Jul 19 14:21:23 2024 +0800

    [Chore][NOTICE] Correct year in NOTICE file (apache#7232)

commit 6e33a97
Author: Jia Fan <[email protected]>
Date:   Thu Jul 18 23:03:27 2024 +0800

    [Fix] Fix Hana type converter decimal scale is 0 convert to int error (apache#7167)

commit d1b4a7f
Author: Jia Fan <[email protected]>
Date:   Thu Jul 18 15:31:30 2024 +0800

    [Improve][Zeta] Add check for submit duplicate job id (apache#7021)

commit e44e8b9
Author: hailin0 <[email protected]>
Date:   Thu Jul 18 15:30:49 2024 +0800

    [Improve][Jdbc] Support write unicode text into sqlserver (apache#7159)

commit f40f11a
Author: tcodehuber <[email protected]>
Date:   Thu Jul 18 00:03:33 2024 +0800

    [Docs] Update engine related docs info (apache#7228)
chaorongzhi pushed a commit to chaorongzhi/seatunnel that referenced this pull request Aug 21, 2024
@liugddx @Hisoka-X
[Improve][Doris Connector] Unified serialization method,Use RowToJson…
a6c2494 
…Converter and TextSerializationSchema (apache#7229)

* 1

* 1

* 1

* Update seatunnel-connectors-v2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/source/serialization/RowBatch.java

Co-authored-by: Jia Fan <[email protected]>

---------

Co-authored-by: gdliu3 <[email protected]>
Co-authored-by: Jia Fan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Hisoka-X Hisoka-X Hisoka-X approved these changes

@wuchunfu wuchunfu wuchunfu approved these changes

Assignees
No one assigned
Labels
approved connectors-v2 doris format reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@liugddx @Hisoka-X @wuchunfu