Skip to content

[cdc] Refactor mysql debezium json event parser #2053

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Sep 26, 2023
Merged

[cdc] Refactor mysql debezium json event parser #2053

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
edaeb31
Refactor mysql Debezium json event parser.
zhuangchong Sep 20, 2023
14efabb
fix code style
zhuangchong Sep 20, 2023
44c0062
fix code style
zhuangchong Sep 20, 2023
01c6c52
fix debezium event json deserialize.
zhuangchong Sep 20, 2023
b18a818
fix debezium event json deserialize.
zhuangchong Sep 20, 2023
3c0b6bf
Fix decimal precision
zhuangchong Sep 21, 2023
a9409c1
fix json parser
zhuangchong Sep 21, 2023
656ff67
merge master
zhuangchong Sep 21, 2023
1675dd5
Restore comments
zhuangchong Sep 21, 2023
7919a44
When database DDL operation, the current table is null.
zhuangchong Sep 22, 2023
f9306d2
merge master
zhuangchong Sep 22, 2023
76a08a7
Merge branch 'master' into refactor-mysql-parser
zhuangchong Sep 25, 2023
7b67e55
Merge branch 'master' into refactor-mysql-parser
zhuangchong Sep 25, 2023
055b12f
Test class modification read file.
zhuangchong Sep 26, 2023
27824ee
merge master
zhuangchong Sep 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .../paimon-flink-cdc/src/main/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSchema.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@

import org.apache.paimon.flink.action.cdc.TableNameConverter;
import org.apache.paimon.flink.action.cdc.TypeMapping;
import org.apache.paimon.flink.action.cdc.kafka.formats.DataFormat;
import org.apache.paimon.flink.action.cdc.kafka.formats.RecordParser;
import org.apache.paimon.flink.action.cdc.kafka.format.DataFormat;
import org.apache.paimon.flink.action.cdc.kafka.format.RecordParser;
import org.apache.paimon.types.DataType;

import org.apache.flink.configuration.Configuration;
Expand All @@ -46,7 +46,7 @@
import java.util.stream.StreamSupport;

import static org.apache.paimon.flink.action.cdc.kafka.KafkaActionUtils.kafkaPropertiesGroupId;
import static org.apache.paimon.flink.action.cdc.kafka.formats.DataFormat.getDataFormat;
import static org.apache.paimon.flink.action.cdc.kafka.format.DataFormat.getDataFormat;

/** Utility class to load canal kafka schema. */
public class KafkaSchema {
Expand Down
4 changes: 2 additions & 2 deletions ...k-cdc/src/main/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSyncDatabaseAction.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@
import org.apache.paimon.flink.action.MultiTablesSinkMode;
import org.apache.paimon.flink.action.cdc.TableNameConverter;
import org.apache.paimon.flink.action.cdc.TypeMapping;
import org.apache.paimon.flink.action.cdc.kafka.formats.DataFormat;
import org.apache.paimon.flink.action.cdc.kafka.formats.RecordParser;
import org.apache.paimon.flink.action.cdc.kafka.format.DataFormat;
import org.apache.paimon.flink.action.cdc.kafka.format.RecordParser;
import org.apache.paimon.flink.sink.cdc.EventParser;
import org.apache.paimon.flink.sink.cdc.FlinkCdcSyncDatabaseSinkBuilder;
import org.apache.paimon.flink.sink.cdc.RichCdcMultiplexRecord;
Expand Down
4 changes: 2 additions & 2 deletions ...link-cdc/src/main/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSyncTableAction.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@
import org.apache.paimon.flink.action.cdc.ComputedColumn;
import org.apache.paimon.flink.action.cdc.TableNameConverter;
import org.apache.paimon.flink.action.cdc.TypeMapping;
import org.apache.paimon.flink.action.cdc.kafka.formats.DataFormat;
import org.apache.paimon.flink.action.cdc.kafka.formats.RecordParser;
import org.apache.paimon.flink.action.cdc.kafka.format.DataFormat;
import org.apache.paimon.flink.action.cdc.kafka.format.RecordParser;
import org.apache.paimon.flink.sink.cdc.CdcSinkBuilder;
import org.apache.paimon.flink.sink.cdc.EventParser;
import org.apache.paimon.flink.sink.cdc.RichCdcMultiplexRecord;
Expand Down
8 changes: 4 additions & 4 deletions .../action/cdc/kafka/formats/DataFormat.java → ...k/action/cdc/kafka/format/DataFormat.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,14 @@
* limitations under the License.
*/

package org.apache.paimon.flink.action.cdc.kafka.formats;
package org.apache.paimon.flink.action.cdc.kafka.format;

import org.apache.paimon.flink.action.cdc.ComputedColumn;
import org.apache.paimon.flink.action.cdc.TableNameConverter;
import org.apache.paimon.flink.action.cdc.TypeMapping;
import org.apache.paimon.flink.action.cdc.kafka.formats.canal.CanalRecordParser;
import org.apache.paimon.flink.action.cdc.kafka.formats.maxwell.MaxwellRecordParser;
import org.apache.paimon.flink.action.cdc.kafka.formats.ogg.OggRecordParser;
import org.apache.paimon.flink.action.cdc.kafka.format.canal.CanalRecordParser;
import org.apache.paimon.flink.action.cdc.kafka.format.maxwell.MaxwellRecordParser;
import org.apache.paimon.flink.action.cdc.kafka.format.ogg.OggRecordParser;

import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.connectors.kafka.table.KafkaConnectorOptions;
Expand Down
2 changes: 1 addition & 1 deletion ...ction/cdc/kafka/formats/RecordParser.java → ...action/cdc/kafka/format/RecordParser.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
* limitations under the License.
*/

package org.apache.paimon.flink.action.cdc.kafka.formats;
package org.apache.paimon.flink.action.cdc.kafka.format;

import org.apache.paimon.flink.action.cdc.ComputedColumn;
import org.apache.paimon.flink.action.cdc.TableNameConverter;
Expand Down
2 changes: 1 addition & 1 deletion ...dc/kafka/formats/RecordParserFactory.java → ...cdc/kafka/format/RecordParserFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
* limitations under the License.
*/

package org.apache.paimon.flink.action.cdc.kafka.formats;
package org.apache.paimon.flink.action.cdc.kafka.format;

import org.apache.paimon.flink.action.cdc.ComputedColumn;
import org.apache.paimon.flink.action.cdc.TableNameConverter;
Expand Down
2 changes: 1 addition & 1 deletion ...kafka/formats/canal/CanalFieldParser.java → .../kafka/format/canal/CanalFieldParser.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
* limitations under the License.
*/

package org.apache.paimon.flink.action.cdc.kafka.formats.canal;
package org.apache.paimon.flink.action.cdc.kafka.format.canal;

/** Converts some special types such as enum、set、geometry. */
public class CanalFieldParser {
Expand Down
4 changes: 2 additions & 2 deletions ...afka/formats/canal/CanalRecordParser.java → ...kafka/format/canal/CanalRecordParser.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,12 @@
* limitations under the License.
*/

package org.apache.paimon.flink.action.cdc.kafka.formats.canal;
package org.apache.paimon.flink.action.cdc.kafka.format.canal;

import org.apache.paimon.flink.action.cdc.ComputedColumn;
import org.apache.paimon.flink.action.cdc.TableNameConverter;
import org.apache.paimon.flink.action.cdc.TypeMapping;
import org.apache.paimon.flink.action.cdc.kafka.formats.RecordParser;
import org.apache.paimon.flink.action.cdc.kafka.format.RecordParser;
import org.apache.paimon.flink.action.cdc.mysql.MySqlTypeUtils;
import org.apache.paimon.flink.sink.cdc.RichCdcMultiplexRecord;
import org.apache.paimon.types.DataType;
Expand Down
4 changes: 2 additions & 2 deletions .../formats/maxwell/MaxwellRecordParser.java → ...a/format/maxwell/MaxwellRecordParser.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,12 @@
* limitations under the License.
*/

package org.apache.paimon.flink.action.cdc.kafka.formats.maxwell;
package org.apache.paimon.flink.action.cdc.kafka.format.maxwell;

import org.apache.paimon.flink.action.cdc.ComputedColumn;
import org.apache.paimon.flink.action.cdc.TableNameConverter;
import org.apache.paimon.flink.action.cdc.TypeMapping;
import org.apache.paimon.flink.action.cdc.kafka.formats.RecordParser;
import org.apache.paimon.flink.action.cdc.kafka.format.RecordParser;
import org.apache.paimon.flink.sink.cdc.RichCdcMultiplexRecord;
import org.apache.paimon.types.RowKind;

Expand Down
4 changes: 2 additions & 2 deletions ...dc/kafka/formats/ogg/OggRecordParser.java → ...cdc/kafka/format/ogg/OggRecordParser.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,12 @@
* limitations under the License.
*/

package org.apache.paimon.flink.action.cdc.kafka.formats.ogg;
package org.apache.paimon.flink.action.cdc.kafka.format.ogg;

import org.apache.paimon.flink.action.cdc.ComputedColumn;
import org.apache.paimon.flink.action.cdc.TableNameConverter;
import org.apache.paimon.flink.action.cdc.TypeMapping;
import org.apache.paimon.flink.action.cdc.kafka.formats.RecordParser;
import org.apache.paimon.flink.action.cdc.kafka.format.RecordParser;
import org.apache.paimon.flink.sink.cdc.RichCdcMultiplexRecord;
import org.apache.paimon.types.RowKind;

Expand Down
Loading