[Improve] improve avro format convert by liunaijie · Pull Request #6082 · apache/seatunnel

liunaijie · 2023-12-26T05:27:38Z

Purpose of this pull request

this pr #5084 bring in avro format to kafka.
But there has some issue when type convert, and the previous CI has't verify the convert issue.
So this pr has some improvemant about the avro type convert. and update the CI to check converted data value and type consistent.

Does this PR introduce any user-facing change?

How was this patch tested?

use the exist e2e

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

zhilinli123 · 2023-12-26T05:46:37Z

Add end to end data End to end test , refer to checkDebeziumFormat()
https://github.com/liunaijie/seatunnel/blob/avro-array/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-kafka-e2e/src/test/java/org/apache/seatunnel/e2e/connector/kafka/KafkaFormatIT.java

zhilinli123 · 2023-12-28T08:34:53Z

+          },
+          {
+           field_name = c_string
+           field_type = string
+           field_value = [
+            {
+              rule_type = MIN_LENGTH
+              rule_value = 6
+            },
+            {
+              rule_type = MAX_LENGTH
+              rule_value = 6
+            }
+          ]
+         }


I still hope to do an end-to-end test on all fields to ensure that the data is not only consistent in the amount of data, but also to ensure that each data and field are consistent, and it is best to write the data to the database for a Check
cc @hailin0 @Hisoka-X

assert sink can't do some equal check, so i add the data check in code

assert sink can't do some equal check, so i add the data check in code

But you can write it to a database such as pg and then read it to make sure the input and output are consistent,The type is only one aspect of the test and we must make sure that the data values are exactly the same

yeah, so i add the data check after job finish, read data from kafka topic and verify the data consistent

yeah, so i add the data check after job finish, read data from kafka topic and verify the data consistent

ok for me.

Hi @zhilinli123 I update the code.
In this update, it can verify all field value and type. also can test the serialize (ST row -> Avro Generic Record -> bytes ) and deserialize (bytes -> Avro Generic Record -> ST Row ). I think this test can cover this feature.

zhilinli123 · 2023-12-29T03:39:59Z

Add a reference to the description of the document information:
https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/formats/ogg-json.md

zhilinli123 · 2024-01-02T08:53:11Z

Wait CI, looks good, can you add documentation about avro for this
@liunaijie

zhilinli123

+1

zhilinli123 · 2024-01-04T15:25:05Z

PTAL: @hailin0 @Hisoka-X

Hisoka-X · 2024-01-05T07:18:45Z

Could you give us more detail information?

liunaijie · 2024-01-05T07:59:26Z

@Hisoka-X @zhilinli123 added another commit to fix bytes type convert. and add type check in fake source result 7235aed

hailin0 · 2024-01-05T08:00:48Z

waiting for ci ok

liunaijie marked this pull request as ready for review December 26, 2023 05:27

zhilinli123 mentioned this pull request Dec 26, 2023

[Feature][Kafka] Support multi-table source read #5992

Merged

4 tasks

liunaijie force-pushed the avro-array branch 4 times, most recently from 04ed0db to 4474b70 Compare December 27, 2023 13:44

[Improve] improve avro format convert

2b93dfd

liunaijie force-pushed the avro-array branch 2 times, most recently from 886a416 to f158cf1 Compare December 28, 2023 06:54

zhilinli123 reviewed Dec 28, 2023

View reviewed changes

Comment thread ...-e2e/connector-kafka-e2e/src/test/java/org/apache/seatunnel/e2e/connector/kafka/KafkaIT.java Outdated

liunaijie force-pushed the avro-array branch 2 times, most recently from c4389c8 to 45f1d74 Compare December 28, 2023 13:04

[Improve] improve kafka avro e2e

ac6e2d1

liunaijie force-pushed the avro-array branch from 45f1d74 to ac6e2d1 Compare January 2, 2024 07:56

zhilinli123 approved these changes Jan 4, 2024

View reviewed changes

github-actions Bot added the reviewed label Jan 4, 2024

Hisoka-X reviewed Jan 5, 2024

View reviewed changes

Comment thread docs/en/connector-v2/formats/avro.md Outdated

Comment thread docs/en/connector-v2/formats/avro.md Outdated

[Feature] add avro format document

b891202

liunaijie force-pushed the avro-array branch from f325780 to b891202 Compare January 5, 2024 07:16

Hisoka-X reviewed Jan 5, 2024

View reviewed changes

Comment thread ...-e2e/connector-kafka-e2e/src/test/java/org/apache/seatunnel/e2e/connector/kafka/KafkaIT.java Outdated

Hisoka-X previously approved these changes Jan 5, 2024

View reviewed changes

github-actions Bot added the approved label Jan 5, 2024

[Improve] update avro bytes type convert

7235aed

liunaijie dismissed Hisoka-X’s stale review via 7235aed January 5, 2024 07:57

github-actions Bot removed the approved label Jan 5, 2024

hailin0 approved these changes Jan 5, 2024

View reviewed changes

github-actions Bot added the approved label Jan 5, 2024

Hisoka-X approved these changes Jan 5, 2024

View reviewed changes

Hisoka-X merged commit fd971b9 into apache:dev Jan 5, 2024

liunaijie deleted the avro-array branch March 4, 2024 05:54

chaorongzhi pushed a commit to chaorongzhi/seatunnel that referenced this pull request Aug 21, 2024

[Improve] improve avro format convert (apache#6082)

a72890c

Conversation

liunaijie commented Dec 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Uh oh!

zhilinli123 commented Dec 26, 2023

Uh oh!

zhilinli123 Dec 28, 2023

Choose a reason for hiding this comment

Uh oh!

liunaijie Dec 28, 2023

Choose a reason for hiding this comment

Uh oh!

zhilinli123 Dec 29, 2023

Choose a reason for hiding this comment

Uh oh!

liunaijie Dec 29, 2023

Choose a reason for hiding this comment

Uh oh!

Hisoka-X Jan 2, 2024

Choose a reason for hiding this comment

Uh oh!

liunaijie Jan 2, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zhilinli123 commented Dec 29, 2023

Uh oh!

zhilinli123 commented Jan 2, 2024

Uh oh!

zhilinli123 left a comment

Choose a reason for hiding this comment

Uh oh!

zhilinli123 commented Jan 4, 2024

Uh oh!

Uh oh!

Uh oh!

Hisoka-X commented Jan 5, 2024

Uh oh!

Uh oh!

liunaijie commented Jan 5, 2024

Uh oh!

hailin0 commented Jan 5, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

liunaijie commented Dec 26, 2023 •

edited

Loading