[Feature] support avro format

[liunaijie]

• added avro format and UT
• kafka support avro format and e2e test

Purpose of this pull request

close #4821
same pr with #5064

Check list

• Code changed are covered with tests, or it does not need tests for reason:
• If any new Jar binary package adding in your PR, please add License Notice according
  New License Guide
• If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
• If you are contributing the connector code, please check that the following files are updated:
  1. Update change log that in connector document. For more details you can refer to connector-v2
  2. Update plugin-mapping.properties and add new connector information in it
  3. Update the pom file of seatunnel-dist
• Update the release-note.

[liunaijie]

pls re-trigger the CI, no failed job, only one cancelled job

[liugddx]

Run mvn spotless:apply and fix code style error.

[liunaijie]

can we retrigger the CI as we has fix the code style error

[liunaijie]

run successful when disable on flink 1.15 https://github.com/liunaijie/seatunnel/actions/runs/5617980130

[liunaijie]

merge the conflict, pls re-trigger the CI

[liunaijie]

https://github.com/liunaijie/seatunnel/actions/runs/5676004114/job/15382134463
this pr run successful in my repo's CI. But get NPE in this CI.
And also can't repet the error in local.

[ic4y]

@liunaijie Great commit! Now we just need to resolve the conflicts.

[ic4y]

Once PRs #5303 and #5222 are merged, they will resolve the e2e testing issue.

[liunaijie]

this pr always failed with PaimonIT, and this pr #5419 has fixed this issue, once this issue merged, pls re-trigger the CI, thanks.

[hailin0]

@liunaijie
good pr, please fix the conflict

[liunaijie]

@liunaijie good pr, please fix the conflict

@hailin0 Done. PTAL, thanks.

[FuYouJ]

Good pr, I learned something new from it

[ruanwenjun]

We also need to add NOTICE file https://github.com/apache/avro/blob/main/NOTICE.txt
at https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/release-docs/NOTICE

[zhilinli123]

You can do this by adding e2e uniformly to All formats are integrated in this
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-kafka-e2e/src/test/java/org/apache/seatunnel/e2e/connector/kafka/KafkaFormatIT.java

[liunaijie]

hi @zhilinli123, i use a method in KafkaIT and also KafKaIT has other format e2e test like json, text. so i don't want move to KafkaFormatIT.

[ruanwenjun]

Need update notice file
https://github.com/apache/avro/blob/release-1.11.1/NOTICE.txt

[ruanwenjun]

Basically LGTM, wait ci pass

[liunaijie]

Basically LGTM, wait ci pass

@ruanwenjun PTAL

[ruanwenjun]

LGTM

[zhilinli123]

hi I found some issues in kafka avro e2e testing. My configuration is as follows

source {
  Kafka {
    bootstrap.servers = "kafkaCluster:9092"
    topic = "test_avro_topic"
    result_table_name = "kafka_table"
 // When configured this will pull the consumption from kafka's starting offset kafka.auto.offset.reset = "earliest" does not take effect
    **start_mode = "earliest"** 
    format = avro
    format_error_handle_way = skip
    schema = {
      fields {
        id = bigint
        c_map = "map<string, smallint>"
        c_array = "array<tinyint>"
        c_string = string
        c_boolean = boolean
        c_tinyint = tinyint
        c_smallint = smallint
        c_int = int
        c_bigint = bigint
        c_float = float
        c_double = double
        c_decimal = "decimal(2, 1)"
        c_bytes = bytes
        c_date = date
        c_timestamp = timestamp
      }
    }
  }
}

sink {
  Console {
    source_table_name = "kafka_table"
  }
  Assert {
    source_table_name = "kafka_table"
    rules =
      {
        field_rules = [
          {
            field_name = id
            field_type = long
            field_value = [
              {
                rule_type = NOT_NULL
              },
              {
                rule_type = MIN
                rule_value = 0
              },
              {
                rule_type = MAX
                rule_value = 99
              }
            ]
          }
        ]
      }
  }
}

@liunaijie