[HUDI-3569] Introduce ChainedJsonKafkaSourePostProcessor to support setting multi processors at one time

[wangxianghu]

…etting multi processors at once

Tips

• Thank you very much for contributing to Apache Hudi.
• Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.

What is the purpose of the pull request

Introduce ChainedJsonKafkaSourePostProcessor to support setting multi processors at once

Verify this pull request

This pull request is already covered by existing tests
org.apache.hudi.utilities.sources.TestJsonKafkaSourcePostProcessor#testChainedJsonKafkaSourcePostProcessor

Committer checklist

• Has a corresponding JIRA in PR title & commit

• Commit message is descriptive of the change

• CI is green

• Necessary doc changes done or have another open PR

• For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

[nsivabalan]

Do we really see a need for chained post processor just for jsonKafkaSource. Unless or otherwise required, don't want to add them. if you know of valid use-case, let me know.

[wangxianghu]

Do we really see a need for chained post processor just for jsonKafkaSource. Unless or otherwise required, don't want to add them. if you know of valid use-case, let me know.

@nsivabalan thanks for your review
There might be fewer usage scenarios compare to Transformer

But, in our companay, we use maxwell to subscribe binlog，so I add a MaxwellJsonKafkaPostprocessor to extract the data we need.

sometimes, there might be some data quality problems, for example, some not nullable fileds have null values, we have to fix it through processor,
sometimes, we also need to encode some fields for safety purpose, MaxwellJsonKafkaPostprocessor + user-defined encode processor could be a choice.

So I think somebody else who use maxwell may need this feature too

[wangxianghu]

@hudi-bot run azure

[wangxianghu]

@hudi-bot run azure

[hudi-bot]

CI report:

• 2b7449c Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[wangxianghu]

hi @nsivabalan can we add this processor ? it is very useful in scenarios with diversified data requirements.
In our comany, we have use this feature to add multiple processors to our pipeline:

1. maxwell post processor : extract data from maxwell json string. this is a standard processor
2. Encrypt post processor ： Encrypt some fields for safety purpose
3. flag post processor : this is quite a business related processor.

with ChainedJsonKafkaSourcePostProcessor we can make data processing more flexible, it makes up for the lack of expression ability of Transformer

[nsivabalan]

sounds good 👍