[HUDI-5533] Support spark nested columns comments

[parisni]

feat: support nested doc datasource. improvement of #8683

Goal:

Full comments support for hudi, from source to target

Context:

as of 1.x and previous, the field comment support is partial:

• hive-sync and co, allow to sync the comments only on the first field level
• the comments are retrieved from the schema resolver. while they are stored in the hudi avro schema, they can't yet be extracted, since it ignore the avro docs.
• avro support nested comments while hive metastore and co only support first layer of comment. However spark can store the schema structure in a property, and if exists, use it in priority.
• Hudi currently optionally supports writing the spark structure, but it does not store comments at all, and rely on parquet schema which don't capture the comments anyway

What this PR covers:

1. enable extracting the nested comments out of avro schema
2. replaces the generation of the spark structure with a new util, directly leveraging avro and not parquet schema anymore
3. provides full support for spark schema comments storage
4. enable metastore first level comment (they were partially working because of the lack of support in 1.

Reminder:

• to activate the metastore comment, set: hoodie.datasource.hive_sync.sync_comment
• to activate the spark schema structure comments, set: [hoodie.datasource.hive_sync.sync_as_datasource](https://hudi.apache.org/docs/0.15.0/configurations#hoodiedatasourcehive_syncsync_as_datasource)

Describe the issue this Pull Request addresses

Summary and Changelog

Impact

Risk Level

Documentation Update

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[parisni]

@yihua what about also landing this into 1.1 ?

[parisni]

also maybe @danny0405 as you did follow this topic previously and this touch a bit to flink too

[danny0405]

@parisni The 1.1 RC2 is out, I don't think this is a blocker for it, I can help for the review though, can you check the test failures?

[parisni]

@danny0405 i did fix the tests. still some are failing but sounds like flaky tests ?
confirmed flaky for:

• TestJavaHoodieBackedMetadata.testReattemptOfFailedClusteringCommit
• TestNewHoodieParquetFileFormat.testNewParquetFileFormat

[yihua]

@parisni Could you rebase your branch on top of the latest master? A few fixes on CI flakiness have been merged.

[parisni]

Hi @yihua @danny0405 i did rebase on master again

[hudi-bot]

CI report:

• 35a1bca UNKNOWN
• 552e8fe UNKNOWN
• 61718fd UNKNOWN
• de214ce UNKNOWN
• f9f84ee UNKNOWN
• 493439b UNKNOWN
• a9c29ea UNKNOWN
• 07d908d UNKNOWN
• 28e3135 UNKNOWN
• 8f16b0b UNKNOWN
• 8e62924 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build