[HUDI-3595] Fixing NULL schema provider for empty batch

[nsivabalan]

What is the purpose of the pull request

when Deltastreamer consumes an empty batch, but checkpoint moved ahead, hudi injects NullSchemaProvider as part of consumed InputBatch. Which returns null schema (Schema.create(Schema.Type.NULL)) schema when asked for source or target schema. So, this gets serialized into commit metadata and later runs into issues when table schema is fetched.

Fix: when serializing the schema to commit metadata, we avoid adding the entry if its null schema (Schema.create(Schema.Type.NULL)).

Ideally, we want to fix the schema provider returned as part of empty batch itself. But don't want to fix it right away as it could blow up somewhere else. So, created a ticket for later.

TableSchemaProvider is capable of walking back to previous commits to find the right schema and hence skipping to add schema should be fine.

Brief change log

Fixed serializing the avro schema to commit metadata to avoid adding the entry if its null schema (Schema.create(Schema.Type.NULL))

Verify this pull request

This change added tests and can be verified as follows:

• Added a test to TestHoodieDeltastreamer to test the fix.

Committer checklist

• Has a corresponding JIRA in PR title & commit

• Commit message is descriptive of the change

• CI is green

• Necessary doc changes done or have another open PR

• For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

[XuQianJin-Stars]

hi @nsivabalan rebase this pr, I have added the time out option in the master branch to solve CI the canceled of IT modules.

[hudi-bot]

CI report:

• 96b8587 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build