[HUDI-1129]: AvroConversionUtils unable to handle avro to row transformation when passing evolved schema

[sbernauer]

What is the purpose of the pull request

Specific fix this error: https://issues.apache.org/jira/browse/HUDI-1129
This is needed to partially fix #1845

Brief change log

• Use avro field names and not indices to convert from avro to GenericRow of catalyst at AvroConversionHelper

Verify this pull request

Run maven test suite

Committer checklist

• Has a corresponding JIRA in PR title & commit

• Commit message is descriptive of the change

• CI is green

• Necessary doc changes done or have another open PR

• For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

[bvaradar]

Thanks @sbernauer for raising the diff.

Can you add the schema evolution test from your other diff here so that we can ensure the test passes.

[bvaradar]

Question : Does this work for nested schemas where same name is used in different hierarchy ? For example : "order.rec.rec" (I am just making this up) but wanted to make sure if there are any chances of ambiguity in field resolution that can arise ? Can you add some test-cases to verify this would work fine.

[sbernauer]

Good point, I will try to write some tests

[sbernauer]

Hi @bvaradar the test currently fails because of the EOFException from https://issues.apache.org/jira/browse/HUDI-1128

[nsivabalan]

I found the issue after some debugging. But need your thoughts on whether it is a bug or how to go about fixing it.
@bvaradar @n3nash @vinothchandar

As per the test case linked to reproduce, here is what we are doing.
Generate records with SCHEMA_1 and ingest to Hudi with SCHEMA_1
Generate records with SCHEMA_2 and ingest to Hudi with SCHEMA_2
Generate records with SCHEMA_1 and ingest to Hudi with SCHEMA_2(both source and target schema)// this is where the exception is thrown.

Here is the gist of the issue.
Lets say we have an avro record with SCHEMA_1
byte[] recordBytes = HoodieAvroUtils.avroToBytes(genericRecord);

Converting this back to GenRec with SCHEMA_1 succeeds. HoodieAvroUtils.bytesToAvro(recordBytes, SCHEMA_1)
But converting this back to GenRec with SCHEMA_2 (which has one additional field compared to SCHEMA_1) fails.

[nsivabalan]

This is not required anymore. #2927 handles the schema evol.