[HUDI-3902] Fallback to HadoopFsRelation in cases non-involving Schema Evolution

[alexeykudinkin]

Tips

• Thank you very much for contributing to Apache Hudi.
• Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.

What is the purpose of the pull request

Spark unfortunately predicates some of its optimization Rules (and some other handling) on usage of HadoopFsRelation which leads to those optimizations not being applied when we rely on our custom Relation impls.

To work this around for the 0.11, we fallback to HadoopFsRelation in cases when it's feasible to do so.

NOTE: We're partially restoring logic present in 0.10.1

Brief change log

• Adding toHadoopFsRelation method for BaseFileOnlyRelation
• Fallback to HadoopFsRelation for non-Schema Evolution use-cases

Verify this pull request

This pull request is already covered by existing tests, such as (please describe tests).

Committer checklist

• Has a corresponding JIRA in PR title & commit

• Commit message is descriptive of the change

• CI is green

• Necessary doc changes done or have another open PR

• For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

[nsivabalan]

one clarification. looks good to me.

[vinothchandar]

@xiarixiaoyao Can you please skim this PR as well

[xiarixiaoyao]

@alexeykudinkin
if we support prune nest schema, can we avoid this performance problem
now schema evolution support prune nest schema， maybe we can use that prune function.
LGTM

[alexeykudinkin]

@xiarixiaoyao it will most likely solve this particular issue, but we'll continue to be exposed to this becoming a problem again until we upstream the real fix and make HadoopFsRelation extensible in Spark

[hudi-bot]

CI report:

• 57f622f UNKNOWN
• cabf2cf Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build