[SPARK-27356][SQL] File source V2: Fix the case that data columns overlap with partition schema

[gengliangwang]

What changes were proposed in this pull request?

In the current file source V2 framework, the schema of FileScan is not returned correctly if there are overlap columns between dataSchema and partitionSchema. The actual schema should be
dataSchema - overlapSchema + partitionSchema, which might have different column order from the pushed down requiredSchema in SupportsPushDownRequiredColumns.pruneColumns.

For example, if the data schema is [a: String, b: String, c: String] and the partition schema is [b: Int, d: Int], the result schema is [a: String, b: Int, c: String, d: Int] in current FileTable and HadoopFsRelation. while the actual scan schema is [a: String, c: String, b: Int, d: Int] in FileScan.

To fix the corner case, this PR proposes that the output schema of FileTable should be dataSchema - overlapSchema + partitionSchema, so that the column order is consistent with FileScan.
Putting all the partition columns to the end of table schema is more reasonable.

How was this patch tested?

Unit test.

[gengliangwang]

@cloud-fan

[gengliangwang]

If this PR contains too many changes, I am OK to create a separate PR for the partition value pruning.

[SparkQA]

Test build #104251 has finished for PR 24284 at commit 313eda8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• abstract class FileScanBuilder(

[SparkQA]

Test build #104274 has finished for PR 24284 at commit 518b628.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #104269 has finished for PR 24284 at commit cd236a7.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[gengliangwang]

retest this please.

[SparkQA]

Test build #104280 has finished for PR 24284 at commit 518b628.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

do we need migration guide? it's a behavior change for file source v2, which is new in Spark 3.0.

[gengliangwang]

I am OK with either way. Let me remove this.

[cloud-fan]

are we testing v1 or v2 here?

[gengliangwang]

V2.
For V1 we use OrcV1PartitionDiscoverySuite.

[cloud-fan]

maybe we should put V2 in the test suite name as well.

[gengliangwang]

This is not quite related to this PR. If we are going to use V2 by default, I think the current test suite name is OK.

[cloud-fan]

isn't V1 by default now?

[gengliangwang]

For read path, it is V2 by default now.

[cloud-fan]

let's make a followup PR to put V2 in the test suite name and do not rely on the default config values.

[SparkQA]

Test build #104291 has finished for PR 24284 at commit 8894d93.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

unnecessary change

[SparkQA]

Test build #104299 has finished for PR 24284 at commit a64107d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

thanks, merging to master!