[SPARK-40600] Support recursiveFileLookup for partitioned datasource

[wForget]

What changes were proposed in this pull request?

Support recursiveFileLookup for partitioned datasource to query partitioned datasource containing subdirectories, such as the HIVE_UNION_SUBDIR directory generated when hive tez executes the union statement.

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

1. Unit test added to FileIndexSuite.
2. After adding the following properties, the partition table with subdirectories can be queried correctly.

ALTER TABLE ${partitioned_table_with_subdir} SET SERDEPROPERTIES ('recursiveFileLookup' = 'true');
ALTER TABLE ${partitioned_table_with_subdir} SET SERDEPROPERTIES ('inferRecursivePartition' = 'true');

[AmplabJenkins]

Can one of the admins verify this patch?

[holdenk]

cc @HyukjinKwon

[HyukjinKwon]

This was added in #24830. cc @WeichenXu123 @cloud-fan @gengliangwang FYI

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

[tanvn]

@HyukjinKwon @wForget
Hi, may I know the status of this PR?
Would like to take part in this issue as we are facing this while reading data from an orc partitioned table and do not like to set spark.sql.hive.convertMetastoreOrc back to false.