Add file metadata columns support for spark parquet

[gaoyangxiaozhu]

Spark support people query file metdata as file_size, file_name, file_path, file_modified_time, file_block_start etc for Hive tables as seperated file metadata column. Checking this for all file metadata supported query by spark https://github.com/apache/spark/blob/081c7a7947a47bf0b2bfd478abdd4b78a1db3ddb/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala#L183C2-L193C56.

The PR extends the HiveSplit with a new parameter metadaColumns to let upsteram computing engine as spark to pass the initialized const file metadata columns (if have) to velox connector split when constructed, it use to fix file metadata columns null issue when intergrating with Velox using Spark.

1. It let HiveSplit can get file metadata info from spark
2. The values is populated by by SplitReader into TableScanOperator output.

Checking this issue #8173 for details context.

It is also a dependency of Gluten repository PR apache/gluten#3870

[netlify]

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	6d8b7d8
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/65be35fb7ccc460008a45ce9

[gaoyangxiaozhu]

@majetideepak also help for review

and I saw looks @aditi-pandit also submit a similar PR #8800 for addressing presdo engine $file_path and $file_modify_time query issue. We can see if can make the PR cover both.

[aditi-pandit]

@gaoyangxiaozhu : Have couple of high-level comments. There are several missing pieces :

i) A TableScan output operator test like https://github.com/facebookincubator/velox/pull/8800/files#r1498654870. This would introduce the wiring for metadata columns in HiveConnector TestBase classes.
ii) The metadata columns can be used as filters in Presto queries as well. I would imagine Spark would allow this as well. The code should be enhanced for this.
iii) What is the difference between synthesized and metadata columns ? Its not clear to me.

It might be simpler to change #8800 to use metadata_columns parameters for HiveSplit instead. wdyt ?

[gaoyangxiaozhu]

@gaoyangxiaozhu : Have couple of high-level comments. There are several missing pieces :

i) A TableScan output operator test like https://github.com/facebookincubator/velox/pull/8800/files#r1498654870. This would introduce the wiring for metadata columns in HiveConnector TestBase classes. ii) The metadata columns can be used as filters in Presto queries as well. I would imagine Spark would allow this as well. The code should be enhanced for this. iii) What is the difference between synthesized and metadata columns ? Its not clear to me.

It might be simpler to change #8800 to use metadata_columns parameters for HiveSplit instead. wdyt ?

thanks @aditi-pandit

for i) Yes, a test is needed and I saw your PR already have it.
for ii) Actually I add filter part support and just remove it temporily with this commit df5e279, but we can added it back if it still need for making both presto and spark work.

for iii) I actually before don't know what's the synthesized means and also i don't found anywhere code path use it, so i just added a KMetadata to easily mark it is metadata column, but we can still use synthesized it should be ok, I'd like metadata naming due to I don't understand what synthesized means.

Sure @aditi-pandit , it's OK if you can update your PR to move using metadata_columns parameters and remove any hard code checking for know it is metadata or not. For filter part just reference change

if (auto name = subfield.toString();
        metadataColumns.count(name) == 0 && partitionKeys.count(name) == 0) {
      filterSubfields[getColumnName(subfield)].push_back(&subfield);
    }

[gaoyangxiaozhu]

closed with @aditi-pandit 's PR is merged do the same thing.