Experimental native scan test failures

[mbutrovich]

We have two experimental native scans based on DataFusion's ParquetExec operator. This issue will track the remaining test failures and any related notes as we bring the test failures down to zero.

ParquetReadV1Suite:

• test multiple pages with different sizes and nulls (prefetch enabled)
  • Unsigned columns on _9 and _10 exhibiting Parquet scan NATIVE_DATAFUSION and NATIVE_ICEBERG_COMPAT fail to read uint8, uint16 negative values correctly #1348
• test multiple pages with different sizes and nulls
  • Unsigned columns on _9 and _10 exhibiting Parquet scan NATIVE_DATAFUSION and NATIVE_ICEBERG_COMPAT fail to read uint8, uint16 negative values correctly #1348
• scan metrics (prefetch enabled) (native_iceberg_compat only)
  • Implement metrics similar to feat: CometNativeScan metrics from ParquetFileMetrics and FileStreamMetrics #1172.
• scan metrics (native_iceberg_compat only)
  • Implement metrics similar to feat: CometNativeScan metrics from ParquetFileMetrics and FileStreamMetrics #1172.

CometJoinSuite:

• HashJoin struct key (native_datafusion only)

CometCastSuite:

• cast TimestampType to LongType
• cast TimestampType to StringType
• cast TimestampType to DateType
  • Arrow-rs PR as the first step for support: Support different TimeUnits and timezones when reading Timestamps from INT96 arrow-rs#7285. DataFusion PR will follow.
• cast StructType to StringType
  • Unsigned columns on _9 and _10 exhibiting Parquet scan NATIVE_DATAFUSION and NATIVE_ICEBERG_COMPAT fail to read uint8, uint16 negative values correctly #1348

CometArrayExpressionSuite:

• array_intersect
  • Unsigned columns on _9 and _10 exhibiting Parquet scan NATIVE_DATAFUSION and NATIVE_ICEBERG_COMPAT fail to read uint8, uint16 negative values correctly #1348

CometExecSuite:

• Comet native metrics: scan (native_iceberg_compat only)
  • Implement metrics similar to feat: CometNativeScan metrics from ParquetFileMetrics and FileStreamMetrics #1172.

CometExpressionSuite:

• round
  • Unsigned columns on _9 and _10 exhibiting Parquet scan NATIVE_DATAFUSION and NATIVE_ICEBERG_COMPAT fail to read uint8, uint16 negative values correctly #1348
• hex
  • Unsigned columns on _9 and _10 exhibiting Parquet scan NATIVE_DATAFUSION and NATIVE_ICEBERG_COMPAT fail to read uint8, uint16 negative values correctly #1348

SparkFunSuite:

• ParquetEncryptionITCase

[andygrove]

One more for the list: #1488

[andygrove]

We now have more detailed issues tracking failures for each scan type, so I think we can close this issue now.

• Spark SQL test failures in native_iceberg_compat mode #1542
• Spark SQL test failures in native_datafusion scan #1545