[SPARK-38182][SQL] Fix NoSuchElementException if pushed filter does not contain any references

[ulysses-you]

What changes were proposed in this pull request?

skip non-references filter during binding metadata-based filiter

Why are the changes needed?

this issue is from #35055.

reproduce:

CREATE TABLE t (c1 int) USING PARQUET;

SET spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.BooleanSimplification;

SELECT * FROM t WHERE c1 = 1 AND 2 > 1;

and the error msg:

java.util.NoSuchElementException: next on empty iterator
	at scala.collection.Iterator$$anon$2.next(Iterator.scala:41)
	at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
	at scala.collection.mutable.LinkedHashSet$$anon$1.next(LinkedHashSet.scala:89)
	at scala.collection.IterableLike.head(IterableLike.scala:109)
	at scala.collection.IterableLike.head$(IterableLike.scala:108)
	at org.apache.spark.sql.catalyst.expressions.AttributeSet.head(AttributeSet.scala:69)
	at org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex.$anonfun$listFiles$3(PartitioningAwareFileIndex.scala:85)
	at scala.Option.map(Option.scala:230)
	at org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex.listFiles(PartitioningAwareFileIndex.scala:84)
	at org.apache.spark.sql.execution.FileSourceScanExec.selectedPartitions$lzycompute(DataSourceScanExec.scala:249)

Does this PR introduce any user-facing change?

yes, a bug fix

How was this patch tested?

add a new test

[ulysses-you]

cc @Yaohua628 @cloud-fan

[Yaohua628]

Thanks for the fix! LGTM

[HyukjinKwon]

LGTM otherwise.

[ulysses-you]

thank you @HyukjinKwon , addressed the comment

[cloud-fan]

If we use rand() > 0.5, can we trigger the bug without disabling the optimizer rule?

[ulysses-you]

It can not trigger this bug since we can not push down indeterministic filter. see code in FileSourceStrategy

spark/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala

Lines 157 to 173 in e0bc977

	val normalizedFilters = DataSourceStrategy.normalizeExprs(
	filters.filter(_.deterministic), l.output)
	val partitionColumns =
	l.resolve(
	fsRelation.partitionSchema, fsRelation.sparkSession.sessionState.analyzer.resolver)
	val partitionSet = AttributeSet(partitionColumns)
	// this partitionKeyFilters should be the same with the ones being executed in
	// PruneFileSourcePartitions
	val partitionKeyFilters = DataSourceStrategy.getPushedDownFilters(partitionColumns,
	normalizedFilters)
	// subquery expressions are filtered out because they can't be used to prune buckets or pushed
	// down as data filters, yet they would be executed
	val normalizedFiltersWithoutSubqueries =
	normalizedFilters.filterNot(SubqueryExpression.hasSubquery)

[dongjoon-hyun]

Could you resolve the conflict, @ulysses-you ?

[dongjoon-hyun]

And, if you don't mind, could you add a minimal test coverage at FileIndexSuite.scala instead of SQLQuerySuite, @ulysses-you ?

[ulysses-you]

thank you @dongjoon-hyun for the reminder !

[dongjoon-hyun]

+1, LGTM. Thank you, @ulysses-you and all.
Merged to master.

[ulysses-you]

thank you all !