[SPARK-47897][SQL][3.5] Fix ExpressionSet performance regression in scala 2.12

[wForget]

What changes were proposed in this pull request?

Fix ExpressionSet performance regression in scala 2.12.

Why are the changes needed?

The implementation of the SetLike.++ method in scala 2.12 is to iteratively execute the + method. The ExpressionSet.+ method first clones a new object and then adds element, which is very expensive.

https://github.com/scala/scala/blob/ceaf7e68ac93e9bbe8642d06164714b2de709c27/src/library/scala/collection/SetLike.scala#L186

After #36121, the ++ and -- methods in ExpressionSet of scala 2.12 were removed, causing performance regression.

Does this PR introduce any user-facing change?

How was this patch tested?

Benchmark code:

import org.apache.spark.benchmark.Benchmark
import org.apache.spark.sql.catalyst.dsl.expressions._
import org.apache.spark.sql.catalyst.expressions.{AttributeReference, ExpressionSet, ExprId}
import org.apache.spark.sql.types.IntegerType

object TestBenchmark {
  def main(args: Array[String]): Unit = {
    val count = 300
    val benchmark = new Benchmark("Test ExpressionSetV2 ++ ", count)
    val aUpper = AttributeReference("A", IntegerType)(exprId = ExprId(1))

    var initialSet = ExpressionSet((0 until 300).map(i => aUpper + i))
    val setToAddWithSameDeterministicExpression = ExpressionSet((0 until 300).map(i => aUpper + i))

    benchmark.addCase("Test ++", 10) { _: Int =>
      for (_ <- 0L until count) {
        initialSet ++= setToAddWithSameDeterministicExpression
      }
    }
    benchmark.run()
  }
}

before this change:

OpenJDK 64-Bit Server VM 1.8.0_222-b10 on Linux 3.10.0-957.el7.x86_64
Intel Core Processor (Skylake, IBRS)
Test ExpressionSetV2 ++ :                 Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Test ++                                            1577           1691          61          0.0     5255516.0       1.0X

after this change:

OpenJDK 64-Bit Server VM 1.8.0_222-b10 on Linux 3.10.0-957.el7.x86_64
Intel Core Processor (Skylake, IBRS)
Test ExpressionSetV2 ++ :                 Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Test ++                                              14             14           0          0.0       45395.2       1.0X

Was this patch authored or co-authored using generative AI tooling?

No

[wForget]

cc @minyyy @cloud-fan could you please take a look?

[dongjoon-hyun]

If this is a regression at SPARK-38836, do we need to to fix this at branch-3.4, too, @wForget ?

[dongjoon-hyun]

cc @LuciferYang , @viirya , too

[viirya]

Good catch.

[minyyy]

Could you add a comment on the method about why we don't use the SetLike.default here? Thanks.

[wForget]

thanks, added.

[wForget]

If this is a regression at SPARK-38836, do we need to to fix this at branch-3.4, too, @wForget ?

I think it's needed.

[yaooqinn]

Thank you @wForget, and @dongjoon-hyun @viirya @minyyy @cloud-fan

Merged to '3.5.2', '3.4.4'

[LuciferYang]

Thanks for your fix @wForget

But for the TestBenchmark shown in the current pr description, there are some compilation errors when I manually copy it for testing:

[ERROR] [Error] /spark/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/TestBenchmark.scala:29: type mismatch;
 found   : Int
 required: String
[ERROR] [Error] /spark/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/TestBenchmark.scala:31: type mismatch;
 found   : Int
 required: String
[ERROR] two errors found

Could you please correct it in pr description?

[wForget]

But for the TestBenchmark shown in the current pr description, there are some compilation errors when I manually copy it for testing:

I guess you may be missing import org.apache.spark.sql.catalyst.dsl.expressions._

Could you please correct it in pr description?

I have added imports.

[LuciferYang]

But for the TestBenchmark shown in the current pr description, there are some compilation errors when I manually copy it for testing:

I guess you may be missing import org.apache.spark.sql.catalyst.dsl.expressions._

Thanks. Could you please add this import to the benchmark code in the PR description?

[LuciferYang]

Thanks @wForget