[SPARK-40880][SQL] Reimplement summary with dataframe operations

[zhengruifeng]

What changes were proposed in this pull request?

Reimplement summary with dataframe operations

Why are the changes needed?

1, do not truncate the sql plan any more;
2, enable sql optimization like column pruning:

scala> val df = spark.range(0, 3, 1, 10).withColumn("value", lit("str"))
df: org.apache.spark.sql.DataFrame = [id: bigint, value: string]

scala> df.summary("max", "50%").show
+-------+---+-----+
|summary| id|value|
+-------+---+-----+
|    max|  2|  str|
|    50%|  1| null|
+-------+---+-----+


scala> df.summary("max", "50%").select("id").show
+---+
| id|
+---+
|  2|
|  1|
+---+


scala> df.summary("max", "50%").select("id").queryExecution.optimizedPlan
res4: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan =
Project [element_at(id#367, summary#376, None, false) AS id#371]
+- Generate explode([max,50%]), false, [summary#376]
   +- Aggregate [map(max, cast(max(id#153L) as string), 50%, cast(percentile_approx(id#153L, [0.5], 10000, 0, 0)[0] as string)) AS id#367]
      +- Range (0, 3, step=1, splits=Some(10))

Does this PR introduce any user-facing change?

No

How was this patch tested?

existing UTs and manually check

[zhengruifeng]

cc @HyukjinKwon

[HyukjinKwon]

Merged to master.

[zhengruifeng]

thank you @HyukjinKwon !

[zhengruifeng]

CI is green in this PR, but it may have conflict with the previous one #38340

then the build in master is broken, I fix it in #38362