[SPARK-34031][SQL] Union operator missing rowCount when CBO enabled

[wangyum]

What changes were proposed in this pull request?

This pr add row count to Union operator when CBO enabled.

spark.sql("CREATE TABLE t1 USING parquet AS SELECT id FROM RANGE(10)")
spark.sql("CREATE TABLE t2 USING parquet AS SELECT id FROM RANGE(10)")
spark.sql("ANALYZE TABLE t1 COMPUTE STATISTICS FOR ALL COLUMNS")
spark.sql("ANALYZE TABLE t2 COMPUTE STATISTICS FOR ALL COLUMNS")
spark.sql("set spark.sql.cbo.enabled=true")
spark.sql("SELECT * FROM t1 UNION ALL SELECT * FROM t2").explain("cost")

Before this pr:

== Optimized Logical Plan ==
Union false, false, Statistics(sizeInBytes=320.0 B)
:- Relation[id#5880L] parquet, Statistics(sizeInBytes=160.0 B, rowCount=10)
+- Relation[id#5881L] parquet, Statistics(sizeInBytes=160.0 B, rowCount=10)

After this pr:

== Optimized Logical Plan ==
Union false, false, Statistics(sizeInBytes=320.0 B, rowCount=20)
:- Relation[id#2138L] parquet, Statistics(sizeInBytes=160.0 B, rowCount=10)
+- Relation[id#2139L] parquet, Statistics(sizeInBytes=160.0 B, rowCount=10)

Why are the changes needed?

Improve query performance, JoinEstimation.estimateInnerOuterJoin need the row count.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit test.

[wangyum]

Same logic, just add row count:

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/SizeInBytesOnlyStatsPlanVisitor.scala

Lines 148 to 150 in 6c5ba81

	override def visitUnion(p: Union): Statistics = {
	Statistics(sizeInBytes = p.children.map(_.stats.sizeInBytes).sum)
	}

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38334/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38334/

[SparkQA]

Test build #133746 has finished for PR 31068 at commit c0dbbe4.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

Seems needing to update query plan files.

[maropu]

Looks fine except for the @viirya comment.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38358/

[HyukjinKwon]

@wangyum, sorry but can you push an empty commit to retrigger the GA build? I would like to keep the result of the test failure because it looks like a flaky test.

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38358/

[HyukjinKwon]

The flaky tests are known and being fixed at https://github.com/apache/spark/pull/31076. Let me just merge this in

[HyukjinKwon]

Merged to master.

[SparkQA]

Test build #133770 has finished for PR 31068 at commit 3c5af90.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.