[SPARK-29450][SS][2.4] Measure the number of output rows for streaming aggregation with append mode

[HeartSaVioR]

What changes were proposed in this pull request?

This patch addresses missing metric, the number of output rows for streaming aggregation with append mode. Other modes are correctly measuring it.

Why are the changes needed?

Without the patch, the value for such metric is always 0.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit test added. Also manually tested with below query:

query

import spark.implicits._

spark.conf.set("spark.sql.shuffle.partitions", "5")

val df = spark.readStream
  .format("rate")
  .option("rowsPerSecond", 1000)
  .load()
  .withWatermark("timestamp", "5 seconds")
  .selectExpr("timestamp", "mod(value, 100) as mod", "value")
  .groupBy(window($"timestamp", "10 seconds"), $"mod")
  .agg(max("value").as("max_value"), min("value").as("min_value"), avg("value").as("avg_value"))

val query = df
  .writeStream
  .format("memory")
  .option("queryName", "test")
  .outputMode("append")
  .start()

query.awaitTermination()

before the patch

after the patch

[HeartSaVioR]

cc. @gatorsmile
also cc. to @dongjoon-hyun who leads the efforts on releasing 2.4.5.

To provide some context, @gatorsmile found that #26104 actually fixed a regression (I wasn't aware) which seems to be broken at Spark 2.3.0, and asked about porting back the patch to 2.4 version line.
(#26104 (comment))

I'm not sure whether @gatorsmile wanted to deal with this in Spark 2.4.5, or it's just to ensure the new bugfix release of Spark 2.4 will include this. @gatorsmile Could you please clarify?

[SparkQA]

Test build #116749 has finished for PR 27209 at commit 25b2769.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[HeartSaVioR]

retest this, please

[SparkQA]

Test build #116774 has finished for PR 27209 at commit 25b2769.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

thanks, merging to 2.4!

[HeartSaVioR]

Thanks for reviewing and merging!

[dongjoon-hyun]

+1, late LGTM.