[SPARK-29450][SS] Measure the number of output rows for streaming aggregation with append mode

[HeartSaVioR]

What changes were proposed in this pull request?

This patch addresses missing metric, the number of output rows for streaming aggregation with append mode. Other modes are correctly measuring it.

Why are the changes needed?

Without the patch, the value for such metric is always 0.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit test added. Also manually tested with below query:

query

import spark.implicits._

spark.conf.set("spark.sql.shuffle.partitions", "5")

val df = spark.readStream
  .format("rate")
  .option("rowsPerSecond", 1000)
  .load()
  .withWatermark("timestamp", "5 seconds")
  .selectExpr("timestamp", "mod(value, 100) as mod", "value")
  .groupBy(window($"timestamp", "10 seconds"), $"mod")
  .agg(max("value").as("max_value"), min("value").as("min_value"), avg("value").as("avg_value"))

val query = df
  .writeStream
  .format("memory")
  .option("queryName", "test")
  .outputMode("append")
  .start()

query.awaitTermination()

before the patch

after the patch

[HeartSaVioR]

cc. @tdas @zsxwing @jose-torres @gaborgsomogyi
also cc. to @jaceklaskowski as reporter of the issue

[SparkQA]

Test build #111992 has finished for PR 26104 at commit bf14f96.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HeartSaVioR]

I've added the query and screenshots to show which issue this PR fixes.

[HeartSaVioR]

retest this, please

[SparkQA]

Test build #113983 has finished for PR 26104 at commit bf14f96.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HeartSaVioR]

@tdas @zsxwing @jose-torres @gaborgsomogyi Kindly reminder.

[HyukjinKwon]

Looks fine to me.

[dongjoon-hyun]

Retest this please.

[SparkQA]

Test build #115529 has finished for PR 26104 at commit bf14f96.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Merged to master.

[HeartSaVioR]

Thanks all for reviewing and merging!

[gatorsmile]

A regression introduced in #18107 ?

[HeartSaVioR]

Yes, looks like so. The API seemed to be revised completely in #18107 and I don't have background though.

[gatorsmile]

@HeartSaVioR Could you help backport this to 2.4?

[HeartSaVioR]

Ah yes I didn't get the intention on comment and now I'm seeing the intention. Happy to submit a PR for porting back of this. Thanks!