[SPARK-41709][CORE][SQL][UI] Explicitly define `Seq` as `collection.Seq` to avoid `toSeq` when create ui objects from protobuf objects for Scala 2.13 #39215

LuciferYang · 2022-12-26T06:25:07Z

What changes were proposed in this pull request?

This pr explicitly define Seq as collection.Seq in the ui related class definitions to avoid collection conversion when creating ui objects from protobuf objects to make no performance difference between Scala 2.13 and Scala 2.12.

Why are the changes needed?

Avoid collection conversion when creating ui objects from protobuf objects for Scala 2.13.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass GitHub Actions

core/src/main/scala/org/apache/spark/status/api/v1/api.scala

codecov-commenter · 2022-12-26T10:33:12Z

Codecov Report

Merging #39215 (1af0a51) into master (1af0a51) will not change coverage.
The diff coverage is n/a.

❗ Current head 1af0a51 differs from pull request most recent head 755a01e. Consider uploading reports for the commit 755a01e to get more accurate results

@@           Coverage Diff           @@
##           master   #39215   +/-   ##
=======================================
  Coverage   90.95%   90.95%           
=======================================
  Files         342      342           
  Lines       76706    76706           
  Branches    10460    10460           
=======================================
  Hits        69765    69765           
  Misses       5352     5352           
  Partials     1589     1589

Flag	Coverage Δ
unittests	`90.93% <0.00%> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

srowen · 2022-12-26T19:34:24Z

sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala

  var physicalPlanDescription: String = null
  var modifiedConfigs: Map[String, String] = _
-  var metrics = Seq[SQLPlanMetric]()
+  var metrics = scala.collection.Seq[SQLPlanMetric]()


Just collection.Seq?

LuciferYang · 2022-12-27T09:58:14Z

project/MimaExcludes.scala

+    ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.storage.BlockManagerMessages#RegisterBlockManager.apply"),
+
+    // [SPARK-41709][CORE][SQL][UI] Explicitly define Seq as collection.Seq to avoid toSeq when create ui objects from protobuf objects for Scala 2.13
+    ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.status.api.v1.ApplicationEnvironmentInfo.resourceProfiles"),


Need to add MimaExcludes for Scala 2.13, should we close this one

If this is acceptable, I will make more changes, otherwise I will close this pr

I think those are all internal methods right? people wouldn't call these classes in code

Yes, the mima filters to be added are all internal apis

LuciferYang · 2022-12-27T10:31:03Z

sql/core/src/main/scala/org/apache/spark/status/protobuf/sql/SQLExecutionUIDataSerializer.scala

    val errorMessage = getOptional(ui.hasErrorMessage, () => ui.getErrorMessage)
    val metrics =
-      ui.getMetricsList.asScala.map(m => SQLPlanMetricSerializer.deserialize(m)).toSeq
+      ui.getMetricsList.asScala.map(m => SQLPlanMetricSerializer.deserialize(m))


If no toSeq, one operation is about 10ns. After adding toSeq, the delay will increase linearly with the length of the original data:

240ns when input length is 10

1740ns when input length is 100

16600ns when input length is 1000

…tatus-pb-toSeq

LuciferYang · 2022-12-28T05:03:02Z

project/MimaExcludes.scala

+    ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.storage.BlockManagerMessages#RegisterBlockManager.apply"),
+
+    // [SPARK-41709][CORE][SQL][UI] Explicitly define Seq as collection.Seq to avoid toSeq when create ui objects from protobuf objects for Scala 2.13
+    ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.status.api.v1.ApplicationEnvironmentInfo.sparkProperties"),


Scala 2.12 does not need to add these exclude filters, all changes only involve Scala 2.13.

srowen · 2022-12-28T05:47:31Z

Seems OK to me, un-mark it as Draft to let it test

LuciferYang · 2022-12-28T13:20:54Z

Local maven test with Scala 2.13, all passed

srowen · 2022-12-28T23:26:32Z

Merged to master

LuciferYang · 2022-12-29T07:09:35Z

Thanks @srowen

LuciferYang added 2 commits December 26, 2022 14:09

Explicitly define Seq to collection.Seq

f9ef3c8

sql module

90e6f6f

github-actions bot added CORE SQL WEB UI labels Dec 26, 2022

LuciferYang added 2 commits December 26, 2022 17:37

avoid change public apis

f5865c2

more

ad179a1

LuciferYang commented Dec 26, 2022

View reviewed changes

core/src/main/scala/org/apache/spark/status/api/v1/api.scala Show resolved Hide resolved

LuciferYang marked this pull request as draft December 26, 2022 10:46

Merge branch 'apache:master' into status-pb-toSeq

755a01e

srowen reviewed Dec 26, 2022

View reviewed changes

just use collection.Seq

02d6e0c

LuciferYang marked this pull request as ready for review December 27, 2022 02:24

LuciferYang marked this pull request as draft December 27, 2022 09:46

add mima excludes

fb4c23c

github-actions bot added the BUILD label Dec 27, 2022

LuciferYang commented Dec 27, 2022

View reviewed changes

fix more

d3659eb

LuciferYang commented Dec 27, 2022

View reviewed changes

add more change

2453322

LuciferYang added 4 commits December 28, 2022 11:34

fix compile

6c46a0a

fix 2.13 compile

464ce87

add more mima excludes

9a682a5

Merge branch 'status-pb-toSeq' of github.com:LuciferYang/spark into s…

43acbcf

…tatus-pb-toSeq

LuciferYang commented Dec 28, 2022

View reviewed changes

fix catalyst compile

5b500d4

LuciferYang marked this pull request as ready for review December 28, 2022 05:50

srowen closed this in 8cceb39 Dec 28, 2022

[SPARK-41709][CORE][SQL][UI] Explicitly define Seq as collection.Seq to avoid toSeq when create ui objects from protobuf objects for Scala 2.13 #39215

[SPARK-41709][CORE][SQL][UI] Explicitly define Seq as collection.Seq to avoid toSeq when create ui objects from protobuf objects for Scala 2.13 #39215

Uh oh!

Conversation

LuciferYang commented Dec 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Uh oh!

codecov-commenter commented Dec 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

srowen commented Dec 28, 2022

Uh oh!

LuciferYang commented Dec 28, 2022

Uh oh!

srowen commented Dec 28, 2022

Uh oh!

LuciferYang commented Dec 29, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[SPARK-41709][CORE][SQL][UI] Explicitly define `Seq` as `collection.Seq` to avoid `toSeq` when create ui objects from protobuf objects for Scala 2.13 #39215

[SPARK-41709][CORE][SQL][UI] Explicitly define `Seq` as `collection.Seq` to avoid `toSeq` when create ui objects from protobuf objects for Scala 2.13 #39215

LuciferYang commented Dec 26, 2022 •

edited

Loading

codecov-commenter commented Dec 26, 2022 •

edited

Loading