[SPARK-41097][CORE][SQL][SS][PROTOBUF] Remove redundant collection conversion base on Scala 2.13 code #38598
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
Will manually test Scala 2.13 |
LuciferYang
changed the title
LuciferYang
changed the title
LuciferYang
changed the title
Member
|
These are probably fine. Some may be hold-overs from earlier versions of Scala. I'm slightly worried that in some cases they cause a copy and we actually rely on that, though this is a bad way to rely on it. SKimming it, I am not sure that's true in these cases. |
Contributor
Author
|
Let me check again |
LuciferYang
commented
Nov 21, 2022
| // We don't want to change our target number of executors, because we already did that | ||
| // when the task backlog decreased. | ||
| if (decommissionEnabled) { | ||
| val executorIdsWithoutHostLoss = executorIdsToBeRemoved.toSeq.map( |
Contributor
Author
There was a problem hiding this comment.
No need to toSeq here because executorIdsToBeRemoved(ArrayBuffer).map finally calls toArray
LuciferYang
commented
Nov 21, 2022
|
|
||
| def apply(fields: java.util.List[StructField]): StructType = { | ||
| import scala.collection.JavaConverters._ | ||
| StructType(fields.asScala.toSeq) |
Contributor
Author
There was a problem hiding this comment.
This represents fields.asScala.toSeq.toArray, seems toSeq is redundant
LuciferYang
commented
Nov 21, 2022
| case None => | ||
| properties.filterKeys(!_.startsWith(CatalogTable.VIEW_PREFIX)) | ||
| .toSeq.sortBy(_._1).map(p => Row(p._1, p._2)).toSeq | ||
| .toSeq.sortBy(_._1).map(p => Row(p._1, p._2)) |
Contributor
Author
There was a problem hiding this comment.
toSeq has been called previously
LuciferYang
commented
Nov 21, 2022
| case None => | ||
| properties.toSeq.sortBy(_._1).map(kv => | ||
| toCatalystRow(kv._1, kv._2)).toSeq | ||
| toCatalystRow(kv._1, kv._2)) |
Contributor
Author
There was a problem hiding this comment.
toSeq has been called previously
Contributor
Author
|
Seems there is no API involved in lazy computing. I think it should be safe. Let me verify full UTs with 2.13 again |
Member
|
Let me know when it seems ready to merge |
Contributor
Author
|
ready to merge |
Member
|
Merged to master |
beliefer
pushed a commit
to beliefer/spark
that referenced
this pull request
Dec 15, 2022
…nversion base on Scala 2.13 code ### What changes were proposed in this pull request? This pr aims clean up redundant collection conversion base on Scala 2.13 code ### Why are the changes needed? Code clean up. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GitHub Actions Closes apache#38598 from LuciferYang/redundant-collection-conversion. Lead-authored-by: YangJie <[email protected]> Co-authored-by: yangjie01 <[email protected]> Signed-off-by: Sean Owen <[email protected]>
beliefer
pushed a commit
to beliefer/spark
that referenced
this pull request
Dec 18, 2022
…nversion base on Scala 2.13 code ### What changes were proposed in this pull request? This pr aims clean up redundant collection conversion base on Scala 2.13 code ### Why are the changes needed? Code clean up. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GitHub Actions Closes apache#38598 from LuciferYang/redundant-collection-conversion. Lead-authored-by: YangJie <[email protected]> Co-authored-by: yangjie01 <[email protected]> Signed-off-by: Sean Owen <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This pr aims clean up redundant collection conversion base on Scala 2.13 code
Why are the changes needed?
Code clean up.
Does this PR introduce any user-facing change?
No
How was this patch tested?