Skip to content

Conversation

@LuciferYang
Copy link
Contributor

What changes were proposed in this pull request?

The optimization result of CostBasedJoinReorder maybe different with same input in Scala 2.12 and Scala 2.13 if there are more than one same cost candidate plans.

In this pr give a way to make the optimization result deterministic as much as possible with same input in Scala 2.12 and Scala 2.13, the main change of this pr as follow:

  • Change to use LinkedHashMap instead of Map to store foundPlans in JoinReorderDP.search method to ensure same iteration order with same insert order because iteration order of Map behave differently under Scala 2.12 and 2.13

  • Fixed StarJoinCostBasedReorderSuite affected by the above change

  • Regenerate golden files affected by the above change.

Why are the changes needed?

We need to support a Scala 2.13 build.

Does this PR introduce any user-facing change?

No

How was this patch tested?

  • Scala 2.12: Pass the Jenkins or GitHub Action

  • Scala 2.13: All tests passed.

Do the following:

dev/change-scala-version.sh 2.13
mvn clean install -DskipTests  -pl sql/core -Pscala-2.13 -am
mvn test -pl sql/core -Pscala-2.13

Before

Tests: succeeded 8485, failed 13, canceled 1, ignored 52, pending 0
*** 13 TESTS FAILED ***

After

Tests: succeeded 8498, failed 0, canceled 1, ignored 52, pending 0
All tests passed.

@LuciferYang LuciferYang changed the title Spark 32848 [SPAKR-32848][SQL] Let CostBasedJoinReorder produce deterministic result with Scala 2.12 and 2.13 Sep 10, 2020
@LuciferYang LuciferYang changed the title [SPAKR-32848][SQL] Let CostBasedJoinReorder produce deterministic result with Scala 2.12 and 2.13 [SPAKR-32848][SQL] Let CostBasedJoinReorder produce same result in Scala 2.12 and 2.13 Sep 10, 2020
@LuciferYang LuciferYang changed the title [SPAKR-32848][SQL] Let CostBasedJoinReorder produce same result in Scala 2.12 and 2.13 [SPARK-32848][SQL] Let CostBasedJoinReorder produce same result in Scala 2.12 and 2.13 Sep 10, 2020
@LuciferYang LuciferYang deleted the SPARK-32848 branch June 6, 2022 03:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant