[SPARK-30556][SQL][FOLLOWUP] Reset the status changed in SQLExecution.withThreadLocalCaptured #27516

xuanyuanking · 2020-02-10T04:54:04Z

What changes were proposed in this pull request?

Follow up for #27267, reset the status changed in SQLExecution.withThreadLocalCaptured.

Why are the changes needed?

For code safety.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing UT.

xuanyuanking · 2020-02-10T04:54:43Z

cc @ajithme @cloud-fan @hvanhovell @gatorsmile

sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala

cloud-fan · 2020-02-10T05:06:52Z

sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala

    val localProps = Utils.cloneProperties(sc.getLocalProperties)
    Future {
+      val originalContext = SparkSession.getActiveSession
+      val originalLocalProps = Utils.cloneProperties(sc.getLocalProperties)


do we need to clone?

We can assure the LocalProps is used in current thread here, getLocalProperties is enough, done in a33154c, thanks.

I think clone is required. Please read
#27267 (comment)

Also i dont see why we need to reset. By default pool threads do not have any localproperties or localsession variable set. That means after the thread is used for first time, this change will always try to retain those properties and session by always finally resetting it to same, which is kind of stale already. Right.?

I think clone is required.

The two scenarios are different, the first one needs clone since we want a snapshot of local properties during submit. But for this one, we need to reset by the variable self. If use clone here, changes in other places might be dropped.

Also i dont see why we need to reset.

For the function added in SQLExecution here, I think we need to clean up all the side effects as far as possible, even though it works in this case.

SparkQA · 2020-02-10T05:37:44Z

Test build #118115 has finished for PR 27516 at commit 0128a37.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala

SparkQA · 2020-02-10T08:05:02Z

Test build #118119 has finished for PR 27516 at commit a33154c.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-02-10T09:02:24Z

sql/core/src/test/scala/org/apache/spark/sql/internal/ExecutorSideSQLConfSuite.scala


        // set local configuration and assert
-        val confValue1 = "e"
+        val confValue1 = UUID.randomUUID().toString()


I don't think this is particularly needed. The value here doesn't matter at all. It doesn't test if the given value itself was correct or not. It just tests if the value is set or not. But .. I am okay.

SparkQA · 2020-02-10T12:27:15Z

Test build #118142 has finished for PR 27516 at commit 51f7fc8.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-02-10T12:28:00Z

retest this please

SparkQA · 2020-02-10T17:27:10Z

Test build #118160 has finished for PR 27516 at commit 51f7fc8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell

LGTM - thanks for addressing this!

hvanhovell · 2020-02-10T21:16:57Z

Merging in master/3.0.

….withThreadLocalCaptured ### What changes were proposed in this pull request? Follow up for #27267, reset the status changed in SQLExecution.withThreadLocalCaptured. ### Why are the changes needed? For code safety. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Existing UT. Closes #27516 from xuanyuanking/SPARK-30556-follow. Authored-by: Yuanjian Li <[email protected]> Signed-off-by: herman <[email protected]> (cherry picked from commit a6b91d2) Signed-off-by: herman <[email protected]>

xuanyuanking · 2020-02-11T02:36:31Z

Thanks all for review.

gatorsmile · 2020-02-19T00:13:13Z

The original PR #27340 was merged to 2.4 too. Should we fix 2.4?

xuanyuanking · 2020-02-19T01:16:23Z

Copy, will send a backport.

….withThreadLocalCaptured Follow up for apache#27267, reset the status changed in SQLExecution.withThreadLocalCaptured. For code safety. No. Existing UT. Closes apache#27516 from xuanyuanking/SPARK-30556-follow. (cherry picked from commit a6b91d2)

….withThreadLocalCaptured ### What changes were proposed in this pull request? Follow up for apache#27267, reset the status changed in SQLExecution.withThreadLocalCaptured. ### Why are the changes needed? For code safety. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Existing UT. Closes apache#27516 from xuanyuanking/SPARK-30556-follow. Authored-by: Yuanjian Li <[email protected]> Signed-off-by: herman <[email protected]>

address comments

0128a37

xuanyuanking force-pushed the SPARK-30556-follow branch from 210b195 to 0128a37 Compare February 10, 2020 04:54

cloud-fan reviewed Feb 10, 2020

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Feb 10, 2020

View reviewed changes

address comments

a33154c

cloud-fan reviewed Feb 10, 2020

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala Show resolved Hide resolved

adress comment

51f7fc8

cloud-fan approved these changes Feb 10, 2020

View reviewed changes

HyukjinKwon reviewed Feb 10, 2020

View reviewed changes

HyukjinKwon approved these changes Feb 10, 2020

View reviewed changes

hvanhovell approved these changes Feb 10, 2020

View reviewed changes

dongjoon-hyun added the SQL label Feb 10, 2020

hvanhovell closed this in a6b91d2 Feb 10, 2020

xuanyuanking deleted the SPARK-30556-follow branch February 11, 2020 02:36

[SPARK-30556][SQL][FOLLOWUP] Reset the status changed in SQLExecution.withThreadLocalCaptured #27516

[SPARK-30556][SQL][FOLLOWUP] Reset the status changed in SQLExecution.withThreadLocalCaptured #27516

Uh oh!

Conversation

xuanyuanking commented Feb 10, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

xuanyuanking commented Feb 10, 2020

Uh oh!

Uh oh!

cloud-fan Feb 10, 2020

Choose a reason for hiding this comment

Uh oh!

xuanyuanking Feb 10, 2020

Choose a reason for hiding this comment

Uh oh!

ajithme Feb 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuanyuanking Feb 10, 2020

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Feb 10, 2020

Uh oh!

Uh oh!

SparkQA commented Feb 10, 2020

Uh oh!

HyukjinKwon Feb 10, 2020

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Feb 10, 2020

Uh oh!

cloud-fan commented Feb 10, 2020

Uh oh!

SparkQA commented Feb 10, 2020

Uh oh!

hvanhovell left a comment

Choose a reason for hiding this comment

Uh oh!

hvanhovell commented Feb 10, 2020

Uh oh!

xuanyuanking commented Feb 11, 2020

Uh oh!

gatorsmile commented Feb 19, 2020

Uh oh!

xuanyuanking commented Feb 19, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

ajithme Feb 10, 2020 •

edited

Loading