[SPARK-11068][SQL] add callback to query execution #9078

cloud-fan · 2015-10-12T19:03:03Z

With this feature, we can track the query plan, time cost, exception during query execution for spark users.

cloud-fan · 2015-10-12T19:03:13Z

cc @yhuai

SparkQA · 2015-10-12T19:16:08Z

Test build #43579 has finished for PR 9078 at commit ea523b9.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait QueryExecutionListener
- class ExecutionListenerManager extends Logging

SparkQA · 2015-10-12T21:26:57Z

Test build #43583 has finished for PR 9078 at commit 4e9f27a.

This patch fails to build.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
- trait QueryExecutionListener
- class ExecutionListenerManager extends Logging

rxin · 2015-10-12T22:30:57Z

Does this need to be a public interface?

If yes, can we add experimental label to it.

rxin · 2015-10-12T22:32:04Z

Actually if QueryExecution is not a public interface (it is in execution - I think it's incorrectly labeled developer api), this should probably be in execution as well as a semi private interface?

SparkQA · 2015-10-12T22:35:43Z

Test build #43592 has finished for PR 9078 at commit 8f9bb8d.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait QueryExecutionListener
- class ExecutionListenerManager extends Logging

cloud-fan · 2015-10-13T00:48:47Z

retest this please.

SparkQA · 2015-10-13T01:09:08Z

Test build #43600 has finished for PR 9078 at commit 8f9bb8d.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait QueryExecutionListener
- class ExecutionListenerManager extends Logging

SparkQA · 2015-10-13T01:38:04Z

Test build #43604 has finished for PR 9078 at commit e5c2120.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-10-13T07:25:44Z

Test build #43626 has finished for PR 9078 at commit e43c955.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait QueryExecutionListener
- class ExecutionListenerManager extends Logging

cloud-fan · 2015-10-13T08:10:54Z

retest this please.

SparkQA · 2015-10-13T10:20:52Z

Test build #43637 has finished for PR 9078 at commit e43c955.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait QueryExecutionListener
- class ExecutionListenerManager extends Logging

davies · 2015-10-13T17:25:05Z

sql/core/src/main/scala/org/apache/spark/sql/QueryExecutionListener.scala

This should be thread-safe.

cloud-fan · 2015-10-13T17:53:15Z

retest this please.

SparkQA · 2015-10-13T18:10:52Z

Test build #43652 has finished for PR 9078 at commit b7927d8.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait QueryExecutionListener
- class ExecutionListenerManager extends Logging

SparkQA · 2015-10-13T19:55:34Z

Test build #43660 has finished for PR 9078 at commit 903d65e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait QueryExecutionListener
- class ExecutionListenerManager extends Logging

SparkQA · 2015-10-13T22:15:45Z

Test build #43667 has finished for PR 9078 at commit 2d81674.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait QueryExecutionListener
- class ExecutionListenerManager extends Logging

SparkQA · 2015-10-13T23:01:45Z

Test build #43674 has finished for PR 9078 at commit 0846260.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait QueryExecutionListener
- class ExecutionListenerManager extends Logging

davies · 2015-10-13T23:32:38Z

LGTM, waiting for tests.

SparkQA · 2015-10-14T00:57:59Z

Test build #43685 has finished for PR 9078 at commit 9943fea.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait QueryExecutionListener
- class ExecutionListenerManager extends Logging

davies · 2015-10-14T00:59:57Z

Merged into master, thanks!

…on thread ## What changes were proposed in this pull request? The SQL execution listener framework was created from scratch(see apache#9078). It didn't leverage what we already have in the spark listener framework, and one major problem is, the listener runs on the spark execution thread, which means a bad listener can block spark's query processing. This PR re-implements the SQL execution listener framework. Now `ExecutionListenerManager` is just a normal spark listener, which watches the `SparkListenerSQLExecutionEnd` events and post events to the user-provided SQL execution listeners. ## How was this patch tested? existing tests. Closes apache#22674 from cloud-fan/listener. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

cloud-fan force-pushed the callback branch from ea523b9 to 4e9f27a Compare October 12, 2015 20:59

cloud-fan force-pushed the callback branch from 4e9f27a to 8f9bb8d Compare October 12, 2015 22:14

cloud-fan force-pushed the callback branch from 8f9bb8d to e5c2120 Compare October 13, 2015 01:24

cloud-fan force-pushed the callback branch from e5c2120 to e43c955 Compare October 13, 2015 06:55

cloud-fan force-pushed the callback branch from e43c955 to b7927d8 Compare October 13, 2015 17:19

davies reviewed Oct 13, 2015
View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/QueryExecutionListener.scala Outdated

Copy link

Contributor

davies Oct 13, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be thread-safe.

add callback to query execution

903d65e

cloud-fan force-pushed the callback branch from b7927d8 to 903d65e Compare October 13, 2015 19:03

add javadoc

0846260

cloud-fan force-pushed the callback branch from 2d81674 to 0846260 Compare October 13, 2015 20:53

do not share listeners between seesions

9943fea

asfgit closed this in 15ff85b Oct 14, 2015

yhuai mentioned this pull request Oct 26, 2015

[SPARK-11253][SQL] reset all accumulators in physical operators before execute an action #9215

Closed

cloud-fan mentioned this pull request Oct 8, 2018

[SPARK-25680][SQL] SQL execution listener shouldn't happen on execution thread #22674

Closed

[SPARK-11068][SQL] add callback to query execution #9078

[SPARK-11068][SQL] add callback to query execution #9078

Uh oh!

Conversation

cloud-fan commented Oct 12, 2015

Uh oh!

cloud-fan commented Oct 12, 2015

Uh oh!

SparkQA commented Oct 12, 2015

Uh oh!

SparkQA commented Oct 12, 2015

Uh oh!

rxin commented Oct 12, 2015

Uh oh!

rxin commented Oct 12, 2015

Uh oh!

SparkQA commented Oct 12, 2015

Uh oh!

cloud-fan commented Oct 13, 2015

Uh oh!

SparkQA commented Oct 13, 2015

Uh oh!

SparkQA commented Oct 13, 2015

Uh oh!

SparkQA commented Oct 13, 2015

Uh oh!

cloud-fan commented Oct 13, 2015

Uh oh!

SparkQA commented Oct 13, 2015

Uh oh!

davies Oct 13, 2015

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Oct 13, 2015

Uh oh!

SparkQA commented Oct 13, 2015

Uh oh!

SparkQA commented Oct 13, 2015

Uh oh!

SparkQA commented Oct 13, 2015

Uh oh!

SparkQA commented Oct 13, 2015

Uh oh!

davies commented Oct 13, 2015

Uh oh!

SparkQA commented Oct 14, 2015

Uh oh!

davies commented Oct 14, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants