[SPARK-4871][SQL] Show sql statement in spark ui when run sql with spark-sql by scwf · Pull Request #3718 · apache/spark

What about just using the function that only sets the job description. I know it is deprecated, but @rxin is the one who deprecated it and says that we won't remove it.

@rxin should we undeprecate that function too BTW?

Or to use setLocalProperty(SparkContext.SPARK_JOB_DESCRIPTION, description) ?setJobDescription essentially calls it.

SparkQA · 2014-12-18T03:37:35Z

Test build #24570 has started for PR 3718 at commit 4d2038a.

This patch merges cleanly.

SparkQA · 2014-12-18T05:17:39Z

Test build #24570 has finished for PR 3718 at commit 4d2038a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-18T05:17:43Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24570/
Test FAILed.

scwf · 2014-12-18T08:39:07Z

Retest this please.

SparkQA · 2014-12-18T08:42:31Z

Test build #24580 has started for PR 3718 at commit 4d2038a.

This patch merges cleanly.

SparkQA · 2014-12-18T10:36:47Z

Test build #24580 has finished for PR 3718 at commit 4d2038a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-18T10:36:51Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24580/
Test PASSed.

scwf · 2014-12-19T03:06:11Z

Updated

SparkQA · 2014-12-19T04:47:28Z

Test build #24625 has started for PR 3718 at commit e0d6b5d.

This patch merges cleanly.

SparkQA · 2014-12-19T06:39:56Z

Test build #24625 has finished for PR 3718 at commit e0d6b5d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-19T06:40:00Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24625/
Test PASSed.

scwf · 2014-12-19T06:46:45Z

@marmbrus, is this ok to go?

marmbrus · 2014-12-19T20:47:39Z

Lets wait for some feedback from @rxin or @pwendell on the APIs in question here.

rxin · 2014-12-20T07:19:35Z

LGTM - but are we sure we don't want to set the job group id? We can use that to cancel running commands.

scwf · 2014-12-20T12:04:26Z

i think this is ok. Even if we set the job group id, we can not use the cancel api to cancel job in spark-sql or beeline.

marmbrus · 2014-12-20T18:54:31Z

In principle I agree with @rxin. We probably should be setting the job group id and saving it so that if the users presses ctrl-c (or calls Statement#cancel() from JDBC) we could use it to cancel the running job. What seemed odd to me was to set it to something random and throw away that id. Either way though, I'm okay with adding cancelation in a followup PR since having the query in the UI is nice.

marmbrus · 2014-12-20T18:56:05Z

One other thought: how does this handle really long queries, or ones with a lots of new lines? Does it mess up the UI? Do we need to truncate the statement? Can you post a screen shot?

scwf · 2014-12-20T19:33:41Z

1 My locally test it seems ctrl-c stops(not cancel) a running sql.

2 The sql statement wraps in ui auto.

markhamstra · 2014-12-20T21:22:49Z

That wrapping in the UI is barely tolerable for what are still fairly modest-length queries. We're really going to need some kind of elided query in the main Jobs table with a link or pop-up containing the full, pretty-printed query.

scwf · 2014-12-22T02:56:52Z

Ok, i will add a pop-up containing the full

liancheng · 2014-12-22T06:28:23Z

@marmbrus Maybe I missed something here, but why do we prefer setJobDescription over setJobGroup, especially when the former is deprecated?

@scwf For job cancellation/stopping, under which shell did you try CTRL-C? Beeline doesn't install a proper signal handler, thus CTRL-C simply kills beeline without stopping/canceling any running job. Actually there is a signal handler for JDBC statement cancellation, but it's commented out in beeline code. If you were trying spark-sql or spark-shell scripts, a CTRL-C kills the main SparkContext, thus all statements are stopped by force.

Would like to add that there is a known job description issue related to CTAS statements, see SPARK-2472. This PR shares exactly the same issue. Main point of SPARK-2472 quoted below:

... The reason is that InsertIntoHiveTable.execute() submits a job to perform the insertion eagerly at the end of the query planning phase. At that time, the job description hasn't been updated, thus shows the previous one.

marmbrus · 2014-12-22T20:36:45Z

It's not deprecated for any good reason and I think we should undeprecate it.

I'm not opposed to setting the job group, but I am opposed to setting it to a random value and then throwing that away. That seems to have no point, and would override a value that might have been set for some reason elsewhere.

liancheng · 2014-12-23T05:44:52Z

I see. Yes, a random group ID string is not intuitive nor useful. And just realize another reason that we shouldn't set group ID in SparkExecuteStatementOperation.run: this class is also used in the Thrift server, which can be accessed by multiple concurrent connections. Interleaved concurrent SQL jobs may mess up the group IDs.

scwf · 2014-12-29T06:42:20Z

Filed a PR to add pop-up for full job description #3819

scwf · 2014-12-30T23:02:56Z

@marmbrus, can this go since i have filed a PR to add pop-up for full job desc?

liancheng · 2015-01-08T13:57:19Z

This LGTM.

scwf · 2015-01-10T23:27:18Z

ping @marmbrus

marmbrus · 2015-01-11T01:05:12Z

Thanks, merged to master

…cription when it is very long In some case the job description will be very long, such as a long sql. refer to #3718 This PR add a pop-up for job description when it is long. ![image](https://cloud.githubusercontent.com/assets/7018048/5847400/c757cbbc-a207-11e4-891f-528821c2e68d.png) ![image](https://cloud.githubusercontent.com/assets/7018048/5847409/d434b2b4-a207-11e4-8813-03a74b43d766.png) Author: wangfei <wangfei1@huawei.com> Closes #3819 from scwf/popup-descrip-ui and squashes the following commits: ba02b83 [wangfei] address comments a7c5e7b [wangfei] spot that it's been truncated fbf6162 [wangfei] Merge branch 'master' into popup-descrip-ui 0bca96d [wangfei] remove no use val 4b55c3b [wangfei] fix style issue 353c6f4 [wangfei] pop up the description of job with a styled read-only text form field

…cription when it is very long In some case the job description will be very long, such as a long sql. refer to apache#3718 This PR add a pop-up for job description when it is long. ![image](https://cloud.githubusercontent.com/assets/7018048/5847400/c757cbbc-a207-11e4-891f-528821c2e68d.png) ![image](https://cloud.githubusercontent.com/assets/7018048/5847409/d434b2b4-a207-11e4-8813-03a74b43d766.png) Author: wangfei <wangfei1@huawei.com> Closes apache#3819 from scwf/popup-descrip-ui and squashes the following commits: ba02b83 [wangfei] address comments a7c5e7b [wangfei] spot that it's been truncated fbf6162 [wangfei] Merge branch 'master' into popup-descrip-ui 0bca96d [wangfei] remove no use val 4b55c3b [wangfei] fix style issue 353c6f4 [wangfei] pop up the description of job with a styled read-only text form field

show sql statement in spark ui when run sql use spark-sql

92ce834

fix compile error

df79837

marmbrus reviewed Dec 17, 2014
View reviewed changes

using setJobDescription

4d2038a

scwf added 2 commits December 19, 2014 12:41

fix conflicts

383b505

format fix

e0d6b5d

scwf mentioned this pull request Dec 29, 2014

[SPARK-4984][CORE][WEBUI] Adding a pop-up containing the full job description when it is very long #3819

Closed

asfgit closed this in 92d9a70 Jan 11, 2015

scwf deleted the sparksqlui branch January 11, 2015 01:28

tianyi mentioned this pull request Jan 14, 2015

[SPARK-5100][SQL] add thriftserver-ui support #3946

Closed

Conversation

scwf commented Dec 17, 2014

Uh oh!

SparkQA commented Dec 17, 2014

Uh oh!

SparkQA commented Dec 17, 2014

Uh oh!

AmplabJenkins commented Dec 17, 2014

Uh oh!

scwf commented Dec 17, 2014

Uh oh!

SparkQA commented Dec 17, 2014

Uh oh!

SparkQA commented Dec 17, 2014

Uh oh!

AmplabJenkins commented Dec 17, 2014

Uh oh!

scwf commented Dec 17, 2014

Uh oh!

SparkQA commented Dec 17, 2014

Uh oh!

SparkQA commented Dec 17, 2014

Uh oh!

AmplabJenkins commented Dec 17, 2014

Uh oh!

SparkQA commented Dec 17, 2014

Uh oh!

SparkQA commented Dec 17, 2014

Uh oh!

AmplabJenkins commented Dec 17, 2014

Uh oh!

marmbrus Dec 17, 2014

Choose a reason for hiding this comment

Uh oh!

scwf Dec 18, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 18, 2014

Uh oh!

SparkQA commented Dec 18, 2014

Uh oh!

AmplabJenkins commented Dec 18, 2014

Uh oh!

scwf commented Dec 18, 2014

Uh oh!

SparkQA commented Dec 18, 2014

Uh oh!

SparkQA commented Dec 18, 2014

Uh oh!

AmplabJenkins commented Dec 18, 2014

Uh oh!

scwf commented Dec 19, 2014

Uh oh!

SparkQA commented Dec 19, 2014

Uh oh!

SparkQA commented Dec 19, 2014

Uh oh!

AmplabJenkins commented Dec 19, 2014

Uh oh!

scwf commented Dec 19, 2014

Uh oh!

marmbrus commented Dec 19, 2014

Uh oh!

rxin commented Dec 20, 2014

Uh oh!

scwf commented Dec 20, 2014

Uh oh!

marmbrus commented Dec 20, 2014

Uh oh!

marmbrus commented Dec 20, 2014

Uh oh!

scwf commented Dec 20, 2014

Uh oh!

markhamstra commented Dec 20, 2014

Uh oh!

scwf commented Dec 22, 2014

Uh oh!

liancheng commented Dec 22, 2014

Uh oh!

marmbrus commented Dec 22, 2014