[SPARK-23653][SQL] Capture sql statements user input and show them in…

[LantaoJin]

… SQL UI

What changes were proposed in this pull request?

SPARK-4871 had already added the sql statement in job description for using spark-sql. But it has some problems:

1. long sql statement cannot be displayed in description column.
   

2. variables like ${var}, ${env:var} in sql cannot be resolved.
   spark-sql --hiveconf a=avalue --hivevar b=bvalue
   spark-sql> select '${a}', '${b}';

3. sql statement submitted in spark-shell or spark-submit cannot be covered.

In eBay, most spark sql applications like ETL, reporting using spark-submit to schedule their jobs with a few sql files. The sql statement in those applications cannot be saw in current spark UI.

More detail a scenario is team A developed a framework to submit application with sql sentences in a file

spark-submit --master yarn-cluster --class com.ebay.SQLFramework -s biz.sql

In the biz.sql, there are many sql sentences like

create or replace temporary view view_a select xx from table ${old_db}.table_a where dt=${check_date};
insert overwrite table ${new_db}.table_a select xx from view_a join ${new_db}.table_b;
...

Team B (Platform) need to capture the really sql sentences which are executed in whole cluster, as the sql files from Team A contains many variables. A better way is recording the really sql sentence in EventLog.

How was this patch tested?

[LantaoJin]

In #20803, the implementation is to bind sql text to DF. That's not good and will introduce many unexpected issues. I open this PR with new implementation. @jerryshao @cloud-fan and @dongjoon-hyun, sorry for that need you move to here to review.

[LantaoJin]

Hi, @jerryshao @cloud-fan, may I have some update?

[AmplabJenkins]

Can one of the admins verify this patch?

[LantaoJin]

Could anyone else attend to review this? Or should it be closed?