[SPARK-15730][SQL] Respect the --hiveconf in the spark-sql command line

[chenghao-intel]

What changes were proposed in this pull request?

We should respect the --hiveconf in the spark-sql command line, otherwise, the existing applications based on the spark 1.6 and earlier will broke, as the configurations specified via --hiveconf are missing

How was this patch tested?

I've added the unit test, but still need to be verified with real application.

[SparkQA]

Test build #60127 has finished for PR 13542 at commit bbf0ac4.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #60144 has finished for PR 13542 at commit 1fc8a30.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[chenghao-intel]

cc @yhuai

[chenghao-intel]

Currently, the SparkSQL cli will ignore the configuration passed from commandline via --hiveconf, this will break lots of existing application, it's not by design, isn't it? @yhuai @rxin

We are still verifying if this PR by running the TPCx-BB, will remove the WIP once it passed the test.

[yhuai]

@chenghao-intel Does --hiveconf ${ConfVars.METASTORECONNECTURLKEY}=$jdbcUrl work?

[chenghao-intel]

yes, it works, that's intention, right?

But seems the below code in SparkSQLCliDriver will not work as we expected.

      if (key != "javax.jdo.option.ConnectionURL") {
        conf.set(key, value)
        sessionState.getOverriddenConfigurations.put(key, value)
      }

Why do we have to ignore the connection url?

[chenghao-intel]

@yhuai any concern for this?

[yhuai]

Do you know why those work but something like --hiveconf conf1=conftest does not?

[jameszhouyi]

Hi @chenghao-intel
This PR is tested and it's OK for my case.

[chenghao-intel]

Thanks @jameszhouyi , I've removed the WIP from the title.

[jameszhouyi]

Hi Spark community ，
Could you please help to review this PR to merge in 2.0.0 ? Since this bug has broken our real cases. Thanks in advance !

[yhuai]

Can you provide more information on the root cause? Seems it is not clear why it does not work.

[yhuai]

also, can you try --conf and see if it works?

[chenghao-intel]

@yhuai I couldn't find any piece of code to copy the HiveConf(from SessionState) to SqlConf? Can you confirm this?

Probably that's the reason why --hiveconf doesn't work.

[jameszhouyi]

Hi,
Could you please help to review this PR to merge it in 2.0.0 ? this broken thing blocked our testing. Thanks!

[yhuai]

I have opened https://github.com/apache/spark/pull/14058/files (it has one update).