[SPARK-11089] [SQL] Adds option for disabling multi-session in Thrift server

[liancheng]

This PR adds a new option spark.sql.hive.thriftServer.singleSession for disabling multi-session support in the Thrift server.

Note that this option is added as a Spark configuration (retrieved from SparkConf) rather than Spark SQL configuration (retrieved from SQLConf). This is because all SQL configurations are session-ized. Since multi-session support is by default on, no JDBC connection can modify global configurations like the newly added one.

[liancheng]

Another approach for this problem is to have "global" options. For example, SET -g key=value sets key to value for all sessions. This adds a new concept and requires more work, but seems to be cleaner and in general more useful. What need to be done are:

1. Add a new syntax SET -g
2. Each HiveContext should remember its parent HiveContext so that we can retrieve the "root" one.
3. When a global option is set, it's always set in the root HiveContext.

Another benefit of this approach is that we don't need to restart the Thrift server to switch between single-/multi-session modes.

@marmbrus @davies @yhuai Any suggestions?

[davies]

Is it possible that we only introduce a command line option here? (Not sure how the JDBC server is used).

[liancheng]

Do you mean something like --single-session? The benefit of SparkConf option is that users can add it in spark-defaults.conf.

Command line options of start-thriftserver.sh are parsed by both SparkSubmit and HiveServer2.ServerOptionsProcessor. The latter isn't extensible. One option is to add a --single-session option in SparkSubmit and make a synonymous of --conf spark.sql.hive.thriftServer.singleSession=true. However, this option doesn't seem to be general enough to be added in SparkSubmit.

[davies]

I see, thanks!

[davies]

@liancheng I like the this approach more, all global configurations should be placed in SparkConf (could be override in current session)

[davies]

@liancheng This option is only for backward compatibility purpose, could we document it somewhere? We should also mention this change (isolated sessions) in release note.

[liancheng]

Yeah, I'll add it to the programming guide.

[marmbrus]

+1 to this approach instead of adding a new concept.

[SparkQA]

Test build #46000 has finished for PR 9740 at commit eeeb194.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

@liancheng Will you update the programming guide in this PR?

[liancheng]

Yeah, updating this one soon.

[SparkQA]

Test build #46081 has finished for PR 9740 at commit 8b70477.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[marmbrus]

Merging to master and 1.6.