[SPARK-31522][SQL] Hive metastore client initialization related configurations should be static

[yaooqinn]

What changes were proposed in this pull request?

HiveClient instance is cross-session, the following configurations which are defined in HiveUtils and used to create it should be considered static:

1. spark.sql.hive.metastore.version - used to determine the hive version in Spark
2. spark.sql.hive.metastore.jars - hive metastore related jars location which is used by spark to create hive client
3. spark.sql.hive.metastore.sharedPrefixes and spark.sql.hive.metastore.barrierPrefixes - package names of classes that are shared or separated between SparkContextLoader and hive client class loader

Those are used only once when creating the hive metastore client. They should be static in SQLConf for retrieving them correctly. We should avoid them being changed by users with SET/RESET command.

Speaking of spark.sql.hive.version - the fake of the spark.sql.hive.metastore.version, it is used by jdbc/thrift client for backward compatibility.

Why are the changes needed?

bugfix, these configurations should not be changed.

Does this PR introduce any user-facing change?

Yes, the following set of configs are not allowed to change.

Seq("spark.sql.hive.metastore.version ",
      "spark.sql.hive.metastore.jars",
      "spark.sql.hive.metastore.sharedPrefixes",
      "spark.sql.hive.metastore.barrierPrefixes")

How was this patch tested?

add unit test

[yaooqinn]

cc: @cloud-fan @dongjoon-hyun @HyukjinKwon, many thanks.

[SparkQA]

Test build #121635 has finished for PR 28302 at commit 344f558.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #121634 has finished for PR 28302 at commit d0cf0ec.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #121644 has finished for PR 28302 at commit 5c15a98.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #121652 has finished for PR 28302 at commit c72ba70.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #121650 has finished for PR 28302 at commit 09b87ff.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[yaooqinn]

retest this please

[SparkQA]

Test build #121658 has finished for PR 28302 at commit c72ba70.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #121668 has finished for PR 28302 at commit 7a26872.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

thanks, merging to master/3.0!

cc @juliuszsompolski as well.

[dongjoon-hyun]

+1, late LGTM.