Skip to content

Commit bf3f6d2

Browse files
brkyvzyhuai
authored andcommitted
[SPARK-17531][BACKPORT] Don't initialize Hive Listeners for the Execution Client
## What changes were proposed in this pull request? If a user provides listeners inside the Hive Conf, the configuration for these listeners are passed to the Hive Execution Client as well. This may cause issues for two reasons: 1. The Execution Client will actually generate garbage 2. The listener class needs to be both in the Spark Classpath and Hive Classpath This PR empties the listener configurations in HiveUtils.newTemporaryConfiguration so that the execution client will not contain the listener confs, but the metadata client will. ## How was this patch tested? Unit tests Author: Burak Yavuz <[email protected]> Closes #15087 from brkyvz/overwrite-hive-listeners.
1 parent 047bc3f commit bf3f6d2

File tree

2 files changed

+39
-0
lines changed

2 files changed

+39
-0
lines changed

sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -753,6 +753,13 @@ private[hive] object HiveContext {
753753
// hive.metastore.uris is not set.
754754
propMap.put(ConfVars.METASTOREURIS.varname, "")
755755

756+
// The execution client will generate garbage events, therefore the listeners that are generated
757+
// for the execution clients are useless. In order to not output garbage, we don't generate
758+
// these listeners.
759+
propMap.put(ConfVars.METASTORE_PRE_EVENT_LISTENERS.varname, "")
760+
propMap.put(ConfVars.METASTORE_EVENT_LISTENERS.varname, "")
761+
propMap.put(ConfVars.METASTORE_END_FUNCTION_LISTENERS.varname, "")
762+
756763
propMap.toMap
757764
}
758765

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one or more
3+
* contributor license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright ownership.
5+
* The ASF licenses this file to You under the Apache License, Version 2.0
6+
* (the "License"); you may not use this file except in compliance with
7+
* the License. You may obtain a copy of the License at
8+
*
9+
* http://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
18+
package org.apache.spark.sql.hive
19+
20+
import org.apache.hadoop.hive.conf.HiveConf.ConfVars
21+
22+
import org.apache.spark.sql.hive.test.TestHiveSingleton
23+
import org.apache.spark.sql.QueryTest
24+
25+
class HiveContextSuite extends QueryTest with TestHiveSingleton {
26+
test("newTemporaryConfiguration overwrites listener configurations") {
27+
val conf = HiveContext.newTemporaryConfiguration()
28+
assert(conf(ConfVars.METASTORE_PRE_EVENT_LISTENERS.varname) === "")
29+
assert(conf(ConfVars.METASTORE_EVENT_LISTENERS.varname) === "")
30+
assert(conf(ConfVars.METASTORE_END_FUNCTION_LISTENERS.varname) === "")
31+
}
32+
}

0 commit comments

Comments
 (0)