Skip to content

Commit 58fcb5a

Browse files
dilipbiswalgatorsmile
authored andcommitted
[SPARK-23275][SQL] hive/tests have been failing when run locally on the laptop (Mac) with OOM
## What changes were proposed in this pull request? hive tests have been failing when they are run locally (Mac Os) after a recent change in the trunk. After running the tests for some time, the test fails with OOM with Error: unable to create new native thread. I noticed the thread count goes all the way up to 2000+ after which we start getting these OOM errors. Most of the threads seem to be related to the connection pool in hive metastore (BoneCP-xxxxx-xxxx ). This behaviour change is happening after we made the following change to HiveClientImpl.reset() ``` SQL def reset(): Unit = withHiveState { try { // code } finally { runSqlHive("USE default") ===> this is causing the issue } ``` I am proposing to temporarily back-out part of a fix made to address SPARK-23000 to resolve this issue while we work-out the exact reason for this sudden increase in thread counts. ## How was this patch tested? Ran hive/test multiple times in different machines. (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review http://spark.apache.org/contributing.html before opening a pull request. Author: Dilip Biswal <[email protected]> Closes #20441 from dilipbiswal/hive_tests.
1 parent 31c00ad commit 58fcb5a

File tree

1 file changed

+11
-15
lines changed

1 file changed

+11
-15
lines changed

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala

Lines changed: 11 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -825,23 +825,19 @@ private[hive] class HiveClientImpl(
825825
}
826826

827827
def reset(): Unit = withHiveState {
828-
try {
829-
client.getAllTables("default").asScala.foreach { t =>
830-
logDebug(s"Deleting table $t")
831-
val table = client.getTable("default", t)
832-
client.getIndexes("default", t, 255).asScala.foreach { index =>
833-
shim.dropIndex(client, "default", t, index.getIndexName)
834-
}
835-
if (!table.isIndexTable) {
836-
client.dropTable("default", t)
837-
}
828+
client.getAllTables("default").asScala.foreach { t =>
829+
logDebug(s"Deleting table $t")
830+
val table = client.getTable("default", t)
831+
client.getIndexes("default", t, 255).asScala.foreach { index =>
832+
shim.dropIndex(client, "default", t, index.getIndexName)
838833
}
839-
client.getAllDatabases.asScala.filterNot(_ == "default").foreach { db =>
840-
logDebug(s"Dropping Database: $db")
841-
client.dropDatabase(db, true, false, true)
834+
if (!table.isIndexTable) {
835+
client.dropTable("default", t)
842836
}
843-
} finally {
844-
runSqlHive("USE default")
837+
}
838+
client.getAllDatabases.asScala.filterNot(_ == "default").foreach { db =>
839+
logDebug(s"Dropping Database: $db")
840+
client.dropDatabase(db, true, false, true)
845841
}
846842
}
847843
}

0 commit comments

Comments
 (0)