Commit 58fcb5a
[SPARK-23275][SQL] hive/tests have been failing when run locally on the laptop (Mac) with OOM
## What changes were proposed in this pull request?
hive tests have been failing when they are run locally (Mac Os) after a recent change in the trunk. After running the tests for some time, the test fails with OOM with Error: unable to create new native thread.
I noticed the thread count goes all the way up to 2000+ after which we start getting these OOM errors. Most of the threads seem to be related to the connection pool in hive metastore (BoneCP-xxxxx-xxxx ). This behaviour change is happening after we made the following change to HiveClientImpl.reset()
``` SQL
def reset(): Unit = withHiveState {
try {
// code
} finally {
runSqlHive("USE default") ===> this is causing the issue
}
```
I am proposing to temporarily back-out part of a fix made to address SPARK-23000 to resolve this issue while we work-out the exact reason for this sudden increase in thread counts.
## How was this patch tested?
Ran hive/test multiple times in different machines.
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
Please review http://spark.apache.org/contributing.html before opening a pull request.
Author: Dilip Biswal <[email protected]>
Closes #20441 from dilipbiswal/hive_tests.1 parent 31c00ad commit 58fcb5a
File tree
1 file changed
+11
-15
lines changed- sql/hive/src/main/scala/org/apache/spark/sql/hive/client
1 file changed
+11
-15
lines changedLines changed: 11 additions & 15 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
825 | 825 | | |
826 | 826 | | |
827 | 827 | | |
828 | | - | |
829 | | - | |
830 | | - | |
831 | | - | |
832 | | - | |
833 | | - | |
834 | | - | |
835 | | - | |
836 | | - | |
837 | | - | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
838 | 833 | | |
839 | | - | |
840 | | - | |
841 | | - | |
| 834 | + | |
| 835 | + | |
842 | 836 | | |
843 | | - | |
844 | | - | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
845 | 841 | | |
846 | 842 | | |
847 | 843 | | |
| |||
0 commit comments