[SPARK-32062][SQL] Reset listenerRegistered in SparkSession #28899
Conversation
| assert(session.conf.get(WAREHOUSE_PATH) === "SPARK-31532-db-2") | ||
| } | ||
|
|
||
| test("SPARK-32062: reset listenerRegistered in SparkSession") { |
There was a problem hiding this comment.
This test relies on other tests to create and stop spark session, which is fragile.
Can we explicitly create and stop a spark session in this test?
| val conf = new SparkConf() | ||
| .setMaster("local") | ||
| .setAppName(s"test-SPARK-32062-$i") | ||
| val context = new SparkContext(conf) |
There was a problem hiding this comment.
does this work? The test doesn't stop the spark context, and AFAIK we don't support having multiple spark context instance at the same time.
|
Test build #124389 has finished for PR 28899 at commit
|
|
Test build #124375 has finished for PR 28899 at commit
|
|
Test build #124390 has finished for PR 28899 at commit
|
|
retest this please |
|
Test build #124416 has finished for PR 28899 at commit
|
|
@ulysses-you - quick question for you (mostly for my own knowledge), do people usually destroy and re-create a spark context in the lifetime of a JVM? I was actually thinking about including this, but assumed that most people would create the singleton context and then use session instances for these types of short lives operations. |
|
@vinooganesh |
|
thanks, merging to master! |
|
@cloud-fan thanks for merging ! |
|
It's not a bug that people can hit in practice, I think it's OK to leave 3.0. |
4 participants
What changes were proposed in this pull request?
Reset listenerRegistered when application end.
Why are the changes needed?
Within a jvm, stop and create
SparkContextmulti times will cause the bug.Does this PR introduce any user-facing change?
No.
How was this patch tested?
Add UT.