Skip to content

Commit a7f400b

Browse files
committed
Added spark conf to enable cloning for threadlocal properties when streamingcontext is used
1 parent e7b77e4 commit a7f400b

File tree

1 file changed

+6
-0
lines changed

1 file changed

+6
-0
lines changed

streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -202,6 +202,12 @@ class StreamingContext private[streaming] (
202202

203203
private var shutdownHookRef: AnyRef = _
204204

205+
// The streaming scheduler and other threads started by the StreamingContext
206+
// should not inherit jobs group and job descriptions from the thread that
207+
// start the context. This configuration allows jobs group and job description
208+
// to be cleared in threads related to streaming. See SPARK-10649.
209+
sparkContext.conf.set("spark.localProperties.clone", "true")
210+
205211
conf.getOption("spark.streaming.checkpoint.directory").foreach(checkpoint)
206212

207213
/**

0 commit comments

Comments
 (0)