[SPARK-29352][SQL][SS] Track active streaming queries in the SparkSession.sharedState #26018

brkyvz · 2019-10-04T00:03:46Z

What changes were proposed in this pull request?

This moves the tracking of active queries from a per SparkSession state, to the shared SparkSession for better safety in isolated Spark Session environments.

Why are the changes needed?

We have checks to prevent the restarting of the same stream on the same spark session, but we can actually make that better in multi-tenant environments by actually putting that state in the SharedState instead of SessionState. This would allow a more comprehensive check for multi-tenant clusters.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Added tests to StreamingQueryManagerSuite

brkyvz · 2019-10-04T00:03:58Z

cc @zsxwing

SparkQA · 2019-10-04T00:11:07Z

Test build #111760 has finished for PR 26018 at commit 08be288.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-10-04T17:24:20Z

Test build #111783 has started for PR 26018 at commit b95474a.

… into sharedStreamingQueryManager

HeartSaVioR · 2019-10-05T00:54:37Z

sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryManagerSuite.scala

+    }
+  }
+
+  testQuietly(


Just a sake of understanding, this patch is intended to prevent starting multiple instances of the same streaming query in the different sessions (while it was allowed to do this and it would probably incur some problem), right?

yes. This is the specific test which would have previously failed.

Technically, this test case doesn't fail without this patch because this is for single session test case. The above test case (line 276) fails without this patch correctly.

HeartSaVioR

LGTM. It would keep isolation of active queries between SparkSession and only add restriction of running same streaming query concurrently across multiple SparkSessions.

SparkQA · 2019-10-05T01:48:44Z

Test build #111791 has finished for PR 26018 at commit 0ddb7e5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-10-05T01:55:01Z

Test build #111790 has finished for PR 26018 at commit a91f017.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing

LGTM

HeartSaVioR · 2019-10-11T13:44:05Z

sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala

      val activeOption =
        Option(sparkSession.sharedState.activeStreamingQueries.putIfAbsent(query.id, this))
-      if (activeOption.isDefined) {
+      if (activeOption.isDefined || activeQueries.values.exists(_.id == query.id) {


Just curious: are you seeing the actual case where activeOption is None but activeQueries contain such query? I'm not seeing the case, though I don't think adding this would hurt.

dongjoon-hyun · 2019-10-12T23:05:42Z

Retest this please.

SparkQA · 2019-10-12T23:11:49Z

Test build #111984 has finished for PR 26018 at commit b808d33.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-10-13T00:31:10Z

sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala

+      // Make sure no other query with same id is active across all sessions
+      val activeOption =
+        Option(sparkSession.sharedState.activeStreamingQueries.putIfAbsent(query.id, this))
+      if (activeOption.isDefined || activeQueries.values.exists(_.id == query.id) {


It seems that one ) is missed after query.id). Style check and compilation failure.

[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala: Expected token RPAREN but got Token(DOT,.,14050,.) [error] Total time: 20 s, completed Oct 12, 2019 4:11:49 PM

SparkQA · 2019-10-13T14:59:25Z

Test build #111996 has finished for PR 26018 at commit d275ec6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-10-15T02:40:45Z

Retest this please.

dongjoon-hyun · 2019-10-15T03:16:01Z

sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryManagerSuite.scala

+    query2.stop()
+  }
+
+  testQuietly("can't start multiple instances of the same streaming query in the same session") {


The test case names seems to be switched, @brkyvz .
in the same session -> in the different sessions?

dongjoon-hyun · 2019-10-15T03:17:54Z

sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryManagerSuite.scala

+  }
+
+  testQuietly(
+    "can't start multiple instances of the same streaming query in the different sessions") {


According to the test body, in the different sessions -> in the same session?

good catch. I named them wrong :)

dongjoon-hyun

+1, LGTM (except the switched test case names). Thank you for adding more missed test cases, @brkyvz .

SparkQA · 2019-10-15T06:17:29Z

Test build #112083 has finished for PR 26018 at commit d275ec6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-10-22T13:27:30Z

Test build #112455 has finished for PR 26018 at commit 37b43ec.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR

LGTM again.

brkyvz · 2019-10-23T08:55:39Z

Merging to master! Thanks all :)

Track active streaming queries in the SparkSession.sharedState

08be288

brkyvz changed the title ~~Track active streaming queries in the SparkSession.sharedState~~ [SPARK-29352][SQL][SS] Track active streaming queries in the SparkSession.sharedState Oct 4, 2019

Update SharedState.scala

b95474a

brkyvz added 3 commits October 4, 2019 13:41

Add tests

b5eb762

Merge branch 'sharedStreamingQueryManager' of github.com:brkyvz/spark…

a91f017

… into sharedStreamingQueryManager

add docs

0ddb7e5

dongjoon-hyun added the SQL label Oct 4, 2019

HeartSaVioR reviewed Oct 5, 2019

View reviewed changes

HeartSaVioR approved these changes Oct 5, 2019

View reviewed changes

Update StreamingQueryManager.scala

b808d33

zsxwing approved these changes Oct 10, 2019

View reviewed changes

HeartSaVioR reviewed Oct 11, 2019

View reviewed changes

dongjoon-hyun reviewed Oct 13, 2019

View reviewed changes

Update StreamingQueryManager.scala

d275ec6

dongjoon-hyun reviewed Oct 15, 2019

View reviewed changes

dongjoon-hyun approved these changes Oct 15, 2019

View reviewed changes

Update StreamingQueryManagerSuite.scala

37b43ec

HeartSaVioR approved these changes Oct 22, 2019

View reviewed changes

asfgit closed this in cbe6ead Oct 23, 2019

[SPARK-29352][SQL][SS] Track active streaming queries in the SparkSession.sharedState #26018

[SPARK-29352][SQL][SS] Track active streaming queries in the SparkSession.sharedState #26018

Uh oh!

Conversation

brkyvz commented Oct 4, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

brkyvz commented Oct 4, 2019

Uh oh!

SparkQA commented Oct 4, 2019

Uh oh!

SparkQA commented Oct 4, 2019

Uh oh!

HeartSaVioR Oct 5, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brkyvz Oct 7, 2019

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Oct 15, 2019

Choose a reason for hiding this comment

Uh oh!

HeartSaVioR left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 5, 2019

Uh oh!

SparkQA commented Oct 5, 2019

Uh oh!

zsxwing left a comment

Choose a reason for hiding this comment

Uh oh!

HeartSaVioR Oct 11, 2019

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Oct 12, 2019

Uh oh!

SparkQA commented Oct 12, 2019

Uh oh!

dongjoon-hyun Oct 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 13, 2019

Uh oh!

dongjoon-hyun commented Oct 15, 2019

Uh oh!

dongjoon-hyun Oct 15, 2019

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Oct 15, 2019

Choose a reason for hiding this comment

Uh oh!

brkyvz Oct 22, 2019

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 15, 2019

Uh oh!

SparkQA commented Oct 22, 2019

Uh oh!

HeartSaVioR left a comment

Choose a reason for hiding this comment

Uh oh!

brkyvz commented Oct 23, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

brkyvz commented Oct 4, 2019 •

edited

Loading

HeartSaVioR Oct 5, 2019 •

edited

Loading

dongjoon-hyun Oct 13, 2019 •

edited

Loading