Skip to content
Closed
Show file tree
Hide file tree
Changes from 49 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
c019474
First draft of StateStore
tdas Mar 10, 2016
4f8dade
Updated tests
tdas Mar 11, 2016
f417bde
Added basic unit test for StateStoreRDD
tdas Mar 11, 2016
d8cee54
Style fix
tdas Mar 11, 2016
7d74c67
Fixed test
tdas Mar 12, 2016
c5dd061
Fixed versioning in StateStoreRDD, and made store updates thread-safe
tdas Mar 14, 2016
7adca70
Style fixes
tdas Mar 14, 2016
a0ba498
Added docs
tdas Mar 14, 2016
48afbe6
Merge remote-tracking branch 'apache-github/master' into state-store
tdas Mar 14, 2016
bee673c
Refactored for new design
tdas Mar 14, 2016
d963efa
Merge remote-tracking branch 'apache-github/master' into HEAD
tdas Mar 14, 2016
22d7e66
Fixed a lot of things
tdas Mar 15, 2016
34ae7ff
Fixed style
tdas Mar 15, 2016
13c29a2
Fixed updates iterator
tdas Mar 15, 2016
d5e2b10
Added unit tests for Coordinator
tdas Mar 15, 2016
f5660d2
Added unit tests and fixed scala style
tdas Mar 15, 2016
d313683
Added unit test for preferred location
tdas Mar 15, 2016
7ea847c
Fixed style
tdas Mar 15, 2016
b8b4632
Added unit test for StateStore background management
tdas Mar 16, 2016
e89f4d0
Updated unit test to test instance unloading in state store
tdas Mar 16, 2016
b5e2421
Updated unit test
tdas Mar 16, 2016
76dd988
Fixed StateStoreRDD unit test
tdas Mar 16, 2016
8123818
Added docs
tdas Mar 16, 2016
2fb5b85
Minor fixes
tdas Mar 16, 2016
dee7a0e
Updated store to UnsafeRow instead of InternalRow
tdas Mar 16, 2016
15e1780
Added custom serialization for delta and added schema to state stores
tdas Mar 17, 2016
b0bd043
state-store-squashed
tdas Mar 17, 2016
e6c7016
Updates based on PR comments
tdas Mar 18, 2016
3fe34e9
Style fixes
tdas Mar 18, 2016
8cb0da8
Some more cleanup
tdas Mar 19, 2016
9fb9c43
Reverting unnecessary changes
tdas Mar 19, 2016
e6b1fb3
Fixed style
tdas Mar 19, 2016
e6f5ab8
Merge remote-tracking branch 'apache-github/master' into state-store
tdas Mar 19, 2016
2bd6cbd
Fixed logging
tdas Mar 19, 2016
25afe31
Updated serialization of snapshot files
tdas Mar 21, 2016
29c2af0
Updated StateStoreCoordinator lifecycle
tdas Mar 21, 2016
32c0139
Added StateStoreCoordinator to ContinuousQueryManager
tdas Mar 21, 2016
3824053
Updated logging
tdas Mar 21, 2016
534ad48
Minor updates
tdas Mar 21, 2016
19a60a6
Addressed comments
tdas Mar 21, 2016
5b7cf53
Added StateStoreConf to address comments
tdas Mar 22, 2016
f4f3838
Updated StateStoreConf
tdas Mar 22, 2016
502e5a5
Addressed comments
tdas Mar 22, 2016
2f29d9a
Merge remote-tracking branch 'apache-github/master' into state-store
tdas Mar 22, 2016
24fb325
Minor style fix
tdas Mar 22, 2016
4752d73
Renamed for consistency
tdas Mar 22, 2016
756762a
Updated comment
tdas Mar 22, 2016
63fad92
Replaced Timer with Scheduled Executor
tdas Mar 22, 2016
b147f59
Style fix
tdas Mar 22, 2016
819ca17
Fixed coordinator bug and added distributed test
tdas Mar 23, 2016
70cc7b1
Addressed more comments
tdas Mar 23, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ import scala.collection.mutable

import org.apache.spark.annotation.Experimental
import org.apache.spark.sql.execution.streaming.{ContinuousQueryListenerBus, Sink, StreamExecution}
import org.apache.spark.sql.execution.streaming.state.StateStoreCoordinatorRef
import org.apache.spark.sql.util.ContinuousQueryListener

/**
Expand All @@ -33,6 +34,7 @@ import org.apache.spark.sql.util.ContinuousQueryListener
@Experimental
class ContinuousQueryManager(sqlContext: SQLContext) {

private[sql] val stateStoreCoordinator = StateStoreCoordinatorRef(sqlContext.sparkContext.env)
private val listenerBus = new ContinuousQueryListenerBus(sqlContext.sparkContext.listenerBus)
private val activeQueries = new mutable.HashMap[String, ContinuousQuery]
private val activeQueriesLock = new Object
Expand Down
Loading