Support for creating a store-service for Storm platform #563
Conversation
| new CombinedServiceStoreFactory[K, V] { | ||
| def mergeableStore = mStore | ||
| def mergeableBatcher = b | ||
| def serviceStore = () => ReadableStore.empty |
There was a problem hiding this comment.
is this a sane constructor to have? left join would always come back None?
There was a problem hiding this comment.
hm, yea, we should only have the constructor with both stores defined
| @@ -25,5 +25,5 @@ import com.twitter.storehaus.ReadableStore | |||
|
|
|||
| case class ReadableServiceFactory[-K, +V]( | |||
| store: () => ReadableStore[K, V]) extends OnlineServiceFactory[K, V] { | |||
| def create = store() | |||
| def serviceStore = store | |||
There was a problem hiding this comment.
can you write a return type on this. It is ambiguous (are we applying the function or no?).
Also, all public methods should have types.
| @@ -25,5 +25,5 @@ import com.twitter.storehaus.ReadableStore | |||
|
|
|||
| case class ReadableServiceFactory[-K, +V]( | |||
| store: () => ReadableStore[K, V]) extends OnlineServiceFactory[K, V] { | |||
There was a problem hiding this comment.
I think this can just be:
case class ReadableServiceFactory[-K, +V](override val serviceStore: () => ReadableStore[K, V]) extends OnlineServiceFactory[K, V]There was a problem hiding this comment.
no love for my simplification. :) ?
There was a problem hiding this comment.
didn't see this note :) will add the simplification
|
I think this is almost ready to merge, I just want to see the use case as easy as possible to do, and come with very clear comments. I think it is easy to get confused trying to use this currently (the wiring up of the clientstore, why it is needed, etc...) |
…ds; ClientStore Monoid->Semigroup
|
@johnynek this is ready for merge |
Support for creating a store-service for Storm platform
|
Nice work here! |
To be able to do leftJoin on a store we need the same object to serve as a store and a service. This is already supported in the Scalding Platform. This adds a trait to the Storm Platform that is of both P#Store and P#Service types. This can be used in jobs that use leftJoin against store. Users in their jobs can call:
where the mergeableStore is the store for aggregation (delta store) and serviceStore is the store that we use as a service (can be delta store or the total store). For read-modify-write behavior the serviceStore should include the delta store, i.e. serviceStore can be a client store that wraps the delta mergeableStore or a client store that wraps the delta mergeableStore and the offline store (i.e. the total store).