Memory platform counters #548
Conversation
| @@ -35,6 +37,9 @@ class Memory extends Platform[Memory] { | |||
| private type Prod[T] = Producer[Memory, T] | |||
| private type JamfMap = Map[Prod[_], Stream[_]] | |||
|
|
|||
| def counter(group: String, name: String): Option[Long] = | |||
There was a problem hiding this comment.
should this be added to Platform? what is the contract here? When is it expected to be accurate?
There was a problem hiding this comment.
hm, I think this can work in scalding, but I am not sure about storm. The contract here is that the users create Counter objects, increment them in their job and then look at the final value after the job is done. With scalding, we can query the hadoop counters, but with storm, I don't see how it would work. Right now we look at storm counters in the viz graphs for the running topology.
|
@julienledem could you take a look? I made some modifications to the JMemory platform in summingbird-core-java |
| */ | ||
| class MemoryCounter { | ||
|
|
||
| private var count: AtomicLong = new AtomicLong() |
There was a problem hiding this comment.
this should be a private val now.
|
Thanks all for your comments, this is ready for another look. |
|
merge when green. Nice work. |
Adding summingbird counter support for the Memory platform.