[SPARK-14503][ML] spark.ml API for FPGrowth by hhbyyh · Pull Request #15415 · apache/spark

hhbyyh · 2016-10-10T06:25:48Z

What changes were proposed in this pull request?

jira: https://issues.apache.org/jira/browse/SPARK-14503
Function parity: Add FPGrowth and AssociationRules to ML.

design doc: https://docs.google.com/document/d/1bVhABn5DiEj8bw0upqGMJT2L4nvO_0_cXdwu4uMT6uU/pub

Currently I make FPGrowthModel a transformer. For each association rule, it will just examine the input items against antecedents and summarize the consequents.

Update:
Thinking again, FPGrowth is only the algorithm to find the frequent itemsets, and can be replaced by other algorithms. The frequent itemsets are used by AssociationRules to generate the association rules. Then we can use the association rules to predict with other records.

For reviewers, Let's first decide if the current transform function meets your expectation.

Current options:

Current implementation: Use Estimator and Transformer pattern in ML, the transform function will examine the input items against all the association rules and summarize the consequents. Users can also access frequent items and association rules via other model members.
Keep the Estimator and Transformer pattern. But AssociationRulesModel and FPGrowthModel will have empty transform function, meaning DataFrame has no change after transform. But users can access frequent items and association rules via other model members.
(mentioned by @zhengruifeng) Keep the Estimator and Transformer pattern. But FPGrowthModel and AssociationRulesModel will just return frequent itemsets and association rules DataFrame in the transform function. Meaning the resulting DataFrame after transform will not be related to the input DataFrame.
Discard the Estimator and Transformer pattern. Both FPGrowth and FPGrowthModel will directly extend from PipelineStage, thus we don't need to have a transform function.

I'd like to hear more concrete suggestions. I would prefer option 1 or 2.

update 2:

As discussed in the jira, we will not expose AssociationRules as a public API for now.

How was this patch tested?

new unit test suites

SparkQA · 2016-10-10T07:28:18Z

Test build #66630 has finished for PR 15415 at commit 2f1a08c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-11-02T17:20:05Z

Test build #67995 has finished for PR 15415 at commit e5574be.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

hhbyyh · 2016-11-02T17:28:32Z

@yanboliang @jkbradley This should be ready for review. Thanks.

SparkQA · 2016-11-08T12:40:05Z

Test build #68337 has finished for PR 15415 at commit e5574be.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-12-17T00:34:05Z

Test build #70283 has finished for PR 15415 at commit 63eaf08.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class FPGrowth @Since(\"2.2.0\") (

zhengruifeng · 2017-01-05T05:26:35Z

mllib/src/main/scala/org/apache/spark/ml/fpm/AssociationRules.scala

+   * Param for minimum confidence, range [0.0, 1.0].
+   * @group param
+   */
+  final val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence", "min confidence")


there should be a ParamValidators.inRange(...)

zhengruifeng · 2017-01-05T05:32:13Z

mllib/src/main/scala/org/apache/spark/ml/fpm/AssociationRules.scala

+  /**
+   * Computes the association rules with confidence above [[minConfidence]].
+   * @param freqItemsets DataFrame containing frequent itemset obtained from algorithms like
+   *                     [[FPGrowth]]. Users can set itemsCol (frequent itemSet, Array[String])


Array[String] confilct with Array[Int] in https://github.com/apache/spark/pull/15415/files#diff-0a641720038f962d333ef38402a02207R41
and is there some way to support general types?

zhengruifeng · 2017-01-05T05:36:03Z

mllib/src/main/scala/org/apache/spark/ml/fpm/AssociationRules.scala

+    val freqItemSetRdd = freqItemsets.select($(itemsCol), $(freqCol)).rdd
+      .map(row => new FreqItemset(row.getSeq[String](0).toArray, row.getLong(1)))
+
+    val sqlContext = SparkSession.builder().getOrCreate()


Since val sqlContext is of type SparkSession, what about rename it spark?

zhengruifeng · 2017-01-05T05:39:43Z

mllib/src/main/scala/org/apache/spark/ml/fpm/AssociationRules.scala

+ */
+@Since("2.1.0")
+@Experimental
+class AssociationRules(override val uid: String) extends Params {


Since AssociationRules transform DataFrame freqItemsets to DataFrame rules, can it be a subclass of Transformer?

freqItemsets and rules does not have a one-to-one mapping relation and will probably violates the primitives of Transformer.

zhengruifeng · 2017-01-05T05:41:24Z

mllib/src/main/scala/org/apache/spark/ml/fpm/AssociationRules.scala

+   *
+   */
+  @Since("2.1.0")
+  def run(freqItemsets: Dataset[_]): DataFrame = {


If inheriting Transformer, here should be override def transform(dataset: Dataset[_]): DataFrame

zhengruifeng · 2017-01-05T05:45:18Z

mllib/src/main/scala/org/apache/spark/ml/fpm/AssociationRules.scala

+ * Generates association rules from frequent itemsets ("items", "freq"). This method only generates
+ * association rules which have a single item as the consequent.
+ */
+@Since("2.1.0")


should be 2.2.0

zhengruifeng · 2017-01-05T05:52:30Z

mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala

+   */
+  @Since("2.2.0")
+  val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+    "the minimal support level of the frequent pattern (Default: 0.3)")


also need a ParamValidator here

zhengruifeng · 2017-01-05T05:55:09Z

mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala

+  /** @group getParam */
+  @Since("2.2.0")
+  def getMinSupport: Double = $(minSupport)
+


MLLib's FPGrowth have a param numPartitions, will it be included here?

hhbyyh · 2017-02-20T18:30:43Z

@jkbradley Sent an update to refine the transform code and address the comments.

Regarding to the behavior changing concern, I think different partition strategy will only affect the overall efficiency and maybe the order of the frequent itemsets and association rules. As long as the result set does not change, I don't think it will disturb the users. Let me know if I miss anything.

jkbradley

Thanks for the updates!

Regarding to the behavior changing concern

I think you're right. I'd never read through the algorithm carefully, but now that I do, I think you're right.

refine the transform code

I'd prefer we hold off on this. If we're changing the transform code (rather than wrapping the old code), then we'll need to spend more time writing new tests and QAing it. We don't really have time to do that before the 2.2 release. Also, when we get around to optimizing it, I'd prefer to do everything using DataFrame operations if possible. Could you please save your code but revert the transform() code to wrap the old AssociationRules code? Thanks!

jkbradley · 2017-02-21T17:54:19Z

mllib/src/test/scala/org/apache/spark/ml/fpm/FPGrowthSuite.scala

+
+  test("FPGrowth fit and transform with different data types") {
+    Array(IntegerType, StringType, ShortType, LongType, ByteType).foreach { dt =>
+      val intData = dataset.withColumn("features", col("features").cast(ArrayType(dt)))


rename intData -> data

jkbradley · 2017-02-21T17:54:20Z

mllib/src/test/scala/org/apache/spark/ml/fpm/FPGrowthSuite.scala

+    assert(checkDF.count() == 3 && checkDF.filter(col("freq") === col("freqExp")).count() == 3)
+  }
+
+  test("FPGrowth getFreqItems with Null") {


In FPGrowth, document that null values are treated as empty sequences.

Btw, I could imagine us wanting to change this later. If we're recommending items a user could add to their basket, then we might want to suggest the most frequent item rather than nothing.

hhbyyh · 2017-02-21T18:49:43Z

Hi @jkbradley
We can hold the transform code.

wrap the old AssociationRules code

Do you mean to make transform return the Association Rules DataFrame, like the currect getAssociationRules ?

jkbradley · 2017-02-22T22:48:35Z

wrap the old AssociationRules code

Sorry, forget this comment from me; I was thinking that something like FPGrowthModel.transform had already been implemented, but it's new.

Btw, I'm adding more comments now, so please hold off on changes for an hour. Thanks!

jkbradley

OK, done with another pass. I'll think more about genericTransform too. Thanks!

jkbradley · 2017-02-22T16:25:48Z

mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala

+   * @return a DataFrame("antecedent", "consequent", "confidence") containing the association
+   *         rules.
+   */
+  def getAssociationRulesFromFP[T: ClassTag](dataset: Dataset[_],


fix style of function args

jkbradley · 2017-02-22T18:24:19Z

mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala

+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP: Parallel FP-Growth for Query


Could you please go ahead and copy the relevant text and links from the Scaladoc string for mllib.fpm.FPGrowth?

jkbradley · 2017-02-22T18:24:21Z

mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala

+   */
+  @Since("2.2.0")
+  val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+    "the minimal support level of the frequent pattern (Default: 0.3)",


"of the frequent pattern" -> "of a frequent pattern"

jkbradley · 2017-02-22T18:24:23Z

mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala

+   */
+  @Since("2.2.0")
+  val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+    "minimal confidence for generating Association Rule (Default: 0.8)",


Don't state default value in built-in Param doc

jkbradley · 2017-02-22T18:24:25Z

mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala

+  def setNumPartitions(value: Int): this.type = set(numPartitions, value)
+
+  /** @group setParam
+   *  Note that minConfidence has no effect during fitting.


Remove this line; it's already in the minConfidence doc

jkbradley · 2017-02-22T23:04:26Z

mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala

+      // Save metadata and Params
+      DefaultParamsWriter.saveMetadata(instance, path, sc)
+      val dataPath = new Path(path, "data").toString
+      instance.freqItemsets.write.save(dataPath)


specify parquet explicitly

jkbradley · 2017-02-22T23:05:09Z

mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala

+    override def load(path: String): FPGrowthModel = {
+      val metadata = DefaultParamsReader.loadMetadata(path, sc, className)
+      val dataPath = new Path(path, "data").toString
+      val frequentItems = sparkSession.read.load(dataPath)


specify parquet

jkbradley · 2017-02-22T23:06:36Z

mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala

+   *         rules.
+   */
+  def getAssociationRulesFromFP[T: ClassTag](dataset: Dataset[_],
+        itemsCol: String = "items",


No need to specify default arg values. If we make this public, we will need to avoid default args to make it Java friendly.

jkbradley · 2017-02-22T23:07:54Z

mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala

+ * :: Experimental ::
+ * Model fitted by FPGrowth.
+ *
+ * @param freqItemsets frequent items in the format of DataFrame("items", "freq")


Specify schema (Seq, Long)

jkbradley · 2017-02-22T23:11:32Z

mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala

+  private def genericFit[T: ClassTag](dataset: Dataset[_]): FPGrowthModel = {
+    val data = dataset.select($(featuresCol))
+    val items = data.where(col($(featuresCol)).isNotNull).rdd.map(r => r.getSeq[T](0).toArray)
+    val parentModel = new MLlibFPGrowth().setMinSupport($(minSupport)).run(items)


Specify numPartitions

SparkQA · 2017-02-23T18:13:51Z

Test build #73354 has finished for PR 15415 at commit d8e4884.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jkbradley

Thanks for the updates! I think this is almost done. We should try to optimize transform in the future, but I think it's OK for an MVP. It'd be interesting to write it using DataFrames instead of RDDs and benchmark (in the future).

jkbradley · 2017-02-23T19:08:55Z

mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala

+          Seq.empty
+        }
+        (id, consequents)
+    }.aggregateByKey(new ArrayBuffer[T])((ar, seq) => ar ++= seq, (ar, seq) => ar ++= seq)


How about using an OpenHashSet here to avoid collecting duplicates during aggregation?

I tried OpenHashSet and it's about %15 slower than ArrayBuffer.

.aggregateByKey(new OpenHashSet[T])((set, seq) => {
seq.foreach(t => set.add(t))
set
} , (set1, set2) => set1.union(set2))

Huh, that's surprising to me. Maybe it depends on how many duplicates are introduced. Let's leave it as is then.

Just curious: What dataset did you test it on?

random number... The duplicate number should be small.

jkbradley · 2017-02-23T19:09:46Z

mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala

+     .map { case (index, cons) => (index, cons.distinct) }
+
+    val rowAndConsequents = dataset.toDF().rdd.zipWithIndex().map(_.swap)
+      .join(indexToConsequents).sortByKey(ascending = true, dataset.rdd.getNumPartitions)


No need to sortByKey

Shall we try to keep the original order of the input dataset? The time cost is about 5% of the total transform time.

DataFrames don't really have a row ordering, so we don't need to maintain one.

If that's the case, does zipWithIndex() guarantees the same result for multiple calls?
And if can ignore the ordering, perhaps I have a faster way to write the transform.

I think zipWithIndex guarantees the order if the data have not been shuffled between load/creation and the zipWithIndex. AFAIK, shuffling can reorder rows in a partition arbitrarily.

We can definitely ignore the ordering for DataFrames.

Checked again and the current implementation is as quick. I will just remove the sort.

jkbradley · 2017-02-23T19:09:53Z

mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala

+ * independent group of mining tasks. The FP-Growth algorithm is described in
+ * <a href="http://dx.doi.org/10.1145/335191.335372">Han et al., Mining frequent patterns without
+ * candidate generation</a>.
+ *


Here or elsewhere, comment that null featuresCol values are ignored during fit() and are treated as empty sets during transform().

hhbyyh · 2017-02-23T23:22:03Z

I tried a few different ways to implement the transform. https://gist.github.com/hhbyyh/889b88ae2176d1263fdc9dd3e29d1c2d.

The performance actually are similiar, while the current one can maintain the original order of the input dataset. I would be glad to see a more optimized version.

SparkQA · 2017-02-24T02:45:22Z

Test build #73384 has finished for PR 15415 at commit bfcef4a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

hhbyyh · 2017-02-24T18:52:47Z

Hi @jkbradley After further performance comparison, I found using broadcast would give much better performance for the transform.

I tested with some public data from http://fimi.ua.ac.be/data/.
For kosarak (.gz) data (300K records), the current transform would take more than 3 hours for only 2 rules, while the broadcast version only cost 0.15 sec with 900 rules. ( I adjusted support and confidence)

  private def genericTransform4[T: Manifest](dataset: Dataset[_]): DataFrame = {
    val rules = associationRules.rdd.map(r =>
      (r.getSeq[Int](0), r.getSeq[Int](1))
    ).collect()
    val brRules = dataset.sparkSession.sparkContext.broadcast(rules)

    // For each rule, examine the input items and summarize the consequents
    val predictUDF = udf { (items: Seq[Int]) =>
      val itemset = items.toSet
      brRules.value.flatMap { r =>
        if (r._1.forall(itemset.contains)) r._2.filterNot(itemset.contains) else Seq.empty[Int]
      }.distinct
    }
    dataset.withColumn($(predictionCol), predictUDF(col($(featuresCol))))
  }

The test can be verified by the code:
https://gist.github.com/hhbyyh/06fcf3fdc8f6edda971847bcb5783d99
https://gist.github.com/hhbyyh/889b88ae2176d1263fdc9dd3e29d1c2d

Thinking again, the broadcast implementation may have much better performance in any case, especially when the rule number is large. The major issue is how to support generic with the UDF, the only way I found is to enum the datatype in a match phrase. Any suggestion?

jkbradley · 2017-02-24T21:28:25Z

I agree that, if the set of rules is small (1-2 GB max), then collecting and broadcasting it is best. But for larger sets of rules, we'd have to keep it distributed.

I'm very surprised by the time difference in your comparison. I'll experiment a little myself and get back soon.

jkbradley

Thinking about it more, I'm OK with keeping the broadcast. Later on, we could write this using DataFrames and give a broadcast hint. Can you please make the warning for transform() more distinct (maybe with a "WARNING" label) and specific about broadcasting if we're going with this option?

I got this to work:

  private def genericTransform4(dataset: Dataset[_]): DataFrame = {
    val rules: Array[(Seq[Any], Seq[Any])] = associationRules.rdd.map(r =>
      (r.getSeq(0), r.getSeq(1))
    ).collect().asInstanceOf[Array[(Seq[Any], Seq[Any])]]
    val brRules = dataset.sparkSession.sparkContext.broadcast(rules)

    val dt = dataset.schema($(featuresCol)).dataType

    // For each rule, examine the input items and summarize the consequents
    val predictUDF = udf((items: Seq[_]) => brRules.value.flatMap( rule =>
      if (items != null && rule._1.forall(item => items.contains(item))) {
        rule._2.filter(item => !items.contains(item))
      } else {
        Seq.empty
      }
    ).distinct, dt)
    dataset.withColumn($(predictionCol), predictUDF(col($(featuresCol))))
  }

Thanks!

jkbradley · 2017-02-24T22:56:22Z

mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala

+   */
+  @Since("2.2.0")
+  @transient lazy val associationRules: DataFrame = {
+    val freqItems = freqItemsets


No need for this temp val

hhbyyh · 2017-02-25T00:18:45Z

Thanks @jkbradley for contributing the code. That helps a lot. I'll merge and send an update now.

hhbyyh · 2017-02-25T01:24:15Z

@jkbradley I changed a little from your version to first convert Seq to Set, to speedup contains operations.

Btw, I could imagine us wanting to change this later. If we're recommending items a user could add to their basket, then we might want to suggest the most frequent item rather than nothing.

Do we need to support this now? That may not be expected for all the scenarios. And it seems an Imputer can help with the issue.

SparkQA · 2017-02-25T02:00:41Z

Test build #73452 has finished for PR 15415 at commit 3d7ed0b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-02-25T02:02:24Z

Test build #73453 has finished for PR 15415 at commit 9940c47.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jkbradley · 2017-02-25T02:44:04Z

I don't think we need to support the default prediction (for empty/null inputs) now. I agree we could use an inputer or add something as an option later on.

Will take a final look now

jkbradley

Looks good, just small comments

jkbradley · 2017-02-25T02:48:45Z

mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala

+      if (items != null) {
+        val itemset = items.toSet
+        brRules.value.flatMap(rule =>
+          if (items != null && rule._1.forall(item => itemset.contains(item))) {


No need to check items != null here since you put that above.

jkbradley · 2017-02-25T02:49:30Z

mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala

+    val predictUDF = udf((items: Seq[_]) => {
+      if (items != null) {
+        val itemset = items.toSet
+        brRules.value.flatMap(rule =>


style:

brRules.value.flatMap { rule => ... }

jkbradley · 2017-02-28T22:51:35Z

test this please

jkbradley · 2017-02-28T22:52:08Z

I'm going to go ahead and merge this after tests to make sure it's in 2.2, but can you please send a follow-up for my last 2 comments? Thanks!

hhbyyh · 2017-02-28T23:22:31Z

Sorry to miss your comments. I can send a follow-up together with document.

SparkQA · 2017-02-28T23:52:31Z

Test build #73616 has finished for PR 15415 at commit 9940c47.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jkbradley · 2017-02-28T23:53:15Z

No problem, thanks! Could you please create a subtask for docs?

Merging with master

YY-OnCall and others added 9 commits June 15, 2016 17:12

initial fpm

710f257

add ass rule

b8d117d

Merge remote-tracking branch 'upstream/master' into mlfpm

bc9b830

Merge remote-tracking branch 'upstream/master' into mlfpm

5764e59

Merge remote-tracking branch 'upstream/master' into mlfpm

1ca4e45

refine code

c30dd7c

Merge remote-tracking branch 'upstream/master' into mlfpm

c0b5291

Merge remote-tracking branch 'upstream/master' into mlfpm

77e1f93

add ut

2f1a08c

YY-OnCall and others added 5 commits October 10, 2016 15:41

Merge remote-tracking branch 'upstream/master' into mlfpm

388adaf

Merge remote-tracking branch 'upstream/master' into mlfpm

7eabd31

Merge remote-tracking branch 'upstream/master' into mlfpm

3730e1b

Merge remote-tracking branch 'upstream/master' into mlfpm

3a000df

refine and add unit test

e5574be

YY-OnCall added 3 commits December 16, 2016 14:39

Merge remote-tracking branch 'upstream/master' into mlfpm

ed1f91e

Merge branch 'mlfpm' of https://github.com/hhbyyh/spark into mlfpm

2afdf48

fpgrowth version

63eaf08

zhengruifeng reviewed Jan 5, 2017

View reviewed changes

Merge remote-tracking branch 'upstream/master' into mlfpm

453ae5b

jkbradley reviewed Feb 21, 2017

View reviewed changes

Merge remote-tracking branch 'upstream/master' into mlfpm

5050bd3

jkbradley reviewed Feb 22, 2017

View reviewed changes

add numpartitions

d8e4884

jkbradley reviewed Feb 23, 2017

View reviewed changes

remove sort

bfcef4a

jkbradley reviewed Feb 25, 2017

View reviewed changes

YY-OnCall added 2 commits February 24, 2017 17:00

back to broadcast

3d7ed0b

ut minor rename

9940c47

jkbradley reviewed Feb 25, 2017

View reviewed changes

asfgit closed this in 0fe8020 Feb 28, 2017

Conversation

hhbyyh commented Oct 10, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Oct 10, 2016

Uh oh!

SparkQA commented Nov 2, 2016

Uh oh!

hhbyyh commented Nov 2, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Nov 8, 2016

Uh oh!

SparkQA commented Dec 17, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhengruifeng Jan 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hhbyyh commented Feb 20, 2017

Uh oh!

jkbradley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hhbyyh commented Feb 21, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jkbradley commented Feb 22, 2017

Uh oh!

jkbradley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Feb 23, 2017

Uh oh!

jkbradley left a comment

Choose a reason for hiding this comment

hhbyyh commented Oct 10, 2016 •

edited

Loading

hhbyyh commented Nov 2, 2016 •

edited

Loading

zhengruifeng Jan 5, 2017 •

edited

Loading

hhbyyh commented Feb 21, 2017 •

edited

Loading

jkbradley Feb 23, 2017 •

edited

Loading

hhbyyh commented Feb 23, 2017 •

edited

Loading

hhbyyh commented Feb 24, 2017 •

edited

Loading