[SPARK-11171][SPARK-11237][SPARK-11241][ML] Try adding PMMLExportable to ML with KMeans

[holdenk]

Add PMMLExportable to ML & use it for KMeans in ML pipeline. This is a bit different than the PMMLExportable in MLLib since the default trait doesn't go through a factory but instead depends on the model implementing it itself (and at the start that will mostly just be calling the parent model's method).
Still a WIP but wanted to see if anyone had comments about the slight change.

[SparkQA]

Test build #44104 has finished for PR 9207 at commit 368879f.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * trait PMMLExportable\n

[SparkQA]

Test build #44105 has finished for PR 9207 at commit 1749aec.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * trait PMMLExportable\n

[SparkQA]

Test build #44111 has finished for PR 9207 at commit bc1b508.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * trait PMMLExportable\n

[holdenk]

cc @jkbradley

[SparkQA]

Test build #44241 has finished for PR 9207 at commit adf0b36.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * trait PMMLExportable\n

[holdenk]

re-ping @jkbradley

[holdenk]

re-ping @jkbradley

[holdenk]

Also maybe @dbtsai if you have a chance to look at this that would be great.

[holdenk]

So I've updated this against master @jkbradley

[SparkQA]

Test build #46990 has finished for PR 9207 at commit 494ecbf.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dbtsai]

Seems this is a copy-paste of org.apache.spark.mllib.pmml, should we deprecate the mllib one, and use the new one in ml package?

[holdenk]

I think that might be good, the main difference is this avoids using the factory implementation that the MLLib API was.

[holdenk]

(Uses the same public facing API as per the JIRA discussion re: lack of complaints from users with old API)

[holdenk]

ping @jkbradley

[SparkQA]

Test build #48499 has finished for PR 9207 at commit 41611b8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #49179 has finished for PR 9207 at commit 9525283.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[holdenk]

jenkins retest this please

[SparkQA]

Test build #49214 has finished for PR 9207 at commit 461c1ce.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #49819 has finished for PR 9207 at commit b514421.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[holdenk]

Jenkins retest this please

[SparkQA]

Test build #49837 has finished for PR 9207 at commit b514421.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #61683 has finished for PR 9207 at commit 49f8a8d.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #61684 has finished for PR 9207 at commit e6845f1.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #61784 has finished for PR 9207 at commit 9170b3f.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[holdenk]

HiveSubmit failures seem unrelated, jenkins retest this please.

[SparkQA]

Test build #61799 has finished for PR 9207 at commit 9170b3f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #62478 has finished for PR 9207 at commit 8579c1b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[holdenk]

Now that 2.0 RC5 has passed - maybe it would be an OK time to revisit this? I think it can also be an ok base for adding more formats if we need to.

[SparkQA]

Test build #63086 has finished for PR 9207 at commit 8103b76.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #63193 has finished for PR 9207 at commit bdcfbd1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Least(children: Seq[Expression]) extends Expression
  • case class Greatest(children: Seq[Expression]) extends Expression
  • implicit class SchemaAttribute(f: StructField)

[SparkQA]

Test build #63295 has finished for PR 9207 at commit 00173aa.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • public class ShuffleIndexInformation
  • public class ShuffleIndexRecord
  • case class CreateTable(tableDesc: CatalogTable, mode: SaveMode, query: Option[LogicalPlan])
  • case class PreprocessDDL(conf: SQLConf) extends Rule[LogicalPlan]

[holdenk]

ping @MLnick now that 2.0 is out for awhile.

[SparkQA]

Test build #65123 has finished for PR 9207 at commit 0e8c523.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[holdenk]

Ping @jkbradley / @MLnick - we are coming up on a year since we told people we would add PMML export to Spark in about a year so I figured I'd try and update this again. I'll update this PR tonight against the latest master.

[SparkQA]

Test build #68724 has finished for PR 9207 at commit 9cb8994.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[holdenk]

@MLnick: Do you have the bandwith to revisit this? I'm open to refactoring to a more plug-gable approach if we've got the review bandwidth for it.

[SparkQA]

Test build #82102 has finished for PR 9207 at commit 9cb8994.

• This patch fails Spark unit tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[holdenk]

Closing this in favor of the approach taken in #19876