[SPARK-11612] [ML] Pipeline and PipelineModel persistence

[jkbradley]

Pipeline and PipelineModel extend Readable and Writable. Persistence succeeds only when all stages are Writable.

Note: This PR reinstates tests for other read/write functionality. It should probably not get merged until [https://issues.apache.org/jira/browse/SPARK-11672] gets fixed.

CC: @mengxr

[SparkQA]

Test build #45764 has finished for PR 9674 at commit 3700091.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * class Pipeline(override val uid: String) extends Estimator[PipelineModel] with Writable\n

[mengxr]

test this please

[SparkQA]

Test build #45897 has finished for PR 9674 at commit caf57c2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * class Pipeline(override val uid: String) extends Estimator[PipelineModel] with Writable\n

[mengxr]

test this please

[SparkQA]

Test build #46012 has finished for PR 9674 at commit caf57c2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * class Pipeline(override val uid: String) extends Estimator[PipelineModel] with Writable\n

[mengxr]

Should users be able to save an incomplete pipeline? For example, I could make a template pipeline, send it to other users, and they only need to fill in some required params like inputCol after they load it back.

[mengxr]

One suggestion is to merge PipelineShardWriter and PipelineSharedReader into a single object under object Pipeline, e.g., called SharedReadWrite. Then move PipelineReader, PipelineWriter to object Pipeline, and PipelineModelReader and PipelineModelWriter to object PipelineModel. The main purpose is to not pollute the package space in Java. Otherwise, they are all visible under org.apache.spark.ml in Java.

[jkbradley]

@mengxr Thanks for reviewing! I believe I addressed everything, except where I quibbled in responses above.

[mengxr]

LGTM pending Jenkins.

[SparkQA]

Test build #46026 has finished for PR 9674 at commit f791010.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * class Pipeline(override val uid: String) extends Estimator[PipelineModel] with Writable\n

[jkbradley]

@mengxr Thank you for reviewing! Merging with master and branch-1.6