[SPARK-14906][ML] Move VectorUDT and MatrixUDT in PySpark to new ML package

[viirya]

What changes were proposed in this pull request?

Move the VectorUDT and MatrixUDT in PySpark to new ML package.

How was this patch tested?

Existing tests.

[SparkQA]

Test build #57642 has finished for PR 12870 at commit 24ec53e.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jkbradley]

@viirya Thanks for this PR. I'll make a few high-level comments.

We need to keep the old pyspark.mllib.linalg APIs. We also want to freeze the pyspark.mllib.linalg APIs, which means we need to keep that old code (not just alias it).

Can you please modify the PR to do the following:

• Keep the pyspark.mllib.linalg code
• Copy the linalg code to pyspark.ml.linalg, including unit tests
• Update the pyspark.ml.linalg code as needed to handle UDTs and make tests run

Thanks!

[SparkQA]

Test build #57723 has finished for PR 12870 at commit b504a60.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class VectorUDT(UserDefinedType):
  • class MatrixUDT(UserDefinedType):
  • class Vector(object):
  • class DenseVector(Vector):
  • class SparseVector(Vector):
  • class Vectors(object):
  • class Matrix(object):
  • class DenseMatrix(Matrix):
  • class SparseMatrix(Matrix):
  • class Matrices(object):
  • class QRDecomposition(object):

[SparkQA]

Test build #57725 has finished for PR 12870 at commit 7314444.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

@jkbradley Thanks for your comments. I want to make sure that you said we will freeze the pyspark.mllib.linalg APIs, so do I need to change the current PySpark ml/mllib codes which use pyspark.mllib.linalg APIs to use new pyspark.ml.linalg?

I just did that. So if you mean we don't touch that, I need to revert it back. Thanks.

[SparkQA]

Test build #57728 has finished for PR 12870 at commit 7df46dc.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

ping @jkbradley @dbtsai @mengxr

[SparkQA]

Test build #58489 has started for PR 12870 at commit 33afa2e.

[shaneknapp]

i will retrigger this build once maintenance is over.

[shaneknapp]

jenkins, test this please

[shaneknapp]

jenkins, test this please

[mengxr]

@viirya Sorry for late response! The changes should be similar to mllib-local. The scope is not moving UDTs but copying the entire linalg package to spark.ml. We should leave spark.mllib unchanged and turn it to maintenance mode in 2.0. Do you have time to make this change? Or I can send a PR based on yours. Thanks!

[SparkQA]

Test build #58500 has finished for PR 12870 at commit 33afa2e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

@mengxr Currently I copyed only Vector/Matrix and their UDTs to pyspark.ml.linalg and made pyspark.ml and pyspark.mllib codes to use moved pyspark.ml.linalg Vector/Matrix. Besides, pyspark.mllib.linalg is untouched.

Do you mean that we want to keep pyspark.mllib codes using pyspark.mllib.linalg and let pyspark.ml codes using new pyspark.ml.linalg?

[viirya]

@mengxr @dbtsai Separating the pickle paths for pyspark.ml and pyspark.mllib is not possible without the changes in #12627, as it requires the Scala ML code changes as well.

I will submit another PR for this jira that only copies pyspark.mllib.linalg to pyspark.ml.linalg. When that PR is merged, I will apply the changes of python codes in this PR against #12627 by submitting a PR for it. Then I think it should solve the problem.

[viirya]

@mengxr @dbtsai That PR is at #13099.

[dbtsai]

Ping @mengxr