[SPARK-28140][MLLIB][PYTHON] Accept DataFrames in RowMatrix and IndexedRowMatrix constructors

[henrydavidge]

What changes were proposed in this pull request?

In both cases, the input DataFrame schema must contain only the information that's required for the matrix object, so a vector column in the case of RowMatrix and long and vector columns for IndexedRowMatrix.

How was this patch tested?

Unit tests that verify:

• RowMatrix and IndexedRowMatrix can be created from DataFrames
• If the schema does not match expectations, we throw an IllegalArgumentException

Please review https://spark.apache.org/contributing.html before opening a pull request.

[jkbradley]

add to whitelist

[srowen]

Does this exist in Scala, even?

[srowen]

Nit: you need to revert the import changes here and above

[henrydavidge]

@srowen In Scala you would use the one liner in the implementation of createRowMatrix. The issue is that from Python this conversion isn't possible without using a Python UDF, which can blow up the execution time.

[srowen]

Likewise I think the changes in this file need to be reverted.

[henrydavidge]

🤦‍♂ oops

[henrydavidge]

Thanks for the initial look @srowen. I fixed the accidental import changes.

@jkbradley Looks like the incantation to enable tests didn't work

[SparkQA]

Test build #4815 has finished for PR 24953 at commit d620b43.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Just some import order issues:

[error] /home/jenkins/workspace/NewSparkPullRequestBuilder/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala:56:0: org.apache.spark.sql. is in wrong order relative to org.apache.spark.sql.types.LongType.
[error] /home/jenkins/workspace/NewSparkPullRequestBuilder/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala:24:21: inv should come before MatrixSingularException.
[error] /home/jenkins/workspace/NewSparkPullRequestBuilder/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala:24:21: axpy should come before SparseVector.

[SparkQA]

Test build #4818 has finished for PR 24953 at commit 4a40143.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Merged to master

[henrydavidge]

Thanks @srowen !