[SPARK-6485] [MLlib] [Python] Add CoordinateMatrix/RowMatrix/IndexedRowMatrix to PySpark. #7554

dusenberrymw · 2015-07-21T02:23:11Z

This PR adds the RowMatrix, IndexedRowMatrix, and CoordinateMatrix distributed matrices to PySpark. Each distributed matrix class acts as a wrapper around the Scala/Java counterpart by maintaining a reference to the Java object. New distributed matrices can be created using factory methods added to DistributedMatrices, which creates the Java distributed matrix and then wraps it with the corresponding PySpark class. This design allows for simple conversion between the various distributed matrices, and lets us re-use the Scala code. Serialization between Python and Java is implemented using DataFrames as needed for IndexedRowMatrix and CoordinateMatrix for simplicity. Associated documentation and unit-tests have also been added. To facilitate code review, this PR implements access to the rows/entries as RDDs, the number of rows & columns, and conversions between the various distributed matrices (not including BlockMatrix), and does not implement the other linear algebra functions of the matrices, although this will be very simple to add now.

SparkQA · 2015-07-21T02:32:39Z

Test build #37893 has finished for PR 7554 at commit 06c39e2.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DistributedMatrix(object):
- class DistributedMatrices(object):
- class RowMatrix(DistributedMatrix):
- class IndexedRow(object):
- class IndexedRowMatrix(DistributedMatrix):
- class MatrixEntry(object):
- class CoordinateMatrix(object):
- trait ExpectsInputTypes extends Expression
- trait ImplicitCastInputTypes extends ExpectsInputTypes
- trait Unevaluable extends Expression
- trait Nondeterministic extends Expression
- trait CodegenFallback extends Expression
- case class Ascii(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Base64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class UnBase64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class FakeFileStatus(

SparkQA · 2015-07-21T02:50:25Z

Test build #37898 has finished for PR 7554 at commit 1b8ea74.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DistributedMatrix(object):
- class DistributedMatrices(object):
- class RowMatrix(DistributedMatrix):
- class IndexedRow(object):
- class IndexedRowMatrix(DistributedMatrix):
- class MatrixEntry(object):
- class CoordinateMatrix(object):
- trait ExpectsInputTypes extends Expression
- trait ImplicitCastInputTypes extends ExpectsInputTypes
- trait Unevaluable extends Expression
- trait Nondeterministic extends Expression
- trait CodegenFallback extends Expression
- case class FakeFileStatus(

MechCoder · 2015-07-21T05:42:22Z

I'll try to give a first pass later today.

mengxr · 2015-07-21T17:16:41Z

@dusenberrymw Thanks for working on this feature! Please fix the following:

Do not introduce methods in Python that are not available on the Scala side, e.g. DistributeMatrices. If we want to add new methods, we should add them to Scala first and then Python wrappers. If those methods are useful, we can do that in a follow-up PR.
Check code style. For Scala code, we follow https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide. For Python, we follow PEP8. I saw many long lines in your Python code. Having a more consistent code style would make code review easier.

SparkQA · 2015-07-21T21:36:22Z

Test build #37981 has finished for PR 7554 at commit 88f28c4.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DistributedMatrix(object):
- class DistributedMatrices(object):
- class RowMatrix(DistributedMatrix):
- class IndexedRow(object):
- class IndexedRowMatrix(DistributedMatrix):
- class MatrixEntry(object):
- class CoordinateMatrix(object):

dusenberrymw · 2015-07-21T22:25:24Z

@mengxr No problem, it has been enjoyable to work on! Here are some thoughts:

For local matrices and vectors, Scala has Matrices and Vectors classes, which each contain factory methods for creating the various local Matrix and Vector types (DenseVector, SparseVector, DenseMatrix, SparseMatrix). These factory methods are the recommended method for creating these matrices & vectors. On the Python side, there are also Matrices and Vectors classes with factory methods, however, rather than calling the Scala counterpart, these just mimic the behavior and create the various Matrix and Vector types directly in Python. For the distributed matrices, I thought it would be best to follow the same idea, so I added a DistributedMatrices class in Scala containing factory methods, and created the equivalent in Python. On the Python side, I think this ends up being a really clean solution, as it allows the specific types of distributed matrices (RowMatrix, IndexedRowMatrix, etc.) to simply be wrappers over their Scala/Java counterpart, similar to how the RDD and DataFrame classes act in Python. This keeps the creation logic within the factory methods, and allows for clean conversions between the distributed matrix types in Python. Really interested in your thoughts on this! I'd definitely be willing to pull that out into a separate pull request though, should that end up being the best idea.
For point number 2, yes, there were still a few long Python doctests, but I have cleaned those up now! Also, looks like I'm now having issues with the unit tests between Python2 & Python3 (2 vs 2L for example), so I need to look into that. The logic is correct, but just need to fix the output so that both Python versions output the same.

SparkQA · 2015-07-21T23:55:27Z

Test build #37991 has finished for PR 7554 at commit f0c19a0.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DistributedMatrix(object):
- class DistributedMatrices(object):
- class RowMatrix(DistributedMatrix):
- class IndexedRow(object):
- class IndexedRowMatrix(DistributedMatrix):
- class MatrixEntry(object):
- class CoordinateMatrix(object):

mengxr · 2015-07-22T01:15:27Z

@dusenberrymw We should split this PR to simplify code review. Try to keep each PR minimal. For example, it would be easier to review if this PR only creates wrappers for existing the Scala APIs. If it is required to add some Scala public API in order to make it work on the Python side, we should have one PR/JIRA for the Scala API, then the Python API. For this PR, I think it should be feasible to have the implementations without DistribbutedMatrices.

dusenberrymw · 2015-07-22T01:33:08Z

@mengxr Thanks for the thoughts! I'll trim this PR down to just the Python wrappers, and then open another JIRA up for further discussion on adding a DistributedMatrices class to Scala.

MechCoder · 2015-07-22T16:52:02Z

docs/mllib-data-types.md

Please line break this. Should not be greater than 100.

dusenberrymw · 2015-07-22T18:02:31Z

@MechCoder Great, thanks for all of the thoughts! I'll incorporate them as I work on refactoring the logic a bit.

MechCoder · 2015-07-22T18:07:51Z

python/pyspark/mllib/linalg.py

There are some rules in PEP8 that even if we limit the line length to 100 in code the docstring length has to be 72. (http://legacy.python.org/dev/peps/pep-0008/#maximum-line-length)

This isn't followed in the rest of the PySpark codebase, but I'd be glad to implement best practices here.

Sure. I realized this from the advice of @mengxr in my previous PR's :)

Yeah, thanks for sharing it! :)

MechCoder · 2015-07-22T18:51:26Z

@dusenberrymw Thanks for your work! I have made some very minor comments. Also should the structure be from pyspark.mllib.linalg.distributed import x ? instead of from pyspark.mllib.linalg import x?

dusenberrymw · 2015-07-22T19:25:22Z

Thanks again for all of the thoughts! As for the project structure, I was wondering the same thing. I know having a package with the same name as the module poses issues in Python, but I suppose we could make a similar change to that of pyspark.mllib.stat that appears to use a _statistics.py file that is imported entirely when the stat package is imported.

@mengxr What are your thoughts on this?

MechCoder · 2015-07-22T19:35:37Z

mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala

I am used to writing this in a line like

rows.map { case Row(ind: Long, vec: Vector) => IndexedRow(ind, vec) }

But that might be personal preference

mengxr · 2015-07-23T00:29:23Z

@dusenberrymw For project structure, we can have a separate PR that turns linalg.py into a folder linalg/. And then you can update this PR (but please address comments first because they become hidden if you move files).

@MechCoder Could you do the refactoring? It is similar to this commit: a3dc618.

SparkQA · 2015-07-23T01:24:05Z

Test build #38135 has finished for PR 7554 at commit bb667c4.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DistributedMatrix(object):
- class DistributedMatrices(object):
- class RowMatrix(DistributedMatrix):
- class IndexedRow(object):
- class IndexedRowMatrix(DistributedMatrix):
- class MatrixEntry(object):
- class CoordinateMatrix(object):

MechCoder · 2015-07-23T14:23:17Z

I have submitted a pull request across your branch, to remove DistributedMatrices. Would be great if we could take the discussion there (for now) . We can do a final clean-up later.

dusenberrymw · 2015-07-24T05:32:13Z

Thanks, @MechCoder! Looking forward to discussing more.

MechCoder · 2015-07-27T12:10:21Z

Alright, I made some changes to the design.

The main problem was:
RowMatrix initialization needs rows, numRows, numCols for creating a RowMatrix. However while converting from other matrices to a RowMatrix, the java model is used directly.
To prevent this, I have used a RowMatrixModel, RowMatrix being a wrapper around it, which allows a non-hackish way of initializing.
A minor drawback is that initializing a RowMatrix will give a RowMatrix instance, but converting it from a CoordinateMatrix will give an instance of RowMatrixModel, but we can live with it?

Just in case, you were not notified, this is my Pull Request dusenberrymw#1 . Could you please review it and merge?

The reason is that the code cutoff is this weekend and it would be great to get this in.

Thanks !

dusenberrymw · 2015-07-27T16:48:55Z

@MechCoder Thanks for the thoughts! I too have been working on changes for the past few days since I last pushed updates, and I have pushed those changes now. Basically, I now have RowMatrix, IndexedRowMatrix, and CoordinateMatrix classes that each take an RDD of rows (or entries for a CoordinateMatrix), optional numRows, and optional numCols. For matrix conversions, each has a _from_java function that can create a PySpark version of a given Java distributed matrix object, which aids in matrix conversions. So, if a PySpark CoordinateMatrix needs to be converted to a RowMatrix, it will first get the resulting RowMatrix Java object, then call RowMatrix._from_java(javaRowMatrix), which will then create a PySpark RowMatrix.

I think we will want to stay away from having both a RowMatrix and a RowMatrixModel, especially since a user that converts a CoordinateMatrix to a RowMatrix will be expecting a RowMatrix. The design I've pushed should keep the API simple and in line with the Scala/Java side.

Interested in any thoughts you may have still!

mengxr · 2015-08-01T09:52:04Z

python/pyspark/mllib/linalg/distributed.py

We shouldn't expose java_matrix in the public API. I suggest the following logic:

If rows is an Python RDD, convert it to DataFrame and create a Java RowMatrix object using numRows and numCols, only keep the reference to the java object.

if rows is a Java RowMatrix, keep the reference directly. (Do not document it in public API.)

+1. And adding a comment will also help for future development.

Yeah, I think that would be really clean, so long as it's documented well internally, as @MechCoder mentioned.

mengxr · 2015-08-03T18:04:11Z

@dusenberrymw Do you have time to update this PR today? Otherwise, we need to postpone it to 1.6.

…t.getOrCreate', as the later doesn't guarantee that the SparkContext will be the same as for the matrix.rows data.

dusenberrymw · 2015-08-03T23:43:22Z

@mengxr Yes, I plan to finish this PR this evening.

…ow, we allow the 'rows' parameter in the constructors to be either an RDD or the Java matrix object. If 'rows' is an RDD, we create a Java matrix object, wrap it, and then store that. If 'rows' is a Java matrix object of the correct type, we just wrap and store that directly. This is only for internal usage, and publicly, we still require 'rows' to be an RDD. We no longer store the 'rows' RDD, and instead just compute it from the Java object when needed. The point of this is that when we do matrix conversions, we do the conversion on the Scala/Java side, which returns a Java object, so we should use that directly, but exposing 'java_matrix' parameter in the public API is not ideal. This non-public feature of allowing 'rows' to be a Java matrix object is documented in the '__init__' constructor docstrings, which are not part of the generated public API, and doctests are also included.

dusenberrymw · 2015-08-04T02:19:52Z

@mengxr Alright, I've updated this PR. Now, the rows (or entries) parameter can (non-publicly) accept a Java matrix object, and we only stored a wrapped Java matrix object for each of the classes.

Also, since we should still internally document and test this for internal usage, I kept the __init__ docstrings and expanded them to explain the non-public feature of the rows (or entries) parameter, as well as to test this feature. The __init__ docstrings are not part of the publicly generated docs, so this is a great place to put this.

Thanks, and please let me know if I can update anything else! Once this is merged, I can finish up #7761.

SparkQA · 2015-08-04T04:19:29Z

Test build #39635 has finished for PR 7554 at commit bb039cb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DistributedMatrix(object):
- class RowMatrix(DistributedMatrix):
- class IndexedRow(object):
- class IndexedRowMatrix(DistributedMatrix):
- class MatrixEntry(object):
- class CoordinateMatrix(DistributedMatrix):

dusenberrymw · 2015-08-04T05:22:01Z

Jenkins, retest this please.

SparkQA · 2015-08-04T07:41:29Z

Test build #198 has finished for PR 7554 at commit bb039cb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-04T07:50:35Z

Test build #39668 has finished for PR 7554 at commit bb039cb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DistributedMatrix(object):
- class RowMatrix(DistributedMatrix):
- class IndexedRow(object):
- class IndexedRowMatrix(DistributedMatrix):
- class MatrixEntry(object):
- class CoordinateMatrix(DistributedMatrix):

dusenberrymw · 2015-08-04T18:11:53Z

@mengxr This is complete, but it looks like there is an unrelated Hive test that keeps failing. I'll keep trying to test it.

dusenberrymw · 2015-08-04T18:11:59Z

Jenkins, retest this please.

mengxr · 2015-08-04T19:31:34Z

LGTM pending Jenkins.

SparkQA · 2015-08-04T20:43:39Z

Test build #39736 has finished for PR 7554 at commit bb039cb.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DistributedMatrix(object):
- class RowMatrix(DistributedMatrix):
- class IndexedRow(object):
- class IndexedRowMatrix(DistributedMatrix):
- class MatrixEntry(object):
- class CoordinateMatrix(DistributedMatrix):

SparkQA · 2015-08-04T20:49:47Z

Test build #211 has finished for PR 7554 at commit bb039cb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DistributedMatrix(object):
- class RowMatrix(DistributedMatrix):
- class IndexedRow(object):
- class IndexedRowMatrix(DistributedMatrix):
- class MatrixEntry(object):
- class CoordinateMatrix(DistributedMatrix):

dusenberrymw · 2015-08-04T21:11:07Z

@mengxr Alright it looks like one of the Jenkins builds passed all of the tests finally.

…owMatrix to PySpark. This PR adds the RowMatrix, IndexedRowMatrix, and CoordinateMatrix distributed matrices to PySpark. Each distributed matrix class acts as a wrapper around the Scala/Java counterpart by maintaining a reference to the Java object. New distributed matrices can be created using factory methods added to DistributedMatrices, which creates the Java distributed matrix and then wraps it with the corresponding PySpark class. This design allows for simple conversion between the various distributed matrices, and lets us re-use the Scala code. Serialization between Python and Java is implemented using DataFrames as needed for IndexedRowMatrix and CoordinateMatrix for simplicity. Associated documentation and unit-tests have also been added. To facilitate code review, this PR implements access to the rows/entries as RDDs, the number of rows & columns, and conversions between the various distributed matrices (not including BlockMatrix), and does not implement the other linear algebra functions of the matrices, although this will be very simple to add now. Author: Mike Dusenberry <[email protected]> Closes #7554 from dusenberrymw/SPARK-6485_Add_CoordinateMatrix_RowMatrix_IndexedMatrix_to_PySpark and squashes the following commits: bb039cb [Mike Dusenberry] Minor documentation update. b887c18 [Mike Dusenberry] Updating the matrix conversion logic again to make it even cleaner. Now, we allow the 'rows' parameter in the constructors to be either an RDD or the Java matrix object. If 'rows' is an RDD, we create a Java matrix object, wrap it, and then store that. If 'rows' is a Java matrix object of the correct type, we just wrap and store that directly. This is only for internal usage, and publicly, we still require 'rows' to be an RDD. We no longer store the 'rows' RDD, and instead just compute it from the Java object when needed. The point of this is that when we do matrix conversions, we do the conversion on the Scala/Java side, which returns a Java object, so we should use that directly, but exposing 'java_matrix' parameter in the public API is not ideal. This non-public feature of allowing 'rows' to be a Java matrix object is documented in the '__init__' constructor docstrings, which are not part of the generated public API, and doctests are also included. 7f0dcb6 [Mike Dusenberry] Updating module docstring. cfc1be5 [Mike Dusenberry] Use 'new SQLContext(matrix.rows.sparkContext)' rather than 'SQLContext.getOrCreate', as the later doesn't guarantee that the SparkContext will be the same as for the matrix.rows data. 687e345 [Mike Dusenberry] Improving conversion performance. This adds an optional 'java_matrix' parameter to the constructors, and pulls the conversion logic out into a '_create_from_java' function. Now, if the constructors are given a valid Java distributed matrix object as 'java_matrix', they will store those internally, rather than create a new one on the Scala/Java side. 3e50b6e [Mike Dusenberry] Moving the distributed matrices to pyspark.mllib.linalg.distributed. 308f197 [Mike Dusenberry] Using properties for better documentation. 1633f86 [Mike Dusenberry] Minor documentation cleanup. f0c13a7 [Mike Dusenberry] CoordinateMatrix should inherit from DistributedMatrix. ffdd724 [Mike Dusenberry] Updating doctests to make documentation cleaner. 3fd4016 [Mike Dusenberry] Updating docstrings. 27cd5f6 [Mike Dusenberry] Simplifying input conversions in the constructors for each distributed matrix. a409cf5 [Mike Dusenberry] Updating doctests to be less verbose by using lists instead of DenseVectors explicitly. d19b0ba [Mike Dusenberry] Updating code and documentation to note that a vector-like object (numpy array, list, etc.) can be used in place of explicit Vector object, and adding conversions when necessary to RowMatrix construction. 4bd756d [Mike Dusenberry] Adding param documentation to IndexedRow and MatrixEntry. c6bded5 [Mike Dusenberry] Move conversion logic from tuples to IndexedRow or MatrixEntry types from within the IndexedRowMatrix and CoordinateMatrix constructors to separate _convert_to_indexed_row and _convert_to_matrix_entry functions. 329638b [Mike Dusenberry] Moving the Experimental tag to the top of each docstring. 0be6826 [Mike Dusenberry] Simplifying doctests by removing duplicated rows/entries RDDs within the various tests. c0900df [Mike Dusenberry] Adding the colons that were accidentally not inserted. 4ad6819 [Mike Dusenberry] Documenting the and parameters. 3b854b9 [Mike Dusenberry] Minor updates to documentation. 10046e8 [Mike Dusenberry] Updating documentation to use class constructors instead of the removed DistributedMatrices factory methods. 119018d [Mike Dusenberry] Adding static methods to each of the distributed matrix classes to consolidate conversion logic. 4d7af86 [Mike Dusenberry] Adding type checks to the constructors. Although it is slightly verbose, it is better for the user to have a good error message than a cryptic stacktrace. 93b6a3d [Mike Dusenberry] Pulling the DistributedMatrices Python class out of this pull request. f6f3c68 [Mike Dusenberry] Pulling the DistributedMatrices Scala class out of this pull request. 6a3ecb7 [Mike Dusenberry] Updating pattern matching. 08f287b [Mike Dusenberry] Slight reformatting of the documentation. a245dc0 [Mike Dusenberry] Updating Python doctests for compatability between Python 2 & 3. Since Python 3 removed the idea of a separate 'long' type, all values that would have been outputted as a 'long' (ex: '4L') will now be treated as an 'int' and outputed as one (ex: '4'). The doctests now explicitly convert to ints so that both Python 2 and 3 will have the same output. This is fine since the values are all small, and thus can be easily represented as ints. 4d3a37e [Mike Dusenberry] Reformatting a few long Python doctest lines. 7e3ca16 [Mike Dusenberry] Fixing long lines. f721ead [Mike Dusenberry] Updating documentation for each of the distributed matrices. ab0e8b6 [Mike Dusenberry] Updating unit test to be more useful. dda2f89 [Mike Dusenberry] Added wrappers for the conversions between the various distributed matrices. Added logic to be able to access the rows/entries of the distributed matrices, which requires serialization through DataFrames for IndexedRowMatrix and CoordinateMatrix types. Added unit tests. 0cd7166 [Mike Dusenberry] Implemented the CoordinateMatrix API in PySpark, following the idea of the IndexedRowMatrix API, including using DataFrames for serialization. 3c369cb [Mike Dusenberry] Updating the architecture a bit to make conversions between the various distributed matrix types easier. The different distributed matrix classes are now only wrappers around the Java objects, and take the Java object as an argument during construction. This way, we can call for example on an , which returns a reference to a Java RowMatrix object, and then construct a PySpark RowMatrix object wrapped around the Java object. This is analogous to the behavior of PySpark RDDs and DataFrames. We now delegate creation of the various distributed matrices from scratch in PySpark to the factory methods on . 4bdd09b [Mike Dusenberry] Implemented the IndexedRowMatrix API in PySpark, following the idea of the RowMatrix API. Note that for the IndexedRowMatrix, we use DataFrames to serialize the data between Python and Scala/Java, so we accept PySpark RDDs, then convert to a DataFrame, then convert back to RDDs on the Scala/Java side before constructing the IndexedRowMatrix. 23bf1ec [Mike Dusenberry] Updating documentation to add PySpark RowMatrix. Inserting newline above doctest so that it renders properly in API docs. b194623 [Mike Dusenberry] Updating design to have a PySpark RowMatrix simply create and keep a reference to a wrapper over a Java RowMatrix. Updating DistributedMatrices factory methods to accept numRows and numCols with default values. Updating PySpark DistributedMatrices factory method to simply create a PySpark RowMatrix. Adding additional doctests for numRows and numCols parameters. bc2d220 [Mike Dusenberry] Adding unit tests for RowMatrix methods. d7e316f [Mike Dusenberry] Implemented the RowMatrix API in PySpark by doing the following: Added a DistributedMatrices class to contain factory methods for creating the various distributed matrices. Added a factory method for creating a RowMatrix from an RDD of Vectors. Added a createRowMatrix function to the PythonMLlibAPI to interface with the factory method. Added DistributedMatrix, DistributedMatrices, and RowMatrix classes to the pyspark.mllib.linalg api. (cherry picked from commit 571d5b5) Signed-off-by: Xiangrui Meng <[email protected]>

mengxr · 2015-08-04T23:30:47Z

Great. Merged into master and branch-1.5. Thanks! Could you also update the block matrix PR? I'm not sure whether we have time to merge it into 1.5, but we can try.

dusenberrymw · 2015-08-04T23:36:15Z

Thanks, @mengxr! Yes, I will go finish the block matrix one as well.

mengxr This adds the `BlockMatrix` to PySpark. I have the conversions to `IndexedRowMatrix` and `CoordinateMatrix` ready as well, so once PR #7554 is completed (which relies on PR #7746), this PR can be finished. Author: Mike Dusenberry <[email protected]> Closes #7761 from dusenberrymw/SPARK-6486_Add_BlockMatrix_to_PySpark and squashes the following commits: 27195c2 [Mike Dusenberry] Adding one more check to _convert_to_matrix_block_tuple, and a few minor documentation changes. ae50883 [Mike Dusenberry] Minor update: BlockMatrix should inherit from DistributedMatrix. b8acc1c [Mike Dusenberry] Moving BlockMatrix to pyspark.mllib.linalg.distributed, updating the logic to match that of the other distributed matrices, adding conversions, and adding documentation. c014002 [Mike Dusenberry] Using properties for better documentation. 3bda6ab [Mike Dusenberry] Adding documentation. 8fb3095 [Mike Dusenberry] Small cleanup. e17af2e [Mike Dusenberry] Adding BlockMatrix to PySpark.

mengxr This adds the `BlockMatrix` to PySpark. I have the conversions to `IndexedRowMatrix` and `CoordinateMatrix` ready as well, so once PR #7554 is completed (which relies on PR #7746), this PR can be finished. Author: Mike Dusenberry <[email protected]> Closes #7761 from dusenberrymw/SPARK-6486_Add_BlockMatrix_to_PySpark and squashes the following commits: 27195c2 [Mike Dusenberry] Adding one more check to _convert_to_matrix_block_tuple, and a few minor documentation changes. ae50883 [Mike Dusenberry] Minor update: BlockMatrix should inherit from DistributedMatrix. b8acc1c [Mike Dusenberry] Moving BlockMatrix to pyspark.mllib.linalg.distributed, updating the logic to match that of the other distributed matrices, adding conversions, and adding documentation. c014002 [Mike Dusenberry] Using properties for better documentation. 3bda6ab [Mike Dusenberry] Adding documentation. 8fb3095 [Mike Dusenberry] Small cleanup. e17af2e [Mike Dusenberry] Adding BlockMatrix to PySpark. (cherry picked from commit 34dcf10) Signed-off-by: Xiangrui Meng <[email protected]>

MechCoder reviewed Jul 22, 2015
View reviewed changes

docs/mllib-data-types.md Outdated

Copy link

Contributor

MechCoder Jul 22, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please line break this. Should not be greater than 100.

MechCoder reviewed Jul 22, 2015
View reviewed changes

mengxr reviewed Aug 1, 2015
View reviewed changes

dusenberrymw added 2 commits August 3, 2015 16:34

Use 'new SQLContext(matrix.rows.sparkContext)' rather than 'SQLContex…

cfc1be5

…t.getOrCreate', as the later doesn't guarantee that the SparkContext will be the same as for the matrix.rows data.

Updating module docstring.

7f0dcb6

dusenberrymw added 2 commits August 3, 2015 18:31

Minor documentation update.

bb039cb

asfgit closed this in 571d5b5 Aug 4, 2015

[SPARK-6485] [MLlib] [Python] Add CoordinateMatrix/RowMatrix/IndexedRowMatrix to PySpark. #7554

[SPARK-6485] [MLlib] [Python] Add CoordinateMatrix/RowMatrix/IndexedRowMatrix to PySpark. #7554

Uh oh!

Conversation

dusenberrymw commented Jul 21, 2015

Uh oh!

SparkQA commented Jul 21, 2015

Uh oh!

SparkQA commented Jul 21, 2015

Uh oh!

MechCoder commented Jul 21, 2015

Uh oh!

mengxr commented Jul 21, 2015

Uh oh!

SparkQA commented Jul 21, 2015

Uh oh!

dusenberrymw commented Jul 21, 2015

Uh oh!

SparkQA commented Jul 21, 2015

Uh oh!

mengxr commented Jul 22, 2015

Uh oh!

dusenberrymw commented Jul 22, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dusenberrymw commented Jul 22, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MechCoder commented Jul 22, 2015

Uh oh!

dusenberrymw commented Jul 22, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mengxr commented Jul 23, 2015

Uh oh!

SparkQA commented Jul 23, 2015

Uh oh!

MechCoder commented Jul 23, 2015

Uh oh!

dusenberrymw commented Jul 24, 2015

Uh oh!

MechCoder commented Jul 27, 2015

Uh oh!

dusenberrymw commented Jul 27, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mengxr commented Aug 3, 2015

Uh oh!

dusenberrymw commented Aug 3, 2015

Uh oh!

dusenberrymw commented Aug 4, 2015

Uh oh!

SparkQA commented Aug 4, 2015

Uh oh!

dusenberrymw commented Aug 4, 2015

Uh oh!

SparkQA commented Aug 4, 2015

Uh oh!

SparkQA commented Aug 4, 2015

Uh oh!

dusenberrymw commented Aug 4, 2015

Uh oh!

dusenberrymw commented Aug 4, 2015

Uh oh!

mengxr commented Aug 4, 2015

Uh oh!