[SPARK-6485] [MLlib] [Python] Add CoordinateMatrix/RowMatrix/IndexedRowMatrix to PySpark.#7554

Closed

dusenberrymw wants to merge 41 commits intoapache:masterfrom

dusenberrymw:SPARK-6485_Add_CoordinateMatrix_RowMatrix_IndexedMatrix_to_PySpark

Commits on Jul 31, 2015

Implemented the RowMatrix API in PySpark by doing the following: Added a DistributedMatrices class to contain factory methods for creating the various distributed matrices. Added a factory method …
dusenberrymw
committed
Adding unit tests for RowMatrix methods.
dusenberrymw
committed
Updating design to have a PySpark RowMatrix simply create and keep a reference to a wrapper over a Java RowMatrix. Updating DistributedMatrices factory methods to accept numRows and numCols with d…
dusenberrymw
committed
Updating documentation to add PySpark RowMatrix. Inserting newline above doctest so that it renders properly in API docs.
dusenberrymw
committed
Implemented the IndexedRowMatrix API in PySpark, following the idea of the RowMatrix API. Note that for the IndexedRowMatrix, we use DataFrames to serialize the data between Python and Scala/Java,…
dusenberrymw
committed
Updating the architecture a bit to make conversions between the various distributed matrix types easier. The different distributed matrix classes are now only wrappers around the Java objects, and…
dusenberrymw
committed
Implemented the CoordinateMatrix API in PySpark, following the idea of the IndexedRowMatrix API, including using DataFrames for serialization.
dusenberrymw
committed
Added wrappers for the conversions between the various distributed matrices. Added logic to be able to access the rows/entries of the distributed matrices, which requires serialization through Dat…
dusenberrymw
committed
Updating unit test to be more useful.
dusenberrymw
committed
Updating documentation for each of the distributed matrices.
dusenberrymw
committed
Fixing long lines.
dusenberrymw
committed
Reformatting a few long Python doctest lines.
dusenberrymw
committed
Updating Python doctests for compatability between Python 2 & 3. Since Python 3 removed the idea of a separate 'long' type, all values that would have been outputted as a 'long' (ex: '4L') will now…
dusenberrymw
committed
Slight reformatting of the documentation.
dusenberrymw
committed
Updating pattern matching.
dusenberrymw
committed
Pulling the DistributedMatrices Scala class out of this pull request.
dusenberrymw
committed
Pulling the DistributedMatrices Python class out of this pull request.
dusenberrymw
committed
Adding type checks to the constructors. Although it is slightly verbose, it is better for the user to have a good error message than a cryptic stacktrace.
dusenberrymw
committed
Adding static methods to each of the distributed matrix classes to consolidate conversion logic.
dusenberrymw
committed
Updating documentation to use class constructors instead of the removed DistributedMatrices factory methods.
dusenberrymw
committed
Minor updates to documentation.
dusenberrymw
committed
Documenting the and parameters.
dusenberrymw
committed
Adding the colons that were accidentally not inserted.
dusenberrymw
committed
Simplifying doctests by removing duplicated rows/entries RDDs within the various tests.
dusenberrymw
committed
Moving the Experimental tag to the top of each docstring.
dusenberrymw
committed
Move conversion logic from tuples to IndexedRow or MatrixEntry types from within the IndexedRowMatrix and CoordinateMatrix constructors to separate _convert_to_indexed_row and _convert_to_matrix_en…
dusenberrymw
committed
Adding param documentation to IndexedRow and MatrixEntry.
dusenberrymw
committed
Updating code and documentation to note that a vector-like object (numpy array, list, etc.) can be used in place of explicit Vector object, and adding conversions when necessary to RowMatrix constr…
dusenberrymw
committed
Updating doctests to be less verbose by using lists instead of DenseVectors explicitly.
dusenberrymw
committed
Simplifying input conversions in the constructors for each distributed matrix.
dusenberrymw
committed
Updating docstrings.
dusenberrymw
committed
Updating doctests to make documentation cleaner.
dusenberrymw
committed
CoordinateMatrix should inherit from DistributedMatrix.
dusenberrymw
committed
Minor documentation cleanup.
dusenberrymw
committed
Using properties for better documentation.
dusenberrymw
committed
Moving the distributed matrices to pyspark.mllib.linalg.distributed.
dusenberrymw
committed

Commits on Aug 1, 2015

Improving conversion performance. This adds an optional 'java_matrix' parameter to the constructors, and pulls the conversion logic out into a '_create_from_java' function. Now, if the constructor…
dusenberrymw
committed

Commits on Aug 3, 2015

Commits on Aug 4, 2015