[SPARK-6309] [SQL] [MLlib] Implement MatrixUDT

[MechCoder]

Utilities to serialize and deserialize Matrices in MLlib

[MechCoder]

cc @mengxr @rxin It would be great if you could have a look! Btw, should I wait for this to get merged, so that I can rebase the GaussianMixtureModel on top of this?

[SparkQA]

Test build #28658 has finished for PR 5048 at commit 18facec.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class MatrixUDT extends UserDefinedType[Matrix]

[SparkQA]

Test build #28660 has finished for PR 5048 at commit 0433fc6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class MatrixUDT extends UserDefinedType[Matrix]

[mengxr]

organize imports

[mengxr]

@MechCoder Btw, please add new commits instead of squashing commits during the code review. It makes easier to see what you've changed in each update.

[SparkQA]

Test build #28667 has finished for PR 5048 at commit 7fa7a2c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MechCoder]

@mengxr I have removed equals and now the testing is done using conversion to Arrays.

Let us please not wait for the other PR and try to get this merged (so that it does not delay the Gaussian save - load PR) Thanks!

[SparkQA]

Test build #28807 has finished for PR 5048 at commit 6e67020.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #28808 has finished for PR 5048 at commit 16d5d47.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

organize imports (put this group under breeze)

[mengxr]

LGTM except minor inline comments.

[MechCoder]

@mengxr fixed! I put the hash as my Year of Birth, so that my permanent mark is left on Spark :P

[SparkQA]

Test build #28934 has finished for PR 5048 at commit 05dc6f2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[karthikvadla]

Hi,

Can anyone please share me a example for using MatrixUDT as datatype column in Spark SQL dataframe. I'm trying to load bunch of images into dataframe. Each image as matrix in dataframe column.

Really appreciate your help. @MechCoder

Thanks
Karthik