Skip to content

[SPARK-9750][MLlib] Improve equals on SparseMatrix and DenseMatrix #8042

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
feynmanliang wants to merge 6 commits into from
Closed

[SPARK-9750][MLlib] Improve equals on SparseMatrix and DenseMatrix #8042

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
6416fa0
Add failing sparse matrix equals tests
Aug 7, 2015
43d28fa
Fix failing test
Aug 7, 2015
78f9426
Add casts
Aug 7, 2015
22782df
Add equality based on matrix semantics, not representation
Aug 10, 2015
ab6f3c8
Sparse matrix compare for equals
Aug 11, 2015
bb70d5e
Breeze compare for dense matrices as well, in case other is sparse
Aug 11, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,7 +257,7 @@ class DenseMatrix(
this(numRows, numCols, values, false)

override def equals(o: Any): Boolean = o match {
case m: DenseMatrix =>
case m: Matrix =>

@jkbradley jkbradley Aug 11, 2015

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could replace toArray with toBreeze here too, in case "o" is sparse

@feynmanliang feynmanliang Aug 11, 2015

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

m.numRows == numRows && m.numCols == numCols && Arrays.equals(toArray, m.toArray)
case _ => false
}
Expand Down Expand Up @@ -519,6 +519,12 @@ class SparseMatrix(
rowIndices: Array[Int],
values: Array[Double]) = this(numRows, numCols, colPtrs, rowIndices, values, false)

override def equals(o: Any): Boolean = o match {
case m: Matrix =>
m.numRows == numRows && m.numCols == numCols && Arrays.equals(toArray, m.toArray)

@jkbradley jkbradley Aug 10, 2015

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like what you had before better. Using Breeze, I think you'd avoid converting to dense representations. With toArray, it becomes dense.

@feynmanliang feynmanliang Aug 11, 2015

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Totally forgot about that, thanks.

case _ => false
}

private[mllib] def toBreeze: BM[Double] = {
if (!isTransposed) {
new BSM[Double](values, numRows, numCols, colPtrs, rowIndices)
Expand Down
18 changes: 18 additions & 0 deletions mllib/src/test/scala/org/apache/spark/mllib/linalg/MatricesSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,24 @@ class MatricesSuite extends SparkFunSuite {
}
}

test("equals") {

@jkbradley jkbradley Aug 10, 2015

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you expand this to compare dense with sparse?

@feynmanliang feynmanliang Aug 10, 2015

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK

val dm1 = Matrices.dense(2, 2, Array(0.0, 1.0, 2.0, 3.0))
assert(dm1 === dm1)
assert(dm1 !== dm1.transpose)

val dm2 = Matrices.dense(2, 2, Array(0.0, 2.0, 1.0, 3.0))
assert(dm1 === dm2.transpose)

val sm1 = dm1.asInstanceOf[DenseMatrix].toSparse
assert(sm1 === sm1)
assert(sm1 === dm1)
assert(sm1 !== sm1.transpose)

val sm2 = dm2.asInstanceOf[DenseMatrix].toSparse
assert(sm1 === sm2.transpose)
assert(sm1 === dm2.transpose)
}

test("matrix copies are deep copies") {
val m = 3
val n = 2
Expand Down