Skip to content

[SPARK-16369][MLlib] tallSkinnyQR of RowMatrix should aware of empty partition #14049

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yinxusen wants to merge 3 commits into from
Closed

[SPARK-16369][MLlib] tallSkinnyQR of RowMatrix should aware of empty partition #14049

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
72991db
fix empty partition issue
yinxusen Jul 4, 2016
f6c25d9
change to filter map
yinxusen Jul 5, 2016
6705a38
merge with master
yinxusen Jul 7, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 19 additions & 10 deletions mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -538,20 +538,29 @@ class RowMatrix @Since("1.0.0") (
val col = numCols().toInt
// split rows horizontally into smaller matrices, and compute QR for each of them
val blockQRs = rows.glom().map { partRows =>
val bdm = BDM.zeros[Double](partRows.length, col)
var i = 0
partRows.foreach { row =>
bdm(i, ::) := row.asBreeze.t
i += 1
if (partRows.length == 0) {

@srowen srowen Jul 5, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you just flatMap this? so that you don't have to handle None below? or filter then map?

@yinxusen yinxusen Jul 5, 2016

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I changed it to filter then map.

None
} else {
val bdm = BDM.zeros[Double](partRows.length, col)
var i = 0
partRows.foreach { row =>
bdm(i, ::) := row.asBreeze.t
i += 1
}
Some(breeze.linalg.qr.reduced(bdm).r)
}
breeze.linalg.qr.reduced(bdm).r
}

// combine the R part from previous results vertically into a tall matrix
val combinedR = blockQRs.treeReduce{ (r1, r2) =>
val stackedR = BDM.vertcat(r1, r2)
breeze.linalg.qr.reduced(stackedR).r
}
val combinedR = blockQRs.treeReduce {
case (Some(r1), Some(r2)) =>
val stackedR = BDM.vertcat(r1, r2)
Some(breeze.linalg.qr.reduced(stackedR).r)
case (Some(r1), None) => Some(r1)
case (None, Some(r2)) => Some(r2)
case _ => None
}.get

val finalR = Matrices.fromBreeze(combinedR.toDenseMatrix)
val finalQ = if (computeQ) {
try {
Expand Down
17 changes: 17 additions & 0 deletions mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ import org.apache.spark.SparkFunSuite
import org.apache.spark.mllib.linalg.{Matrices, Vector, Vectors}
import org.apache.spark.mllib.random.RandomRDDs
import org.apache.spark.mllib.util.{LocalClusterSparkContext, MLlibTestSparkContext}
import org.apache.spark.mllib.util.TestingUtils._

class RowMatrixSuite extends SparkFunSuite with MLlibTestSparkContext {

Expand Down Expand Up @@ -281,6 +282,22 @@ class RowMatrixSuite extends SparkFunSuite with MLlibTestSparkContext {
assert(cov(i, j) === cov(j, i))
}
}

test("QR decomposition should aware of empty partition (SPARK-16369)") {
val mat: RowMatrix = new RowMatrix(sc.parallelize(denseData, 1))
val qrResult = mat.tallSkinnyQR(true)

val matWithEmptyPartition = new RowMatrix(sc.parallelize(denseData, 8))
val qrResult2 = matWithEmptyPartition.tallSkinnyQR(true)

assert(qrResult.Q.numCols() === qrResult2.Q.numCols(), "Q matrix ncol not match")
assert(qrResult.Q.numRows() === qrResult2.Q.numRows(), "Q matrix nrow not match")
qrResult.Q.rows.collect().zip(qrResult2.Q.rows.collect())
.foreach(x => assert(x._1 ~== x._2 relTol 1E-8, "Q matrix not match"))

qrResult.R.toArray.zip(qrResult2.R.toArray)
.foreach(x => assert(x._1 ~== x._2 relTol 1E-8, "R matrix not match"))
}
}

class RowMatrixClusterSuite extends SparkFunSuite with LocalClusterSparkContext {
Expand Down