Skip to content
Closed
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -277,39 +277,38 @@ object MatrixFactorizationModel extends Loader[MatrixFactorizationModel] {
num: Int): RDD[(Int, Array[(Int, Double)])] = {
val srcBlocks = blockify(rank, srcFeatures)
val dstBlocks = blockify(rank, dstFeatures)
val ratings = srcBlocks.cartesian(dstBlocks).flatMap {
case (users, items) =>
val m = users.size
val n = math.min(items.size, num)
val ratings = srcBlocks.cartesian(dstBlocks).flatMap { case (srcIter, dstIter) =>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to more detail to the doc string comment for this method to explain the approach used for efficiency.

val m = srcIter.size
val n = math.min(dstIter.size, num)
val output = new Array[(Int, (Int, Double))](m * n)
var j = 0
users.foreach (user => {
srcIter.foreach { case (srcId, srcFactor) =>
def order(a: (Int, Double)) = a._2
val pq: BoundedPriorityQueue[(Int, Double)] =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could remove the type sig from the val definition here to make it fit on one line

new BoundedPriorityQueue[(Int, Double)](n)(Ordering.by(order))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe you can just do Ordering.by(_._2) without needing to define def order(... above

items.foreach (item => {
dstIter.foreach { case (dstId, dstFactor) =>
/**
* blas.ddot (F2jBLAS) is the same performance with the following code.
* the performace of blas.ddot with NativeBLAS is very bad.
* blas.ddot (F2jBLAS) is about 10% improvement comparing with linalg.dot.
* val rate = blas.ddot(rank, user._2, 1, item._2, 1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can perhaps say here instead "The below code is equivalent to val score = blas.ddot(rank, srcFactor, 1, dstFactor, 1)"

*/
var rate: Double = 0
var score: Double = 0
var k = 0
while(k < rank) {
rate += user._2(k) * item._2(k)
while (k < rank) {
score += srcFactor(k) * dstFactor(k)
k += 1
}
pq += ((item._1, rate))
})
pq += ((dstId, score))
}
val pqIter = pq.iterator
var i = 0
while(i < n) {
output(j + i) = (user._1, pqIter.next())
while (i < n) {
output(j + i) = (srcId, pqIter.next())
i += 1
}
j += n
})
}
output.toSeq
}
ratings.topByKey(num)(Ordering.by(_._2))
Expand Down