Skip to content

[SPARK-24213][ML] Fix for Int id type for PowerIterationClustering in spark.ml #21274

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jkbradley wants to merge 3 commits into from
Closed

[SPARK-24213][ML] Fix for Int id type for PowerIterationClustering in spark.ml #21274

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
cd02df5
fix for Int id type for PowerIterationClustering in spark.ml
jkbradley May 8, 2018
e504c3e
cleaned up code
jkbradley May 9, 2018
3e40a92
code cleanup per review
jkbradley May 9, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion mllib/src/main/scala/org/apache/spark/ml/clustering/PowerIterationClustering.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -231,8 +231,12 @@ class PowerIterationClustering private[clustering] (
dataset.schema($(idCol)).dataType match {
case _: LongType =>
uncastPredictions
case _: IntegerType =>
uncastPredictions.withColumn($(idCol), col($(idCol)).cast(LongType))

@shahidki31 shahidki31 May 10, 2018 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't it be

case _: IntegerType => uncastPredictions.withColumn($(idCol), col($(idCol)).cast(IntegerType))

Otherwise it is not necessary for casting. right? Because prediction already has id as Long type and dataset has id as IntegerType. So, we need to cast prediction.id to IntegerType. right?
Correct me if I am wrong.

case otherType =>

@WeichenXu123 WeichenXu123 May 9, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not directly use
case intType: IntType: so that make a stronger restriction ?
Or do we need to support other types besides int/long ?

uncastPredictions.select(col($(idCol)).cast(otherType).alias($(idCol)))
throw new IllegalArgumentException(s"PowerIterationClustering had an unexpected error: " +
s"ID col was found to be of type $otherType, despite initial schema checks. Please " +

@mgaido91 mgaido91 May 9, 2018 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: ${otherType.simpleString}

s"report this bug.")
}
}

Expand Down
13 changes: 5 additions & 8 deletions mllib/src/test/scala/org/apache/spark/ml/clustering/PowerIterationClusteringSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,24 +103,21 @@ class PowerIterationClusteringSuite extends SparkFunSuite
.setK(2)
.setMaxIter(1)

def runTest(idType: DataType, neighborType: DataType, similarityType: DataType): Unit = {
def runTest(idType: DataType, similarityType: DataType): Unit = {
val typedData = data.select(
col("id").cast(idType).alias("id"),
col("neighbors").cast(ArrayType(neighborType, containsNull = false)).alias("neighbors"),
col("neighbors").cast(ArrayType(idType, containsNull = false)).alias("neighbors"),
col("similarities").cast(ArrayType(similarityType, containsNull = false))
.alias("similarities")
)
model.transform(typedData).collect()
model.transform(typedData).select("id", "prediction").collect()
}

for (idType <- Seq(IntegerType, LongType)) {
runTest(idType, LongType, DoubleType)
}
for (neighborType <- Seq(IntegerType, LongType)) {
runTest(LongType, neighborType, DoubleType)
runTest(idType, DoubleType)
}
for (similarityType <- Seq(FloatType, DoubleType)) {
runTest(LongType, LongType, similarityType)
runTest(LongType, similarityType)
}
}

Expand Down