Skip to content

[SPARK-11679][SQL] Invoking method " apply(fields: java.util.List[StructField])" in "StructType" gets ClassCastException #9649

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jackieMaKing wants to merge 5 commits into from
Closed

[SPARK-11679][SQL] Invoking method " apply(fields: java.util.List[StructField])" in "StructType" gets ClassCastException #9649

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
1208354
fixed SPARK-11679
jackieMaKing Nov 12, 2015
647bf2b
using scala.collection.JavaConverters to create StructType
jackieMaKing Nov 13, 2015
c639ab4
add test case for SPARK-11679
jackieMaKing Nov 14, 2015
4496753
imporve test for SPARK-11679
jackieMaKing Nov 16, 2015
2ebeb4f
remove wildcard import and change variables name
jackieMaKing Nov 17, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -328,7 +328,8 @@ object StructType extends AbstractDataType {
def apply(fields: Seq[StructField]): StructType = StructType(fields.toArray)

def apply(fields: java.util.List[StructField]): StructType = {
Copy link
Contributor

@marmbrus marmbrus Nov 17, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related, we should just deprecate this. Its weird to have an apply for java.

Copy link
Contributor Author

@jackieMaKing jackieMaKing Nov 18, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Yes, a java friendly API is preferred.

StructType(fields.toArray.asInstanceOf[Array[StructField]])
import scala.collection.JavaConverters._
StructType(fields.asScala)
Copy link
Contributor

@yhuai yhuai Nov 17, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without this change, was there any issue when we use org.apache.spark.sql.types.DataTypes.createStructType? Or, the problem only appear when we use this apply method directly?

Copy link
Contributor

@cloud-fan cloud-fan Nov 17, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DataTypes.createStructType works well because it use fields.toArray(new StructField[0]). However, I think it's bad to create an empty StructField array every time we call that method.
You remind me that we should also update DataTypes.createStructType to use this apply method.

}

protected[sql] def fromAttributes(attributes: Seq[Attribute]): StructType =
Expand Down
13 changes: 13 additions & 0 deletions sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.ArrayList;

import scala.collection.JavaConverters;
import scala.collection.Seq;
Expand Down Expand Up @@ -209,6 +210,18 @@ public void testCreateDataFromFromList() {
Assert.assertEquals(1, result.length);
}

@Test
public void testCreateStructTypeFromList(){
List<StructField> fields1 = new ArrayList<>();
fields1.add(new StructField("id", DataTypes.StringType, true, Metadata.empty()));
StructType schema1 = StructType$.MODULE$.apply(fields1);
Assert.assertEquals(0, schema1.fieldIndex("id"));

List<StructField> fields2 = Arrays.asList(new StructField("id", DataTypes.StringType, true, Metadata.empty()));
StructType schema2 = StructType$.MODULE$.apply(fields2);
Assert.assertEquals(0, schema2.fieldIndex("id"));
}

private static final Comparator<Row> crosstabRowComparator = new Comparator<Row>() {
@Override
public int compare(Row row1, Row row2) {
Expand Down