Skip to content

[SPARK-11679][SQL] Invoking method " apply(fields: java.util.List[StructField])" in "StructType" gets ClassCastException #9649

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jackieMaKing wants to merge 5 commits into from
Closed

[SPARK-11679][SQL] Invoking method " apply(fields: java.util.List[StructField])" in "StructType" gets ClassCastException #9649

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
1208354
fixed SPARK-11679
jackieMaKing Nov 12, 2015
647bf2b
using scala.collection.JavaConverters to create StructType
jackieMaKing Nov 13, 2015
c639ab4
add test case for SPARK-11679
jackieMaKing Nov 14, 2015
4496753
imporve test for SPARK-11679
jackieMaKing Nov 16, 2015
2ebeb4f
remove wildcard import and change variables name
jackieMaKing Nov 17, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -328,7 +328,8 @@ object StructType extends AbstractDataType {
def apply(fields: Seq[StructField]): StructType = StructType(fields.toArray)

def apply(fields: java.util.List[StructField]): StructType = {

@marmbrus marmbrus Nov 17, 2015

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related, we should just deprecate this. Its weird to have an apply for java.

@jackieMaKing jackieMaKing Nov 18, 2015

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Yes, a java friendly API is preferred.

StructType(fields.toArray.asInstanceOf[Array[StructField]])
import scala.collection.JavaConverters._
StructType(fields.asScala)

@yhuai yhuai Nov 17, 2015

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without this change, was there any issue when we use org.apache.spark.sql.types.DataTypes.createStructType? Or, the problem only appear when we use this apply method directly?

@cloud-fan cloud-fan Nov 17, 2015

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DataTypes.createStructType works well because it use fields.toArray(new StructField[0]). However, I think it's bad to create an empty StructField array every time we call that method.
You remind me that we should also update DataTypes.createStructType to use this apply method.

}

protected[sql] def fromAttributes(attributes: Seq[Attribute]): StructType =
Expand Down
6 changes: 6 additions & 0 deletions sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,12 @@ public void testCreateDataFromFromList() {
Assert.assertEquals(1, result.length);
}

@Test
public void testCreateStructTypeFromList(){
StructType schema = StructType$.MODULE$.apply(Arrays.asList(createStructField("i", IntegerType, true)));
Assert.assertEquals(0, schema.fieldIndex("i"));

@cloud-fan cloud-fan Nov 14, 2015

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried this test case locally, but can't reproduce the bug. However, the code snippet you provided in the JIRA do reproduce the bug. Can you investigate this and improve the test? Thanks!

}

private static final Comparator<Row> crosstabRowComparator = new Comparator<Row>() {
@Override
public int compare(Row row1, Row row2) {
Expand Down