[SPARK-11679][SQL] Invoking method " apply(fields: java.util.List[StructField])" in "StructType" gets ClassCastException #9649
Conversation
|
Please fix the title of this PR. Can you also not create a zero-length array each time? keep one instance; it's immutable |
jackieMaKing
changed the title
|
Call toArray(T[] a) in Java List to cast List to a specific type Array, thus I create a zero-length array. @srowen what do you mean, better to create an array with the same length of the list to pass to toArray? Like this "StructType(fields.toArray(new Array'[StructField]'(fields.size())))" |
There was a problem hiding this comment.
I think we can import scala.collection.JavaConverters._ and use StructType(fields.asScala) here.
|
As suggested by cloud-fan, I pushed the new patch. I use scala.collection.JavaConverters to create StructType. |
|
can you also add a test for it in |
|
cloud-fun, I have add a test for SPARK-11679 in JavaDataFrameSuite. |
|
ok to test |
|
Test build #45927 has finished for PR 9649 at commit
|
There was a problem hiding this comment.
I tried this test case locally, but can't reproduce the bug. However, the code snippet you provided in the JIRA do reproduce the bug. Can you investigate this and improve the test? Thanks!
|
Oops！When create List using Arrays.asList, we get a java.util.Arrays$ArrayList instead of java.util.ArrayList. The previous method works fine on java.util.Arrays$ArrayList but gets exception on java.util.ArrayList. This patch works fine on both. |
|
Test build #45990 has finished for PR 9649 at commit
|
There was a problem hiding this comment.
I generally wouldn't bother collapsing these imports. But don't change it unless you need to make other changes for this PR.
|
srowen, I have restored the original import sentences, but I imported "java.util.ArrayList". cloud-fan, I have already changed f_1 and f_2 to fields1 and field2. |
|
Test build #46042 has finished for PR 9649 at commit
|
|
LGTM, cc @yhuai |
There was a problem hiding this comment.
Without this change, was there any issue when we use org.apache.spark.sql.types.DataTypes.createStructType? Or, the problem only appear when we use this apply method directly?
There was a problem hiding this comment.
DataTypes.createStructType works well because it use fields.toArray(new StructField[0]). However, I think it's bad to create an empty StructField array every time we call that method.
You remind me that we should also update DataTypes.createStructType to use this apply method.
|
Thanks, I'm going to merge this to 1.6 and master. |
There was a problem hiding this comment.
Related, we should just deprecate this. Its weird to have an apply for java.
There was a problem hiding this comment.
Thanks. Yes, a java friendly API is preferred.
…uctField])" in "StructType" gets ClassCastException In the previous method, fields.toArray will cast java.util.List[StructField] into Array[Object] which can not cast into Array[StructField], thus when invoking this method will throw "java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to [Lorg.apache.spark.sql.types.StructField;" I directly cast java.util.List[StructField] into Array[StructField] in this patch. Author: mayuanwen <[email protected]> Closes #9649 from jackieMaKing/Spark-11679. (cherry picked from commit e8833dd) Signed-off-by: Michael Armbrust <[email protected]>
6 participants
In the previous method, fields.toArray will cast java.util.List[StructField] into Array[Object] which can not cast into Array[StructField], thus when invoking this method will throw "java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to [Lorg.apache.spark.sql.types.StructField;"
I directly cast java.util.List[StructField] into Array[StructField] in this patch.