Skip to content

[SPARK-30413][SQL] Avoid WrappedArray roundtrip in GenericArrayData constructor, plus related optimization in ParquetMapConverter #27088

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
JoshRosen wants to merge 6 commits into from
Closed

[SPARK-30413][SQL] Avoid WrappedArray roundtrip in GenericArrayData constructor, plus related optimization in ParquetMapConverter #27088

Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d6e478e
Revert "[SPARK-16634][SQL] Workaround JVM bug by moving some code out…
JoshRosen Jan 3, 2020
c25bd4d
Avoid wraping and unwrapping when seqOrArray is already an Array[Any]
JoshRosen Jan 3, 2020
45e2ef0
Merge remote-tracking branch 'origin/master' into joshrosen/GenericAr…
JoshRosen Jan 19, 2020
a86be1c
Add benchmark for GenericArrayData's constructors
JoshRosen Jan 19, 2020
ad93f68
Avoid indirectly calling the new GenericArrayData(Any) constructor in…
JoshRosen Jan 19, 2020
512eb53
Add JDK11 benchmark results
JoshRosen Jan 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 5 additions & 11 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/GenericArrayData.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,16 +23,6 @@ import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.types.{DataType, Decimal}
import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}

private object GenericArrayData {

// SPARK-16634: Workaround for JVM bug present in some 1.7 versions.
def anyToSeq(seqOrArray: Any): Seq[Any] = seqOrArray match {
case seq: Seq[Any] => seq
case array: Array[_] => array.toSeq
}

}

class GenericArrayData(val array: Array[Any]) extends ArrayData {

def this(seq: Seq[Any]) = this(seq.toArray)
Expand All @@ -47,7 +37,11 @@ class GenericArrayData(val array: Array[Any]) extends ArrayData {
def this(primitiveArray: Array[Byte]) = this(primitiveArray.toSeq)
def this(primitiveArray: Array[Boolean]) = this(primitiveArray.toSeq)

def this(seqOrArray: Any) = this(GenericArrayData.anyToSeq(seqOrArray))
def this(seqOrArray: Any) = this(seqOrArray match {
case seq: Seq[Any] => seq.toArray
case array: Array[Any] => array // array of objects, so no need to convert
case array: Array[_] => array.toSeq.toArray[Any] // array of primitives, so box them
})

override def copy(): ArrayData = {
val newValues = new Array[Any](array.length)
Expand Down