Skip to content

Commit 2c6a51e

Browse files
navisrxin
authored andcommitted
[SPARK-10684] [SQL] StructType.interpretedOrdering need not to be serialized
Kryo fails with buffer overflow even with max value (2G). {noformat} org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 1 Serialization trace: containsChild (org.apache.spark.sql.catalyst.expressions.BoundReference) child (org.apache.spark.sql.catalyst.expressions.SortOrder) array (scala.collection.mutable.ArraySeq) ordering (org.apache.spark.sql.catalyst.expressions.InterpretedOrdering) interpretedOrdering (org.apache.spark.sql.types.StructType) schema (org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema). To avoid this, increase spark.kryoserializer.buffer.max value. at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:263) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:240) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) {noformat} Author: navis.ryu <[email protected]> Closes #8808 from navis/SPARK-10684. (cherry picked from commit e3b5d6c) Signed-off-by: Reynold Xin <[email protected]>
1 parent f97db94 commit 2c6a51e

File tree

1 file changed

+3
-1
lines changed

1 file changed

+3
-1
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -305,7 +305,9 @@ case class StructType(fields: Array[StructField]) extends DataType with Seq[Stru
305305
f(this) || fields.exists(field => field.dataType.existsRecursively(f))
306306
}
307307

308-
private[sql] val interpretedOrdering = InterpretedOrdering.forSchema(this.fields.map(_.dataType))
308+
@transient
309+
private[sql] lazy val interpretedOrdering =
310+
InterpretedOrdering.forSchema(this.fields.map(_.dataType))
309311
}
310312

311313
object StructType extends AbstractDataType {

0 commit comments

Comments
 (0)