Skip to content

Commit a4f4fbc

Browse files
ash211rxin
authored andcommitted
Include reference to twitter/chill in tuning docs
Author: Andrew Ash <[email protected]> Closes apache#647 from ash211/doc-tuning and squashes the following commits: b87de0a [Andrew Ash] Include reference to twitter/chill in tuning docs
1 parent 4d88030 commit a4f4fbc

File tree

1 file changed

+6
-3
lines changed

1 file changed

+6
-3
lines changed

docs/tuning.md

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,10 @@ This setting configures the serializer used for not only shuffling data between
4444
nodes but also when serializing RDDs to disk. The only reason Kryo is not the default is because of the custom
4545
registration requirement, but we recommend trying it in any network-intensive application.
4646

47-
Finally, to register your classes with Kryo, create a public class that extends
47+
Spark automatically includes Kryo serializers for the many commonly-used core Scala classes covered
48+
in the AllScalaRegistrar from the [Twitter chill](https://github.com/twitter/chill) library.
49+
50+
To register your own custom classes with Kryo, create a public class that extends
4851
[`org.apache.spark.serializer.KryoRegistrator`](api/core/index.html#org.apache.spark.serializer.KryoRegistrator) and set the
4952
`spark.kryo.registrator` config property to point to it, as follows:
5053

@@ -72,8 +75,8 @@ If your objects are large, you may also need to increase the `spark.kryoserializ
7275
config property. The default is 2, but this value needs to be large enough to hold the *largest*
7376
object you will serialize.
7477

75-
Finally, if you don't register your classes, Kryo will still work, but it will have to store the
76-
full class name with each object, which is wasteful.
78+
Finally, if you don't register your custom classes, Kryo will still work, but it will have to store
79+
the full class name with each object, which is wasteful.
7780

7881
# Memory Tuning
7982

0 commit comments

Comments
 (0)