Update documentation and add to SparkContext

Author: sryza

core/src/main/scala/org/apache/spark/SparkContext.scala

@@ -2263,6 +2263,20 @@ object SparkContext extends Logging {
    */
   def jarOfObject(obj: AnyRef): Option[String] = jarOfClass(obj.getClass)
 
+  /**
+   * :: DeveloperApi ::
+   * Estimate the number of bytes that the given object takes up on the JVM heap. The estimate
+   * includes space taken up by objects referenced by the given object, their references, and so on
+   * and so forth.
+   *
+   * This is useful for determining the amount of heap space a broadcast variable will occupy on
+   * each executor or the amount of space each object will take when caching objects in
+   * deserialized form. This is not the same as the serialized size of the object, which will
+   * typically be much smaller.
+   */
+  @DeveloperApi
+  def estimateSizeOf(obj: AnyRef): Long = SizeEstimator.estimate(obj)
+
   /**
    * Creates a modified version of a SparkConf with the parameters that can be passed separately
    * to SparkContext, to make it easier to write SparkContext's constructors. This ignores

core/src/main/scala/org/apache/spark/util/SizeEstimator.scala

@@ -36,8 +36,7 @@ import org.apache.spark.util.collection.OpenHashSet
  * Based on the following JavaWorld article:
  * http://www.javaworld.com/javaworld/javaqa/2003-12/02-qa-1226-sizeof.html
  */
-@DeveloperApi
-object SizeEstimator extends Logging {
+private[spark] object SizeEstimator extends Logging {
 
   // Sizes of primitive types
   private val BYTE_SIZE    = 1

docs/tuning.md

@@ -94,11 +94,13 @@ We will then cover tuning Spark's cache size and the Java garbage collector.
 
 ## Determining Memory Consumption
 
-The best way to size the amount of memory consumption your dataset will require is to create an RDD, put it into cache, and look at the SparkContext logs on your driver program. The logs will tell you how much memory each partition is consuming, which you can aggregate to get the total size of the RDD. You will see messages like this:
+The best way to size the amount of memory consumption a dataset will require is to create an RDD, put it
+into cache, and look at the "Storage" page in the web UI. The page will tell you how much memory the RDD
+is occupying.
 
-    INFO BlockManagerMasterActor: Added rdd_0_1 in memory on mbk.local:50311 (size: 717.5 KB, free: 332.3 MB)
-
-This means that partition 1 of RDD 0 consumed 717.5 KB.
+To estimate the memory consumption of a particular object, use the `SparkContext`'s `estimateSizeOf`
+method. This is useful for experimenting with different data layouts to trim memory usage, as well as
+determining the amount of space a broadcast variable will occupy on each executor heap.
 
 ## Tuning Data Structures