diff --git a/docs/configuration.md b/docs/configuration.md index 7d3bbf93ae96..0b35cfeafcfb 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -190,8 +190,10 @@ of the most common options to set are: and it is up to the application to avoid exceeding the overhead memory space shared with other non-JVM processes. When PySpark is run in YARN or Kubernetes, this memory is added to executor resource requests. - - NOTE: Python memory usage may not be limited on platforms that do not support resource limiting, such as Windows. +
+ Note: This feature is dependent on Python's `resource` module; therefore, the behaviors and + limitations are inherited. For instance, Windows does not support resource limiting and actual + resource is not limited on MacOS. @@ -223,7 +225,8 @@ of the most common options to set are: stored on disk. This should be on a fast, local disk in your system. It can also be a comma-separated list of multiple directories on different disks. - NOTE: In Spark 1.0 and later this will be overridden by SPARK_LOCAL_DIRS (Standalone), MESOS_SANDBOX (Mesos) or +
+ Note: This will be overridden by SPARK_LOCAL_DIRS (Standalone), MESOS_SANDBOX (Mesos) or LOCAL_DIRS (YARN) environment variables set by the cluster manager.