Skip to content

[SPARK-2652] [PySpark] Turning some default configs for PySpark #1568

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
davies wants to merge 5 commits into from
Closed

[SPARK-2652] [PySpark] Turning some default configs for PySpark #1568

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c04a83d
some default configs for PySpark
davies Jul 23, 2014
8bc9f08
fix unittest
davies Jul 24, 2014
8f63f45
Merge branch 'master' into conf
davies Jul 25, 2014
f71a355
rebase to master, add spark.rdd.compress = True
davies Jul 25, 2014
cd316f1
remove duplicated line
davies Jul 26, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion python/pyspark/context.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,15 @@
from py4j.java_collections import ListConverter


# These are special default configs for PySpark, they will overwrite
# the default ones for Spark if they are not configured by user.
DEFAULT_CONFIGS = {
"spark.serializer": "org.apache.spark.serializer.KryoSerializer",
"spark.serializer.objectStreamReset": 100,
"spark.rdd.compress": True,
}


class SparkContext(object):
"""
Main entry point for Spark functionality. A SparkContext represents the
Expand Down Expand Up @@ -101,7 +110,7 @@ def __init__(self, master=None, appName=None, sparkHome=None, pyFiles=None,
else:
self.serializer = BatchedSerializer(self._unbatched_serializer,
batchSize)
self._conf.setIfMissing("spark.rdd.compress", "true")

# Set any parameters passed directly to us on the conf
if master:
self._conf.setMaster(master)
Expand All @@ -112,6 +121,8 @@ def __init__(self, master=None, appName=None, sparkHome=None, pyFiles=None,
if environment:
for key, value in environment.iteritems():
self._conf.setExecutorEnv(key, value)
for key, value in DEFAULT_CONFIGS.items():
self._conf.setIfMissing(key, value)
Copy link
Contributor

@mateiz mateiz Jul 25, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davies, you also need to remove the self._conf.setIfMissing("spark.rdd.compress", "true") line above. Otherwise it looks good.


# Check that we have at least the required parameters
if not self._conf.contains("spark.master"):
Expand Down