Skip to content

[SPARK-2652] [PySpark] Turning some default configs for PySpark #1568

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
davies wants to merge 5 commits into from
+12 −1
Closed

[SPARK-2652] [PySpark] Turning some default configs for PySpark #1568

Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c04a83d
some default configs for PySpark
davies Jul 23, 2014
8bc9f08
fix unittest
davies Jul 24, 2014
8f63f45
Merge branch 'master' into conf
davies Jul 25, 2014
f71a355
rebase to master, add spark.rdd.compress = True
davies Jul 25, 2014
cd316f1
remove duplicated line
davies Jul 26, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions python/pyspark/context.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,14 @@
from py4j.java_collections import ListConverter


# These are special default configs for PySpark, they will overwrite
# the default ones for Spark if they are not configured by user.
DEFAULT_CONFIGS = {
"spark.serializer": "org.apache.spark.serializer.KryoSerializer",
"spark.serializer.objectStreamReset": 100,
}


class SparkContext(object):
"""
Main entry point for Spark functionality. A SparkContext represents the
Expand Down Expand Up @@ -112,6 +120,9 @@ def __init__(self, master=None, appName=None, sparkHome=None, pyFiles=None,
if environment:
for key, value in environment.iteritems():
self._conf.setExecutorEnv(key, value)
for key, value in DEFAULT_CONFIGS.items():
if self._conf.get(key) is None:
self._conf.set(key, value)

@mateiz mateiz Jul 25, 2014

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've now merged #1051, so update this to do _conf.setIfMissing().

@mateiz mateiz Jul 25, 2014

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also you may want to move the "spark.rdd.compress" that that one set into your map above


# Check that we have at least the required parameters
if not self._conf.contains("spark.master"):
Expand Down