Skip to content

[SPARK-2014] Make PySpark store RDDs in MEMORY_ONLY_SER with compression by default #1051

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ScrapCodes wants to merge 2 commits into from
Closed

[SPARK-2014] Make PySpark store RDDs in MEMORY_ONLY_SER with compression by default #1051

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2a2f43f
[SPARK-2014] Make PySpark store RDDs in MEMORY_ONLY_SER with compress…
ScrapCodes Jun 11, 2014
f192df7
Code Review
ScrapCodes Jul 24, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions python/pyspark/conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,12 @@ def set(self, key, value):
self._jconf.set(key, unicode(value))
return self

def setIfMissing(self, key, value):
"""Set a configuration property, if not already set."""
if self.get(key) == None:
Copy link
Contributor

@davies davies Jul 11, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

if key not in self:

Copy link
Member Author

@ScrapCodes ScrapCodes Jul 24, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may have to make SparkConf iterable for this to work. It does not work as is.

Copy link
Contributor

@davies davies Jul 24, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, yes. but "self.get(key) is None" sounds more Pythonic :-)

self.set(key, value)
return self

def setMaster(self, value):
"""Set master URL to connect to."""
self._jconf.setMaster(value)
Expand Down
2 changes: 1 addition & 1 deletion python/pyspark/context.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ def __init__(self, master=None, appName=None, sparkHome=None, pyFiles=None,
else:
self.serializer = BatchedSerializer(self._unbatched_serializer,
batchSize)

self._conf.setIfMissing("spark.rdd.compress", "true")
# Set any parameters passed directly to us on the conf
if master:
self._conf.setMaster(master)
Expand Down
4 changes: 2 additions & 2 deletions python/pyspark/rdd.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,10 +194,10 @@ def context(self):

def cache(self):
"""
Persist this RDD with the default storage level (C{MEMORY_ONLY}).
Persist this RDD with the default storage level (C{MEMORY_ONLY_SER}).
"""
self.is_cached = True
self._jrdd.cache()
self.persist(StorageLevel.MEMORY_ONLY_SER)
return self

def persist(self, storageLevel):
Expand Down