Skip to content

[SPARK-2014] Make PySpark store RDDs in MEMORY_ONLY_SER with compression by default #1051

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ScrapCodes wants to merge 2 commits into from
Closed

[SPARK-2014] Make PySpark store RDDs in MEMORY_ONLY_SER with compression by default #1051

ScrapCodes wants to merge 2 commits into from

Conversation

@ScrapCodes
Copy link
Member

@ScrapCodes ScrapCodes commented Jun 11, 2014

No description provided.

@rxin
Copy link
Contributor

rxin commented Jun 12, 2014

Now I think about it, perhaps compression should be a setting for individual RDDs?

@ScrapCodes
Copy link
Member Author

ScrapCodes commented Jun 12, 2014

I agree !, perhaps part of Storage Levels ?

davies
davies reviewed Jul 11, 2014
View reviewed changes
python/pyspark/conf.py Outdated
Copy link
Contributor

@davies davies Jul 11, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

if key not in self:

Copy link
Member Author

@ScrapCodes ScrapCodes Jul 24, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may have to make SparkConf iterable for this to work. It does not work as is.

Copy link
Contributor

@davies davies Jul 24, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, yes. but "self.get(key) is None" sounds more Pythonic :-)

@mateiz
Copy link
Contributor

mateiz commented Jul 23, 2014

Jenkins, test this please

@SparkQA
Copy link

SparkQA commented Jul 23, 2014

QA tests have started for PR 1051. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17058/consoleFull

@SparkQA
Copy link

SparkQA commented Jul 23, 2014

QA results for PR 1051:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17058/consoleFull

@SparkQA
Copy link

SparkQA commented Jul 24, 2014

QA tests have started for PR 1051. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17104/consoleFull

@SparkQA
Copy link

SparkQA commented Jul 24, 2014

QA results for PR 1051:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17104/consoleFull

@SparkQA
Copy link

SparkQA commented Jul 24, 2014

QA tests have started for PR 1051. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17111/consoleFull

@SparkQA
Copy link

SparkQA commented Jul 24, 2014

QA results for PR 1051:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17111/consoleFull

@mateiz
Copy link
Contributor

mateiz commented Jul 25, 2014

Thanks Prashant, I've merged this.

@asfgit asfgit closed this in eff9714 Jul 25, 2014
@mateiz mateiz mentioned this pull request Jul 25, 2014
Closed
xiliu82 pushed a commit to xiliu82/spark that referenced this pull request Sep 4, 2014
@ScrapCodes @conviva-zz
[SPARK-2014] Make PySpark store RDDs in MEMORY_ONLY_SER with compress…
43e841e 
…ion by default

Author: Prashant Sharma <[email protected]>

Closes apache#1051 from ScrapCodes/SPARK-2014/pyspark-cache and squashes the following commits:

f192df7 [Prashant Sharma] Code Review
2a2f43f [Prashant Sharma] [SPARK-2014] Make PySpark store RDDs in MEMORY_ONLY_SER with compression by default
@ScrapCodes ScrapCodes deleted the SPARK-2014/pyspark-cache branch June 3, 2015 06:01
@Taco-W
Copy link

Taco-W commented Feb 2, 2017 •
edited
Loading

Hi guys,

Could anyone please explain why we don't want to use MEMORY_ONLY in PySpark by default?

Thanks a lot

wangyum pushed a commit that referenced this pull request May 26, 2023
[CARMEL-6121][Followup] Skip subsequent checks after the first author…
4cab0ab 
…ization is passed (#1051)

* [CARMEL-6121] Skip subsequent checks after the first authorization is passed

* [CARMEL-6121] Skip subsequent checks after the first authorization is passed
udaynpusa pushed a commit to mapr/spark that referenced this pull request Jan 30, 2024
@d-popkov @ekrivokonmapr
MapR [SPARK-1106] Regulate dependencies in dep-blacklist.txt via conf…
388c0de 
…igure.sh(fix/refactor) (apache#1051)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@ScrapCodes @rxin @mateiz @SparkQA @Taco-W @davies