[SPARK-19064][PySpark]Fix pip installing of sub components #16465
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…stribution scripts to be more explicit about cleanup. Also add pypandoc to dev requirements file since we want it for publishing
|
Test build #70848 has finished for PR 16465 at commit
|
Contributor
Author
|
cc @JoshRosen who reviewed the original PR and is probably the most familiar. |
viirya
reviewed
Jan 5, 2017
|
|
||
| from pyspark.sql import SparkSession | ||
| from pyspark.ml.param import Params | ||
| from pyspark.mllib.linalg import * |
Member
There was a problem hiding this comment.
Is it better to import pyspark.ml.linalg or both?
Contributor
Author
There was a problem hiding this comment.
This just checks one sub component from each, we could import each with rename I suppose but not sure it would do much?
Member
There was a problem hiding this comment.
ok. i think this should be enough.
Contributor
Author
|
Gentle ping for @JoshRosen to @davies maybe? |
Contributor
Author
|
Small follow up ping for @JoshRosen to @davies maybe? |
srowen
approved these changes
Jan 24, 2017
Member
srowen
left a comment
srowen
left a comment
There was a problem hiding this comment.
I'd say you can merge it yourself !
Contributor
Author
|
Sounds like a plan, I think should should probably be on the 2.1 branch as well so I'll go bug someone who has done backports to make sure I do that part right :) |
Member
|
Holden, https://github.com/apache/spark/blob/master/dev/merge_spark_pr.py works well for merging to master and back port to any branch :) unless there is conflict then it would be easier with a separate PR.
Have fun!
|
asfgit
pushed a commit
that referenced
this pull request
Jan 25, 2017
## What changes were proposed in this pull request? Fix instalation of mllib and ml sub components, and more eagerly cleanup cache files during test script & make-distribution. ## How was this patch tested? Updated sanity test script to import mllib and ml sub-components. Author: Holden Karau <[email protected]> Closes #16465 from holdenk/SPARK-19064-fix-pip-install-sub-components. (cherry picked from commit 965c82d) Signed-off-by: Holden Karau <[email protected]>
uzadude
pushed a commit
to uzadude/spark
that referenced
this pull request
Jan 27, 2017
## What changes were proposed in this pull request? Fix instalation of mllib and ml sub components, and more eagerly cleanup cache files during test script & make-distribution. ## How was this patch tested? Updated sanity test script to import mllib and ml sub-components. Author: Holden Karau <[email protected]> Closes apache#16465 from holdenk/SPARK-19064-fix-pip-install-sub-components.
Contributor
Author
|
Merged into master & branch-2.1 |
cmonkey
pushed a commit
to cmonkey/spark
that referenced
this pull request
Feb 15, 2017
## What changes were proposed in this pull request? Fix instalation of mllib and ml sub components, and more eagerly cleanup cache files during test script & make-distribution. ## How was this patch tested? Updated sanity test script to import mllib and ml sub-components. Author: Holden Karau <[email protected]> Closes apache#16465 from holdenk/SPARK-19064-fix-pip-install-sub-components.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Fix instalation of mllib and ml sub components, and more eagerly cleanup cache files during test script & make-distribution.
How was this patch tested?
Updated sanity test script to import mllib and ml sub-components.