Skip to content

[SPARK-23776][DOC] Update instructions for running PySpark after building with SBT #21628

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

[SPARK-23776][DOC] Update instructions for running PySpark after building with SBT #21628

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9fcd05d
Update build doc for running pyspark after building with sbt
bersprockets Jun 25, 2018
be281e9
Build all tests
bersprockets Jun 25, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions docs/building-spark.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -215,19 +215,23 @@ If you are building Spark for use in a Python environment and you wish to pip in

Alternatively, you can also run make-distribution with the --pip option.

## PySpark Tests with Maven
## PySpark Tests with Maven or SBT

If you are building PySpark and wish to run the PySpark tests you will need to build Spark with Hive support.

./build/mvn -DskipTests clean package -Phive
./python/run-tests

If you are building PySpark with SBT and wish to run the PySpark tests, you will need to build Spark with Hive support and also build the test components:

./build/sbt -Phive clean package
Copy link
Contributor Author

@bersprockets bersprockets Jun 25, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed that the pyspark tests were recently changed so that -Phive is no longer strictly necessary to run pyspark tests, but I decided not to address that in this update.

Copy link
Member

@HyukjinKwon HyukjinKwon Jun 25, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, I think we don't necessarily mention it now.

./build/sbt sql/test:compile
Copy link
Member

@HyukjinKwon HyukjinKwon Jun 25, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, shouldn't we better compile other tests too?

./python/run-tests

The run-tests script also can be limited to a specific Python version or a specific module

./python/run-tests --python-executables=python --modules=pyspark-sql

**Note:** You can also run Python tests with an sbt build, provided you build Spark with Hive support.

## Running R Tests

To run the SparkR tests you will need to install the [knitr](https://cran.r-project.org/package=knitr), [rmarkdown](https://cran.r-project.org/package=rmarkdown), [testthat](https://cran.r-project.org/package=testthat), [e1071](https://cran.r-project.org/package=e1071) and [survival](https://cran.r-project.org/package=survival) packages first:
Expand Down