Skip to content

[SPARK-23776][python][test] Check for needed components/files before running pyspark-sql tests #20909

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bersprockets wants to merge 7 commits into from
Closed

[SPARK-23776][python][test] Check for needed components/files before running pyspark-sql tests #20909

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
8a965a5
initial commit: check for test udf and hive assembly
bersprockets Mar 23, 2018
01857e1
Move component checks from pyspark.sql.tests to the modules that care
bersprockets Mar 29, 2018
4eeca66
Cleanup
bersprockets Mar 29, 2018
0f830e2
Skip HiveSparkSubmitTests if Hive not available
bersprockets Mar 30, 2018
85c3a21
If Hive not enabled, skip doctests that will fail
bersprockets Apr 3, 2018
28f0548
If test udf files not compiled, skip doctests that will fail
bersprockets Apr 3, 2018
db14acb
Clean up
bersprockets Apr 3, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions python/pyspark/sql/tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
individual modules.
"""
import os
import glob
import sys
import subprocess
import pydoc
Expand Down Expand Up @@ -84,6 +85,31 @@
from pyspark.sql.utils import AnalysisException, ParseException, IllegalArgumentException


def found_file(pattern):
SPARK_HOME = os.environ["SPARK_HOME"]
files = glob.glob(os.path.join(SPARK_HOME, pattern))
return len(files) > 0


def search_hive_assembly_jars():
Copy link
Member

@HyukjinKwon HyukjinKwon Mar 28, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quick note: I think check_hive_assembly_jars would make more sense.

pattern = "assembly/target/scala-*/jars/spark-hive_*-*.jar"
if not found_file(pattern):
raise Exception(
("Failed to find Hive assembly jar. ") +
"You need to build Spark with "
"'build/sbt -Phive package' or "
"'build/mvn -DskipTests -Phive package' before running this test.")


def search_test_udf_classes():
pattern = "sql/core/target/scala-*/test-classes/" + \
"test/org/apache/spark/sql/JavaStringLength.class"
if not found_file(pattern):
raise Exception(
("Failed to find test udf classes. ") +
"You need to build Spark with 'build/sbt sql/test:compile'")


class UTCOffsetTimezone(datetime.tzinfo):
"""
Specifies timezone in UTC offset
Expand Down Expand Up @@ -5205,6 +5231,8 @@ def test_invalid_args(self):

if __name__ == "__main__":
from pyspark.sql.tests import *
search_hive_assembly_jars()
search_test_udf_classes()
if xmlrunner:
unittest.main(testRunner=xmlrunner.XMLTestRunner(output='target/test-reports'))
else:
Expand Down