Skip to content

[SPARK-22811][pyspark][ml] Fix pyspark.ml.tests failure when Hive is not available. #19997

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MrBago wants to merge 1 commit into from
Closed

[SPARK-22811][pyspark][ml] Fix pyspark.ml.tests failure when Hive is not available. #19997

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions python/pyspark/ml/tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@
import numpy as np
from numpy import abs, all, arange, array, array_equal, inf, ones, tile, zeros
import inspect
import py4j
Copy link
Member

@HyukjinKwon HyukjinKwon Dec 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, mind elaborating how importing this fixes an issue? It sounds orthogonal to me.

Copy link
Contributor Author

@MrBago MrBago Dec 16, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the line below, we catch py4j.protocol.Py4JError so that we can then raise SkipTest instead, but if we don't import py4j we get a NameError instead of skipping the test. Furthermore, because we don't trigger tearDownClass() on the following line we leave behind stale state which causes other tests to fail. The except line is only ever triggered in environments that don't have Hive and should skip this test.

https://github.com/apache/spark/pull/19997/files#diff-4a75aace12688903bc8f97e7930622f4R1868

Copy link
Member

@HyukjinKwon HyukjinKwon Dec 16, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, it was my bad. Yup, you are right.

It was also written in the JIRA as well. Sorry, I just got up (I am in Korea :) ..) and rushed to leave some comments.


from pyspark import keyword_only, SparkContext
from pyspark.ml import Estimator, Model, Pipeline, PipelineModel, Transformer, UnaryTransformer
Expand Down Expand Up @@ -1859,8 +1860,9 @@ class ImageReaderTest2(PySparkTestCase):

@classmethod
def setUpClass(cls):
PySparkTestCase.setUpClass()
super(ImageReaderTest2, cls).setUpClass()
# Note that here we enable Hive's support.
cls.spark = None
try:
cls.sc._jvm.org.apache.hadoop.hive.conf.HiveConf()
except py4j.protocol.Py4JError:
Expand All @@ -1873,8 +1875,10 @@ def setUpClass(cls):

@classmethod
def tearDownClass(cls):
PySparkTestCase.tearDownClass()
cls.spark.sparkSession.stop()
super(ImageReaderTest2, cls).tearDownClass()
if cls.spark is not None:
cls.spark.sparkSession.stop()
cls.spark = None

def test_read_images_multiple_times(self):
# This test case is to check if `ImageSchema.readImages` tries to
Expand Down