[SPARK-3867][PySpark] ./python/run-tests failed when it run with Python 2.6 and unittest2 is not installed

[cocoatomo]

./python/run-tests search a Python 2.6 executable on PATH and use it if available.
When using Python 2.6, it is going to import unittest2 module which is not a standard library in Python 2.6, so it fails with ImportError.

[AmplabJenkins]

Can one of the admins verify this patch?

[AmplabJenkins]

Can one of the admins verify this patch?

[JoshRosen]

Jenkins, add to whitelist. This is ok to test.

[SparkQA]

QA tests have started for PR 2759 at commit f068eb5.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2759 at commit f068eb5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21625/Test PASSed.

[mattf]

+1 lgtm

[JoshRosen]

This looks good to me. Thanks!

[scwf]

locally test even after pip install unittest2 got this error
Running test: pyspark/tests.py Please install unittest2 to test with Python 2.6 or earlier

i missed something?

[cocoatomo]

Hi, @scwf

You may use an unexpected python executable.
./python/run-tests checks whether "which python2.6" command has a return code 0, and uses $(which python2.6) as an executable to run tests.
c.f. https://github.com/apache/spark/blob/master/python/run-tests#L95

Would you like to check a result of the command "which python2.6"?

[JoshRosen]

@scwf Did pip install unittest2 to your Python 2.6 packages directory? Try running python2.6 to open a Python shell and check that you're able to import unittest2.

[scwf]

I think you are right

/usr/bin/python2.6
JackydeMacBook-Pro:~ jackylee$ python2.6
Python 2.6.9 (unknown, Sep  9 2014, 15:05:12) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

> > > import unittest2
> > > Traceback (most recent call last):
> > >   File "<stdin>", line 1, in <module>
> > > ImportError: No module named unittest2

JackydeMacBook-Pro:~ jackylee$ python
Python 2.7.6 (default, Sep  9 2014, 15:04:36) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.