[PySpark] Fix tests with Python 2.6 in 1.0 branch

[SPARK-2951] [PySpark] support unpickle array.array for Python 2.6

Pyrolite can not unpickle array.array which pickled by Python 2.6, this patch fix it by extend Pyrolite.

There is a bug in Pyrolite when unpickle array of float/double, this patch workaround it by reverse the endianness for float/double. This workaround should be removed after Pyrolite have a new release to fix this issue.

[PySpark] [SPARK-2954] [SPARK-2948] [SPARK-2910] [SPARK-2101] Python 2.6 Fixes

    - Modify python/run-tests to test with Python 2.6
    - Use unittest2 when running on Python 2.6.
    - Fix issue with namedtuple.
    - Skip TestOutputFormat.test_newhadoop on Python 2.6 until SPARK-2951 is fixed.
    - Fix MLlib _deserialize_double on Python 2.6.

[SPARK-3867][PySpark] ./python/run-tests failed when it run with Python 2.6 and unittest2 is not installed
    ./python/run-tests search a Python 2.6 executable on PATH and use it if available.
    When using Python 2.6, it is going to import unittest2 module which is not a standard library in Python 2.6, so it fails with Import

Author: cocoatomo <[email protected]>
Author: Josh Rosen <[email protected]>
Author: Davies Liu <[email protected]>
Author: Davies Liu <[email protected]>

Closes #3668 from davies/port_2365 and squashes the following commits:

b32583d [Davies Liu] rollback _common.py
bda1c72 [cocoatomo] [SPARK-3867][PySpark] ./python/run-tests failed when it run with Python 2.6 and unittest2 is not installed
14ad3d9 [Josh Rosen] [PySpark] [SPARK-2954] [SPARK-2948] [SPARK-2910] [SPARK-2101] Python 2.6 Fixes
7c55cff [Davies Liu] [SPARK-2951] [PySpark] support unpickle array.array for Python 2.6

Conflicts:
	python/pyspark/mllib/tests.py
	python/pyspark/tests.py
	python/run-tests

Author: cocoatomo

python/pyspark/tests.py

@@ -26,7 +26,16 @@
 import sys
 from tempfile import NamedTemporaryFile
 import time
-import unittest
+
+if sys.version_info[:2] <= (2, 6):
+    try:
+        import unittest2 as unittest
+    except ImportError:
+        sys.stderr.write('Please install unittest2 to test with Python 2.6 or earlier')
+        sys.exit(1)
+else:
+    import unittest
+
 
 from pyspark.context import SparkContext
 from pyspark.files import SparkFiles

python/run-tests

@@ -33,6 +33,16 @@ function run_test() {
     FAILED=$((PIPESTATUS[0]||$FAILED))
 }
 
+echo "Running PySpark tests. Output is in python/unit-tests.log."
+
+# Try to test with Python 2.6, since that's the minimum version that we support:
+if [ $(which python2.6) ]; then
+    export PYSPARK_PYTHON="python2.6"
+fi
+
+echo "Testing with Python version:"
+$PYSPARK_PYTHON --version
+
 run_test "pyspark/rdd.py"
 run_test "pyspark/context.py"
 run_test "pyspark/conf.py"