[SPARK-9793] [MLlib] [PySpark] PySpark DenseVector, SparseVector implement __eq__ and __hash__ correctly

[yanboliang]

PySpark DenseVector, SparseVector __eq__ method should use semantics equality, and DenseVector can compared with SparseVector.
Implement PySpark DenseVector, SparseVector __hash__ method based on the first 16 entries. That will make PySpark Vector objects can be used in collections.

[SparkQA]

Test build #40766 has finished for PR 8166 at commit 1b4ed66.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yanboliang]

Jenkins, test this please.

[SparkQA]

Test build #40949 has finished for PR 8166 at commit 2a85d09.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[feynmanliang]

nit: since k1 will be at most == v1_size due to the earlier while, checking for == here will suffice and is easier to read

[feynmanliang]

ditto for k2

[yanboliang]

Actually I think checking k1 >= v1_size is more robust than k1 == v1_size, and Scala code also use the former one.

[feynmanliang]

OK, that's fine with me

[feynmanliang]

LGTM after docstring change

[SparkQA]

Test build #41666 has finished for PR 8166 at commit d63d54e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

Should it return False?

[mengxr]

@yanboliang Please update the PR to use the first 128 nonzeros entries to compute hash.

[SparkQA]

Test build #42420 has finished for PR 8166 at commit 3b8ac7a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

We can make the code more readable:

if isnan(value):
  value = float('nan')
return struct.unpack('Q', struct.pack('d', value))[0]

[SparkQA]

Test build #42465 has finished for PR 8166 at commit b58d1bb.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

LGTM. Merged into master. @yanboliang assertEquals is deprecated. Could you make a pass over existing unit tests and make a new PR that changes assertEquals to assertEqual? Thanks!

[yanboliang]

@mengxr OK, I opened SPARK-10615 to track the assertEquals to assertEqual issue. I will submit a PR in a few days.