Use Murmur3 hash function as default, remove hashobj parameter.

[ekzhu]

@aastafiev @ae-foster @vmarkovtsev

I had to explain to more than a few people why MinHash by default use SHA1 hash function, or any of the secure hash functions, because these functions tend to be much slower than non-secure ones like Murmur3. The hashlib built-in library only offers secure hash functions, and the digest method outputs bytes, which is awkward, as I have to use struct.unpack to convert to integer.

# This is how we are doing the hashing right now.
import hashlib, struct
timeit struct.unpack("<I", hashlib.sha1(b"foo").digest()[:4])[0]
# 1 µs ± 27.2 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

# This is what I am trying to do.
import mmh3
timeit mmh3.hash("foo") & 0xffffffff
# 199 ns ± 6.04 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

I am looking at using mmh3 python library to replace the use of SHA1, and removing hashobj parameter all together (MinHash and HyperLogLog) to make the default fast. Do you think the performance gain is worth it? How much of your code am I going to break?

[vmarkovtsev]

@ekzhu I wonder why murmur and not xxhash which is faster, or Google farmhash which is even faster than xx? Both have tested Python packages...

[aastafiev]

Good idea and I agree with @vmarkovtsev. But, installing farmhash on windows is not as easy as on Linux. For example, I couldn't easy install it just from "pip install pyfarmhash". So our users will have troubles during installing too and we should support more on this. So, if this is blocking, I suggest using xxhash. Moreover it's already has "intdigest() – return the current digest as an integer".

[vmarkovtsev]

We could have swappable hashing backends and try to load the best package which is installed. It would also help to measure the performance gains.

[ekzhu]

Thanks guys. I will look more into it.

[ekzhu]

Python built-in SHA1:

Murmur3 from mmh3.

xxhash

farmhash

Take aways:

1. The performance is mainly determined by num_perm. The relative performance difference is not that big, however. Of course I haven't tested on bigger dataset.
2. The main difference is in accuracy, and farmhash is crazily good -- likely much more random.
3. I think the accuracy difference is big enough to justify a code change, especially for num_perm under 50. This is going to save a lot of CPU time since a much smaller num_perm is required to achieve the same accuracy.

I think making the hash function swappable is a good idea. Need to replace hashobj with a more generic interface.