Fix incorrect results for large numbers of points

[lkeegan]

• replace all 32-bit integers with 64-bit integers
• previously the results would silently be incorrect if num_points * k > 2^32-1
  • caused by integer overflow (see pydktree breaks down for very large number of data points #38)
  • for e.g. k=20 this occurred for ~200 million points (~a few GB of RAM)
• now the results are correct for any (practical) number of points
  • overflow would now only occur for k=20 with ~10^17 points (> trillions of GB of RAM)
  • resolves pydktree breaks down for very large number of data points #38

[mraspaud]

Thank you for you contribution!

I’m wondering what the effect on performance (speed, memory usage) can be when 32 bits would be enough?

[djhoese]

I am also concerned about performance. I think the "safer" solution, although much more difficult to implement, would be to use something like cython fused types and then also generate the C code (from the .mako template) with both 32-bit and 64-bit types. I could be wrong, but I think overloading should work fine if two functions in the C code have the same name but accept different input types.

[lkeegan]

I did some local benchmarks and this PR was ~5% slower than the master branch, and used ~50% more RAM.
Are there any standard benchmarks I should try running?

I agree the better solution would be to have both 32-bit int and 64-bit int implementations, and to choose the appropriate one at runtime based on the number of points and value of k provided by the user, although I also agree it would be more difficult to implement :-)

[djhoese]

This user just did some updated benchmarks it seems: #121 (comment)

[lkeegan]

See #135 for an alternative with both 32 and 64 bit implementations which avoids making existing performance / ram use worse.