Speed up downsample_counts

[ivirshup]

On master (3785143) from the base of the repo, I haven't seen the following code finish running:

import scanpy.api as sc
adata = sc.read("./data/pbmc3k_raw.h5ad")
%time sc.pp.downsample_counts(adata, 1500)

This PR implements an optimized version of the same thing, which gives:

%time sc.pp.downsample_counts(adata, 1500)                                                   
CPU times: user 2.25 s, sys: 44.7 ms, total: 2.29 s
Wall time: 2.32 s

What's changed

• I've rewritten the function to use numba along with fewer allocations
• Added a test for the function
• Added argument replace, which indicates whether subsampling should happen with replacement

Notes

To me, it makes more sense to sample without replacement, since for small changes in total counts you'll have more similar profiles. However, I've set the default for replacement to True to preserve the current behavior.

Neither this or the previous method scale well with sampling depth, and it's maybe worth using a call to sample a multinomial or multivariate hypergeometric distribution instead.

[ivirshup]

Um, it wasn't me.

CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/conda-forge/linux-64/xorg-libxdmcp-1.1.2-h470a237_7.tar.bz2>

Also, downsampling from 3785143 finished after an hour, but definitely had the wrong answer (all counts in one gene). I'm not sure what to make of this, since it's given reasonable results for smaller tests.

[LuckyMD]

Hey!
I wrote this function a while ago... it was definitely not the cleanest or quickest implementation. And it did take a while to run on ~5k cells at the time, but I thought it would be useful to have this functionality in scanpy.

Just wanted to note that the intention was definitely to implement this without resampling. I clearly missed that the default was to use resampling in np.random.choice. Thanks for spotting this.

[falexwolf]

That's really cool, thank you!

I'll add a logging output about that replace=False is the more natural choice and we'll make it the default in the next major release.