Write pickle to file-like without intermediate in-memory buffer

[ig248]

Before this change, calling pickle.dumps() created an in-memory byte buffer, negating the advantage
of zero-copy pickle protocol 5. After this change, pickle.dump writes directly to open file(-like),
cutting peak memory in half in most cases.

Profiling was done with pandas@master and python 3.8.5

Related issues:

• ENH: Support out-of-band pickling (protocol 5) #34244

Update: ASV results

$ asv continuous -f 1.1 origin/master HEAD -b pickle yields:

       before           after         ratio
     [58f74686]       [3c1ee270]
                      <zero-copy-pickle>
-           91.9M            80.1M     0.87  io.pickle.Pickle.peakmem_write_pickle

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.

• closes ENH: Support out-of-band pickling (protocol 5) #34244
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[jreback]

nice, can you add a memory asv and show the results of this & a whatsnew note for 1.2 perf section.

[jreback]

also to actually close this issue i think would be good to add some tests that explicity set the pickle protocal to 5

[ig248]

also to actually close this issue i think would be good to add some tests that explicity set the pickle protocal to 5

Isn't that covered by py38 tests where pickle.HIGHEST_PROTOCOL (==5) is used by default?

[jreback]

also to actually close this issue i think would be good to add some tests that explicity set the pickle protocal to 5

Isn't that covered by py38 tests where pickle.HIGHEST_PROTOCOL (==5) is used by default?

probably but let's make it explict as well (e.g. parameterize on 4,5,highest when PY38 or greater)

[ig248]

Added ASV peakmem_ benchmark. The benchmark currently uses an ~11MB dataframe - I have no experience with ASV to tell how reliable memory benchmarking is for small-ish objects, any advice appreciated.

[jreback]

Added ASV peakmem_ benchmark. The benchmark currently uses an ~11MB dataframe - I have no experience with ASV to tell how reliable memory benchmarking is for small-ish objects, any advice appreciated.

it should show a diff here in any event. what kind of results are you seeing?

[ig248]

Added ASV peakmem_ benchmark. The benchmark currently uses an ~11MB dataframe - I have no experience with ASV to tell how reliable memory benchmarking is for small-ish objects, any advice appreciated.

it should show a diff here in any event. what kind of results are you seeing?

Yep, added results above - peakmem_write_pickle shows the 11MB drop nicely - in practice the relative gain is obv much more significant than shown for larger dataframes.

[pep8speaks]

Hello @ig248! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-10-12 22:51:15 UTC

[jreback]

lgtm ping on green. can you post the results of the asv (in the top of the PR, just edit it)

[jreback]

lgtm @ig248 just resolved the conflict and can merge on green.

[jreback]

thanks @ig248 very nice!