Pickling error on OS X / Python 3.8 (but not in 3.7)

[avirshup]

I am getting very similar errors to those described in #54, but in a different context:
I'm using Python 3.8 on MacOS.

Note that everything works fine with python 3.7!

$ py.test -x
========================================== test session starts ==========================================
platform darwin -- Python 3.8.1, pytest-5.3.2, py-1.8.1, pluggy-0.13.1
rootdir: [...], inifile: pytest.ini
plugins: typeguard-2.7.1, flask-0.15.0

test_cli.py E


================================================ ERRORS =================================================
_______________________________ ERROR at setup of test_idempotent_upload ________________________________

[...]
/Users/avirshup/.pyenv/versions/anaconda3-2019.10/envs/wcdb-py38/lib/python3.8/site-packages/pytest_flask/fixtures.py:72: in start
    self._process.start()
/Users/avirshup/.pyenv/versions/anaconda3-2019.10/envs/wcdb-py38/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Users/avirshup/.pyenv/versions/anaconda3-2019.10/envs/wcdb-py38/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Users/avirshup/.pyenv/versions/anaconda3-2019.10/envs/wcdb-py38/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Users/avirshup/.pyenv/versions/anaconda3-2019.10/envs/wcdb-py38/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Users/avirshup/.pyenv/versions/anaconda3-2019.10/envs/wcdb-py38/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Users/avirshup/.pyenv/versions/anaconda3-2019.10/envs/wcdb-py38/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <Process name='Process-1' parent=61378 initial>, file = <_io.BytesIO object at 0x7ff570abf9f0>
protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'LiveServer.start.<locals>.worker'

/Users/avirshup/.pyenv/versions/anaconda3-2019.10/envs/wcdb-py38/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
======================================== short test summary info ========================================
ERROR test_cli.py::test_idempotent_upload - AttributeError: Can't pickle local object 'LiveServer.star...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=================================== 393 deselected, 1 error in 4.41s ====================================

[avirshup]

This is probably because the defaults for multiprocessing changed in python 3.8. Now, by default, OS X multiprocessing starts processes with spawn instead of fork (like Windows). And with spawning, data is sent by pickling.

The workaround is simply to run multiprocessing.set_start_method("fork") exactly once per test session before calling the live_server fixture.

[nicoddemus]

@avirshup thanks for posting the solution so others can find it.

Closing it for now.