Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PicklingError using live_server fixture on Windows #54

Open
martinpengellyphillips opened this issue Dec 9, 2016 · 13 comments
Open

PicklingError using live_server fixture on Windows #54

martinpengellyphillips opened this issue Dec 9, 2016 · 13 comments
Labels
bug stale This issue has not seen activity for a while

Comments

@martinpengellyphillips
Copy link

Not sure if Windows is supposed to be supported, but for me it fails on Windows 10 Python 2.7.12 with the following error when live_server fixture used:

request = <SubRequest '_push_request_context' for <Function 'test_get_url'>>                                                                                       
                                                                                                                                                                   
    @pytest.fixture(autouse=True)                                                                                                                                  
    def _push_request_context(request):                                                                                                                            
        """During tests execution request context has been pushed, e.g. `url_for`,                                                                                 
        `session`, etc. can be used in tests as is::                                                                                                               
                                                                                                                                                                   
            def test_app(app, client):                                                                                                                             
                assert client.get(url_for('myview')).status_code == 200                                                                                            
                                                                                                                                                                   
        """                                                                                                                                                        
        if 'app' not in request.fixturenames:                                                                                                                      
            return                                                                                                                                                 
                                                                                                                                                                   
        app = request.getfuncargvalue('app')                                                                                                                       
                                                                                                                                                                   
        # Get application bound to the live server if ``live_server`` fixture                                                                                      
        # is applyed. Live server application has an explicit ``SERVER_NAME``,                                                                                     
        # so ``url_for`` function generates a complete URL for endpoint which                                                                                      
        # includes application port as well.                                                                                                                       
        if 'live_server' in request.fixturenames:                                                                                                                  
>           app = request.getfuncargvalue('live_server').app                                                                                                       
                                                                                                                                                                   
..\..\..\..\python-virtualenv\shadow\Lib\site-packages\pytest_flask\plugin.py:85:                                                                                  
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _                                                                                    
..\..\..\..\python-virtualenv\shadow\Lib\site-packages\pytest_flask\fixtures.py:127: in live_server                                                                
    server.start()                                                                                                                                                 
..\..\..\..\python-virtualenv\shadow\Lib\site-packages\pytest_flask\fixtures.py:64: in start                                                                       
    self._process.start()                                                                                                                                          
c:\python27\Lib\multiprocessing\process.py:130: in start                                                                                                           
    self._popen = Popen(self)                                                                                                                                      
c:\python27\Lib\multiprocessing\forking.py:277: in __init__                                                                                                        
    dump(process_obj, to_child, HIGHEST_PROTOCOL)                                                                                                                  
c:\python27\Lib\multiprocessing\forking.py:199: in dump                                                                                                            
    ForkingPickler(file, protocol).dump(obj)                                                                                                                       
c:\python27\Lib\pickle.py:224: in dump                                                                                                                             
    self.save(obj)                                                                                                                                                 
c:\python27\Lib\pickle.py:331: in save                                                                                                                             
    self.save_reduce(obj=obj, *rv)                                                                                                                                 
c:\python27\Lib\pickle.py:425: in save_reduce                                                                                                                      
    save(state)                                                                                                                                                    
c:\python27\Lib\pickle.py:286: in save                                                                                                                             
    f(self, obj) # Call unbound method with explicit self                                                                                                          
c:\python27\Lib\pickle.py:655: in save_dict                                                                                                                        
    self._batch_setitems(obj.iteritems())                                                                                                                          
c:\python27\Lib\pickle.py:687: in _batch_setitems                                                                                                                  
    save(v)                                                                                                                                                        
c:\python27\Lib\pickle.py:286: in save                                                                                                                             
    f(self, obj) # Call unbound method with explicit self                                                                                                          
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _                                                                                    
                                                                                                                                                                   
self = <multiprocessing.forking.ForkingPickler instance at 0x0000000007EDBA08>                                                                                     
obj = <function <lambda> at 0x00000000081B9DD8>, name = '<lambda>'                                                                                                 
pack = <built-in function pack>                                                                                                                                    
                                                                                                                                                                   
    def save_global(self, obj, name=None, pack=struct.pack):                                                                                                       
        write = self.write                                                                                                                                         
        memo = self.memo                                                                                                                                           
                                                                                                                                                                   
        if name is None:                                                                                                                                           
            name = obj.__name__                                                                                                                                    
                                                                                                                                                                   
        module = getattr(obj, "__module__", None)                                                                                                                  
        if module is None:                                                                                                                                         
            module = whichmodule(obj, name)                                                                                                                        
                                                                                                                                                                   
        try:                                                                                                                                                       
            __import__(module)                                                                                                                                     
            mod = sys.modules[module]                                                                                                                              
            klass = getattr(mod, name)                                                                                                                             
        except (ImportError, KeyError, AttributeError):                                                                                                            
            raise PicklingError(                                                                                                                                   
                "Can't pickle %r: it's not found as %s.%s" %                                                                                                       
>               (obj, module, name))                                                                                                                               
E           PicklingError: Can't pickle <function <lambda> at 0x00000000081B9DD8>: it's not found as pytest_flask.fixtures.<lambda>                                
@vitalk
Copy link
Collaborator

vitalk commented Dec 9, 2016

Hi @martinpengellyphillips!

Thank you for the report. Sadly I have no possibility to reproduce or test your issue, but looks like it can be easily fixed. Please, test against 54-pickle-error-on-windows branch.

@martinpengellyphillips
Copy link
Author

Hi @vitalk - thanks for the quick response. I've tried that branch now and it fixes the previous error. However, a new error is given now:

request = <SubRequest '_push_request_context' for <Function 'test_get_url'>>

    @pytest.fixture(autouse=True)
    def _push_request_context(request):
        """During tests execution request context has been pushed, e.g. `url_for`,
        `session`, etc. can be used in tests as is::

            def test_app(app, client):
                assert client.get(url_for('myview')).status_code == 200

        """
        if 'app' not in request.fixturenames:
            return

        app = request.getfuncargvalue('app')

        # Get application bound to the live server if ``live_server`` fixture
        # is applyed. Live server application has an explicit ``SERVER_NAME``,
        # so ``url_for`` function generates a complete URL for endpoint which
        # includes application port as well.
        if 'live_server' in request.fixturenames:
>           app = request.getfuncargvalue('live_server').app

..\..\..\..\python-virtualenv\shadow\Lib\site-packages\pytest_flask\plugin.py:85:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
..\..\..\..\python-virtualenv\shadow\Lib\site-packages\pytest_flask\fixtures.py:130: in live_server
    server.start()
..\..\..\..\python-virtualenv\shadow\Lib\site-packages\pytest_flask\fixtures.py:67: in start
    self._process.start()
c:\python27\Lib\multiprocessing\process.py:130: in start
    self._popen = Popen(self)
c:\python27\Lib\multiprocessing\forking.py:277: in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
c:\python27\Lib\multiprocessing\forking.py:199: in dump
    ForkingPickler(file, protocol).dump(obj)
c:\python27\Lib\pickle.py:224: in dump
    self.save(obj)
c:\python27\Lib\pickle.py:331: in save
    self.save_reduce(obj=obj, *rv)
c:\python27\Lib\pickle.py:425: in save_reduce
    save(state)
c:\python27\Lib\pickle.py:286: in save
    f(self, obj) # Call unbound method with explicit self
c:\python27\Lib\pickle.py:655: in save_dict
    self._batch_setitems(obj.iteritems())
c:\python27\Lib\pickle.py:687: in _batch_setitems
    save(v)
c:\python27\Lib\pickle.py:286: in save
    f(self, obj) # Call unbound method with explicit self
c:\python27\Lib\pickle.py:554: in save_tuple
    save(element)
c:\python27\Lib\pickle.py:331: in save
    self.save_reduce(obj=obj, *rv)
c:\python27\Lib\pickle.py:425: in save_reduce
    save(state)
c:\python27\Lib\pickle.py:286: in save
    f(self, obj) # Call unbound method with explicit self
c:\python27\Lib\pickle.py:655: in save_dict
    self._batch_setitems(obj.iteritems())
c:\python27\Lib\pickle.py:687: in _batch_setitems
    save(v)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <multiprocessing.forking.ForkingPickler instance at 0x00000000075EA588>
obj = <thread.lock object at 0x00000000071A1130>

    def save(self, obj):
        # Check for persistent id (defined by a subclass)
        pid = self.persistent_id(obj)
        if pid is not None:
            self.save_pers(pid)
            return

        # Check the memo
        x = self.memo.get(id(obj))
        if x:
            self.write(self.get(x[0]))
            return

        # Check the type dispatch table
        t = type(obj)
        f = self.dispatch.get(t)
        if f:
            f(self, obj) # Call unbound method with explicit self
            return

        # Check copy_reg.dispatch_table
        reduce = dispatch_table.get(t)
        if reduce:
            rv = reduce(obj)
        else:
            # Check for a class with a custom metaclass; treat as regular class
            try:
                issc = issubclass(t, TypeType)
            except TypeError: # t is not a class (old Boost; see SF #502085)
                issc = 0
            if issc:
                self.save_global(obj)
                return

            # Check for a __reduce_ex__ method, fall back to __reduce__
            reduce = getattr(obj, "__reduce_ex__", None)
            if reduce:
>               rv = reduce(self.proto)
E               TypeError: can't pickle thread.lock objects                                                                         

@vitalk
Copy link
Collaborator

vitalk commented Dec 9, 2016

I‘m sorry, but I‘m not sure I can be of much help (I don‘t run windows), but if you come up with a solution (or someone else), I‘ll happily review and merge.

@martinpengellyphillips
Copy link
Author

No worries, thanks for taking a look.

@Nagasaki45
Copy link

I don't know if another report of the issue will help in anyway, but here it is:

============================= test session starts =============================
platform win32 -- Python 3.6.1, pytest-3.1.3, py-1.4.34, pluggy-0.4.0
rootdir: C:\Users\tgurion\unity\Unsocial VR\backchannel, inifile:
plugins: flask-0.10.0
collected 1 item

test_server.py E

=================================== ERRORS ====================================
_____________ ERROR at setup of test_server_not_completely_broken _____________

request = <SubRequest '_push_request_context' for <Function 'test_server_not_completely_broken'>>

    @pytest.fixture(autouse=True)
    def _push_request_context(request):
        """During tests execution request context has been pushed, e.g. `url_for`,
        `session`, etc. can be used in tests as is::
    
            def test_app(app, client):
                assert client.get(url_for('myview')).status_code == 200
    
        """
        if 'app' not in request.fixturenames:
            return
    
        app = request.getfuncargvalue('app')
    
        # Get application bound to the live server if ``live_server`` fixture
        # is applyed. Live server application has an explicit ``SERVER_NAME``,
        # so ``url_for`` function generates a complete URL for endpoint which
        # includes application port as well.
        if 'live_server' in request.fixturenames:
>           app = request.getfuncargvalue('live_server').app

..\..\..\AppData\Local\Continuum\Anaconda3\envs\backchannel\lib\site-packages\pytest_flask\plugin.py:85: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
..\..\..\AppData\Local\Continuum\Anaconda3\envs\backchannel\lib\site-packages\pytest_flask\fixtures.py:127: in live_server
    server.start()
..\..\..\AppData\Local\Continuum\Anaconda3\envs\backchannel\lib\site-packages\pytest_flask\fixtures.py:64: in start
    self._process.start()
..\..\..\AppData\Local\Continuum\Anaconda3\envs\backchannel\lib\multiprocessing\process.py:105: in start
    self._popen = self._Popen(self)
..\..\..\AppData\Local\Continuum\Anaconda3\envs\backchannel\lib\multiprocessing\context.py:223: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
..\..\..\AppData\Local\Continuum\Anaconda3\envs\backchannel\lib\multiprocessing\context.py:322: in _Popen
    return Popen(process_obj)
..\..\..\AppData\Local\Continuum\Anaconda3\envs\backchannel\lib\multiprocessing\popen_spawn_win32.py:65: in __init__
    reduction.dump(process_obj, to_child)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <Process(Process-1, initial)>, file = <_io.BufferedWriter name=8>
protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'LiveServer.start.<locals>.<lambda>'

..\..\..\AppData\Local\Continuum\Anaconda3\envs\backchannel\lib\multiprocessing\reduction.py:60: AttributeError
=========================== 1 error in 0.89 seconds ===========================

@Glazz87
Copy link

Glazz87 commented Jul 20, 2017

@Nagasaki45 @vitalk
Same problem on Windows 7 64 bit, python 3.5.3.
Very sad because fixture looks very useful...

(venv) C:\Git\GitHub\pytest-flask>pytest -v tests
============================= test session starts =============================
platform win32 -- Python 3.5.3, pytest-3.1.3, py-1.4.34, pluggy-0.4.0 -- c:\git\git
hub\pytest-flask\venv\scripts\python.exe
cachedir: .cache
rootdir: C:\Git\GitHub\pytest-flask, inifile: tox.ini
plugins: flask-0.10.0
collected 22 items

tests/test_fixtures.py::TestFixtures::test_config_access PASSED
tests/test_fixtures.py::TestFixtures::test_client PASSED
tests/test_fixtures.py::TestFixtures::test_accept_json PASSED
tests/test_fixtures.py::TestFixtures::test_accept_jsonp PASSED
tests/test_fixtures.py::TestFixtures::test_request_ctx PASSED
tests/test_fixtures.py::TestFixtures::test_request_ctx_is_kept_around PASSED
tests/test_fixtures.py::TestJSONResponse::test_json_response PASSED
tests/test_fixtures.py::TestJSONResponse::test_dont_rewrite_existing_implementation
 PASSED
tests/test_fixtures.py::TestClientClass::test_client_attribute PASSED
tests/test_live_server.py::TestLiveServer::test_server_is_alive ERROR
tests/test_live_server.py::TestLiveServer::test_server_url ERROR
tests/test_live_server.py::TestLiveServer::test_server_listening ERROR
tests/test_live_server.py::TestLiveServer::test_url_for ERROR
tests/test_live_server.py::TestLiveServer::test_set_application_server_name ERROR
tests/test_live_server.py::TestLiveServer::test_rewrite_application_server_name ERR
OR
tests/test_live_server.py::TestLiveServer::test_prevent_starting_live_server PASSED

tests/test_live_server.py::TestLiveServer::test_start_live_server FAILED
tests/test_live_server.py::TestLiveServer::test_add_endpoint_to_live_server FAILED
tests/test_live_server.py::TestLiveServer::test_concurrent_requests_to_live_server
FAILED
tests/test_markers.py::TestOptionMarker::test_not_debug_app PASSED
tests/test_markers.py::TestOptionMarker::test_update_application_config PASSED
tests/test_markers.py::TestOptionMarker::test_application_config_teardown PASSED

=================================== ERRORS ====================================
____________ ERROR at setup of TestLiveServer.test_server_is_alive ____________

request = <SubRequest '_push_request_context' for <Function 'test_server_is_alive'>
>

    @pytest.fixture(autouse=True)
    def _push_request_context(request):
        """During tests execution request context has been pushed, e.g. `url_for`,
        `session`, etc. can be used in tests as is::

            def test_app(app, client):
                assert client.get(url_for('myview')).status_code == 200

        """
        if 'app' not in request.fixturenames:
            return

        app = request.getfuncargvalue('app')

        # Get application bound to the live server if ``live_server`` fixture
        # is applyed. Live server application has an explicit ``SERVER_NAME``,
        # so ``url_for`` function generates a complete URL for endpoint which
        # includes application port as well.
        if 'live_server' in request.fixturenames:
>           app = request.getfuncargvalue('live_server').app

venv\lib\site-packages\pytest_flask\plugin.py:85:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
venv\lib\site-packages\pytest_flask\fixtures.py:127: in live_server
    server.start()
venv\lib\site-packages\pytest_flask\fixtures.py:64: in start
    self._process.start()
C:\Users\a.glazkov\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\pr
ocess.py:105: in start
    self._popen = self._Popen(self)
C:\Users\a.glazkov\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\co
ntext.py:212: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
C:\Users\a.glazkov\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\co
ntext.py:313: in _Popen
    return Popen(process_obj)
C:\Users\a.glazkov\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\po
pen_spawn_win32.py:66: in __init__
    reduction.dump(process_obj, to_child)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <Process(Process-1, initial)>, file = <_io.BufferedWriter name=8>
protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'LiveServer.start.<locals>.<lambd
a>'

C:\Users\a.glazkov\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\re
duction.py:59: AttributeError

@Jitsusama
Copy link

I'm running into the same thing. Makes me very :(

I guess I'll just roll my own.

@Jitsusama
Copy link

This seems to work for me fine:

import subprocess
import pytest
import requests

@pytest.fixture(scope="module", autouse=True)
def server():
    server = subprocess.Popen(['python', '-m', 'app'])
    time.sleep(1)
    yield server
    server.terminate()

def test_request():
    response = requests.get('http://localhost:5000')
    assert response.status_code == 200

@brandonschabell
Copy link

I'm running into the same issue with Windows 10 and Python 3.6.7.

@psdon
Copy link

psdon commented Jun 10, 2019

Having this issue too in Windows 10 and Python 3.6

@psdon
Copy link

psdon commented Jun 10, 2019

Any workaround for this?

@ankostis
Copy link

Improving on @Jitsusama workaround, using flask run command to launch app to properly configure, and moved server-launch in ./tests/conftest.py:

import pytest
import socket
import subprocess

@pytest.fixture(scope="session")
def flask_port():
    ## Ask OS for a free port.
    #
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.bind(("", 0))
        addr = s.getsockname()
        port = addr[1]
        return port

@pytest.fixture(scope="session", autouse=True)
def live_server(flask_port):
    env = os.environ.copy()
    env["FLASK_APP"] = "your_package.app_module"
    server = subprocess.Popen(['flask', 'run', '--port', str(flask_port)], env=env)
    try:
        yield server
    finally:
        server.terminate()

And then write your TestCases in test-files (e.g. ./tests/test_my_app.py) like this:

import pytest
import requests

def test_request(flask_port):
    response = requests.get(f"http://localhost:{flask_port}"')
    assert response.status_code == 200

@VCleemput
Copy link

Hey, I'm new here. I get the same error on Windows 10, Python 3.12 trying to set up the live-server for use with selenium.
I've been looking arround, and found something about pickle not being able to serialize enough different types. A workaround using "dill" was suggested (https://stackoverflow.com/questions/73570416/python-multiprocessing-cant-pickle-local-object#comment129918523_73570416), for which you can use the "multiprocess" fork of multiprocessing (https://pypi.org/project/multiprocess). I'm going to toy arround with that, but I don't have much experience on the matter...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug stale This issue has not seen activity for a while
Projects
None yet
Development

No branches or pull requests

10 participants