ServiceFramework incompatible with venv

[tfishr]

Windows 10 build 1903, Python 3.8.0, pywin32 227, an activated venv and the following simple Windows service implementation:

#!/usr/bin/env python
import win32serviceutil, win32service, win32event, servicemanager
from multiprocessing import Process
import my_http_server

class WindowsService(win32serviceutil.ServiceFramework):
	_svc_name_ = "FMP_API"
	_svc_display_name_ = "FMP API"

	def __init__(self, args):
		win32serviceutil.ServiceFramework.__init__(self, args)
		self.stop_event = win32event.CreateEvent(None, 0, 0, None)

	def SvcDoRun(self):
		servicemanager.LogMsg(servicemanager.EVENTLOG_INFORMATION_TYPE,
			servicemanager.PYS_SERVICE_STARTED,
			(self._svc_name_, ""))
		self.server = Process(target=my_http_server.start)
		self.server.start()
		self.server.join() # block here until stop requested

	def SvcStop(self):
		self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
		win32event.SetEvent(self.stop_event)
		self.server.terminate()
		self.server.join()

if __name__ == "__main__":
	win32serviceutil.HandleCommandLine(WindowsService)

If you pip install pywin32 into your venv and run pywin32_postinstall.py, then install and start the service will not work. The event log contains ModuleNotFoundError: No module named 'win32serviceutil' %2: %3

If you pip install pywin32 globally, it sees the pywin32 modules, but not any other package dependencies unless you also install those packages globally.

So, it really only works correctly if you install all packages globally and not use venv at all.

[sm-Fifteen]

The service is not run by your account, but by the System account, so it won't be aware of the virtual environment you were in when you installed the service.

You need to manually add the packages from your venv to your environment before you try loading them.

import sys, os, site

# Required for pythonservice.exe to find venv packages
# Will also load pywin32.pth, so the win32* packages will work as normal
script_dir = os.path.dirname(os.path.realpath(__file__))
site_packages = os.path.join(script_dir, ".pyenv", 'Lib', 'site-packages')
site.addsitedir(site_packages)

[tfishr]

You need to manually add the packages from your venv to your environment before you try loading them.

I'm not sure what this means. Copy the venv to System's home directory?

copy .\venv C:\windows\system32

Like that?

[mhammond]

I would strongly suggest you simply don't try and implement a service from inside a venv. If I take any action here it's likely to be to make it more obvious it doesn't work (ie, to fail even earlier, possibly even with a message to say you shouldn't be doing it)

[sm-Fifteen]

Like that?

No, just add the code I included at the very start of your service and you should be fine.
*EDIT: * Make sure site_packages is actually pointing at your virtual environment's site_packages

[tfishr]

Oh I see. So the site_packages folder (that contains the pywin32.pth file) is what needs to be fed to site.addsitedir() which adds it to the environment (aka sys.path).

In the example, ".pyenv" would be whatever you named your venv.

So then you could have all your service's modules in one place, but your site_packages could be in a completely different place, if you wanted to organize it that way.

[vernondcole]

The example program contains an error. The hash-bang line should not reference the systems python. It ought to reference the python interpreter in the virtual environment, and the Python Launcher for Windows must be installed. If those two things are done, then the service ought to operate correctly. If it does not, then we need to make a fix. see <https://docs.python.org/3/library/venv.html>

…

scripts installed into virtual environments have a “shebang” line which points to the virtual environment’s Python interpreter. This means that the script will run with that interpreter regardless of the value of PATH. On Windows, “shebang” line processing is supported if you have the Python Launcher for Windows installed (this was added to Python in 3.3 - see *PEP 397* <https://www.python.org/dev/peps/pep-0397> for more details). Thus, double-clicking an installed script in a Windows Explorer window should run the script with the correct interpreter without there needing to be any reference to its virtual environment in PATH. On Wed, Dec 11, 2019 at 3:36 PM Tim Fisher ***@***.***> wrote: Oh I see. So the site_packages folder (that contains the pywin32.pth file) is what needs to be fed to site.addsitedir(), which adds it to the environment (aka sys.path). In your code ".pyenv" would be whatever you named your venv. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1450?email_source=notifications&email_token=AAEZOBKQ4JO6KFC7ZDBKKPDQYFTQBA5CNFSM4JZTNLSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGU2A3A#issuecomment-564764780>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEZOBJQFFK3OGVKRCUQG2TQYFTQBANCNFSM4JZTNLSA> .

[tfishr]

The example program contains an error. The hash-bang line should not reference the systems python. It ought to reference the python interpreter in the virtual environment, and the Python Launcher for Windows must be installed.

Thanks! Yes I didn't notice this. You are correct. There ought to be an explicit path there that references the correct Python interpreter so in case the user doesn't have the venv activated, things should still work correctly.

I recently converted from C# to Python so still trying to learn all the nuances.

[sm-Fifteen]

The example program contains an error. The hash-bang line should not reference the systems python. It ought to reference the python interpreter in the virtual environment, and the Python Launcher for Windows must be installed.

That's extremely interesting! There's no mention of relative paths, but setting the shebang to #!./.pyenv/Scripts/python and then running py service.py works (though it's relative to the working directory, not the script's path like I'd hoped) removes the need for the site_packages manipulation.

EDIT: I spoke too soon, package import still fails while running the service proper, both in debug and in service mode, so it doesn't seem to follow the same rules as the py launcher at all. It also doesn't fix the searchpath for PythonXX.dll while running as a service either, so I still get the error from #1451.

If those two things are done, then the service ought to operate correctly. If it does not, then we need to make a fix. see https://docs.python.org/3/library/venv.html

So you're saying pythonservice.exe should be fixed to behave like py.exe and use whatever python interpreter the shebang is refering to, then?

[tfishr]

Sounds like you hit on the key, which is that pythonservice.exe doesn't use py.exe method of finding the interpreter, and it doesn't provide any way to select the interpreter explicitly. If the correct interpreter were launched then that would take care of finding the venv and resolving packages. If it just relies on the current system interpreter from the system path however, that's not going to have a happy ending with respect to venv.

It sounds like modifying how pythonserver.exe selects the interpreter would be a big new feature for pythonservice.exe and not likely to happen soon. So for now, I'll have to stay with globals.

For me having venv support isn't so much about dependency isolation but about making deployment easier for end-users. Not only pip install -r is required. Some packages require build tools. Some packages are not on pypi so you have to provide a url to pip. That's a lot of extra steps for them to screw up. Having venv support would be a really nice way to spread the adoption of Python Windows services.

With venv the installation becomes: (1) Install Python, (2) copy these files and (3) run commands .\service.py install and .\service.py start. If I can keep an installation process to three steps, that's valuable.

[sm-Fifteen]

I did some research, and it would seem that the way Python venv dependencies usually work is by looking in the directory above sys.executable (which would be, say, myvenv/Scripts/python.exe) to see if there is a pyvenv.cfg file in there, and use that to configure the available site packages. Since pythonservice.exe is inside of myvenv/Lib/site-packages/win32, that lookup fails. Copying pyenv.cfg over to site_packages "fixes" the package import issue.

https://github.com/python/cpython/blob/9048c49322a5229ff99610aba35913ffa295ebb7/Lib/site.py#L16-L23

As for picking and loading the correct python core DLL, the way py.exe seems to do it when a shabang is specified is to simply call that python interpreter with no extra logic. The venv's copy of Python.exe will just try loading the same pyvenv.cfg use the specified home directory to load its DLLs and libraries. You can try this by changing the home key to a nonexistent path before calling that interpreter:

.pyenv/pyvenv.cfg

home = C:\Program Files\Python\Python388
include-system-site-packages = false
version = 3.8.0

>.pyenv\Scripts\python.exe
No Python at 'C:\Program Files\Python\Python388\python.exe'

Supposedly the default behavior for py.exe is to look at the list of registered python installations in the Windows registry (HKEY_CURRENT_USER and HKEY_LOCAL_MACHINE have different sets) instead of relying on the PATH, which would explain how it can run regardless of where the core DLL is, though there appears to be more to it than just that since it can also find Python versions that are neither in my path nor my registry. Nevermind that, that doesn't seem to factor in with the current issue.

EDIT: This section of the official doc goes through most of the process, including how to get debugging information out of py.exe with PYLAUNCH_DEBUG and the various special files that will affect CPython's behavior when loading.

[alex-eri]

I spent day with pythonservice code and found solution!

import win32serviceutil
import win32service
import servicemanager
import sys
import os
import os.path
import multiprocessing

# 

def main():
    import time
    time.sleep(600)  

class ProcessService(win32serviceutil.ServiceFramework):
    _svc_name_ = "SleepService"
    _svc_display_name_ = "Sleep Service"
    _svc_description_ = "Sleeps for 600"
    _exe_name_ = sys.executable # python.exe from venv
    _exe_args_ = '-u -E "' + os.path.abspath(__file__) + '"'

    proc = None

    def SvcStop(self):
        self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
        if self.proc:
            self.proc.terminate()

    def SvcRun(self):
        self.proc = multiprocessing.Process(target=main)
        self.proc.start()        
        self.ReportServiceStatus(win32service.SERVICE_RUNNING)
        self.SvcDoRun()
        self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)

    def SvcDoRun(self):
        self.proc.join()

def start():
    if len(sys.argv)==1:
        import win32traceutil
        servicemanager.Initialize()
        servicemanager.PrepareToHostSingle(ProcessService)
        servicemanager.StartServiceCtrlDispatcher()
    elif '--fg' in sys.argv:
        main()
    else:
        win32serviceutil.HandleCommandLine(ProcessService)

if __name__ == '__main__':
    try:
        start()
    except (SystemExit, KeyboardInterrupt):
        raise
    except:
        import traceback
        traceback.print_exc()

Modern python found its venv auto -E removes installed python`s vars. -u for not to die without stdin.

Service registred as "C:\Python\Python310\python310.exe" -u -E "C:\path\to\service.py"

Worker in Process not blocking Framework and so start-stop works fast.

[mojodevops]

@alex-eri When using pyinstaller and package the code to exe, it does not work. Can you fix it? Thanks