ModuleNotFound when creating a new plugin.

[phlnx]

• Version: 5.2.0+ 6f31e18
• Game Version: Odyssey
• OS: Windows 10 Pro
• OS Locale: Italian
• If applicable: Browser N/A

Describe the bug
Disclaimer: I am not an expert Python coder.

I am in the process of creating a plugin that will ultimately interface with a document in my google drive.
I followed a tutorial regarding how to use the python module gspread, and I created, outside of EDMC, a working python script that authenticates and handles a document remotely. It consists of four lines of code and it works as I want it to; no problem so far and nothing that concerns EDMC.
I created a directory for my new plugin, within C:\Users\myuser\AppData\Local\EDMarketConnector\plugins\, and inside that directory, I created a requirements.txt file, containing gspread. After that, I proceeded with a pip install -r requirements.txt -target . which downloaded gspread and all of its (multiple) dependencies and saved them in the directory itself (I was under the impression that this is the required modus operandi if a plugin needs additional modules).

My load.py file contains among the first lines, import gspread.
This caused the following error in EDMarketConnector.log:

2021-11-02 02:01:10.037 - INFO - 15976:5452:5452 plug.Plugin.__init__:47: loading plugin "myplugin" from "C:\Users\myuser\AppData\Local\EDMarketConnector\plugins\myplugin\load.py"
2021-11-02 02:01:10.099 - ERROR - 15976:5452:5452 plug.Plugin.__init__:62: : Failed for Plugin "myplugin"
Traceback (most recent call last):
  File "plug.pyc", line 49, in __init__
  File "<frozen importlib._bootstrap_external>", line 529, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 1029, in load_module
  File "<frozen importlib._bootstrap_external>", line 854, in load_module
  File "<frozen importlib._bootstrap>", line 274, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 711, in _load
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "C:\Users\myuser\AppData\Local\EDMarketConnector\plugins\myplugin\load.py", line 4, in <module>
    import gspread
  File "C:\Users\myuser\AppData\Local\EDMarketConnector\plugins\myplugin\gspread\__init__.py", line 16, in <module>
    from .auth import oauth, service_account, service_account_from_dict
  File "C:\Users\myuser\AppData\Local\EDMarketConnector\plugins\myplugin\gspread\auth.py", line 16, in <module>
    from google_auth_oauthlib.flow import InstalledAppFlow
  File "C:\Users\myuser\AppData\Local\EDMarketConnector\plugins\myplugin\google_auth_oauthlib\__init__.py", line 21, in <module>
    from .interactive import get_user_credentials
  File "C:\Users\myuser\AppData\Local\EDMarketConnector\plugins\myplugin\google_auth_oauthlib\interactive.py", line 24, in <module>
    import google_auth_oauthlib.flow
  File "C:\Users\myuser\AppData\Local\EDMarketConnector\plugins\myplugin\google_auth_oauthlib\flow.py", line 65, in <module>
    import wsgiref.simple_server
ModuleNotFoundError: No module named 'wsgiref'

I was puzzled that this wsgiref module was not found so I tried to understand what it exactly does. If I am not mistaken, apparently it is included in the Python Standard Library (https://docs.python.org/3/library/) which, according to this file (https://github.com/EDCD/EDMarketConnector/blob/main/PLUGINS.md) should render it automatically available. My very limited powers of investigation stopped here and at that point I decided to compile this issue report. I am grateful for your help.

To Reproduce
Steps to reproduce the behavior:
Not applicable.

Expected behavior
The plugin is successfully activated by EDMC.

Screenshots
Not applicable.

Additional context
Nothing to add.

[A-UNDERSCORE-D]

When we package for windows we don't include the entire python stdlib (It's quite large and would balloon our binaries quite a lot) -- You will either want to run EDMC from source while testing or ask us to include the library in our next release (Though an entire WSGI frontend might be a bit much).

[Syzuna]

In that case the docs maybe should be updated to reflect that.
Because right now they state that you can literally use/import anything in the python stdlib you may need.

[A-UNDERSCORE-D]

Yeah its not amazingly clear. I'll see about modifying it when Im a bit more awake.

[phlnx]

Thank you for your help, on one hand I'm relieved it was not me botching everything with a rookie mistake, on the other hand I guess it might have been easier to solve that way :-)

I had tried adding wsgiref to requirements.txt but pip install in that case returned a plethora of syntax errors (!) and I didn't know what was wrong. I guess you can't install that module with pip.

Re: asking you to include the module, I wouldn't want to inconvenience you and I am happy to explore alternative solutions.

Even if I ran EDMC from source, which I am not sure I am capable of doing with my limited skills, would that in any way ensure that my plugin will run when deposited in the plugin folder of someone else's installation?

[Athanasius]

Even if I ran EDMC from source, which I am not sure I am capable of doing with my limited skills, would that in any way ensure that my plugin will run when deposited in the plugin folder of someone else's installation?

No, as if they're using our provided installer then they only get the modules we ship.

More specifically we only ship what py2exe decides needs to be in the .exe and library.zip, based on:

1. What our core code actually uses.
2. Anything extra we specify in the setup.py - this is where we could add 'small enough' things for use by everyone.

Outside of that it's up to a third-party plugin developer to include the necessary modules in their plugin.

For some things it might end up being better to implement the 'meat' of such a plugin as its own standalone installation with some IPC mechanism for the 'stub' EDMC plugin to talk to it. Another possible issue with anything 'meaty' is if your plugin code is trying to do too much in the main EDMC thread with possible impact on other parts of the program (i.e. causing laggy UI during long-running functions).

[Athanasius]

In that case the docs maybe should be updated to reflect that. Because right now they state that you can literally use/import anything in the python stdlib you may need.

Indeed, that's an error, probably on my part, thinking I was referring to only the core of Python, and not all the extra modules that come with a standard python install.

Now I'm not quite sure how to word it so as to be:

1. Accurate.
2. Comprehensive.
3. Not give the false impression that it includes things it doesn't.

[phlnx]

No, as if they're using our provided installer then they only get the modules we ship.

Alright thank you, that was my guess as well. At this point then I think what I would need is a way to have wsgiref, as a module, available locally to my plugin, whether by pip installation or by copying manually certain files, and hoping it doesn't in turn require other modules that are not provided by EDMC.

[Athanasius]

I think, in the first instance, we should look at the 'Lib' directory in a standard Windows python install and then add all the modules there into setup.py packages or includes as appropriate and see what the size difference is. If it's trivial enough then the problem becomes one of checking we've properly updated this for any new version of Python we migrate to.

[phlnx]

Just wanted to say that by copying the entire directory wsgiref from an existing python 3.8.10 lib folder onto my plugin's folder, it worked fine. So from my point of view the issue can be closed but if you want to keep it as a reminder for the possible inclusion of wsgiref and/or the amendments to the documentation, that's fine with me.

[Athanasius]

Just wanted to say that by copying the entire directory wsgiref from an existing python 3.8.10 lib folder onto my plugin's folder, it worked fine. So from my point of view the issue can be closed but if you want to keep it as a reminder for the possible inclusion of wsgiref and/or the amendments to the documentation, that's fine with me.

You should use a copy from python 3.9.7, just in case there are any differences, and track the version we're building against.

But, yes, we'll keep this open as a reminder for future enhancement.

[phlnx]

Thank you for your reply. I used a 3.8.10 base dir because I had either that or the latest version, and I thought it safer to use an older version. As it seems to be working fine, I'll leave that for now.

[Athanasius]

A note for when we address this, you can find all the available modules in Python 3.10 using

sys.stdlib_module_names

Ref: https://realpython.com/python310-new-features/#more-information-about-your-python-interpreter

We still need to determine which are single-file modules and which are packages, because they need putting in separate arrays in the setup.py configuration. That and current py2exe needs any sub-part of a package listing separately, you can no longer just list the top level name.

[Athanasius]

Space considerations for including all of stdlib:

0$ ls -hl ~/AppData/Local/Temp/python310-32-Lib.zip
-rw-r--r-- 1 Athan 197121 43M Feb 14 16:52 /c/Users/Athan/AppData/Local/Temp/python310-32-Lib.zip
(venv)
0$ du -shc /c/Program\ Files\ \(x86\)/Python310-32/Lib
138M    /c/Program Files (x86)/Python310-32/Lib
138M    total

[Athanasius]

NB: If we effectively do "zip up the main python install's Lib into the py2exe library.zip" then it would need to be aware of if a pip install had pulled in a greater version of something. Yes, I'm thinking about setuptools there specifically, but it could potentially affect other things that plugins are more likely to use.

[A-UNDERSCORE-D]

Yeah Ive been puzzling over if we did that how to order sys.path

[Athanasius]

NB: We'd need all of the DLL directory as well, if py2exe doesn't bring contents in as appropriate.