Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3.10 compatiblity #145

Open
7 tasks
sashkab opened this issue Oct 6, 2021 · 8 comments
Open
7 tasks

Python 3.10 compatiblity #145

sashkab opened this issue Oct 6, 2021 · 8 comments
Assignees

Comments

@sashkab
Copy link
Contributor

sashkab commented Oct 6, 2021

Questions

  • Which operating system are you using (if Linux, please provide flavour of it, i.e RedHat, CentOS or Ubuntu), is it 32-bit, or 64-bit?

    macOS 11.6, 64-bit
    
  • Which version of PyQ are you running? Please provide output of pyq --versions, if PyQ isn't operational, please provide Python interpreter version and PyQ version python -V; python3 -V; pip list | grep pyq:

    5.00
    
  • Which version of kdb+ are you using, is it 32-bit or 64-bit?

    64-bit
    
    • If on 64-bit, is your QLIC set? Please provide output env | grep QLIC on linux/macOS, or set|grep QLIC on Windows.

      $ env | grep QLIC
      QLIC=/Users/user/.lic
      
  • Did you use virtual environment to install PyQ? If not, why?

    • yes
  • Where is your QHOME? Please provide output env | grep QHOME on linux/macOS, or set|grep QHOME on Windows.

    $ env | grep QHOME
    $
    
  • Do you use Conda? If so, what version?

    • no

Steps to reproduce the issue

$ brew install [email protected]
$  /usr/local/opt/[email protected]/bin/python3 -mvenv .virtualenvs/310
$ source .virtualenvs/310/bin/activate
$ python3.10 -mpip install -U pip setuptools wheel
$ unzip ~/Downloads/m64.zip -d "${VIRTUAL_ENV}/q"
$ (unset PIP_NO_INDEX; python3.10 -mpip install pyq)
...
Successfully built pyq
Installing collected packages: pyq
Successfully installed pyq-5.0.0
$ pyq --versions
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py", line 4, in <module>
    from collections import Mapping as _Mapping
ImportError: cannot import name 'Mapping' from 'collections' (/usr/local/Cellar/[email protected]/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/collections/__init__.py)
'2021.10.06T12:38:40.403 python

Expected result

pyq --versions will output versions installed on system

Actual result

$ pyq --versions
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py", line 4, in <module>
    from collections import Mapping as _Mapping
ImportError: cannot import name 'Mapping' from 'collections' (/usr/local/Cellar/[email protected]/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/collections/__init__.py)
'2021.10.06T12:38:40.403 python

Workaround

$ cp -pv /Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py{,.orig}
/Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py -> /Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py.orig
$ vi /Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py
$ diff /Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py{.orig,}
4c4,7
< from collections import Mapping as _Mapping
---
> try:
>     from collections import Mapping as _Mapping
> except ImportError:
>     from collections.abc import Mapping as _Mapping
$ pyq --versions
PyQ 5.0.0
KDB+ 3.5 (2019.01.31) m64
Python 3.10.0 (default, Oct  6 2021, 01:11:32) [Clang 13.0.0 (clang-1300.0.29.3)]
@sashkab
Copy link
Contributor Author

sashkab commented Oct 6, 2021

After first problem fixed, there is second problem.

$ python3.10 -mpip list
Package    Version
---------- -------
attrs      21.2.0
coverage   6.0
iniconfig  1.1.1
numpy      1.21.2
packaging  21.0
pip        21.2.4
pluggy     1.0.0
py         1.10.0
pyparsing  2.4.7
pyq        5.0.0
pytest     6.2.5
pytest-cov 3.0.0
pytest-pyq 1.2.0
setuptools 58.2.0
toml       0.10.2
tomli      1.2.1
wheel      0.37.0
$ pyq -mpip list
/Users/user/.virtualenvs/310/bin/pyq: No module named pip

I believe we saw this kind of issues before, but can't recall if it was addressed. It was related to the virutalenv and venv modules, how they are now implemented... which breaks logic PyQ relies on.

@sashkab
Copy link
Contributor Author

sashkab commented Oct 6, 2021

Interestingly:

$ QHOME=$VIRTUAL_ENV/q $VIRTUAL_ENV/q/m64/q
...
q)p)import numpy
q)p)import pytest
q)p)import pip
q)p)print(pip.__file__)
/Users/user/.virtualenvs/310/lib/python3.10/site-packages/pip/__init__.py
q)p)print(sys.executable)
/Users/user/.virtualenvs/310/bin/pyq
q)p)print(sys.path)
['/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python310.zip', '/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10', '/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload', '/Users/user/.virtualenvs/310/lib/python3.10/site-packages']
q)p)print(sys.prefix)
/Users/user/.virtualenvs/310
q)p)print(sys.exec_prefix)
/Users/user/.virtualenvs/310

but

$ pyq
Python 3.10.0 (default, Oct  6 2021, 01:11:32) [Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'/Users/user/.virtualenvs/310/bin/pyq'
>>> sys.path
['', '/usr/local/Cellar/[email protected]/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python310.zip', '/usr/local/Cellar/[email protected]/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10', '/usr/local/Cellar/[email protected]/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload']
>>> sys.prefix
'/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10'
>>> sys.exec_prefix
'/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10'

@sashkab
Copy link
Contributor Author

sashkab commented Oct 12, 2021

NB: mapping import was fixed in the internal MR 746, which is planned to be released as 5.0.1...

@MurrMack
Copy link

MurrMack commented Aug 5, 2022

Hi Aleks - is there a plan for 5.0.1 to be released any time soon? the default version of python on ubuntu 22.04 LTS is 3.10 so others may start running into this issue as upgrades begin to occur. Cheers!

@b-nln
Copy link

b-nln commented Aug 5, 2022

There are no plans to release 5.0.1, as there is no full python 3.10 support yet.

@MurrMack
Copy link

MurrMack commented Aug 15, 2022

Is there anyway to force pyq to use a different version of python? My default version of python is 3.9 using update-alternatives however when I run pyq it tries to still use 3.10

CPUS=24 pyq                                                                                                                                                                         
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/murray/.local/lib/python3.10/site-packages/pyq/__init__.py", line 4, in <module>
    from collections import Mapping as _Mapping
ImportError: cannot import name 'Mapping' from 'collections' (/usr/lib/python3.10/collections/__init__.py)
'2022.08.15T08:58:06.931 python
  [5]  /home/murray/q/p.k:7: p)from pyq import q
                             ^
  [2]  /home/murray/q/python.q:1: \l p.k

python --version  
Python 3.9.13

python3 --version
Python 3.9.13

@b-nln
Copy link

b-nln commented Aug 15, 2022

Use virtualenv with python3.9 and install PyQ in there as described in the installation guide.

@sashkab
Copy link
Contributor Author

sashkab commented Oct 23, 2022

After spending hours earlier this week in order to rewrite Python initialization for 3.10 and failing, I came up with a quick workaround for the importing modules installed in the virtual environment when using Python 3.10 and PyQ:

$ pyq
Python 3.10.5 (main, Jul 25 2022, 15:52:08) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import markdown
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/.virtualenvs/3/lib/python3.10/site-packages/pyq/__init__.py", line 820, in __import__
    m = _imp(name, globals, locals, fromlist, level)
ModuleNotFoundError: No module named 'markdown'

Unfortunately, due to Python Initialization changes in 3.10, sys.path is not properly initialized when Python is loaded from kdb+. Dirty workaround is to extend sys.path to what it should have been:

>>> import sys
>>> sys.path.extend(['/home/user/.virtualenvs/3/lib/python3.10/site-packages', '/usr/lib/python3.10/site-packages'])
>>> import markdown
>>> markdown.__version__
'3.3.4'

Basically, first path is your $VIRTUAL_ENV/lib/python3.10/site-packages, and second path is where system python 3 packages are installed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants