Fix CI builds and add pytests to HDK test suite

[alexbaden]

No description provided.

[alexbaden]

Update: CI builds fixed with #28. Need to add pyhdk tests still. Roughly this should correspond to:

1. Make sure python module is properly installed in Docker container (should just be adding make install to the build process).
2. Currently running pytest from the python/tests directory fails because of JVM Initialization preventing back to back test runs  #32. So, we will need to run pytest xxx.py separately for each test. This should ensure full shutdown and removal of all contexts between each test.

[leshikus]

I've made some progress with enabling python tests, see
https://github.com/intel-ai/hdk/runs/8287088156?check_suite_focus=true

The simple test import pyhdk fails (segfault) when tries to load pyarrow.lib:

__init__.py(65): import pyarrow.lib as _lib
 --- modulename: _bootstrap, funcname: _find_and_load
<frozen importlib._bootstrap>(988):  --- modulename: _bootstrap, funcname: __init__
<frozen importlib._bootstrap>(144): <frozen importlib._bootstrap>(145):  --- modulename: _bootstrap, funcname: __enter__
<frozen importlib._bootstrap>(148):  --- modulename: _bootstrap, funcname: _get_module_lock
<frozen importlib._bootstrap>(163): <frozen importlib._bootstrap>(164): <frozen importlib._bootstrap>(165): <frozen importlib._bootstrap>(166): <frozen importlib._bootstrap>(170): <frozen importlib._bootstrap>(189): <frozen importlib._bootstrap>(191): <frozen importlib._bootstrap>(149):  --- modulename: _bootstrap, funcname: acquire
<frozen importlib._bootstrap>(84): <frozen importlib._bootstrap>(85): <frozen importlib._bootstrap>(86): <frozen importlib._bootstrap>(88): <frozen importlib._bootstrap>(89): <frozen importlib._bootstrap>(90): <frozen importlib._bootstrap>(91): <frozen importlib._bootstrap>(92): <frozen importlib._bootstrap>(101): <frozen importlib._bootstrap>(92): <frozen importlib._bootstrap>(989): <frozen importlib._bootstrap>(990): <frozen importlib._bootstrap>(991):  --- modulename: _bootstrap, funcname: _find_and_load_unlocked
<frozen importlib._bootstrap>(957): <frozen importlib._bootstrap>(958): <frozen importlib._bootstrap>(959): <frozen importlib._bootstrap>(960): <frozen importlib._bootstrap>(963): <frozen importlib._bootstrap>(965): <frozen importlib._bootstrap>(966): <frozen importlib._bootstrap>(967): <frozen importlib._bootstrap>(971):  --- modulename: _bootstrap, funcname: _find_spec
<frozen importlib._bootstrap>(892): <frozen importlib._bootstrap>(893): <frozen importlib._bootstrap>(898): <frozen importlib._bootstrap>(904): <frozen importlib._bootstrap>(905): <frozen importlib._bootstrap>(906):  --- modulename: _bootstrap, funcname: __enter__
<frozen importlib._bootstrap>(865): <frozen importlib._bootstrap>(907): <frozen importlib._bootstrap>(908): <frozen importlib._bootstrap>(914):  --- modulename: _bootstrap, funcname: find_spec
<frozen importlib._bootstrap>(727): <frozen importlib._bootstrap>(728):  --- modulename: _bootstrap, funcname: __exit__
<frozen importlib._bootstrap>(869): <frozen importlib._bootstrap>(915): <frozen importlib._bootstrap>(905): <frozen importlib._bootstrap>(906):  --- modulename: _bootstrap, funcname: __enter__
<frozen importlib._bootstrap>(865): <frozen importlib._bootstrap>(907): <frozen importlib._bootstrap>(908): <frozen importlib._bootstrap>(914):  --- modulename: _bootstrap, funcname: find_spec
<frozen importlib._bootstrap>(802): <frozen importlib._bootstrap>(805):  --- modulename: _bootstrap, funcname: __exit__
<frozen importlib._bootstrap>(869): <frozen importlib._bootstrap>(915): <frozen importlib._bootstrap>(905): <frozen importlib._bootstrap>(906):  --- modulename: _bootstrap, funcname: __enter__
<frozen importlib._bootstrap>(865): <frozen importlib._bootstrap>(907): <frozen importlib._bootstrap>(908): <frozen importlib._bootstrap>(914):  --- modulename: _bootstrap_external, funcname: find_spec
Error: Process completed with exit code 139.

Do you have an idea how to fix that?

[Garra1980]

Is it for docker build?

[leshikus]

Yes, this is a docker build; python tests in conda env were implemented earlier

[Garra1980]

So I guess we need to use some Python mechanisms to set up everything needed to run Python tests. @alexbaden do you know what needs to be done for that?

[alexbaden]

Not specifically, only that we need to determine how to install pyHDK in userspace without requiring Conda. I have a branch here that started down that route: https://github.com/intel-ai/hdk/tree/alex/pyhdk_github_actions

[leshikus]

For Conda build pytest mostly works, it gives the following error:
https://github.com/intel-ai/hdk/actions/runs/3114434766/jobs/5050269248#step:9:25

==================================== ERRORS ====================================
________________________ ERROR collecting test session _________________________
/usr/share/miniconda/envs/omnisci-dev/lib/python3.8/site-packages/pluggy/_manager.py:115: in register
    hook._maybe_apply_history(hookimpl)
/usr/share/miniconda/envs/omnisci-dev/lib/python3.8/site-packages/pluggy/_hooks.py:[30](https://github.com/intel-ai/hdk/actions/runs/3114434766/jobs/5050269248#step:9:31)0: in _maybe_apply_history
    res = self._hookexec(self.name, [method], kwargs, False)
/usr/share/miniconda/envs/omnisci-dev/lib/python3.8/site-packages/pluggy/_manager.py:80: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
modin/modin/conftest.py:303: in pytest_configure
    if config.option.extra_test_parameters is not None:
E   AttributeError: 'Namespace' object has no attribute 'extra_test_parameters'
=============================== warnings summary ===============================```

[alexbaden]

Might be a mismatch between our latest version and the latest jit-engine version. I will see about syncing everything this morning.

[alexbaden]

I opened #83 but did not have to make any changes for the latest jit-engine code. My env uses pytest 7.1.3 and python 3.9.13, but I noticed your error messages are coming from python 3.8. Do we have a minimum python version we require?

[Garra1980]

Only for modin which is python 3.8+