Debian packaging for V0.16.2

[rolandmas]

Describe the bug
The "signal" symbol seems to be conflicting with the standard library's "signal" module.

To Reproduce
Running the testsuite in a clean Debian unstable chroot, as part of building the .deb package.

Expected behavior
Testsuite proceeds and succeeds :-)

Screenshots
Here's the part where the package build runs the testsuite:

make[1]: Entering directory '/build/datalab-0.16.1'
set -e;	for v in $(py3versions -vs) ; do HOME=$(mktemp -d);trap "rm -rf $HOME" EXIT;cd /build/datalab-0.16.1/.pybuild/cpython3_${v}_datalab/build ; PYTHONPATH=. xvfb-run python$v ./cdl/tests/all_tests.py && rm -r $HOME ; done
QStandardPaths: error creating runtime directory '/run/user/1001' (No such file or directory)
QStandardPaths: error creating runtime directory '/run/user/1001' (No such file or directory)
Qt: Session management error: None of the authentication protocols specified are supported
Traceback (most recent call last):
  File "/build/datalab-0.16.1/.pybuild/cpython3_3.11_datalab/build/cdl/tests/features/tour_unit_test.py", line 22, in <module>
    test_tour()
  File "/build/datalab-0.16.1/.pybuild/cpython3_3.11_datalab/build/cdl/tests/features/tour_unit_test.py", line 17, in test_tour
    with cdltest_app_context() as win:
  File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
   File "/build/datalab-0.16.1/.pybuild/cpython3_3.11_datalab/build/cdl/tests/__init__.py", line 56, in cdltest_app_context
    win = CDLMainWindow(console=console)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/build/datalab-0.16.1/.pybuild/cpython3_3.11_datalab/build/cdl/core/gui/main.py", line 193, in __init__
    self.setup(console)
  File "/build/datalab-0.16.1/.pybuild/cpython3_3.11_datalab/build/cdl/core/gui/main.py", line 699, in setup
    self.__add_signal_image_panels()
  File "/build/datalab-0.16.1/.pybuild/cpython3_3.11_datalab/build/cdl/core/gui/main.py", line 909, in __add_signal_image_panels
    cdock = self.__add_dockwidget(self.__add_signal_panel(), title=_("Signal View"))
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/build/datalab-0.16.1/.pybuild/cpython3_3.11_datalab/build/cdl/core/gui/main.py", line 863, in __add_signal_panel
    self.signalpanel = signal.SignalPanel(self, dpw, self.signalpanel_toolbar)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/build/datalab-0.16.1/.pybuild/cpython3_3.11_datalab/build/cdl/core/gui/panel/signal.py", line 75, in __init__
    self.processor = SignalProcessor(self, dockableplotwidget.plotwidget)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/build/datalab-0.16.1/.pybuild/cpython3_3.11_datalab/build/cdl/core/gui/processor/base.py", line 176, in __init__
    self.set_process_isolation_enabled(Conf.main.process_isolation_enabled.get())
  File "/build/datalab-0.16.1/.pybuild/cpython3_3.11_datalab/build/cdl/core/gui/processor/base.py", line 193, in set_process_isolation_enabled
    self.worker.create_pool()
  File "/build/datalab-0.16.1/.pybuild/cpython3_3.11_datalab/build/cdl/core/gui/processor/base.py", line 95, in create_pool
    POOL = Pool(processes=1)  # pylint: disable=not-callable,consider-using-with
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 191, in __init__
    self._setup_queues()
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 346, in _setup_queues
    self._inqueue = self._ctx.SimpleQueue()
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/context.py", line 113, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/queues.py", line 341, in __init__
    self._rlock = ctx.Lock()
                  ^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/synchronize.py", line 169, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/usr/lib/python3.11/multiprocessing/synchronize.py", line 79, in __init__
    from .resource_tracker import register
  File "/usr/lib/python3.11/multiprocessing/resource_tracker.py", line 30, in <module>
    _IGNORED_SIGNALS = (signal.SIGINT, signal.SIGTERM)
                        ^^^^^^^^^^^^^
AttributeError: module 'signal' has no attribute 'SIGINT'

Installation information
Building from the 0.16.1 release, no patches.

Additional context
I suspect that the "signal" module that "multiprocessing" tries to use is actually one provided by Datalab rather than the one in the Python standard library. So either something's not tight enough when isolating the scope of the imports, or maybe it would be better to rename the "signal" submodules in Datalab.

[PierreRaybaut]

Hi @rolandmas, thanks for the feedback!

I suspect that this AttributeError: module 'signal' has no attribute 'SIGINT' is only a side effect: this is not what's causing the crash. Indeed, I never saw such name conflict on any of the test platforms (Python 3.8 to 3.12, Centos, Ubuntu, Windows - so quite a lot).

I think that we should begin by fixing this part of the issue:

QStandardPaths: error creating runtime directory '/run/user/1001' (No such file or directory)
QStandardPaths: error creating runtime directory '/run/user/1001' (No such file or directory)
Qt: Session management error: None of the authentication protocols specified are supported

Apparently, this is caused by the fact that the XDG_RUNTIME_DIR environment variable is set to a problematic directory (here /run/user/1001): nonexistent or with wrong permissions?

[PierreRaybaut]

BTW, the "cdl\tests\features\tour_unit_test.py" test is not the first test of the DataLab test suite.
Would it be possible to copy/paste here the full log to better understand what's going on?

Otherwise, I will need to be able to reproduce the build to debug, like we did on the PlotPy Issue.

[PierreRaybaut]

I'm ready to rename the module into signals to avoid this conflict because I think that it's masking the real issue here. But I'm quite convinced that this won't solve the issue - but, it may help understanding it and solving it, so...

[PierreRaybaut]

However, the impact of the renaming is quite significant (for consistency, a lot of modules would be renamed as well), so if you would help me diagnose the problem, it would be great.

[rolandmas]

Thanks for the XDG_RUNTIME_DIR idea, I'll investigate it before any other and keep you informed.

[PierreRaybaut]

Ok, thanks @rolandmas .

BTW, I think that the "signal" module that should be renamed just for testing purpose is this one: "cdl/tests/features/signal".
You may rename this folder without precaution (because modules inside of it are not called from elsewjere in the code base), just to see what happens.

[PierreRaybaut]

My previous comment may still be relevant.

However, in the meantime, I though about one thing that might be the cause.
The test suite is configured for pytest thanks to the conftest.py file which is in the cdl/tests directory.
If this configuration is not taken into account, then it could lead to ignore the fact that it forces pytest to reset the working directory after executing each test. Then, if the working directory is cdl/tests/features when executing tour_unit_test.py, it would explain the signal module name conflict because one of the subpackage in this folder is precisely named signal.

To check if conftest.py has been taken into account, there is an easy way:

1. If conftest.py is loaded by pytest, you would see this header:

============================================================================================= test session starts ==============================================================================================
platform win32 -- Python 3.11.6, pytest-8.0.2, pluggy-1.4.0
DataLab 0.16.2 [Plugins: Données de test 1.0.0]
guidata 3.5.0, PlotPy 2.3.2
PythonQwt 0.12.1, PyQt5 5.15.10 [Qt version: 5.15.2]
NumPy 1.26.2, SciPy 1.11.4, h5py 3.10.0, scikit-image 0.22.0, OpenCV 4.8.1
rootdir: C:\Dev\DataLab
configfile: pyproject.toml
collected 206 items

2. If it is not taken into account, you would see the standard header:

============================================================================================= test session starts ==============================================================================================
platform win32 -- Python 3.11.6, pytest-8.0.2, pluggy-1.4.0
rootdir: C:\Dev\DataLab
configfile: pyproject.toml
collected 206 items

If what you see is described in point 2., then you should try to add the folder name after the pytest command (e.g., pytest cdl/tests).

[PierreRaybaut]

FYI, I've renamed the submodule 'signal' to 'signals' in DataLab V0.16.2, so that this specific error will no longer occur (even though I suspect that it's not the real problem here).

[rolandmas]

I haven't seen the "signal" problem yet (build is still running), but now I get the following error:
ImportError: cannot import name 'AboutTool' from partially initialized module 'plotpy.tools' (most likely due to a circular import) (/usr/lib/python3/dist-packages/plotpy/tools/init.py)
(With plotpy 2.4.0 and datalab 0.16.2)

[PierreRaybaut]

This may (again) be a working directory issue, because some modules are not meant to be executed from some directory (it's sometimes difficult to cover all the conflict import combinations).

So, without any execution context, I can't tell you more about this.

[PierreRaybaut]

In other words, a copy/paste of the original log would certainly help.

[PierreRaybaut]

I'm in the verge of releasing a new bugfix version (V0.16.2).
So now would really be the time to fix all those remaining issues once and for all: I'm available for assisting you, please do not hesitate...

[rolandmas]

Are you on IRC or Matrix by any chance?

[rolandmas]

Current build (in progress):

   debian/rules override_dh_auto_test
make[1]: Entering directory '/build/datalab-0.16.2'
set -e;	for v in $(py3versions -vs) ; do HOME=$(mktemp -d);trap "rm -rf $HOME" EXIT;cd /build/datalab-0.16.2/.pybuild/cpython3_${v}_datalab/build ; PYTHONPATH=. XDG_RUNTIME_DIR=$HOME xvfb-run python$v ./cdl/tests/all_tests.py && rm -r $HOME ; done
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Traceback (most recent call last):
  File "/build/datalab-0.16.2/.pybuild/cpython3_3.11_datalab/build/cdl/tests/features/images/imagetools_app_test.py", line 16, in <module>
    from plotpy.tools import CrossSectionTool
  File "/usr/lib/python3/dist-packages/plotpy/tools/__init__.py", line 24, in <module>
    from .cross_section import (
  File "/usr/lib/python3/dist-packages/plotpy/tools/cross_section.py", line 20, in <module>
    from plotpy.tools.image import update_image_tool_status
  File "/usr/lib/python3/dist-packages/plotpy/tools/image.py", line 37, in <module>
    from plotpy.plot import BasePlot
  File "/usr/lib/python3/dist-packages/plotpy/plot/__init__.py", line 7, in <module>
    from .manager import PlotManager
  File "/usr/lib/python3/dist-packages/plotpy/plot/manager.py", line 16, in <module>
    from plotpy.tools import (
ImportError: cannot import name 'AboutTool' from partially initialized module 'plotpy.tools' (most likely due to a circular import) (/usr/lib/python3/dist-packages/plotpy/tools/__init__.py)
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
[…]