Skip to content

Conversation

@Flamefire
Copy link
Contributor

$PYTHONPATH takes precedence which might lead to unexpected or wrong packages being used especially when a dependency includes a newer package than another dependency.
As it is very hard to find this as the cause print a warning when this situation is found.

I ran into this while building scipy of the SciPy-bundle which includes patches that require a newer Cython as the build dependency then what is available in the Python-bundle-PyPI module. As only Python-bundle used $PYTHONPATH the package from the Cython module was ignored even though which cython points to the right binary.

The error wasn't helpful:

Error compiling Cython file:
------------------------------------------------------------
...
    cdef __Pyx_memviewslice memslice
    cdef Py_ssize_t itemsize
    cdef bint dtype_signed
    cdef char kind
    itemsize = -1
    cdef bint ___pyx_const int8_is_signed
                          ^
------------------------------------------------------------

(tree fragment):16:27: Syntax error in C variable declaration

The least we can do is to detect such issues and warn the user.

The check is triggered for direct dependencies when both $PYTHONPATH and $EBPYTHONPREFIXES are set and the Python module is loaded.
I only check for modules that have a lib/python*/site-packages folder and specifically exclude the Python module which sets $PYTHONPATH for the site-customize.py
This should catch most false positives while still catching enough real issues

@Flamefire Flamefire force-pushed the check-ebpythonprefixes branch 2 times, most recently from 2777a95 to 2323b76 Compare November 28, 2024 13:17
@boegel boegel changed the title Print a warning when mixing $EBPYTHONPREFIXES and $PYTHONPATH modules Print a warning when mixing $EBPYTHONPREFIXES and $PYTHONPATH modules Dec 4, 2024
@boegel boegel added this to the 4.x milestone Dec 4, 2024
conflicts = [dep['name'] for dep in self.cfg.dependencies() if is_in_pythonpaths(dep)]
if conflicts:
print_warning('The following modules set $PYTHONPATH but $%s is also used: %s\n'
'Mixing such modules might cause unexpected results during the build of Python packages. '
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we use problems instead of results here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to be more generic because the result may not be a problem but still not what is expected or even intended

@boegel
Copy link
Member

boegel commented Dec 4, 2024

Looks OK to me at first glance, but I would like @Micket to pitch in here too...

@boegel
Copy link
Member

boegel commented Dec 18, 2024

ping @Micket

Micket
Micket previously approved these changes Dec 18, 2024
Copy link
Contributor

@Micket Micket left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks fine to me, i'll upload a test report

@Micket
Copy link
Contributor

Micket commented Dec 18, 2024

Hm, i wanted to test this in conjunction with --prefer-python-search-path, it would help me if this targeted 5.0.x instead for testing.

@Flamefire Flamefire changed the base branch from develop to 5.0.x December 19, 2024 07:43
@Flamefire Flamefire dismissed Micket’s stale review December 19, 2024 07:43

The base branch was changed.

@Flamefire Flamefire force-pushed the check-ebpythonprefixes branch from 2323b76 to df5a827 Compare December 19, 2024 07:44
@Flamefire
Copy link
Contributor Author

Rebased and retargeted to 5.0x

@boegel boegel changed the base branch from 5.0.x to develop March 19, 2025 11:12
@boegel boegel modified the milestones: 4.x, 5.x Mar 19, 2025
@boegel
Copy link
Member

boegel commented Mar 19, 2025

@Flamefire I changed to target branch in this PR from 5.0.x to develop, you should synchronize your PR branch with current develop branch (which has received a massive update after the release of EasyBuild v5.0.0, see #3670)

@boegel boegel modified the milestones: 5.x.x, 5.x Apr 9, 2025
$PYTHONPATH takes precedence which might lead to unexpected or wrong
packages being used especially when a dependency includes a newer
package than another dependency.
As it is very hard to find this as the cause print a warning when this
situation is found.
@Flamefire Flamefire force-pushed the check-ebpythonprefixes branch from 44104d5 to e5b7430 Compare May 23, 2025 09:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants