Add a patch that enable to use libraries exposed via LIBRARY_PATH or by dagonzalezfo · Pull Request #23042 · easybuilders/easybuild-easyconfigs

dagonzalezfo · 2025-06-05T10:53:00Z

Enabling using LIBRARY_PATH exposed shared libraries will enhance user experience using local installed packages that relies on external libraries. Current solution requires using LD_LIBRARY_PATH or adding the neeeded libraries to python main installation. This is related with an EESSI reported issue.
Further testing and refinement is needed.

relies on prefix provided ldconfig

casparvl · 2025-06-10T10:59:11Z

I'll try to test this patch in a bit, but @ocaisa already remarked it's a "scary big hammer", since it changes a very fundamental standard python library. Previously, when we patched ctypes.util.find_library, we did this as selectively as we could, i.e. only when LD_LIBRARY_PATH is filtered. This means it needed to be done at the python easyblock level, because it involves logic. I think it makes sense to do that again this time - but let me first test if it resolves the issues I'm seeing. In the EESSI support meeting, it came up that LAMMPS with GPU support ran into similar issues, and there was some concern about the libcuda.so stubs library being picked up as a result of this patch, that's another thing that we should test for.

casparvl · 2025-06-10T11:46:20Z

The good news is: the hammer does seem to solve my particular problem.

Before, I got this issue:

  * Found HDF5 headers at ``/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/HDF5/1.14.3-gompi-2023b/include``, library at ``/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/HDF5/1.14.3-gompi-2023b/lib``.
  .. WARNING:: Could not find the HDF5 runtime.
     The HDF5 shared library was *not* found in the default library
     paths. In case of runtime problems, please remember to install it.
  * Found LZO 2 headers at ``/home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen2/software/LZO/2.10-GCCcore-13.2.0/include``, the library is located in the standard system search dirs.
  .. WARNING:: Could not find the LZO 2 runtime.
     The LZO 2 shared library was *not* found in the default library
     paths. In case of runtime problems, please remember to install it.
  * Skipping detection of LZO 1 since LZO 2 has already been found.
  * Found bzip2 headers at ``/usr/include``, library at ``/usr/lib64``.
  * Found blosc headers at ``/home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen2/software/Blosc/1.21.5-GCCcore-13.2.0/include``, the library is located in the standard system search dirs.
  .. WARNING:: Could not find the blosc runtime.
     The blosc shared library was *not* found in the default library
     paths. In case of runtime problems, please remember to install it.
  * Run 'blosc2_find_directories_hook'
  * Found blosc2 headers at ``/home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen2/software/Blosc2/2.13.2-GCCcore-13.2.0/include``, library at ``/home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen2/software/Blosc2/2.13.2-GCCcore-13.2.0/lib``.
    * Copying blosc2 runtime library to 'tables' dir because it was not found in standard locations

Now, PyTables builds succesfully, and I find this in the logs:

  * Using Python 3.11.5 (main, Jun 10 2025, 13:20:30) [GCC 13.2.0]
  * Found cython 3.0.4
  * USE_PKGCONFIG: True
  * Found HDF5 headers at ``/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/HDF5/1.14.3-gompi-2023b/include``, library at ``/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/HDF5/1.14.3-gompi-2023b/lib``.
  * Found LZO 2 headers at ``/home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen2/software/LZO/2.10-GCCcore-13.2.0/include``, the library is located in the standard system search dirs.
  * Skipping detection of LZO 1 since LZO 2 has already been found.
  * Found bzip2 headers at ``/usr/include``, library at ``/usr/lib64``.
  * Found blosc headers at ``/home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen2/software/Blosc/1.21.5-GCCcore-13.2.0/include``, the library is located in the standard system search dirs.
  * Run 'blosc2_find_directories_hook'
  * Found blosc2 headers at ``/home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen2/software/Blosc2/2.13.2-GCCcore-13.2.0/include``, library at ``/home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen2/software/Blosc2/2.13.2-GCCcore-13.2.0/lib``.
  Compiling tables/utilsextension.pyx because it changed.
  [1/1] Cythonizing tables/utilsextension.pyx
  Compiling tables/hdf5extension.pyx because it changed.
  [1/1] Cythonizing tables/hdf5extension.pyx
  Compiling tables/tableextension.pyx because it changed.
  [1/1] Cythonizing tables/tableextension.pyx
  Compiling tables/linkextension.pyx because it changed.
  [1/1] Cythonizing tables/linkextension.pyx
  Compiling tables/_comp_lzo.pyx because it changed.
  [1/1] Cythonizing tables/_comp_lzo.pyx
  Compiling tables/_comp_bzip2.pyx because it changed.
  [1/1] Cythonizing tables/_comp_bzip2.pyx
  Compiling tables/lrucacheextension.pyx because it changed.
  [1/1] Cythonizing tables/lrucacheextension.pyx
  Compiling tables/indexesextension.pyx because it changed.
  [1/1] Cythonizing tables/indexesextension.pyx
  [06/10/25 13:33:50] WARNING  pyproject.toml does not contain a tool.setuptools_scm    setuptools.py:119

Also, before, I got:

module load Blosc2/2.13.2-GCCcore-13.2.0
module load Python/3.11.5-GCCcore-13.2.0
$ python -c "import ctypes; ctypes.CDLL('libblosc2.so')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/Python/3.11.5-GCCcore-13.2.0/lib/python3.11/ctypes/__init__.py", line 376, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: libblosc2.so: cannot open shared object file: No such file or directory

Whereas now I get:

module load Blosc2/2.13.2-GCCcore-13.2.0
module load Python/3.11.5-GCCcore-13.2.0
$ python -c "import ctypes; print(ctypes.CDLL('libblosc2.so'))"
<CDLL '/home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen2/software/Blosc2/2.13.2-GCCcore-13.2.0/lib/libblosc2.so', handle 13748a0 at 0x145c9d1e9990>

As expected

casparvl · 2025-06-10T11:55:31Z

easybuild/easyconfigs/p/Python/Python-3.11.5-custom-ctypes.patch

+
+             try:
+-                proc = subprocess.Popen(('/sbin/ldconfig', '-r'),
+                proc = subprocess.Popen((os.environ.get('EPREFIX') + '/sbin/ldconfig', '-r'),


I think this is another reason why we should resolve things at the EasyBlock level: there, we can query if EasyBuild was configured with a sysroot build option. If so, we get that value, instead of EPREFIX (which I believe is set by Gentoo-prefix, and thus specific to that OS?). That makes this more generic. It also excludes any possible name collisions (someone could set EPREFIX as an environment for a totally different purpose - unrelated to Gentoo-Prefix, but build_option('sysroot') can only mean one thing: someone wanted to set a sysroot).

ocaisa · 2025-06-10T12:27:07Z

@casparvl Can you try

python -c "import ctypes; print(ctypes.CDLL('libcuda.so'))"

with/without the CUDA module loaded on a system with a GPU? I'm curious to see if it picks up the "correct" one

casparvl · 2025-06-10T12:43:16Z

@ocaisa correctly anticipated that this would be a problem:

$ module load CUDA/12.4.0
$ module load Python/3.11.5-GCCcore-13.2.0
$ python -c "import ctypes; print(ctypes.CDLL('libcuda.so'))"
<CDLL '/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/icelake/accel/nvidia/cc80/software/CUDA/12.4.0/stubs/lib64/libcuda.so', handle 13db710 at 0x15413f109c90>

Now, it may or may not actually bite us in practice. If a libxyz.so was built against libcuda.so, and if we import libxyz.so in Python through ctypes, all is probably fine, because the runtime linker will load libcuda.so in the regular way (i.e. no Python shenanigans going on). Since the stubs dir are not on the LD_LIBRARY_PATH, this means the runtime linker will correctly find the driver in the default location (/usr/lib/libcuda.so.1 or something). This (probably) only causes trouble if a code directly loads libcuda.so through the ctypes interface.

One thing we could do - but this is even more invasive - is that we patch ctypes some more and do

library_path = os.getenv('LIBRARY_PATH')
# some code to clean up `stubs` prefixes from `library_path`. Or maybe more specific: clean up any directory matching `CUDA/*/stubs/lib64`
library_path = remove_stubs_from_library_path(library_path)
libpath.append(library_path)

casparvl · 2025-06-10T12:43:49Z

Lol, I was already doing that @ocaisa :D just needed time for the write-up (and got distracted by something on Slack :P)

ocaisa · 2025-06-10T12:46:32Z

For EESSI, this is actually less of a problem as we drop CUDA to a build dependency. This means that at runtime the stubs libcuda.so won't be available via LIBRARY_PATH either.

casparvl · 2025-06-10T12:54:42Z

easybuild/easyconfigs/p/Python/Python-3.11.5-custom-ctypes.patch

+            if name and name.endswith(".so"):
+                s = re.sub(r'lib', '', name)
+                s = re.sub(r'\..*', '', s)
+                self._name=util._findLib_ld(s) 


I'm not sure I understand what's happening here. What issue is this actually resolving? Also, invoking _findLib_ld means we need to make absolutely sure that that searches the correct path (though I guess that's ensured by the changes below as well)

oooh, wait, this is where you overwrite the name with the full path already? I.e. this, together with the change below, makes sure we get something like /home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen2/software/Blosc2/2.13.2-GCCcore-13.2.0/lib/libblosc2.so back, and then the rest of the CDLL call just acts on that full path?

In other words: this is where we are actually pretending the calling code called the function with the full path, thus avoiding the whole issue?

Yes, by overriding self._name

dagonzalezfo · 2025-06-10T14:06:35Z

I will encapsulate all changes using sysroot as condition, as you commented previously.

casparvl · 2025-06-10T14:23:07Z

Summarizing, my suggestion would be:

Comment out the original patch for find_library at https://github.com/easybuilders/easybuild-easyblocks/blob/0f4157641cd21688f48532cd75153ebe65aecb92/easybuild/easyblocks/p/python.py#L351
Replace that with your fix, but done as part of the easyblock
When LD_LIBRARY_PATH is filtered, add libpath.append(os.getenv('LIBRARY_PATH'))
When sysroot is set, do proc = subprocess.Popen(build_option('sysroot') + '/sbin/ldconfig', '-r'), and libpath.append(os.path.join(build_option(sysroot), "path", "to", "default", "libs"))
Test if this fix indeed makes the original patch for find_library obsolete (i.e. check that the original issue for that is still resolved). If not, restore that patch.

dagonzalezfo · 2025-06-26T06:54:07Z

Patch migrated to easybuilders/easybuild-easyblocks#3798.
Thus, I will close this PR

Add a patch that enable to use libraries exposed via LIBRARY_PATH or

f750fa4

relies on prefix provided ldconfig

github-actions bot added the change label Jun 5, 2025

casparvl mentioned this pull request Jun 10, 2025

Add patch to fix permission denied issue during PyTables installation. #22975

Closed

casparvl reviewed Jun 10, 2025

View reviewed changes

dagonzalezfo mentioned this pull request Jun 26, 2025

WIP: update Python easyblock to use custom ctypes for shared libraries easybuilders/easybuild-easyblocks#3798

Closed

dagonzalezfo closed this Jun 26, 2025

boegel added this to the 5.x milestone Oct 8, 2025

Conversation

dagonzalezfo commented Jun 5, 2025

Uh oh!

casparvl commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

casparvl commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

casparvl Jun 10, 2025

Choose a reason for hiding this comment

Uh oh!

ocaisa commented Jun 10, 2025

Uh oh!

casparvl commented Jun 10, 2025

Uh oh!

casparvl commented Jun 10, 2025

Uh oh!

ocaisa commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

casparvl Jun 10, 2025

Choose a reason for hiding this comment

Uh oh!

casparvl Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dagonzalezfo Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dagonzalezfo commented Jun 10, 2025

Uh oh!

casparvl commented Jun 10, 2025

Uh oh!

dagonzalezfo commented Jun 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

casparvl commented Jun 10, 2025 •

edited

Loading

casparvl commented Jun 10, 2025 •

edited

Loading

ocaisa commented Jun 10, 2025 •

edited

Loading

casparvl Jun 10, 2025 •

edited

Loading

dagonzalezfo Jun 10, 2025 •

edited

Loading