Skip to content

update easyblocks to let EasyBuild framework take care of prepend to $PYTHONPATH or $EBPYTHONPREFIXES + make PythonBundle and PythonPackage aware of --prefer-python-search-path EasyBuild configuration option#3343

Merged
boegel merged 10 commits intoeasybuilders:5.0.xfrom
Micket:ebpythonprefixes
Oct 2, 2024
Merged

Conversation

@Micket
Copy link
Copy Markdown
Contributor

@Micket Micket commented May 29, 2024 •
edited by boegel
Loading

@Micket Micket added feature EasyBuild-5.0 EasyBuild 5.0 labels May 29, 2024
@Micket Micket added this to the 5.0 milestone May 29, 2024
@Micket Micket changed the title Use new prefer_ebpythonprefixes Use prefer_ebpythonprefixes flag to use EBPYTHONPREFIXES May 29, 2024
@Micket Micket mentioned this pull request Jul 7, 2024
Merged
@Micket
Copy link
Copy Markdown
Contributor Author

Micket commented Aug 5, 2024 •
edited
Loading

special easyblocks that have been directly modified:

  • amber - Amber-22.0-foss-2021b-AmberTools-22.3.eb - no source
  • cplex - CPLEX-22.1.1-GCCcore-11.2.0.eb - no source
  • gurobi - Gurobi-11.0.0-GCCcore-12.3.0.eb
  • lammps - LAMMPS-2Aug2023_update2-foss-2023a-kokkos.eb
  • mxnet - MXNet-1.9.1-foss-2022a.eb
  • openbabel - OpenBabel-3.1.1-gompi-2023a.eb
  • opencv - OpenCV-4.6.0-foss-2022a-contrib.eb
  • qscintilla - QScintilla-2.11.6-GCCcore-11.2.0.eb
  • tkinter - Tkinter-3.11.5-GCCcore-13.2.0.eb

PythonPackage or PythonBundle or subclasses indirectly modified:

  • Python - Python-3.11.5-GCCcore-13.2.0.eb
  • PythonPackage - PyYAML-6.0.1-GCCcore-13.2.0.eb
  • PythonBundle - Python-bundle-PyPI-2024.06-GCCcore-13.3.0.eb
  • CargoPythonPackage - cryptography-41.0.5-GCCcore-13.2.0.eb
  • CargoPythonBundle - pydantic-2.5.3-GCCcore-12.3.0.eb
  • wxpython - wxPython-4.2.1-foss-2022a.eb
  • dm_reverb - dm-reverb-0.7.0-foss-2021b.eb
  • torchvision - torchvision-0.16.0-foss-2023a.eb
  • tensorrt - TensorRT-8.6.1-foss-2022a-CUDA-11.7.0.eb
  • tensorflow - TensorFlow-2.9.1-foss-2022a.eb
  • tensorflow_compression - tensorflow-compression-2.11.0-foss-2022a-CUDA-11.7.0.eb
  • numpy - part of testing Scipy bundle
  • numexpr - numexpr-2.8.4-foss-2022a.eb
  • netcdf4_python - netcdf4-python-1.6.5-foss-2023b.eb
  • configuremakepythonpackage - PyQt5-5.15.7-GCCcore-12.2.0.eb
  • cmakepythonpackage - vigra-1.11.2-foss-2023a.eb
  • fortranpythonpackage - inherited by scipy easyblock
  • libxml2 - libxml2-2.12.7-GCCcore-13.3.0.eb
  • pybind11 - pybind11-2.11.1-GCCcore-13.2.0.eb
  • pyzmq - PyZMQ-25.1.2-GCCcore-13.2.0.eb (is this even using pyzmq? i don't think so)
  • sepp - SEPP-4.5.1-foss-2022b.eb
  • sympy - sympy-1.12-gfbf-2023b.eb
  • scipy - SciPy-bundle-2023.11-gfbf-2023b.eb
  • eqqlib - EggLib-3.3.0-GCC-13.2.0.eb
  • easybuildmeta - EasyBuild-4.9.2.eb
  • jaxlib - jax-0.4.4-foss-2022a.eb

Nothing to test:

  • vsc_tools - nothing?
  • cryptography - not in use anymore?
  • ufc - nothing
  • nose - nothing? nose3 uses pythonpackage directly.
  • dolfin - DOLFIN-xxx.eb (nothing remains)
  • versionindependentpythonpackage - vsc-base (nothing remains)
  • pyquante - PyQuante (nothing remains)
  • python_meep - python-meep (nothing remains)

(todo: make list of specific easyconfigs to test for the generic easyblocks)

@Micket
Copy link
Copy Markdown
Contributor Author

Micket commented Aug 15, 2024

Slight issue with libxml2; the easyblock inherits from PythonPackage, and PythonPackage has always unconditionally added PYTHONPATH/EBPYTHONPREFIXES, so that behavior is retained here.

I could change this; always let the framework add the environment variable if it detects the corresponding path, but this is not done here (yet?)

@Micket Micket force-pushed the ebpythonprefixes branch 4 times, most recently from 2ed73d5 to 1a4bd52 Compare August 15, 2024 22:23
@Micket
Copy link
Copy Markdown
Contributor Author

Micket commented Aug 22, 2024

Test report by @Micket

Overview of tested easyconfigs (in order)

  • SUCCESS Gurobi-11.0.0-GCCcore-12.3.0.eb
  • SUCCESS LAMMPS-2Aug2023_update2-foss-2023a-kokkos.eb
  • SUCCESS MXNet-1.9.1-foss-2022a.eb
  • SUCCESS OpenBabel-3.1.1-gompi-2023a.eb
  • SUCCESS OpenCV-4.6.0-foss-2022a-contrib.eb
  • SUCCESS QScintilla-2.11.6-GCCcore-11.2.0.eb
  • SUCCESS Tkinter-3.11.5-GCCcore-13.2.0.eb

Build succeeded for 7 out of 7 (7 easyconfigs in total)
alvis-skylake-build - Linux Rocky Linux 8.9, x86_64, Intel Xeon Processor (Skylake, IBRS, no TSX), Python 3.6.8
See https://gist.github.com/Micket/9eef93fc27bd9113f744e9a00c06ba64 for a full test report.

@Micket
Copy link
Copy Markdown
Contributor Author

Micket commented Aug 22, 2024

Test report by @Micket

Overview of tested easyconfigs (in order)

  • SUCCESS Python-3.11.5-GCCcore-13.2.0.eb
  • SUCCESS PyYAML-6.0.1-GCCcore-13.2.0.eb
  • SUCCESS Python-bundle-PyPI-2024.06-GCCcore-13.3.0.eb
  • SUCCESS cryptography-41.0.5-GCCcore-13.2.0.eb
  • SUCCESS pydantic-2.5.3-GCCcore-12.3.0.eb

Build succeeded for 5 out of 5 (5 easyconfigs in total)
alvis-skylake-build - Linux Rocky Linux 8.9, x86_64, Intel Xeon Processor (Skylake, IBRS, no TSX), Python 3.6.8
See https://gist.github.com/Micket/6f25e6b6852273bd08bb082839e84420 for a full test report.

@Micket
Copy link
Copy Markdown
Contributor Author

Micket commented Aug 22, 2024

Test report by @Micket

Overview of tested easyconfigs (in order)

  • SUCCESS wxPython-4.2.1-foss-2022a.eb
  • SUCCESS dm-reverb-0.7.0-foss-2021b.eb
  • SUCCESS torchvision-0.16.0-foss-2023a.eb
  • SUCCESS numexpr-2.8.4-foss-2022a.eb
  • SUCCESS PyQt5-5.15.7-GCCcore-12.2.0.eb
  • SUCCESS libxml2-2.12.7-GCCcore-13.3.0.eb
  • SUCCESS pybind11-2.11.1-GCCcore-13.2.0.eb

Build succeeded for 7 out of 7 (7 easyconfigs in total)
vera-skylake-build - Linux Rocky Linux 8.9, x86_64, Intel Xeon Processor (Skylake, IBRS, no TSX), Python 3.6.8
See https://gist.github.com/Micket/e60e25dd2a7ee72182ebded2138fdf1c for a full test report.

@Micket
Copy link
Copy Markdown
Contributor Author

Micket commented Aug 22, 2024

Test report by @Micket

Overview of tested easyconfigs (in order)

  • SUCCESS netcdf4-python-1.6.5-foss-2023b.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
vera-skylake-build - Linux Rocky Linux 8.9, x86_64, Intel Xeon Processor (Skylake, IBRS, no TSX), Python 3.6.8
See https://gist.github.com/Micket/fe7c70f7a98bf61dfc899fb8cc2cebb2 for a full test report.

@Micket
Copy link
Copy Markdown
Contributor Author

Micket commented Aug 22, 2024

Test report by @Micket

Overview of tested easyconfigs (in order)

  • SUCCESS PyZMQ-25.1.2-GCCcore-13.2.0.eb
  • SUCCESS SEPP-4.5.1-foss-2022b.eb
  • SUCCESS sympy-1.12-gfbf-2023b.eb
  • SUCCESS SciPy-bundle-2023.11-gfbf-2023b.eb
  • SUCCESS EggLib-3.3.0-GCC-13.2.0.eb
  • SUCCESS jax-0.4.4-foss-2022a.eb
  • SUCCESS vigra-1.11.2-foss-2023a.eb

Build succeeded for 7 out of 7 (7 easyconfigs in total)
vera-skylake-build - Linux Rocky Linux 8.9, x86_64, Intel Xeon Processor (Skylake, IBRS, no TSX), Python 3.6.8
See https://gist.github.com/Micket/f65845a0504893de88541d44b48432d1 for a full test report.

@Micket
Copy link
Copy Markdown
Contributor Author

Micket commented Aug 22, 2024

Test report by @Micket

Overview of tested easyconfigs (in order)

  • SUCCESS EasyBuild-4.9.2.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
vera-skylake-build - Linux Rocky Linux 8.9, x86_64, Intel Xeon Processor (Skylake, IBRS, no TSX), Python 3.6.8
See https://gist.github.com/Micket/f9a0187ee654cfafbd899010aaf97132 for a full test report.

@Micket
Copy link
Copy Markdown
Contributor Author

Micket commented Aug 22, 2024

Test report by @Micket

Overview of tested easyconfigs (in order)

  • SUCCESS TensorRT-8.6.1-foss-2022a-CUDA-11.7.0.eb
  • SUCCESS TensorFlow-2.9.1-foss-2022a.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
vera-skylake-build - Linux Rocky Linux 8.9, x86_64, Intel Xeon Processor (Skylake, IBRS, no TSX), Python 3.6.8
See https://gist.github.com/Micket/d15524a675bdb83242a8c84e2a042c99 for a full test report.

@Micket
Copy link
Copy Markdown
Contributor Author

Micket commented Aug 22, 2024

Test report by @Micket

Overview of tested easyconfigs (in order)

  • SUCCESS tensorflow-compression-2.11.0-foss-2022a-CUDA-11.7.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis2-01 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz, 1 x NVIDIA Tesla T4, 550.54.14, Python 3.6.8
See https://gist.github.com/Micket/37055a52691ade6faff20ff9cb4e2e0e for a full test report.

@Micket
Copy link
Copy Markdown
Contributor Author

Micket commented Aug 23, 2024

All the tests above ran fine and produced the correct PYTHONPATH.

@Micket Micket force-pushed the ebpythonprefixes branch 2 times, most recently from a663110 to 4ba7735 Compare September 20, 2024 14:45
@boegel
don't use force_pythonpath + clarify why we're keeping logic on $PYTH…
bc1d2b1 
…ONPATH vs $EBPYTHONPREFIXES in PythonBundle + PythonPackage easyblocks
@boegel boegel mentioned this pull request Sep 30, 2024
Merged
@Micket
Merge pull request #4 from boegel/ebpythonprefixes
64d7ec3 
don't use `force_pythonpath` + clarify why we're keeping logic on `$PYTHONPATH` vs `$EBPYTHONPREFIXES` in `PythonBundle` + `PythonPackage` easyblocks
@boegel boegel changed the title Use prefer_ebpythonprefixes flag to use EBPYTHONPREFIXES update easyblock to let EasyBuild framework take care of prepend to $PYTHONPATH or $EBPYTHONPREFIXES + make PythonBundle and PythonPackage aware of prefer_python_search-path Sep 30, 2024
@boegel boegel changed the title update easyblock to let EasyBuild framework take care of prepend to $PYTHONPATH or $EBPYTHONPREFIXES + make PythonBundle and PythonPackage aware of prefer_python_search-path update easyblock to let EasyBuild framework take care of prepend to $PYTHONPATH or $EBPYTHONPREFIXES + make PythonBundle and PythonPackage aware of --prefer-python-search-path EasyBuild configuration option Sep 30, 2024
@boegel
Copy link
Copy Markdown
Member

boegel commented Sep 30, 2024

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS LAMMPS-29Aug2024-foss-2023b-kokkos.eb
  • SUCCESS CPLEX-22.1.1-GCCcore-11.2.0.eb
  • SUCCESS AmberTools-23.6-foss-2023a.eb
  • SUCCESS ESMF-8.6.1-foss-2023b.eb
  • SUCCESS Gurobi-9.5.2-GCCcore-11.3.0.eb
  • FAIL (build issue) OpenBabel-3.1.1-gompi-2023a.eb (partial log available at https://gist.github.com/boegel/cd8ffd13dc9b540fc541ad5404882472)
  • SUCCESS OpenCV-4.8.1-foss-2023a-contrib.eb
  • SUCCESS QScintilla-2.11.6-GCCcore-11.2.0.eb
  • SUCCESS Tkinter-3.11.5-GCCcore-13.2.0.eb

Build succeeded for 8 out of 9 (9 easyconfigs in total)
node3518.doduo.os - Linux RHEL 8.8, x86_64, AMD EPYC 7552 48-Core Processor, Python 3.6.8
See https://gist.github.com/boegel/349e7b66f40e34aa3e278c54272a8a49 for a full test report.

@boegel
Copy link
Copy Markdown
Member

boegel commented Oct 1, 2024

The failure with the OpenBabel tests has nothing to do with the changes in this PR, so I won't let that block this PR...

@boegel
Copy link
Copy Markdown
Member

boegel commented Oct 1, 2024 •
edited
Loading

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS yaff-1.6.0-foss-2023a.eb
  • SUCCESS matplotlib-3.8.2-gfbf-2023b.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
node3518.doduo.os - Linux RHEL 8.8, x86_64, AMD EPYC 7552 48-Core Processor, Python 3.6.8
See https://gist.github.com/boegel/89b050e16724658b56cfa30bab74a7d1 for a full test report.

edit: using standard configuration, so using $PYTHONPATH in generated module files

@boegel
Copy link
Copy Markdown
Member

boegel commented Oct 1, 2024 •
edited
Loading

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS yaff-1.6.0-foss-2023a.eb
  • SUCCESS matplotlib-3.8.2-gfbf-2023b.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
node3518.doduo.os - Linux RHEL 8.8, x86_64, AMD EPYC 7552 48-Core Processor, Python 3.6.8
See https://gist.github.com/boegel/acbbd44d4cfb4e9ea085bf9c65f76114 for a full test report.

edit: using --prefer-python-search-path=EBPYTHONPREFIXES

boegel
boegel approved these changes Oct 2, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@boegel boegel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@boegel boegel merged commit 01c4005 into easybuilders:5.0.x Oct 2, 2024
@boegel boegel changed the title update easyblock to let EasyBuild framework take care of prepend to $PYTHONPATH or $EBPYTHONPREFIXES + make PythonBundle and PythonPackage aware of --prefer-python-search-path EasyBuild configuration option update easyblocks to let EasyBuild framework take care of prepend to $PYTHONPATH or $EBPYTHONPREFIXES + make PythonBundle and PythonPackage aware of --prefer-python-search-path EasyBuild configuration option Mar 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@boegel boegel boegel approved these changes

Assignees

No one assigned

Labels

EasyBuild-5.0 EasyBuild 5.0 enhancement

Projects

None yet

Milestone

5.0.0

Development

Successfully merging this pull request may close these issues.

2 participants

@Micket @boegel