Skip to content

allow unresolved templates in bundle easyblock when handling components#3939

Closed
Thyre wants to merge 1 commit intoeasybuilders:developfrom
Thyre:20250923153721_new_pr_bundle
Closed

allow unresolved templates in bundle easyblock when handling components#3939
Thyre wants to merge 1 commit intoeasybuilders:developfrom
Thyre:20250923153721_new_pr_bundle

Conversation

@Thyre
Copy link
Collaborator

@Thyre Thyre commented Sep 23, 2025

(created using eb --new-pr)

Fixes regression introduced in #3801, where e.g. %(installdir)s in modextravars was causing an error message like:

== FAILED: Installation ended unsuccessfully: Sanity check failed: loading fake module failed: 'Failed to resolve all templates in "%(installdir)s" using 
template dictionary: {\'arch\': \'x86_64\', \'rpath_enabled\': \'true\', \'sysroot\': \'\', \'software_commit\': \'\', \'nameletter\': \'l\', 
\'toolchain_name\': \'GCCcore\', \'toolchain_version\': \'14.3.0\', \'version_major\': \'1\', \'version_minor\': \'7\', \'version_major_minor\': \'1.7\', 
\'pyver\': \'3.13.5\', \'pymajver\': \'3\', \'pyminver\': \'13\', \'pyshortver\': \'3.13\', \'bitbucket_account\': \'libglvnd\', \'github_account\': 
\'libglvnd\', \'name\': \'libglvnd\', \'version\': \'1.7.0\', \'versionsuffix\': \'\', \'versionprefix\': \'\', \'namelower\': \'libglvnd\', 
\'nameletterlower\': \'l\', \'cuda_compute_capabilities\': \'7.5\', \'cuda_cc_space_sep\': \'7.5\', \'cuda_cc_space_sep_no_period\': \'75\', 
\'cuda_cc_semicolon_sep\': \'7.5\', \'cuda_cc_cmake\': \'75\', \'cuda_int_comma_sep\': \'75\', \'cuda_int_space_sep\': \'75\', \'cuda_int_semicolon_sep\': 
\'75\', \'cuda_sm_comma_sep\': \'sm_75\', \'cuda_sm_space_sep\': \'sm_75\', \'module_name\': \'OpenGL/2025.09\'}. This might cause failures or unexpected 
behavior, check for correct escaping if this is intended!' (took 9 secs)

This can be tested by adding e.g. this to a bundle with at least one component:

modextravars = {
    'TEST': '%(installdir)s',
}

Reproducer:

easyblock = 'Bundle'

name = 'Test'
version = '1.0.0'

description = """
Test modextravars with installdir template
"""
homepage = ''

toolchain = SYSTEM

default_easyblock = 'Binary'

components = [
    ('zlib', '1.3.1', {
        'install_cmd': 'echo "Hello World"',
        'source_urls': ['https://zlib.net/fossils/'],
        'sources': [SOURCELOWER_TAR_GZ],
    }),
]

modextravars = {
    'TEST_DIR': '%(installdir)s',
}

moduleclass = 'lib'

eb ./test.eb works, but doing eb --rebuild --module-only ./test.eb afterwards fails with the error shown above.
With this PR, the installation works and the module looks sane:

help([==[

Description
===========
Test modextravars with installdir template


More information
================
 - Homepage: 
]==])

whatis([==[Description: 
Test modextravars with installdir template
]==])
whatis([==[Homepage: ]==])
whatis([==[URL: ]==])

local root = "/tmp/software/Test/1.0.0"

conflict("Test")

prepend_path("CMAKE_PREFIX_PATH", root)
setenv("EBROOTTEST", root)
setenv("EBVERSIONTEST", "1.0.0")
setenv("EBDEVELTEST", pathJoin(root, "easybuild", "Core-Test-1.0.0-easybuild-devel"))

setenv("TEST_DIR", "/tmp/software/Test/1.0.0")
-- Built with EasyBuild version 5.1.2.dev0

@Thyre Thyre added the bug fix label Sep 23, 2025
@Thyre
Copy link
Collaborator Author

Thyre commented Sep 23, 2025

Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS OpenGL-2025.09-GCCcore-14.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
ZAM054 - Linux Zorin OS 17, x86_64, 12th Gen Intel(R) Core(TM) i7-1260P, 1 x NVIDIA NVIDIA GeForce MX550, 580.65.06, Python 3.10.12
See https://gist.github.com/Thyre/9ce0e7216d7850bb26be0ea1df2e487f for a full test report.

@Thyre
Copy link
Collaborator Author

Thyre commented Sep 23, 2025

Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS FlexiBLAS-3.4.5-GCC-14.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
ZAM054 - Linux Zorin OS 17, x86_64, 12th Gen Intel(R) Core(TM) i7-1260P, 1 x NVIDIA NVIDIA GeForce MX550, 580.65.06, Python 3.10.12
See https://gist.github.com/Thyre/371eee7e9cd835aca534d5081ca07789 for a full test report.


With a slightly adapted EasyConfig:

Click to open
easyblock = 'Bundle'

name = 'FlexiBLAS'
version = '3.4.5'

homepage = 'https://gitlab.mpi-magdeburg.mpg.de/software/flexiblas-release'
description = """FlexiBLAS is a wrapper library that enables the exchange of the BLAS and LAPACK implementation
used by a program without recompiling or relinking it."""

toolchain = {'name': 'GCC', 'version': '14.3.0'}
local_extra_flags = "-fstack-protector-strong -fstack-clash-protection"
toolchainopts = {'pic': True, 'extra_cflags': local_extra_flags, 'extra_fflags': local_extra_flags}

builddependencies = [
    ('CMake', '4.0.3'),
    ('Python', '3.13.5'),  # required for running the tests
    ('BLIS', '2.0'),
]
if ARCH == 'x86_64':
    builddependencies.append(('AOCL-BLAS', '5.1'))
if ARCH == 'aarch64':
    builddependencies.append(('NVPL', '25.5', '', SYSTEM))

dependencies = [
    ('OpenBLAS', '0.3.30'),
]

# note: first listed backend will be used as default by FlexiBLAS,
# unless otherwise specified via easyconfig parameter flexiblas_default
local_backends = ['OpenBLAS', 'BLIS']

# imkl supplies its backend via the imkl module, not as a dependency
if ARCH == 'x86_64':
    local_backends.extend(['AOCL-BLAS', 'imkl'])
if ARCH == 'aarch64':
    local_backends.extend(['NVPL'])

default_component_specs = {'start_dir': '%(namelower)s-%(version)s'}
sanity_check_all_components = True

# Also build and install LAPACKE, which FlexiBLAS does not support yet
components = [
    (name, version, {
        'source_urls':
        ['https://gitlab.mpi-magdeburg.mpg.de/api/v4/projects/386/packages/generic/flexiblas-source/v%(version)s/'],
        'sources': [SOURCELOWER_TAR_GZ],
        'checksums': ['e819949c614c4968919b0ea4e873ab916d95cdc6943e9d091a78d209b7d6ed07'],
        'backends': local_backends,
    }),
    ('LAPACK', '3.12.1', {
        'easyblock': 'CMakeMake',
        'source_urls': ['https://github.com/Reference-LAPACK/lapack/archive/'],
        'sources': ['v%(version)s.tar.gz'],
        'checksums': ['2ca6407a001a474d4d4d35f3a61550156050c48016d949f0da0529c0aa052422'],
        'configopts': ('-DBUILD_SHARED_LIBS=ON -DUSE_OPTIMIZED_BLAS=ON -DLAPACKE=ON '
                       '-DUSE_OPTIMIZED_LAPACK=ON -DBUILD_DEPRECATED=ON '
                       '-DCMAKE_INSTALL_INCLUDEDIR=%(installdir)s/include/flexiblas'),
        'sanity_check_paths': {
            'files': ['lib/liblapacke.%s' % SHLIB_EXT, 'include/flexiblas/lapacke.h'],
            'dirs': [],
        },
    }),
]

modextravars = {
    'TEST': '%(installdir)s',
}

moduleclass = 'lib'

@Thyre
Copy link
Collaborator Author

Thyre commented Sep 23, 2025

@boegelbot please test @ jsc-zen3
EB_ARGS="purge_dups-1.2.6-foss-2023b.eb FlexiBLAS-3.4.5-GCC-14.2.0.eb matplotlib-3.8.2-gfbf-2023b.eb --installpath /tmp/$USER/ebpr-3939"

@Thyre
Copy link
Collaborator Author

Thyre commented Sep 23, 2025

@boegelbot please test @ jsc-zen3
EB_ARGS="buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb gfbf-2024a.eb crypt4gh-1.7-GCC-12.3.0.eb Python-3.11.3-GCCcore-12.3.0.eb setuptools-rust-1.6.0-GCCcore-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb X11-20240607-GCCcore-13.3.0.eb --installpath=/tmp/$USER/ebpr-3939-2"

@boegelbot
Copy link

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3939 EB_ARGS="buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb gfbf-2024a.eb crypt4gh-1.7-GCC-12.3.0.eb Python-3.11.3-GCCcore-12.3.0.eb setuptools-rust-1.6.0-GCCcore-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb X11-20240607-GCCcore-13.3.0.eb --installpath=/tmp/$USER/ebpr-3939-2" EB_CONTAINER= EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3939 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 8038

Test results coming soon (I hope)...

Details

- notification for comment with ID 3324101885 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

comp.make_module_step(fake=True)
# Set fake arg to True, as module components should not try to create their own module
with comp.cfg.allow_unresolved_templates():
comp.make_module_step(fake=True)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How can this be correct? If this tries to create a module without being able to resolve templates that unresolved template ends up in the module, doesn't it?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably don't need it in both places, and just having it in the make_module_step should be sufficient. Needs testing though.
The resulting module still looks fine, so this looks like some weirdness with how our Bundle EasyBlock is working.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed: The result is ignored, so we can just not copy the parent parameters: #3940

@Thyre
Copy link
Collaborator Author

Thyre commented Sep 23, 2025

Superseded by #3940

@Thyre Thyre closed this Sep 23, 2025
@boegel boegel added this to the next release (5.1.2) milestone Sep 23, 2025
@boegelbot
Copy link

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS buildenv-default-foss-2023b.eb
  • SUCCESS OpenSSL-3.eb
  • SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
  • SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb
  • SUCCESS gfbf-2024a.eb
  • SUCCESS Python-3.11.3-GCCcore-12.3.0.eb
  • SUCCESS setuptools-rust-1.6.0-GCCcore-12.3.0.eb
  • SUCCESS jiter-0.4.1-GCCcore-12.3.0.eb
  • SUCCESS X11-20240607-GCCcore-13.3.0.eb
  • SUCCESS crypt4gh-1.7-GCC-12.3.0.eb

Build succeeded for 10 out of 10 (10 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/fee04ed7513bea0959f6bf21c2f10688 for a full test report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants