{lib}[foss/2023a] CuPy v12.3.0 w/ CUDA 12.1.1

[ThomasHoffmann77]

(created using eb --new-pr)

[schiotz]

@ThomasHoffmann77

I am very interested in this one, so I tried to build this, but it fails with

== 2024-03-12 11:38:52,127 easyconfig.py:2034 WARNING Unable to resolve template value CFLAGS="$CFLAGS -Wno-error=deprecated-declarations" CUPY_NUM_BUILD_JOBS=%(parallel)s  CUPY_NVCC_GENERATE_CODE="$(echo '%(cuda_cc_cmake)s'|sed -e "s/[0-9]*/arch=compute_&,code=sm_&/g")" with dict {'arch': 'x86_64', 'sysroot': '', 'nameletter': 'c', 'toolchain_name': 'foss', 'toolchai
n_version': '2023a', 'version_major': '12', 'version_minor': '3', 'version_major_minor': '12.3', 'pyver': '3.11.3', 'pymajver': '3', 'pyminver': '11', 'pyshortver': '3.11', 'cudaver': '12.1.1', 'cudamajver': '12', 'cudaminver': '1', 'cudashortver': '12.1', 'bitbucket_account': 'cupy', 'github_account': 'cupy', 'name': 'cupy', 'version': '12.3.0', 'versionsuffix': '-CUDA-12.1.1', 'versionprefix': '', 'namelower': 'cupy', 'nameletterlower': 'c', 'mpi_cmd_prefix': 'mpirun -n 1', 'module_name': None, 'builddir': '/scratch/schiotz/eb_build/CuPy/12.3.0/foss-2023a-CUDA-12.1.1', 'installdir': '/home/niflheim/schiotz/easybuild_2023a/icelake/software/CuPy/12.3.0-foss-2023a-CUDA-12.1.1', 'parallel': '112', 'start_dir': '/scratch/schiotz/eb_build/CuPy/12.3.0/foss-2023a-CUDA-12.1.1/cupy/cupy-12.3.0'}
== 2024-03-12 11:38:52,127 run.py:247 INFO running cmd: export PYTHONPATH=/tmp/eb-qmj83xaq/tmp_sb5pv7k/lib/python3.11/site-packages:$PYTHONPATH && CFLAGS="$CFLAGS -Wno-error=deprecated-declarations" CUPY_NUM_BUILD_JOBS=%(parallel)s  CUPY_NVCC_GENERATE_CODE="$(echo '%(cuda_cc_cmake)s'|sed -e "s/[0-9]*/arch=compute_&,code=sm_&/g")" /home/modules/software/Python/3.11.3-GCCcore-12.3.0/bin/python -m pip install --prefix=/tmp/eb-qmj83xaq/tmp_sb5pv7k  --no-deps  --ignore-installed  --no-index  --no-build-isolation  . 
== 2024-03-12 11:38:52,190 build_log.py:171 ERROR EasyBuild crashed with an error (at easybuild/base/exceptions.py:126 in __init__): cmd "export PYTHONPATH=/tmp/eb-qmj83xaq/tmp_sb5pv7k/lib/python3.11/site-packages:$PYTHONPATH && CFLAGS="$CFLAGS -Wno-error=deprecated-declarations" CUPY_NUM_BUILD_JOBS=%(parallel)s  CUPY_NVCC_GENERATE_CODE="$(echo '%(cuda_cc_cmake)s'|sed -e "s/[0-9]*/arch=compute_&,code=sm_&/g")" /home/modules/software/Python/3.11.3-GCCcore-12.3.0/bin/python -m pip install --prefix=/tmp/eb-qmj83xaq/tmp_sb5pv7k  --no-deps  --ignore-installed  --no-index  --no-build-isolation  ." exited with exit code 1 and output:
/bin/bash: -c: line 0: syntax error near unexpected token `('

The problem seems to be the lines

_cupy_preininstopts = [
    'CFLAGS="$CFLAGS -Wno-error=deprecated-declarations"',
    'CUPY_NUM_BUILD_JOBS=%(parallel)s ',
    'CUPY_NVCC_GENERATE_CODE=\"$(echo \'%(cuda_cc_cmake)s\'|sed -e "s/[0-9]*/arch=compute_&,code=sm_&/g")\"'
]

where the %(cuda_cc_cmake)s macro is not defined. What is that line trying to accomplish?

[ThomasHoffmann77]

where the %(cuda_cc_cmake)s macro is not defined.
you need to use eb --cuda-compute-capabilities=...

[schiotz]

With --cuda-compute-capabilities=8.0 it builds and mostly passes the self test, there are however two tests that fail. I don't know if that represents a serious issue.

And thank you very much for making this, I had a go at it myself a few months ago, but was nowhere near getting it to work.

=================================== FAILURES ===================================
__________ TestArgsort_param_0_{external=False}.test_argsort_one_dim ___________
/tmp/eb-wgw3svhq/tmpym9kpuvg/lib/python3.11/site-packages/cupy/testing/_loops.py:843: in test_func
    impl(*args, **kw)
/tmp/eb-wgw3svhq/tmpym9kpuvg/lib/python3.11/site-packages/cupy/testing/_loops.py:362: in test_func
    check_func(cupy_r, numpy_r)
/tmp/eb-wgw3svhq/tmpym9kpuvg/lib/python3.11/site-packages/cupy/testing/_loops.py:666: in check_func
    _array.assert_array_equal(x, y, err_msg, verbose, strides_check)
/tmp/eb-wgw3svhq/tmpym9kpuvg/lib/python3.11/site-packages/cupy/testing/_array.py:91: in assert_array_equal
    numpy.testing.assert_array_equal(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (<built-in function eq>, array([9, 4, 6, 0, 3, 2, 5, 1, 7, 8]), array([9, 6, 4, 0, 3, 5, 2, 1, 7, 8]))
kwds = {'err_msg': '', 'header': 'Arrays are not equal', 'strict': False, 'verbose': True}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           AssertionError: 
E           Arrays are not equal
E           
E           Mismatched elements: 4 / 10 (40%)
E           Max absolute difference: 3
E           Max relative difference: 1.5
E            x: array([9, 4, 6, 0, 3, 2, 5, 1, 7, 8])
E            y: array([9, 6, 4, 0, 3, 5, 2, 1, 7, 8])

/home/modules/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/contextlib.py:81: AssertionError
----------------------------- Captured stdout call -----------------------------
dtype is <class 'numpy.int32'>
___________ TestArgsort_param_1_{external=True}.test_argsort_one_dim ___________
/tmp/eb-wgw3svhq/tmpym9kpuvg/lib/python3.11/site-packages/cupy/testing/_loops.py:843: in test_func
    impl(*args, **kw)
/tmp/eb-wgw3svhq/tmpym9kpuvg/lib/python3.11/site-packages/cupy/testing/_loops.py:362: in test_func
    check_func(cupy_r, numpy_r)
/tmp/eb-wgw3svhq/tmpym9kpuvg/lib/python3.11/site-packages/cupy/testing/_loops.py:666: in check_func
    _array.assert_array_equal(x, y, err_msg, verbose, strides_check)
/tmp/eb-wgw3svhq/tmpym9kpuvg/lib/python3.11/site-packages/cupy/testing/_array.py:91: in assert_array_equal
    numpy.testing.assert_array_equal(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (<built-in function eq>, array([9, 4, 6, 0, 3, 2, 5, 1, 7, 8]), array([9, 6, 4, 0, 3, 5, 2, 1, 7, 8]))
kwds = {'err_msg': '', 'header': 'Arrays are not equal', 'strict': False, 'verbose': True}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           AssertionError: 
E           Arrays are not equal
E           
E           Mismatched elements: 4 / 10 (40%)
E           Max absolute difference: 3
E           Max relative difference: 1.5
E            x: array([9, 4, 6, 0, 3, 2, 5, 1, 7, 8])
E            y: array([9, 6, 4, 0, 3, 5, 2, 1, 7, 8])

/home/modules/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/contextlib.py:81: AssertionError
----------------------------- Captured stdout call -----------------------------
dtype is <class 'numpy.int32'>
=============================== warnings summary ===============================

[ThomasHoffmann77]

Test report by @ThomasHoffmann77
SUCCESS
Build succeeded for 6 out of 6 (6 easyconfigs in total)
srv-mahamid-01.embl.de - Linux AlmaLinux 8.8, x86_64, AMD EPYC 7513 32-Core Processor, 2 x NVIDIA NVIDIA GeForce RTX 3090, 535.113.01, Python 3.6.8
See https://gist.github.com/ThomasHoffmann77/c0551c3270c55c2a69259fef50109d01 for a full test report.

[schiotz]

I see the error I posted above on two different architectures, both RTX3090 and A100.

I lean towards just installing with --ignore-test-failure

[Micket]

I had missed this when i made the CuPy 13 easyconfig. The only practical difference is that mine missed out on Optuna. Not very familiar with this library, though I see it was necessary for Optuna.

Would you mind splitting out just Optuna here as a start to get that merged, then we can test adding it into CuPy-13.0.0...eb?

[ThomasHoffmann77]

I had missed this when i made the CuPy 13 easyconfig. The only practical difference is that mine missed out on Optuna. Not very familiar with this library, though I see it was necessary for Optuna.

Would you mind splitting out just Optuna here as a start to get that merged, then we can test adding it into CuPy-13.0.0...eb?

@Micket #20264

[jfgrimm]

closing since #20264 and #20203 have been merged