Skip to content

enhance CUDA support in CP2K easyblock (WIP) #2349

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
boegel wants to merge 5 commits into
base: develop
Choose a base branch
from
Open

enhance CUDA support in CP2K easyblock (WIP) #2349

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b2b9dc6
enhance CUDA support in CP2K easyblock
boegel Feb 26, 2021
9b53c58
Merge branch 'develop' into cp2k_cuda
boegel Mar 30, 2021
f71f992
use -arch in NVFLAGS to compile for specific CUDA compute capabilitie…
boegel Mar 30, 2021
664d91b
require a single CUDA compute capability when building CP2K with CUDA…
boegel Mar 30, 2021
68e15c5
wrap compiler command and flags in single quotes in NVFLAGS value
boegel Mar 30, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 54 additions & 5 deletions easybuild/easyblocks/c/cp2k.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ def extra_options():
extra_vars = {
'extracflags': ['', "Extra CFLAGS to be added", CUSTOM],
'extradflags': ['', "Extra DFLAGS to be added", CUSTOM],
'gpuver': [None, "Value for GPUVER configuration setting, specifies type of GPU to build for", CUSTOM],
'ignore_regtest_fails': [False, ("Ignore failures in regression test "
"(should be used with care)"), CUSTOM],
'library': [False, "Also build CP2K as a library", CUSTOM],
Expand All @@ -98,9 +99,9 @@ def extra_options():
'modinc': [[], ("List of modinc's to use (*.f90], or 'True' to use "
"all found at given prefix"), CUSTOM],
'modincprefix': ['', "Intel MKL prefix for modinc include dir", CUSTOM],
'runtest': [True, "Build and run CP2K tests", CUSTOM],
'omp_num_threads': [None, "Value to set $OMP_NUM_THREADS to during testing", CUSTOM],
'plumed': [None, "Enable PLUMED support", CUSTOM],
'runtest': [True, "Build and run CP2K tests", CUSTOM],
'type': ['popt', "Type of build ('popt' or 'psmp')", CUSTOM],
'typeopt': [True, "Enable optimization", CUSTOM],
}
Expand Down Expand Up @@ -228,12 +229,60 @@ def configure_step(self):
else:
options['DFLAGS'] += ' -D__ELPA3'

# CUDA
# CUDA support
# see https://github.com/cp2k/cp2k/blob/master/INSTALL.md#2j-cuda-optional-improved-performance-on-gpu-systems
cuda = get_software_root('CUDA')
if cuda:
options['DFLAGS'] += ' -D__ACC -D__DBCSR_ACC'
options['LIBS'] += ' -lcudart -lcublas -lcufft -lrt'
options['NVCC'] = ' nvcc'
# determine CUDA compute capability to use based on --cuda-compute-capabilities in EasyBuild configuration,
# or cuda_compute_capabilities easyconfig parameter (fallback);
# must be a single value to build CP2K with CUDA support!
cuda_cc = build_option('cuda_compute_capabilities') or self.cfg.get('cuda_compute_capabilities')
if len(cuda_cc) == 1:
cuda_cc = cuda_cc[0]
elif cuda_cc:
error_msg = "Exactly one CUDA compute capability must be specified, found %d: %s"
raise EasyBuildError(error_msg, len(cuda_cc), ', '.join(cuda_cc))
else:
error_msg = "Exactly one CUDA compute capability must be specified via "
error_msg += "--cuda-compute-capabilities or the cuda_compute_capabilities easyconfig parameter."
raise EasyBuildError(error_msg)

# GPUVER must be set, required by the DBCSR component,
# see exts/dbcsr/Makefile and the parameters_*.json in src/acc/libsmm_acc/libcusmm/;
# determine string value to use based on select CUDA compute capability, unless specified explicitly via
# custom 'gpuver' easyconfig parameter
gpuver = self.cfg['gpuver']

if gpuver is None:
cuda_cc_lv = LooseVersion(cuda_cc)
known_gpuver = [
('7.0', 'V100'),
('6.0', 'P100'),
('3.7', 'K80'),
('3.5', 'K40'),
]
for min_cuda_cc, val in known_gpuver:
if cuda_cc_lv >= LooseVersion(min_cuda_cc):
gpuver = val
break

if gpuver is None:
raise EasyBuildError("Failed to determine value for required GPUVER setting!")
else:
options['GPUVER'] = gpuver

options['DFLAGS'] += ' -D__ACC -D__DBCSR_ACC -D__PW_CUDA -D__GRID_CUDA'
Copy link
Copy Markdown
Member Author

@boegel boegel Feb 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need easyconfig parameters to opt-out of using -D__PW_CUDA and -D__GRID_CUDA?

options['LIBS'] += ' -lcudart -lnvrtc -lcuda -lcublas -lcufft -lrt'
options['NVCC'] = 'nvcc'
options['NVFLAGS'] = ' '.join([
options['DFLAGS'],
'-O3',
'--std=c++11',
'-arch sm_%s' % cuda_cc.replace('.', ''),
# control host compilers + options
"-ccbin='%s'" % os.getenv('CXX'),
"-Xcompiler='%s'" % os.getenv('CXXFLAGS'),
])

# avoid group nesting
options['LIBS'] = options['LIBS'].replace('-Wl,--start-group', '').replace('-Wl,--end-group', '')
Expand Down