Skip to content

Enabled filtering of paths in the CPATH variable generated for the wrapper #1436

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
boegel merged 7 commits into from
Jul 5, 2018
Merged

Enabled filtering of paths in the CPATH variable generated for the wrapper #1436

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
bdb5964
Enabled filtering of paths in the CPATH variable generated for the wr…
Jun 13, 2018
90a98fc
Removed paths also for any toolchain, and included LIBRARY_PATH in th…
Jun 14, 2018
043d756
Fixed non defined cpath variable
Jun 14, 2018
0a54a16
Fixed bug that created a list of lists, instead of a list of filtered…
Jun 15, 2018
1f4c7e5
Added logging, and some extra explanation of what path_filter does an…
Jun 22, 2018
d9bdb3f
Fixed undefined var
Jun 22, 2018
6b46635
Style fixes
Jul 5, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion easybuild/easyblocks/t/tensorflow.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ def extra_options():
extra_vars = {
# see https://developer.nvidia.com/cuda-gpus
'cuda_compute_capabilities': [[], "List of CUDA compute capabilities to build with", CUSTOM],
'path_filter': [[], "List of patterns to be filtered out in paths in $CPATH and $LIBRARY_PATH", CUSTOM],
'with_jemalloc': [None, "Make TensorFlow use jemalloc (usually enabled by default)", CUSTOM],
'with_mkl_dnn': [None, "Make TensorFlow use Intel MKL-DNN (enabled unless cuDNN is used)", CUSTOM],
}
Expand Down Expand Up @@ -99,10 +100,22 @@ def configure_step(self):

tmpdir = tempfile.mkdtemp(suffix='-bazel-configure')

# filter out paths from CPATH and LIBRARY_PATH. This is needed since bazel will pull some dependencies that
# might conflict with dependencies on the system and/or installed with EB. For example: protobuf
path_filter = self.cfg['path_filter']
if path_filter:
self.log.info("Filtering $CPATH and $LIBRARY_PATH with path filter %s", path_filter)
for var in ['CPATH', 'LIBRARY_PATH']:
path = os.getenv(var).split(os.pathsep)
self.log.info("$%s old value was %s" % (var, path))
filtered_path = os.pathsep.join([p for fil in path_filter for p in path if fil not in p])
env.setvar(var, filtered_path)

# put wrapper for Intel C compiler in place (required to make sure license server is found)
# cfr. https://github.com/bazelbuild/bazel/issues/663
if self.toolchain.comp_family() == toolchain.INTELCOMP:
wrapper_dir = os.path.join(tmpdir, 'bin')

icc_wrapper_txt = INTEL_COMPILER_WRAPPER % {
'compiler_path': which('icc'),
'cpath': os.getenv('CPATH'),
Expand All @@ -111,7 +124,7 @@ def configure_step(self):
}
icc_wrapper = os.path.join(wrapper_dir, 'icc')
write_file(icc_wrapper, icc_wrapper_txt)
env.setvar('PATH', ':'.join([os.path.dirname(icc_wrapper), os.getenv('PATH')]))
env.setvar('PATH', os.pathsep.join([os.path.dirname(icc_wrapper), os.getenv('PATH')]))
if self.dry_run:
self.dry_run_msg("Wrapper for 'icc' was put in place: %s", icc_wrapper)
else:
Expand Down