mkl-service should not set MKL_THREADING_LAYER environment variable

[rgommers]

In _mklinitmodule.c the environment variable MKL_THREADING_LAYER is set. A library should never set any env vars.

This is resulting in issues in conda envs when libgomp is already loaded, see pytorch/pytorch#37377.

[rgommers]

@oleksandr-pavlyk is there a way we can get this moving?

[oleksandr-pavlyk]

The message reported in pytorch/pytorch#37377 results when mkl_service is being loaded and

1. MKL_THREADING_LAYER is found have explicit value INTEL
2. Gnu OMP has been detected in the Python process.

The message warns that MKL_THREADING_LAYER=INTEL is being disregarded, and replaced with MKL_THREADING_LAYER=GNU to avoid getting into state where MKL functionality is broken.

To triage the issue I created an environment on an Ubuntu box:

conda create -n debug_pytorch -c pytorch -c defaults --override-channels pytorch ipython

Output of ``conda list --explicit``

# This file may be used to create an environment using:
# $ conda create --name  --file 
# platform: linux-64
@EXPLICIT
https://repo.anaconda.com/pkgs/main/linux-64/_libgcc_mutex-0.1-main.conda
https://repo.anaconda.com/pkgs/main/linux-64/blas-1.0-mkl.conda
https://repo.anaconda.com/pkgs/main/linux-64/ca-certificates-2020.6.24-0.conda
https://repo.anaconda.com/pkgs/main/linux-64/intel-openmp-2020.1-217.conda
https://repo.anaconda.com/pkgs/main/linux-64/ld_impl_linux-64-2.33.1-h53a641e_7.conda
https://repo.anaconda.com/pkgs/main/linux-64/libstdcxx-ng-9.1.0-hdf63c60_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/libgcc-ng-9.1.0-hdf63c60_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/mkl-2020.1-217.conda
https://repo.anaconda.com/pkgs/main/linux-64/cudatoolkit-10.2.89-hfd86e86_1.conda
https://repo.anaconda.com/pkgs/main/linux-64/libffi-3.3-he6710b0_2.conda
https://repo.anaconda.com/pkgs/main/linux-64/ncurses-6.2-he6710b0_1.conda
https://repo.anaconda.com/pkgs/main/linux-64/openssl-1.1.1g-h7b6447c_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/xz-5.2.5-h7b6447c_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/zlib-1.2.11-h7b6447c_3.conda
https://repo.anaconda.com/pkgs/main/linux-64/libedit-3.1.20191231-h14c3975_1.conda
https://repo.anaconda.com/pkgs/main/linux-64/readline-8.0-h7b6447c_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/tk-8.6.10-hbc83047_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/sqlite-3.33.0-h62c20be_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/python-3.7.7-hcff3b4d_5.conda
https://repo.anaconda.com/pkgs/main/noarch/backcall-0.2.0-py_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/certifi-2020.6.20-py37_0.conda
https://repo.anaconda.com/pkgs/main/noarch/decorator-4.4.2-py_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/ipython_genutils-0.2.0-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/ninja-1.10.0-py37hfd86e86_0.conda
https://repo.anaconda.com/pkgs/main/noarch/parso-0.7.0-py_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/pickleshare-0.7.5-py37_1001.conda
https://repo.anaconda.com/pkgs/main/linux-64/ptyprocess-0.6.0-py37_0.conda
https://repo.anaconda.com/pkgs/main/noarch/six-1.15.0-py_0.conda
https://repo.anaconda.com/pkgs/main/noarch/wcwidth-0.2.5-py_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/jedi-0.17.2-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/mkl-service-2.3.0-py37he904b0f_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/pexpect-4.8.0-py37_1.conda
https://repo.anaconda.com/pkgs/main/linux-64/setuptools-49.6.0-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/traitlets-4.3.3-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/numpy-base-1.19.1-py37hfa32c7d_0.conda
https://repo.anaconda.com/pkgs/main/noarch/pygments-2.6.1-py_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/wheel-0.34.2-py37_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/pip-20.2.2-py37_0.conda
https://repo.anaconda.com/pkgs/main/noarch/prompt-toolkit-3.0.5-py_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/ipython-7.17.0-py37h39e3cac_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/mkl_fft-1.1.0-py37h23d657b_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/mkl_random-1.1.1-py37h0573a6f_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/numpy-1.19.1-py37hbc911f0_0.conda
https://conda.anaconda.org/pytorch/linux-64/pytorch-1.6.0-py3.7_cuda10.2.89_cudnn7.6.5_0.tar.bz2

In this environment, I have no trouble importing torch's multiprocessing following by importing numpy:

(debug_pytorch) [08:45:18 vmlin $]$ python -c "from torch import multiprocessing as mp; import numpy as np; np.show_config()"   blas_mkl_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/localdisk/work/miniconda3/envs/debug_pytorch/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/localdisk/work/miniconda3/envs/debug_pytorch/include']
blas_opt_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/localdisk/work/miniconda3/envs/debug_pytorch/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/localdisk/work/miniconda3/envs/debug_pytorch/include']
lapack_mkl_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/localdisk/work/miniconda3/envs/debug_pytorch/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/localdisk/work/miniconda3/envs/debug_pytorch/include']
lapack_opt_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/localdisk/work/miniconda3/envs/debug_pytorch/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/localdisk/work/miniconda3/envs/debug_pytorch/include']

And the same time, if I set the environment variable MKL_THREADING_LAYER explicitly to INTEL, I get an abnormal termination with exit code 1, from _mklinitmodule L103:

(debug_pytorch) [08:39:09 vmlin $]$ MKL_THREADING_LAYER=INTEL python -c "from torch import multiprocessing as mp; import numpy as np; np.show_config()" || echo $?
Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library.
        Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it.
1

So I guess what I am saying is that the code behaves as designed, and the suggested solution is to understand while MKL_THREADING_LAYER is being set to INTEL in the user's code environment.

Please clarify if I misunderstood the issue. Thanks.

[peterbell10]

Here is a simple reproducer that fails for me without setting MKL_THREADING_LAYER.

def do_test(*args):
    import torch; import numpy as np
    x = np.random.rand(100)
    x.dot(x)

if __name__ == '__main__':
    import multiprocessing as mp
    import numpy as np
    mp = mp.get_context('spawn')
    with mp.Pool(2) as p:
        p.map(do_test, range(2))

• The main process imports numpy, which leads to mkl-service linking in libiomp5.so and setting MKL_THREADING_LAYER=INTEL.
• The worker processes are created using spawn, so aren't linked to any openmp runtime yet.
• On the worker process, import torch links to libgomp.so.1. import numpy then tries to load the mkl but fails because libgomp is incompatible to MKL_THREADING_LAYER=INTEL, as set by mkl-service in the parent process.

The problem here is that MKL_THREADING_LAYER=INTEL is being applied to child processes when it shouldn't.