simulations get stuck

[nikohansen]

I have received several reports by word of mouth that simulations get stuck in that they consume 100% CPU time but cease to write output.

One reason for this behavior can be numpys multithreading, see here. In this case, a simple solution is to execute

import mkl
mkl.set_num_threads(1)

before numpy or any module that imports numpy was imported in Python.

However, this seems not to be the only reason for simulations getting stuck. It would be in particular helpful now to know more about such problems which are not due to Python mkl.

[TGlas]

I observed simulations being stuck for far too long time: should have been a week (according to my extrapolation), and I killed the jobs after two months. Trying the above fix revealed that mkl was not installed on the target system. I have no clue yet what may have caused the problem.

[nikohansen]

Is there a remote possibility that while Python mkl is not installed, yet some underlying C code uses the offending library? In this case something like export OPENBLAS_NUM_THREADS=1 or export MKL_NUM_THREADS=1 could be a fix (as also given here).

[brockho]

Thanks to @petrposik who sent another related link to the twitter account of Jeremy Howard:
https://twitter.com/jeremyphoward/status/1185044752753815552 (in which he mentions that you want to set a few more environment variables). Thought that it's worth to keep the link here instead of within an e-mail only.

[nikohansen]

Proposal:

def set_num_threads(nt=1, disp=1):
    """see https://github.com/numbbo/coco/issues/1919
    and https://twitter.com/jeremyphoward/status/1185044752753815552
    """
    try: import mkl
    except ImportError: disp and print("mkl is not installed")
    else:
        mkl.set_num_threads(nt)
    nt = str(nt)
    for name in ['OPENBLAS_NUM_THREADS',
                 'NUMEXPR_NUM_THREADS',
                 'OMP_NUM_THREADS',
                 'MKL_NUM_THREADS']:
        os.environ[name] = nt
    disp and print("setting mkl threads num to", nt)

if sys.platform.lower() not in ('darwin', 'windows'):
    set_num_threads(1)

will be added to example_experiment2.py.