Concurrency and parallelism.

[giulioungaretti]

The architecture implemented at the moment is showing some issues that are non trivial to fix:

• pickling,
• monitoring,
• performance
• in general seems to hard to grasp for the users
• intermittent test failures due to syncing issues

Need to:

• "Foreground" fully functional with live plot
• No multiprocessing as default, keep in the code as "experimental"

And then carefully design how we want to do concurrency and parallelism:

• python or not python ?
  • if not python ØMQ, erlang, f# backends
• servers running on fridges and instruments

Notes:

• @akhmerov and @jbweston from kwant <3 concurrent.futures.Executor

[Rubenknex]

I don't know much about parallelism in python but the pathos (https://github.com/uqfoundation/pathos) and corresponding dill (https://github.com/uqfoundation/dill) projects might be interesting to look at? It's a kind of fork from the python multiprocessing library and fixes a lot of pickling issues among others.

[akhmerov]

Can you please explain what you mean by pickling issues? What is actually pickled currently?

[giulioungaretti]

@akhmerov multiprocessing, in use now, uses Pickle to pass objects to the child processes.
The list of pickable objects is rather small, and numpy f.ex raccomends not to pickle.

The entire server architecture is build to avoid pickling, which seems a bit brittle because of the need of proxying objects.

[akhmerov]

Sure, but what objects are passed around? The instruments? I imagine a much more sustainable approach would be to instead create all these objects in other processes/workers.

[giulioungaretti]

Closing this down as it will be take care of. See #314 and #315 .