Evaluate documents in parallel

[osma]

We could perhaps speed up the evaldir command by making use of multiple CPUs, using the multiprocessing module. There would be a pool of workers (as many as there are CPU cores) and documents would be handed to the workers for evaluation.

[osma]

Logging is a bit of a challenge, but there is https://pypi.python.org/pypi/multiprocessing-logging/ that might help

[kinow]

For me the evaluation of a directory took a while, but I suspect over 80% of the time was loading the vectorizer from disk.

Is the idea here to perhaps first pre-load the model, and then evaluate the documents in parallel?

[osma]

@kinow Yes, the initialization time (loading vectorizer, models etc) tends to dominate when you evaluate with a small amount of documents. Possibly some of this initialization could be parallelized as well...

Anyways, the idea of this feature was to load the model first, then evaluate the documents in parallel. With a large enough set of documents (thousands?) there should be a significant reduction in overall time spent on evaluation.

[osma]

The main thing holding this back currently is that Annif projects are too tightly tied with the Flask current_app object. This is especially problematic for ensemble backends (and vw_multi) that need to access other projects. They have to do that via current_app. This works fine in the main thread, but not in subprocesses launched by e.g. multiprocessing or joblib.Parallel.

I'm working on decoupling the Annif internals from Flask and will prepare a PR on that soon. I also have a mostly-working implementation on parallel evaluation of documents but it needs a bit more work (and perhaps first some more refactorings to avoid passing around large objects).