use gthread instead of gevent to allow multithreading

[kimakan]

We are currently using gevent as a worker-class for Gunicorn which can handle high volume of concurrent HTTP requests. However, it's impossible to use multi-threading in the daiquiri-apps ( for example the threading library).
The alternative is to use the gthread worker which can handle not only the concurrent HTTP request but also the OS-level threads. AFAIK, the downside for using gthread is potentially a higher resource consumption by high number of simultaneous connections . For some apps that use multi-threading, gthread is required to work at all.
IMHO, there are two possible solutions:

• we change gevent to gthread. I'm not sure what would be the appropriate number of workers and threads in this case. atm, this is my preferred solution.
• we make another conf variable (multithreading?). If the variable is set to true then dq-dev uses gthread instead of gevent for Gunicorn. This solution is more flexible but adds another variable to the config which is already somewhat overcrowded.

[triole]

Yeah, I also tend to simply replace gevent by gthread. A potential higher resource consumption is possibly something that we can live with. I don't imagine our relatively small web services to be resource hungry enough to make us feel any serious downsides even if they use gthread. gthread seems more capable and simply always using it is the cleaner option.

[jochenklar]

The are some web specific caveats here. It is in general not advised to perform long lasting operations in one web request. Rule of thumb: one request should not take longer that half a second.

Instead of threading, one usually uses asyncronous tasks which are queued in one request, then run on a worker, and have some URL routes so that a client can check when its done. In Daiquiri this is none with Celery and RabbitMQ.

[kimakan]

@triole did you meant to say gthread in your last two sentences?

@jochenklar Some background. The cutout service on MUSE-Wide uses threading and works synchronously. The task takes less than a second but requires the worker-class gthread in order to work.
The other possible solution, if we want to keep gevent, would be to rewrite the cutout service without using threading. By now, I would probably prefer this solution.

[jochenklar]

Ok makes sense, I actually don't know why we use gevent here, but it could be that something else in Daiquiri does not work with gthread.

[triole]

@kimakan yes. i corrected it.

So we try to switch to ghtread then? Of course applying intensive tests to find out if anything is broken.