Supervisor can kill the main process, but not the scheduler, web server, and xmpp server

[sivy]

Running Will under supervisor, with the following config:

[program:will]
command=/apps/will/venv/bin/python /apps/will/will/run_will.py
process_name=will
autostart=true
stdout_logfile=/var/log/will/bot.log
redirect_stderr=true
priority=500
environment=PYTHONPATH="/apps/will:$PYTHONPATH",
    [lots of other env vars]

The problem I'm seeing is that starting Will under supervisor works fine, it starts 4 processes (main and the scheduler, web server, and xmpp server). But when using supervisorctl stop will, it kills the main process but leaves the subprocesses running (and running as the root user, weirdly).

Any tips on how to run will with supervisor properly?

[skoczen]

Hey @sivy ,

Thanks for your patience with my slow replies here. My knowledge of supervisor is 4 years old and minimal at that, so I'm not much help in that department.

That's very strange behavior though. Maybe supervisor can send a more kind kill event (similar to the keyboard interrupt/etc?)

It sounds (based on zero knowledge) that supervisor is somehow hard-killing the main thread at a really low level, and not giving it time to clean up. Python's supposed to take care of this, but strange things can happen.

Not sure how much that helps - I'm happy to share what I know to help you get it solved!

[sivy]

Solved it! Supervisor has an option for stopsignal - usually that's TERM (SIGTERM) but you can set it to QUIT, or in my case, INT (SIGINT which correlates to ^C). INT works for Will, and setting the stopwaitsecs option lets you give Will time to shut things down before rebooting.

[skoczen]

Glad to hear it! :)

[pepedocs]

Late comment but this might add valuable information;

Unlike SIGINT, looks like python does not have a default signal handler for SIGTERM. This can be observed by inspection using the interpreter v2.7.9;

>>> import signal
>>> signal.signal(signal.SIGTERM, signal.SIG_DFL)
0
>>> signal.signal(signal.SIGINT, signal.SIG_DFL)
<built-in function default_int_handler>

# signal.signal(...) returns the previous handler

This might be the reason why will isn't working nicely with supervisor assuming supervisor's default stopsignal is SIGTERM.

--EDIT--

More pythonic way to retrieve the current signal handler;

>>> import signal
>>> signal.getsignal(signal.SIGTERM)
0
>>> signal.getsignal(signal.SIGINT)
<built-in function default_int_handler>

Cheers

[skoczen]

@pepedocs nice catch on that - any interest in a PR that handles both SIGTERM and SIGINT?

[pepedocs]

@skoczen I will find time to provide a PR for this.

Meanwhile, anyone feel free to provide a PR.

[skoczen]

@pepedocs now that I'm working on it, I'm realizing that this may also bundle really nicely into the 1.0 abstractions - please ping if you're digging into it so we can coordinate!

[skoczen]

Noting that this is on the list for the #258 release - closing and tracking there!