[Libbeat] Gracefully shut down on SIGHUP

[cwurm]

SIGHUP is sent to a process if the user's terminal is disconnected. It is one of the five termination signals. Currently, Beats is going to terminate suddenly, i.e. the log will cut off, no shutdown actions are going to be run, etc.

As we handle more and more state (e.g. writing state to beat.db files) it becomes more important to gracefully shutdown in "less than ideal" situations. Beats should probably not be run directly from a terminal or a terminal multiplexer, but it probably happens, and if so the consequences of a network disconnect or clumsy fingers (I forget the number of times I've hit the wrong key when using shortcuts in tmux/screen) should not be severe if we can help it.

This change catches SIGHUP as it already does SIGINT/SIGTERM and runs the usual shutdown actions. This can include waiting for a period of time specified using the shutdown_timeout option Filebeat and Packetbeat support.

I thought we might want to exit with a non-zero exit code if we receive this unexpected signal, but I don't see a good way to specifiy the exit code in Libbeat at the moment.

[andrewkroh]

I recommend putting some tests in place from the system tests to send in a SIGHUP. This way it gets tested on multiple OSes and we can make sure nothing unusual is happening upstream the Go runtimes signal handling (it has some builtin handlers).

[tsg]

Minor, but this comment is now out of date (and has been for a while). Perhaps: // On termination signals, gracefully stop the Beat

[jsoriano]

Even if it was not intended for that, SIGHUP is commonly used to request a configuration reload in services/daemons in unix systems. One of the reasons to use this signal for that is that daemons are usually not run in terminals, so they are not going to receive it by this reason. I think that Beats should be included in this category.

In any case I think it is fine to merge this by now, I agree that we should handle this signal in some way.

[cwurm]

I recommend putting some tests in place from the system tests to send in a SIGHUP. This way it gets tested on multiple OSes and we can make sure nothing unusually is happening upstream the Go runtimes signal handling (it has some builtin handlers).

Good call, I've added a system test that sends SIGHUP and checks if it shut down correctly. Let me know what you think @andrewkroh.