Implement graceful shutdown

[hashmap]

Currently grin server sets a stop flag and sends a stop message to each peer thread, waits for 1 second and exits. As result of the lack of graceful shutdown we introduced a stop mutex to protect critical phases like chain update etc. This global lock brings some performance penalties, also we need to manually find all such places and protect them.

This PR introduced a graceful shutdown using thread::join, so we track all important threads in grin and make sure that they exit. In this case we are sure that threads stop in safe places (flag or message check).
Also mutex around read/sent stats for peer was removed, should improve perfomance.

This PR doesn't cover Stratum stop (we don't have it).
Also TUI update is needed, we need to inform user that we are shutting down, because shutdown may take a few seconds.

[hashmap]

Ready for review. It passes tests, there is a CI issue, we discussed it in gitter

[ignopeverell]

There are quite a few changes in the lower-level peer and connection locking in here. Have you tried running it for a while?

[hashmap]

@ignopeverell I've been running it for 4 days, no issues so far. Also a few times tested sync from scratch stopping node at different random points. When a node is running with ~ 150 peers stop takes some time, 5-20 seconds.

[DavidBurkett]

Is this still with non blocking io? If it's non-blocking, it should be able to be implemented with only a few millisecond delay on shutdown. Where is the delay coming from?

[hashmap]

@DavidBurkett yes, with non-blocking IO, it depends on state of threads. Eg sync was sleeping for 10 secs, I made some optimization to keep it under 1 sec. Also if a peer is processing a txhashset we would wait couple minutes.

If a node is running the majority of time comes from waiting for a peers lock to be released. We have Peers object, it's like a manager of peers. It's shared and protected by RwLock. When we shutdown the server we take an exclusive write lock because we need to be sure that we stop all peers and no new peers will be added. However some peers may be doing something at that moment (processing a header, pinging peers, serving other peer request), which usually requires some level (usually read, sometimes write) of access to Peers object, which leads to a deadlock, because in the main thread we keep that lock and wait for those threads to finish. To prevent it I used try_lock on Peers object with 2 seconds timeout. During normal operations we take a Peer's lock for very short period of time, if we can't get it in 2 seconds it means that we are shutting down the server so it's safe to fail on taking a new lock. Perhaps we could decrease timeout.

[DavidBurkett]

Thanks for the thorough explanation @hashmap. The syncing delays make sense, but the deadlock when there's a high number of peers is what I would like to be able to avoid. Once you get this merged in, I'm going to take a stab at trying to break the deadlock. In Grin++, I had a similar issue, but was able to deal with it using an additional mutex: https://github.com/GrinPlusPlus/GrinPlusPlus/blob/master/P2P/ConnectionManager.cpp#L247. We may be able to do something similar.

[hashmap]

@DavidBurkett makes sense, I've used a similar strategy in this PR, perhaps we can remove this delay too. To be on the same page we don't have a deadlock now, but a peer's thread may get stuck for 2 seconds during shutdown.

[ignopeverell]

@hashmap can you fix the conflicts? I'd like to have this in 1.1.0 beta2 for testing.

[hashmap]

@ignopeverell sure, in 5-7 hours, on a plane now

[antiochp]

Just been testing this locally with a handful of restarts. 👍
Both q and ctrl-c.