RGS randomly hangs after sometime #32
Comments
Ah yeah, this is bound to fail, we shouldn't ever
|
|
Wait, I'm confused on how this causes a deadlock, the mitigation in this case is called from a tokio thread (there are no non-tokio threads in the RGS server) so it should happily create a new thread for the duration of the block, and I don't immediately see any deadlock in the |
If thre's a thread currently handling `PeerManager` events, the next thread which attempts to handle events will block on the first and then handle events after the first completes. (later threads will return immediately to avoid blocking more than one thread). This works fine as long as the user has a spare thread to leave blocked, but if they don't (e.g. are running with a single-threaded tokio runtime) this can lead to a full deadlock. Instead, here, we never block waiting on another event processing thread, returning immediately after signaling that the first thread should start over once its complete to ensure all events are handled. While this could lead to starvation as we cause one thread to go around and around and around again, the risk of that should be relatively low as event handling should be pretty quick, and it's certainly better than deadlocking. Fixes lightningdevkit/rapid-gossip-sync-server#32
|
I believe this should be fixed by lightningdevkit/rust-lightning#2280 (untested, though). I didn't manage to reproduce this issue with two threads, but with one thread I hit what looks like it should be fixed by that. |
If thre's a thread currently handling `PeerManager` events, the next thread which attempts to handle events will block on the first and then handle events after the first completes. (later threads will return immediately to avoid blocking more than one thread). This works fine as long as the user has a spare thread to leave blocked, but if they don't (e.g. are running with a single-threaded tokio runtime) this can lead to a full deadlock. Instead, here, we never block waiting on another event processing thread, returning immediately after signaling that the first thread should start over once its complete to ensure all events are handled. While this could lead to starvation as we cause one thread to go around and around and around again, the risk of that should be relatively low as event handling should be pretty quick, and it's certainly better than deadlocking. Fixes lightningdevkit/rapid-gossip-sync-server#32
If thre's a thread currently handling `PeerManager` events, the next thread which attempts to handle events will block on the first and then handle events after the first completes. (later threads will return immediately to avoid blocking more than one thread). This works fine as long as the user has a spare thread to leave blocked, but if they don't (e.g. are running with a single-threaded tokio runtime) this can lead to a full deadlock. Instead, here, we never block waiting on another event processing thread, returning immediately after signaling that the first thread should start over once its complete to ensure all events are handled. While this could lead to starvation as we cause one thread to go around and around and around again, the risk of that should be relatively low as event handling should be pretty quick, and it's certainly better than deadlocking. Fixes lightningdevkit/rapid-gossip-sync-server#32
Ah, hum, good point, we're indeed in a tokio context in this case. |
Hm, you are right. My interpretation was wrong. But my fix is still applicable. I checked the backtraces and indeed there is a thread waiting for a lock on |
|
I built a version with @TheBlueMatt fix and it solves the problem! |
|
Cool gonna go ahead and close this since github doesn't do the magic close-on-merge thing across repos. |
|
Oh wait, maybe it does! |
|
And why reopened now? 😁 |
|
Cause I want to find out if GitHub magics here:) |
If thre's a thread currently handling `PeerManager` events, the next thread which attempts to handle events will block on the first and then handle events after the first completes. (later threads will return immediately to avoid blocking more than one thread). This works fine as long as the user has a spare thread to leave blocked, but if they don't (e.g. are running with a single-threaded tokio runtime) this can lead to a full deadlock. Instead, here, we never block waiting on another event processing thread, returning immediately after signaling that the first thread should start over once its complete to ensure all events are handled. While this could lead to starvation as we cause one thread to go around and around and around again, the risk of that should be relatively low as event handling should be pretty quick, and it's certainly better than deadlocking. Fixes lightningdevkit/rapid-gossip-sync-server#32 Atomic lock simplification suggestion from @andrei-21
If thre's a thread currently handling `PeerManager` events, the next thread which attempts to handle events will block on the first and then handle events after the first completes. (later threads will return immediately to avoid blocking more than one thread). This works fine as long as the user has a spare thread to leave blocked, but if they don't (e.g. are running with a single-threaded tokio runtime) this can lead to a full deadlock. Instead, here, we never block waiting on another event processing thread, returning immediately after signaling that the first thread should start over once its complete to ensure all events are handled. While this could lead to starvation as we cause one thread to go around and around and around again, the risk of that should be relatively low as event handling should be pretty quick, and it's certainly better than deadlocking. Fixes lightningdevkit/rapid-gossip-sync-server#32 Atomic lock simplification suggestion from @andrei-21
This includes a fix for the deadlock in lightningdevkit#32.
This includes a fix for the deadlock in lightningdevkit#32.
If thre's a thread currently handling `PeerManager` events, the next thread which attempts to handle events will block on the first and then handle events after the first completes. (later threads will return immediately to avoid blocking more than one thread). This works fine as long as the user has a spare thread to leave blocked, but if they don't (e.g. are running with a single-threaded tokio runtime) this can lead to a full deadlock. Instead, here, we never block waiting on another event processing thread, returning immediately after signaling that the first thread should start over once its complete to ensure all events are handled. While this could lead to starvation as we cause one thread to go around and around and around again, the risk of that should be relatively low as event handling should be pretty quick, and it's certainly better than deadlocking. Fixes lightningdevkit/rapid-gossip-sync-server#32 Atomic lock simplification suggestion from @andrei-21
Current Behavior:
RGS randomly hangs after sometime.
Expected Behavior:
RGS does not hang if the database is slow.
Steps To Reproduce:
#[tokio::main(flavor = "multi_thread", worker_threads = 2)](more peers — higher the changes to hangs)
What Happens
The problem is with
mpsc::channel(100)forGossipMessage. There is a task which receives elements inGossipPersisterand tasks which send elements inGossipRouter.GossipRouterusestry_send()method to send without blocking the thread, but if it fails (when the channel is full) it uses blockingsend(), what blocks the thread. But this thread is a tokio executor thread, if all tokio executor threads get blocked inGossipRouter::new_channel_announcement()orGossipRouter::new_channel_update()then the task to receive elements from the channel will never be executed. Deadlock.Note that there is a code in
GossipRouterwhich tries to minimize the risk of deadlock, but unfortunate it does not eliminate it:The text was updated successfully, but these errors were encountered: