Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

grin server can't stop #2906

Closed
garyyu opened this issue Jun 21, 2019 · 4 comments
Closed

grin server can't stop #2906

garyyu opened this issue Jun 21, 2019 · 4 comments
Assignees
Labels

Comments

@garyyu
Copy link
Contributor

garyyu commented Jun 21, 2019

In 1.1.0 release, look like we still have the peers stop issue? the grin process can't stop. here is the shutdown logs:

shutdown logs

20190621 10:10:10.485 WARN grin::cmd::server - Received SIGINT (Ctrl+C) or SIGTERM (kill).
20190621 10:10:10.485 DEBUG grin_servers::common::types - sync_state: sync_status: NoSync -> Shutdown
20190621 10:10:10.552 INFO grin_servers::grin::server - connect_and_monitor thread stopped
20190621 10:10:11.015 INFO grin_servers::grin::server - sync thread stopped
20190621 10:10:11.015 INFO grin_servers::grin::server - dandelion_monitor thread stopped
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(86.85.139.156:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(178.63.97.197:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(52.59.238.178:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(46.4.91.48:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(157.230.142.11:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(120.78.134.123:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(109.124.18.120:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(18.222.188.55:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(68.183.167.183:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(217.123.171.4:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(178.62.232.49:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(217.182.192.59:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(18.224.59.48:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(35.246.210.108:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(38.109.105.21:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(45.77.150.172:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(119.28.60.52:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(95.216.193.239:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(178.63.16.52:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(54.36.109.161:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(99.250.50.83:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(206.189.196.224:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(139.99.192.176:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(152.44.40.69:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(34.65.152.46:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(47.97.198.21:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(47.97.163.164:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(95.211.48.178:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(52.57.14.225:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(47.244.97.106:13414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(18.179.2.142:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(47.52.131.159:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(109.169.84.77:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(109.74.202.16:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(149.129.102.254:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(54.148.113.184:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(34.80.153.18:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(68.183.33.125:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(157.230.175.104:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(147.135.11.144:13414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(147.160.55.3:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(35.176.195.242:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(34.90.247.67:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(35.222.167.4:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Stopping peer PeerAddr(V4(134.209.195.97:3414))
20190621 10:10:11.015 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(86.85.139.156:3414)) to stop
20190621 10:10:11.015 DEBUG grin_p2p::conn - waiting for thread ThreadId(4182) exit
20190621 10:10:11.017 DEBUG grin_p2p::conn - Shutting down connection with 119.28.60.52:34736
20190621 10:10:11.017 DEBUG grin_p2p::conn - Shutting down connection with 45.77.150.172:3414
20190621 10:10:11.017 DEBUG grin_p2p::conn - Shutting down connection with 152.44.40.69:40054
20190621 10:10:11.017 DEBUG grin_p2p::conn - Shutting down connection with 99.250.50.83:29950
20190621 10:10:11.017 DEBUG grin_p2p::conn - Shutting down connection with 54.36.109.161:35736
20190621 10:10:11.017 DEBUG grin_p2p::conn - Shutting down connection with 18.179.2.142:3414
20190621 10:10:11.017 DEBUG grin_p2p::conn - Shutting down connection with 47.244.97.106:60890
20190621 10:10:11.017 DEBUG grin_p2p::conn - Shutting down connection with 46.4.91.48:3414
20190621 10:10:11.017 DEBUG grin_p2p::conn - Shutting down connection with 52.57.14.225:3414
20190621 10:10:11.017 DEBUG grin_p2p::conn - Shutting down connection with 35.246.210.108:48588
20190621 10:10:11.017 DEBUG grin_p2p::conn - Shutting down connection with 147.135.11.144:13414
20190621 10:10:11.017 DEBUG grin_p2p::conn - Shutting down connection with 109.169.84.77:34124
20190621 10:10:11.017 DEBUG grin_p2p::conn - Shutting down connection with 34.90.247.67:2417
20190621 10:10:11.017 DEBUG grin_p2p::conn - Shutting down connection with 47.97.163.164:3414
20190621 10:10:11.017 DEBUG grin_p2p::conn - Shutting down connection with 157.230.175.104:37898
20190621 10:10:11.018 DEBUG grin_p2p::conn - Shutting down connection with 217.123.171.4:33550
20190621 10:10:11.020 DEBUG grin_p2p::conn - Shutting down connection with 178.62.232.49:3414
20190621 10:10:11.021 DEBUG grin_p2p::conn - Shutting down connection with 178.63.16.52:33240
20190621 10:10:11.021 DEBUG grin_p2p::conn - Shutting down connection with 134.209.195.97:3414
20190621 10:10:11.021 DEBUG grin_p2p::conn - Shutting down connection with 95.216.193.239:3414
20190621 10:10:11.021 DEBUG grin_p2p::conn - Shutting down connection with 109.74.202.16:3414
20190621 10:10:11.022 DEBUG grin_p2p::conn - Shutting down connection with 38.109.105.21:4558
20190621 10:10:11.022 DEBUG grin_p2p::conn - Shutting down connection with 147.160.55.3:56884
20190621 10:10:11.022 DEBUG grin_p2p::conn - Shutting down connection with 34.80.153.18:60776
20190621 10:10:11.022 DEBUG grin_p2p::conn - Shutting down connection with 149.129.102.254:33074
20190621 10:10:11.022 DEBUG grin_p2p::conn - Shutting down connection with 68.183.167.183:60744
20190621 10:10:11.023 DEBUG grin_p2p::conn - Shutting down connection with 86.85.139.156:46192
20190621 10:10:11.023 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(178.63.97.197:3414)) to stop
20190621 10:10:11.023 DEBUG grin_p2p::conn - waiting for thread ThreadId(21588) exit
20190621 10:10:11.023 DEBUG grin_p2p::conn - Shutting down connection with 157.230.142.11:42138
20190621 10:10:11.024 DEBUG grin_p2p::conn - Shutting down connection with 178.63.97.197:3414
20190621 10:10:11.024 DEBUG grin_p2p::conn - Shutting down connection with 18.222.188.55:32800
20190621 10:10:11.024 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(52.59.238.178:3414)) to stop
20190621 10:10:11.024 DEBUG grin_p2p::conn - waiting for thread ThreadId(19252) exit
20190621 10:10:11.024 DEBUG grin_p2p::conn - Shutting down connection with 18.224.59.48:33970
20190621 10:10:11.024 DEBUG grin_p2p::conn - Shutting down connection with 217.182.192.59:59844
20190621 10:10:11.024 DEBUG grin_p2p::conn - Shutting down connection with 34.65.152.46:3414
20190621 10:10:11.025 DEBUG grin_p2p::conn - Shutting down connection with 120.78.134.123:33638
20190621 10:10:11.025 DEBUG grin_p2p::conn - Shutting down connection with 95.211.48.178:49144
20190621 10:10:11.025 DEBUG grin_p2p::conn - Shutting down connection with 47.97.198.21:3414
20190621 10:10:11.025 DEBUG grin_p2p::conn - Shutting down connection with 109.124.18.120:51694
20190621 10:10:11.025 DEBUG grin_p2p::conn - Shutting down connection with 139.99.192.176:38942
20190621 10:10:11.026 DEBUG grin_p2p::conn - Shutting down connection with 35.222.167.4:59776
20190621 10:10:11.026 DEBUG grin_p2p::conn - Shutting down connection with 52.59.238.178:3414
20190621 10:10:11.026 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(46.4.91.48:3414)) to stop
20190621 10:10:11.026 DEBUG grin_p2p::conn - waiting for thread ThreadId(19672) exit
20190621 10:10:11.026 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(157.230.142.11:3414)) to stop
20190621 10:10:11.026 DEBUG grin_p2p::conn - waiting for thread ThreadId(9242) exit
20190621 10:10:11.026 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(120.78.134.123:3414)) to stop
20190621 10:10:11.026 DEBUG grin_p2p::conn - waiting for thread ThreadId(3574) exit
20190621 10:10:11.026 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(109.124.18.120:3414)) to stop
20190621 10:10:11.026 DEBUG grin_p2p::conn - waiting for thread ThreadId(7561) exit
20190621 10:10:11.026 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(18.222.188.55:3414)) to stop
20190621 10:10:11.026 DEBUG grin_p2p::conn - waiting for thread ThreadId(9520) exit
20190621 10:10:11.026 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(68.183.167.183:3414)) to stop
20190621 10:10:11.026 DEBUG grin_p2p::conn - waiting for thread ThreadId(8848) exit
20190621 10:10:11.026 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(217.123.171.4:3414)) to stop
20190621 10:10:11.026 DEBUG grin_p2p::conn - waiting for thread ThreadId(6742) exit
20190621 10:10:11.027 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(178.62.232.49:3414)) to stop
20190621 10:10:11.027 DEBUG grin_p2p::conn - waiting for thread ThreadId(19198) exit
20190621 10:10:11.027 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(217.182.192.59:3414)) to stop
20190621 10:10:11.027 DEBUG grin_p2p::conn - waiting for thread ThreadId(8315) exit
20190621 10:10:11.027 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(18.224.59.48:3414)) to stop
20190621 10:10:11.027 DEBUG grin_p2p::conn - waiting for thread ThreadId(9088) exit
20190621 10:10:11.027 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(35.246.210.108:3414)) to stop
20190621 10:10:11.027 DEBUG grin_p2p::conn - waiting for thread ThreadId(6347) exit
20190621 10:10:11.027 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(38.109.105.21:3414)) to stop
20190621 10:10:11.027 DEBUG grin_p2p::conn - waiting for thread ThreadId(7296) exit
20190621 10:10:11.027 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(45.77.150.172:3414)) to stop
20190621 10:10:11.027 DEBUG grin_p2p::conn - waiting for thread ThreadId(5755) exit
20190621 10:10:11.027 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(119.28.60.52:3414)) to stop
20190621 10:10:11.027 DEBUG grin_p2p::conn - waiting for thread ThreadId(7831) exit
20190621 10:10:11.027 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(95.216.193.239:3414)) to stop
20190621 10:10:11.027 DEBUG grin_p2p::conn - waiting for thread ThreadId(15461) exit
20190621 10:10:11.027 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(178.63.16.52:3414)) to stop
20190621 10:10:11.027 DEBUG grin_p2p::conn - waiting for thread ThreadId(797) exit
20190621 10:10:11.027 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(54.36.109.161:3414)) to stop
20190621 10:10:11.027 DEBUG grin_p2p::conn - waiting for thread ThreadId(10549) exit
20190621 10:10:11.027 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(99.250.50.83:3414)) to stop
20190621 10:10:11.027 DEBUG grin_p2p::conn - waiting for thread ThreadId(154) exit
20190621 10:10:11.027 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(206.189.196.224:3414)) to stop
20190621 10:10:11.027 DEBUG grin_p2p::conn - waiting for thread ThreadId(11264) exit
20190621 10:10:11.027 DEBUG grin_p2p::conn - Shutting down connection with 54.148.113.184:3414
20190621 10:10:11.027 DEBUG grin_p2p::conn - Shutting down connection with 47.52.131.159:44332
20190621 10:10:11.027 DEBUG grin_p2p::conn - Shutting down connection with 35.176.195.242:3414
20190621 10:10:11.028 DEBUG grin_p2p::conn - Shutting down connection with 206.189.196.224:41344
20190621 10:10:11.028 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(139.99.192.176:3414)) to stop
20190621 10:10:11.028 DEBUG grin_p2p::conn - waiting for thread ThreadId(8847) exit
20190621 10:10:11.028 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(152.44.40.69:3414)) to stop
20190621 10:10:11.028 DEBUG grin_p2p::conn - waiting for thread ThreadId(5425) exit
20190621 10:10:11.028 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(34.65.152.46:3414)) to stop
20190621 10:10:11.028 DEBUG grin_p2p::conn - waiting for thread ThreadId(15984) exit
20190621 10:10:11.028 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(47.97.198.21:3414)) to stop
20190621 10:10:11.028 DEBUG grin_p2p::conn - waiting for thread ThreadId(19438) exit
20190621 10:10:11.028 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(47.97.163.164:3414)) to stop
20190621 10:10:11.028 DEBUG grin_p2p::conn - waiting for thread ThreadId(20937) exit
20190621 10:10:11.028 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(95.211.48.178:3414)) to stop
20190621 10:10:11.028 DEBUG grin_p2p::conn - waiting for thread ThreadId(9526) exit
20190621 10:10:11.028 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(52.57.14.225:3414)) to stop
20190621 10:10:11.029 DEBUG grin_p2p::conn - waiting for thread ThreadId(21599) exit
20190621 10:10:11.029 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(47.244.97.106:13414)) to stop
20190621 10:10:11.029 DEBUG grin_p2p::conn - waiting for thread ThreadId(13526) exit
20190621 10:10:11.029 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(18.179.2.142:3414)) to stop
20190621 10:10:11.029 DEBUG grin_p2p::conn - waiting for thread ThreadId(21586) exit
20190621 10:10:11.029 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(47.52.131.159:3414)) to stop
20190621 10:10:11.029 DEBUG grin_p2p::conn - waiting for thread ThreadId(12626) exit
20190621 10:10:11.029 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(109.169.84.77:3414)) to stop
20190621 10:10:11.029 DEBUG grin_p2p::conn - waiting for thread ThreadId(9254) exit
20190621 10:10:11.029 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(109.74.202.16:3414)) to stop
20190621 10:10:11.029 DEBUG grin_p2p::conn - waiting for thread ThreadId(20125) exit
20190621 10:10:11.029 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(149.129.102.254:3414)) to stop
20190621 10:10:11.029 DEBUG grin_p2p::conn - waiting for thread ThreadId(9087) exit
20190621 10:10:11.029 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(54.148.113.184:3414)) to stop
20190621 10:10:11.029 DEBUG grin_p2p::conn - waiting for thread ThreadId(19401) exit
20190621 10:10:11.029 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(34.80.153.18:3414)) to stop
20190621 10:10:11.029 DEBUG grin_p2p::conn - waiting for thread ThreadId(2065) exit
20190621 10:10:11.029 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(68.183.33.125:3414)) to stop
20190621 10:10:11.029 DEBUG grin_p2p::conn - waiting for thread ThreadId(19674) exit
20190621 10:10:13.023 ERROR grin_p2p::peers - get_connected_peer: failed to get peers lock
20190621 10:10:13.023 DEBUG grin_p2p::conn - Shutting down connection with 68.183.33.125:3414
20190621 10:10:13.023 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(157.230.175.104:3414)) to stop
20190621 10:10:13.023 DEBUG grin_p2p::conn - waiting for thread ThreadId(8157) exit
20190621 10:10:13.024 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(147.135.11.144:13414)) to stop
20190621 10:10:13.024 DEBUG grin_p2p::conn - waiting for thread ThreadId(15196) exit
20190621 10:10:13.024 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(147.160.55.3:3414)) to stop
20190621 10:10:13.024 DEBUG grin_p2p::conn - waiting for thread ThreadId(579) exit
20190621 10:10:13.024 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(35.176.195.242:3414)) to stop
20190621 10:10:13.024 DEBUG grin_p2p::conn - waiting for thread ThreadId(21552) exit
20190621 10:10:13.024 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(34.90.247.67:3414)) to stop
20190621 10:10:13.024 DEBUG grin_p2p::conn - waiting for thread ThreadId(10547) exit
20190621 10:10:13.024 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(35.222.167.4:3414)) to stop
20190621 10:10:13.024 DEBUG grin_p2p::conn - waiting for thread ThreadId(8627) exit
20190621 10:10:13.024 DEBUG grin_p2p::peer - Waiting for peer PeerAddr(V4(134.209.195.97:3414)) to stop
20190621 10:10:13.024 DEBUG grin_p2p::conn - waiting for thread ThreadId(20551) exit

@garyyu
Copy link
Contributor Author

garyyu commented Jun 21, 2019

Here is the lldb attach process and get the thread backtrace:

thread backtrace

(lldb) thread backtrace all
* thread #1, name = 'peer_connect', stop reason = signal SIGSTOP
  * frame #0: 0x00007fbe22b5082e libpthread.so.0`__libc_recv + 94
    frame #1: 0x0000562f5481491a grin`read [inlined] recv_with_flags at net.rs:231
    frame #2: 0x0000562f54814907 grin`read [inlined] read at net.rs:240
    frame #3: 0x0000562f54814907 grin`read [inlined] read at net.rs:256
    frame #4: 0x0000562f54814907 grin`read at tcp.rs:570
    frame #5: 0x0000562f546f7a82 grin`grin_util::read_write::read_exact::h53347feb87392601 + 450
    frame #6: 0x0000562f545323c0 grin`grin_p2p::msg::read_header::hdc27fcbe4d45f3fb + 160
    frame #7: 0x0000562f5453451a grin`grin_p2p::msg::read_message::he4dbdba8240e0ff8 + 42
    frame #8: 0x0000562f5454c308 grin`grin_p2p::handshake::Handshake::initiate::hf4ac3acee476a1f2 + 1080
    frame #9: 0x0000562f5455859b grin`grin_p2p::peer::Peer::connect::hb9b0f6a482fd4b69 + 363
    frame #10: 0x0000562f54546005 grin`grin_p2p::serv::Server::connect::h76a620650b5afbf5 + 1877
    frame #11: 0x0000562f5424804e grin`std::sys_common::backtrace::__rust_begin_short_backtrace::h5407bb19743255a4 + 94
    frame #12: 0x0000562f5428003e grin`std::panicking::try::do_call::ha61a66e54fa31ba9 (.llvm.498615535989353943) + 46
    frame #13: 0x0000562f5482388a grin`__rust_maybe_catch_panic at lib.rs:87
    frame #14: 0x0000562f5427def3 grin`std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::h2b1e68bc911d4fe5 + 147
    frame #15: 0x0000562f5480b31f grin`call_once<(),FnBox<()>> at boxed.rs:702
    frame #16: 0x0000562f54822c20 grin`thread_start [inlined] call_once<(),alloc::boxed::Box<FnBox<()>>> at boxed.rs:702
    frame #17: 0x0000562f54822c14 grin`thread_start [inlined] start_thread at thread.rs:14
    frame #18: 0x0000562f54822b9a grin`thread_start at thread.rs:80
    frame #19: 0x00007fbe22b466db libpthread.so.0`start_thread + 219
    frame #20: 0x00007fbe2265788f libc.so.6`clone + 63
  thread #2, name = 'peer_connect', stop reason = signal SIGSTOP
    frame #0: 0x00007fbe22b5082e libpthread.so.0`__libc_recv + 94
    frame #1: 0x0000562f5481491a grin`read [inlined] recv_with_flags at net.rs:231
    frame #2: 0x0000562f54814907 grin`read [inlined] read at net.rs:240
    frame #3: 0x0000562f54814907 grin`read [inlined] read at net.rs:256
    frame #4: 0x0000562f54814907 grin`read at tcp.rs:570
    frame #5: 0x0000562f546f7a82 grin`grin_util::read_write::read_exact::h53347feb87392601 + 450
    frame #6: 0x0000562f545323c0 grin`grin_p2p::msg::read_header::hdc27fcbe4d45f3fb + 160
    frame #7: 0x0000562f5453451a grin`grin_p2p::msg::read_message::he4dbdba8240e0ff8 + 42
    frame #8: 0x0000562f5454c308 grin`grin_p2p::handshake::Handshake::initiate::hf4ac3acee476a1f2 + 1080
    frame #9: 0x0000562f5455859b grin`grin_p2p::peer::Peer::connect::hb9b0f6a482fd4b69 + 363
    frame #10: 0x0000562f54546005 grin`grin_p2p::serv::Server::connect::h76a620650b5afbf5 + 1877
    frame #11: 0x0000562f5424804e grin`std::sys_common::backtrace::__rust_begin_short_backtrace::h5407bb19743255a4 + 94
    frame #12: 0x0000562f5428003e grin`std::panicking::try::do_call::ha61a66e54fa31ba9 (.llvm.498615535989353943) + 46
    frame #13: 0x0000562f5482388a grin`__rust_maybe_catch_panic at lib.rs:87
    frame #14: 0x0000562f5427def3 grin`std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::h2b1e68bc911d4fe5 + 147
    frame #15: 0x0000562f5480b31f grin`call_once<(),FnBox<()>> at boxed.rs:702
    frame #16: 0x0000562f54822c20 grin`thread_start [inlined] call_once<(),alloc::boxed::Box<FnBox<()>>> at boxed.rs:702
    frame #17: 0x0000562f54822c14 grin`thread_start [inlined] start_thread at thread.rs:14
    frame #18: 0x0000562f54822b9a grin`thread_start at thread.rs:80
    frame #19: 0x00007fbe22b466db libpthread.so.0`start_thread + 219
    frame #20: 0x00007fbe2265788f libc.so.6`clone + 63
  thread #3, name = 'peer_connect', stop reason = signal SIGSTOP
    frame #0: 0x00007fbe22b5082e libpthread.so.0`__libc_recv + 94
    frame #1: 0x0000562f5481491a grin`read [inlined] recv_with_flags at net.rs:231
    frame #2: 0x0000562f54814907 grin`read [inlined] read at net.rs:240
    frame #3: 0x0000562f54814907 grin`read [inlined] read at net.rs:256
    frame #4: 0x0000562f54814907 grin`read at tcp.rs:570
    frame #5: 0x0000562f546f7a82 grin`grin_util::read_write::read_exact::h53347feb87392601 + 450
    frame #6: 0x0000562f545323c0 grin`grin_p2p::msg::read_header::hdc27fcbe4d45f3fb + 160
    frame #7: 0x0000562f5453451a grin`grin_p2p::msg::read_message::he4dbdba8240e0ff8 + 42
    frame #8: 0x0000562f5454c308 grin`grin_p2p::handshake::Handshake::initiate::hf4ac3acee476a1f2 + 1080
    frame #9: 0x0000562f5455859b grin`grin_p2p::peer::Peer::connect::hb9b0f6a482fd4b69 + 363
    frame #10: 0x0000562f54546005 grin`grin_p2p::serv::Server::connect::h76a620650b5afbf5 + 1877
    frame #11: 0x0000562f5424804e grin`std::sys_common::backtrace::__rust_begin_short_backtrace::h5407bb19743255a4 + 94
    frame #12: 0x0000562f5428003e grin`std::panicking::try::do_call::ha61a66e54fa31ba9 (.llvm.498615535989353943) + 46
    frame #13: 0x0000562f5482388a grin`__rust_maybe_catch_panic at lib.rs:87
    frame #14: 0x0000562f5427def3 grin`std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::h2b1e68bc911d4fe5 + 147
    frame #15: 0x0000562f5480b31f grin`call_once<(),FnBox<()>> at boxed.rs:702
    frame #16: 0x0000562f54822c20 grin`thread_start [inlined] call_once<(),alloc::boxed::Box<FnBox<()>>> at boxed.rs:702
    frame #17: 0x0000562f54822c14 grin`thread_start [inlined] start_thread at thread.rs:14
    frame #18: 0x0000562f54822b9a grin`thread_start at thread.rs:80
    frame #19: 0x00007fbe22b466db libpthread.so.0`start_thread + 219
    frame #20: 0x00007fbe2265788f libc.so.6`clone + 63
  thread #4, name = 'grin', stop reason = signal SIGSTOP
    frame #0: 0x00007fbe22b47d2d libpthread.so.0`__GI___pthread_timedjoin_ex + 381
    frame #1: 0x0000562f54822cdd grin`join at thread.rs:167
    frame #2: 0x0000562f5427d6e8 grin`std::thread::JoinHandle$LT$T$GT$::join::hce13e644f28edd03 + 72
    frame #3: 0x0000562f54220c33 grin`grin_servers::grin::server::Server::stop::h17eed08e54d11457 + 1251
    frame #4: 0x0000562f5408afbc grin`grin_servers::grin::server::Server::start::hc1a97f07b90f5737 + 1452
    frame #5: 0x0000562f5407eea1 grin`grin::cmd::server::start_server::h291e1bf4872e6640 + 353
    frame #6: 0x0000562f5407f8e3 grin`grin::cmd::server::server_command::h87db2cf18b4cc7cb + 2387
    frame #7: 0x0000562f540b1b72 grin`grin::real_main::h67d3c8a6d2e36e82 + 5138
    frame #8: 0x0000562f540b0756 grin`grin::main::hf381143b11389b36 + 6
    frame #9: 0x0000562f5408bc83 grin`std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h956e9479c92a3326 + 3
    frame #10: 0x0000562f5481d923 grin`do_call<closure,i32> [inlined] {{closure}} at rt.rs:49
    frame #11: 0x0000562f5481d917 grin`do_call<closure,i32> at panicking.rs:293
    frame #12: 0x0000562f5482388a grin`__rust_maybe_catch_panic at lib.rs:87
    frame #13: 0x0000562f5481e42d grin`lang_start_internal [inlined] try<i32,closure> at panicking.rs:272
    frame #14: 0x0000562f5481e3ef grin`lang_start_internal [inlined] catch_unwind<closure,i32> at panic.rs:388
    frame #15: 0x0000562f5481e3ef grin`lang_start_internal at rt.rs:48
    frame #16: 0x0000562f540b4dd2 grin`main + 34
    frame #17: 0x00007fbe22557b97 libc.so.6`__libc_start_main + 231
    frame #18: 0x0000562f5406bd3a grin`_start + 42
  thread #5, name = 'p2p-server', stop reason = signal SIGSTOP
    frame #0: 0x00007fbe22b5082e libpthread.so.0`__libc_recv + 94
    frame #1: 0x0000562f5481491a grin`read [inlined] recv_with_flags at net.rs:231
    frame #2: 0x0000562f54814907 grin`read [inlined] read at net.rs:240
    frame #3: 0x0000562f54814907 grin`read [inlined] read at net.rs:256
    frame #4: 0x0000562f54814907 grin`read at tcp.rs:570
    frame #5: 0x0000562f546f7932 grin`grin_util::read_write::read_exact::h53347feb87392601 + 114
    frame #6: 0x0000562f54532398 grin`grin_p2p::msg::read_header::hdc27fcbe4d45f3fb + 120
    frame #7: 0x0000562f545341fa grin`grin_p2p::msg::read_message::h369dfec4090bc72a + 42
    frame #8: 0x0000562f5454c88b grin`grin_p2p::handshake::Handshake::accept::h909a001e91251a89 + 59
    frame #9: 0x0000562f54557eb8 grin`grin_p2p::peer::Peer::accept::hd7e207f70f7e5c5e + 328
    frame #10: 0x0000562f54545192 grin`grin_p2p::serv::Server::listen::heb68c24f86052dc7 + 2626
    frame #11: 0x0000562f542487ac grin`std::sys_common::backtrace::__rust_begin_short_backtrace::hbcdf0bf297c65c05 + 28
    frame #12: 0x0000562f5482388a grin`__rust_maybe_catch_panic at lib.rs:87
    frame #13: 0x0000562f5427e388 grin`std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::h91e6f8de77a08dfb + 120
    frame #14: 0x0000562f5480b31f grin`call_once<(),FnBox<()>> at boxed.rs:702
    frame #15: 0x0000562f54822c20 grin`thread_start [inlined] call_once<(),alloc::boxed::Box<FnBox<()>>> at boxed.rs:702
    frame #16: 0x0000562f54822c14 grin`thread_start [inlined] start_thread at thread.rs:14
    frame #17: 0x0000562f54822b9a grin`thread_start at thread.rs:80
    frame #18: 0x00007fbe22b466db libpthread.so.0`start_thread + 219
    frame #19: 0x00007fbe2265788f libc.so.6`clone + 63
  thread #6, name = 'apis', stop reason = signal SIGSTOP
    frame #0: 0x00007fbe22b4c9f3 libpthread.so.0`__pthread_cond_wait + 579
    frame #1: 0x0000562f5451737c grin`futures::task_impl::std::ThreadNotify::park::h8d62de686122a3c2 + 220
    frame #2: 0x0000562f543bd41a grin`std::thread::local::LocalKey$LT$T$GT$::with::h43c33f80a0e1ed4b + 90
    frame #3: 0x0000562f543a4098 grin`tokio_executor::enter::Enter::block_on::h98670678e6d7991f + 104
    frame #4: 0x0000562f5430c060 grin`tokio::runtime::threadpool::run::h8efb84af5555bc4b + 400
    frame #5: 0x0000562f542acb47 grin`std::sys_common::backtrace::__rust_begin_short_backtrace::h21efea6c75b924a7 + 295
    frame #6: 0x0000562f5436ce8e grin`std::panicking::try::do_call::h0637720bb426ceae (.llvm.8830947357135432667) + 46
    frame #7: 0x0000562f5482388a grin`__rust_maybe_catch_panic at lib.rs:87
    frame #8: 0x0000562f54392363 grin`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h05acc56b7d4f6cc5 + 147
    frame #9: 0x0000562f5480b31f grin`call_once<(),FnBox<()>> at boxed.rs:702
    frame #10: 0x0000562f54822c20 grin`thread_start [inlined] call_once<(),alloc::boxed::Box<FnBox<()>>> at boxed.rs:702
    frame #11: 0x0000562f54822c14 grin`thread_start [inlined] start_thread at thread.rs:14
    frame #12: 0x0000562f54822b9a grin`thread_start at thread.rs:80
    frame #13: 0x00007fbe22b466db libpthread.so.0`start_thread + 219
    frame #14: 0x00007fbe2265788f libc.so.6`clone + 63
  thread #7, name = 'grin', stop reason = signal SIGSTOP
    frame #0: 0x00007fbe22b50384 libpthread.so.0`__libc_read + 68
    frame #1: 0x0000562f540f4042 grin`nix::unistd::read::h004b620e5cb08672 + 18
    frame #2: 0x0000562f54077880 grin`ctrlc::set_handler::_$u7b$$u7b$closure$u7d$$u7d$::h6ae501340f0f51b5 + 96
    frame #3: 0x0000562f54075f56 grin`std::sys_common::backtrace::__rust_begin_short_backtrace::hcd028a8be794da68 + 6
    frame #4: 0x0000562f5482388a grin`__rust_maybe_catch_panic at lib.rs:87
    frame #5: 0x0000562f540794d8 grin`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hb3fa506df6acb185 + 120
    frame #6: 0x0000562f5480b31f grin`call_once<(),FnBox<()>> at boxed.rs:702
    frame #7: 0x0000562f54822c20 grin`thread_start [inlined] call_once<(),alloc::boxed::Box<FnBox<()>>> at boxed.rs:702
    frame #8: 0x0000562f54822c14 grin`thread_start [inlined] start_thread at thread.rs:14
    frame #9: 0x0000562f54822b9a grin`thread_start at thread.rs:80
    frame #10: 0x00007fbe22b466db libpthread.so.0`start_thread + 219
    frame #11: 0x00007fbe2265788f libc.so.6`clone + 63
  thread #8, name = 'tokio-runtime-w', stop reason = signal SIGSTOP
    frame #0: 0x00007fbe22657bb7 libc.so.6`epoll_wait + 87
    frame #1: 0x0000562f544d3744 grin`mio::sys::unix::epoll::Selector::select::hd0c8b48cc134e988 + 116
    frame #2: 0x0000562f544d1f65 grin`mio::poll::Poll::poll1::h2e541a6887ebfe13 + 933
    frame #3: 0x0000562f544d1bb3 grin`mio::poll::Poll::poll::h49b80b68216f50e4 + 35
    frame #4: 0x0000562f544b342b grin`tokio_reactor::Reactor::turn::h8579c1e3b583b82a + 75
    frame #5: 0x0000562f544b4021 grin`_$LT$tokio_reactor..Reactor$u20$as$u20$tokio_executor..park..Park$GT$::park::hf96db01db7275d4d + 33
    frame #6: 0x0000562f544ab829 grin`_$LT$tokio_timer..timer..Timer$LT$T$C$N$GT$$u20$as$u20$tokio_executor..park..Park$GT$::park::hdca8cfe060c00d45 + 217
    frame #7: 0x0000562f544a6aa4 grin`_$LT$tokio_threadpool..park..boxed..BoxedPark$LT$T$GT$$u20$as$u20$tokio_executor..park..Park$GT$::park::h5cbccc9d333b29eb + 20
    frame #8: 0x0000562f544c048b grin`tokio_threadpool::worker::Worker::run::hbd2b65bf4785b85f + 1947
    frame #9: 0x0000562f544a2347 grin`tokio_trace_core::dispatcher::with_default::hbc3717ab4a4693c2 + 55
    frame #10: 0x0000562f544a37a6 grin`std::thread::local::LocalKey$LT$T$GT$::with::hace55c53ef488449 + 166
    frame #11: 0x0000562f544a354e grin`std::thread::local::LocalKey$LT$T$GT$::with::h5313608b2e558ead + 190
    frame #12: 0x0000562f544a39a4 grin`std::thread::local::LocalKey$LT$T$GT$::with::hb2b7f858eba85f39 + 244
    frame #13: 0x0000562f544a6c97 grin`tokio::runtime::threadpool::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::h361e58e4644f55db + 151
    frame #14: 0x0000562f544c92a3 grin`std::thread::local::LocalKey$LT$T$GT$::with::hc411e33ba50df815 + 179
    frame #15: 0x0000562f544c93de grin`std::thread::local::LocalKey$LT$T$GT$::with::hd0837a251e5431c9 + 190
    frame #16: 0x0000562f544c8151 grin`std::sys_common::backtrace::__rust_begin_short_backtrace::h8e3f9e6230134b29 + 241
    frame #17: 0x0000562f544c842c grin`std::panicking::try::do_call::h10667f3d4d63fd43 (.llvm.17747147481032781157) + 28
    frame #18: 0x0000562f5482388a grin`__rust_maybe_catch_panic at lib.rs:87
    frame #19: 0x0000562f544c2200 grin`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hbfa7375176e4bca4 + 128
    frame #20: 0x0000562f5480b31f grin`call_once<(),FnBox<()>> at boxed.rs:702
    frame #21: 0x0000562f54822c20 grin`thread_start [inlined] call_once<(),alloc::boxed::Box<FnBox<()>>> at boxed.rs:702
    frame #22: 0x0000562f54822c14 grin`thread_start [inlined] start_thread at thread.rs:14
    frame #23: 0x0000562f54822b9a grin`thread_start at thread.rs:80
    frame #24: 0x00007fbe22b466db libpthread.so.0`start_thread + 219
    frame #25: 0x00007fbe2265788f libc.so.6`clone + 63
  thread #9, name = 'tokio-runtime-w', stop reason = signal SIGSTOP
    frame #0: 0x00007fbe22657bb7 libc.so.6`epoll_wait + 87
    frame #1: 0x0000562f544d3744 grin`mio::sys::unix::epoll::Selector::select::hd0c8b48cc134e988 + 116
    frame #2: 0x0000562f544d1f65 grin`mio::poll::Poll::poll1::h2e541a6887ebfe13 + 933
    frame #3: 0x0000562f544d1bb3 grin`mio::poll::Poll::poll::h49b80b68216f50e4 + 35
    frame #4: 0x0000562f544b342b grin`tokio_reactor::Reactor::turn::h8579c1e3b583b82a + 75
    frame #5: 0x0000562f544b4021 grin`_$LT$tokio_reactor..Reactor$u20$as$u20$tokio_executor..park..Park$GT$::park::hf96db01db7275d4d + 33
    frame #6: 0x0000562f544ab829 grin`_$LT$tokio_timer..timer..Timer$LT$T$C$N$GT$$u20$as$u20$tokio_executor..park..Park$GT$::park::hdca8cfe060c00d45 + 217
    frame #7: 0x0000562f544a6aa4 grin`_$LT$tokio_threadpool..park..boxed..BoxedPark$LT$T$GT$$u20$as$u20$tokio_executor..park..Park$GT$::park::h5cbccc9d333b29eb + 20
    frame #8: 0x0000562f544c048b grin`tokio_threadpool::worker::Worker::run::hbd2b65bf4785b85f + 1947
    frame #9: 0x0000562f544a2347 grin`tokio_trace_core::dispatcher::with_default::hbc3717ab4a4693c2 + 55
    frame #10: 0x0000562f544a37a6 grin`std::thread::local::LocalKey$LT$T$GT$::with::hace55c53ef488449 + 166
    frame #11: 0x0000562f544a354e grin`std::thread::local::LocalKey$LT$T$GT$::with::h5313608b2e558ead + 190
    frame #12: 0x0000562f544a39a4 grin`std::thread::local::LocalKey$LT$T$GT$::with::hb2b7f858eba85f39 + 244
    frame #13: 0x0000562f544a6c97 grin`tokio::runtime::threadpool::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::h361e58e4644f55db + 151
    frame #14: 0x0000562f544c92a3 grin`std::thread::local::LocalKey$LT$T$GT$::with::hc411e33ba50df815 + 179
    frame #15: 0x0000562f544c93de grin`std::thread::local::LocalKey$LT$T$GT$::with::hd0837a251e5431c9 + 190
    frame #16: 0x0000562f544c8151 grin`std::sys_common::backtrace::__rust_begin_short_backtrace::h8e3f9e6230134b29 + 241
    frame #17: 0x0000562f544c842c grin`std::panicking::try::do_call::h10667f3d4d63fd43 (.llvm.17747147481032781157) + 28
    frame #18: 0x0000562f5482388a grin`__rust_maybe_catch_panic at lib.rs:87
    frame #19: 0x0000562f544c2200 grin`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hbfa7375176e4bca4 + 128
    frame #20: 0x0000562f5480b31f grin`call_once<(),FnBox<()>> at boxed.rs:702
    frame #21: 0x0000562f54822c20 grin`thread_start [inlined] call_once<(),alloc::boxed::Box<FnBox<()>>> at boxed.rs:702
    frame #22: 0x0000562f54822c14 grin`thread_start [inlined] start_thread at thread.rs:14
    frame #23: 0x0000562f54822b9a grin`thread_start at thread.rs:80
    frame #24: 0x00007fbe22b466db libpthread.so.0`start_thread + 219
    frame #25: 0x00007fbe2265788f libc.so.6`clone + 63
  thread #10, name = 'peer_connect', stop reason = signal SIGSTOP
    frame #0: 0x00007fbe22b5082e libpthread.so.0`__libc_recv + 94
    frame #1: 0x0000562f5481491a grin`read [inlined] recv_with_flags at net.rs:231
    frame #2: 0x0000562f54814907 grin`read [inlined] read at net.rs:240
    frame #3: 0x0000562f54814907 grin`read [inlined] read at net.rs:256
    frame #4: 0x0000562f54814907 grin`read at tcp.rs:570
    frame #5: 0x0000562f546f7a82 grin`grin_util::read_write::read_exact::h53347feb87392601 + 450
    frame #6: 0x0000562f545323c0 grin`grin_p2p::msg::read_header::hdc27fcbe4d45f3fb + 160
    frame #7: 0x0000562f5453451a grin`grin_p2p::msg::read_message::he4dbdba8240e0ff8 + 42
    frame #8: 0x0000562f5454c308 grin`grin_p2p::handshake::Handshake::initiate::hf4ac3acee476a1f2 + 1080
    frame #9: 0x0000562f5455859b grin`grin_p2p::peer::Peer::connect::hb9b0f6a482fd4b69 + 363
    frame #10: 0x0000562f54546005 grin`grin_p2p::serv::Server::connect::h76a620650b5afbf5 + 1877
    frame #11: 0x0000562f5424804e grin`std::sys_common::backtrace::__rust_begin_short_backtrace::h5407bb19743255a4 + 94
    frame #12: 0x0000562f5428003e grin`std::panicking::try::do_call::ha61a66e54fa31ba9 (.llvm.498615535989353943) + 46
    frame #13: 0x0000562f5482388a grin`__rust_maybe_catch_panic at lib.rs:87
    frame #14: 0x0000562f5427def3 grin`std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::h2b1e68bc911d4fe5 + 147
    frame #15: 0x0000562f5480b31f grin`call_once<(),FnBox<()>> at boxed.rs:702
    frame #16: 0x0000562f54822c20 grin`thread_start [inlined] call_once<(),alloc::boxed::Box<FnBox<()>>> at boxed.rs:702
    frame #17: 0x0000562f54822c14 grin`thread_start [inlined] start_thread at thread.rs:14
    frame #18: 0x0000562f54822b9a grin`thread_start at thread.rs:80
    frame #19: 0x00007fbe22b466db libpthread.so.0`start_thread + 219
    frame #20: 0x00007fbe2265788f libc.so.6`clone + 63
  thread #11, name = 'peer_connect', stop reason = signal SIGSTOP
    frame #0: 0x00007fbe22b5082e libpthread.so.0`__libc_recv + 94
    frame #1: 0x0000562f5481491a grin`read [inlined] recv_with_flags at net.rs:231
    frame #2: 0x0000562f54814907 grin`read [inlined] read at net.rs:240
    frame #3: 0x0000562f54814907 grin`read [inlined] read at net.rs:256
    frame #4: 0x0000562f54814907 grin`read at tcp.rs:570
    frame #5: 0x0000562f546f7a82 grin`grin_util::read_write::read_exact::h53347feb87392601 + 450
    frame #6: 0x0000562f545323c0 grin`grin_p2p::msg::read_header::hdc27fcbe4d45f3fb + 160
    frame #7: 0x0000562f5453451a grin`grin_p2p::msg::read_message::he4dbdba8240e0ff8 + 42
    frame #8: 0x0000562f5454c308 grin`grin_p2p::handshake::Handshake::initiate::hf4ac3acee476a1f2 + 1080
    frame #9: 0x0000562f5455859b grin`grin_p2p::peer::Peer::connect::hb9b0f6a482fd4b69 + 363
    frame #10: 0x0000562f54546005 grin`grin_p2p::serv::Server::connect::h76a620650b5afbf5 + 1877
    frame #11: 0x0000562f5424804e grin`std::sys_common::backtrace::__rust_begin_short_backtrace::h5407bb19743255a4 + 94
    frame #12: 0x0000562f5428003e grin`std::panicking::try::do_call::ha61a66e54fa31ba9 (.llvm.498615535989353943) + 46
    frame #13: 0x0000562f5482388a grin`__rust_maybe_catch_panic at lib.rs:87
    frame #14: 0x0000562f5427def3 grin`std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::h2b1e68bc911d4fe5 + 147
    frame #15: 0x0000562f5480b31f grin`call_once<(),FnBox<()>> at boxed.rs:702
    frame #16: 0x0000562f54822c20 grin`thread_start [inlined] call_once<(),alloc::boxed::Box<FnBox<()>>> at boxed.rs:702
    frame #17: 0x0000562f54822c14 grin`thread_start [inlined] start_thread at thread.rs:14
    frame #18: 0x0000562f54822b9a grin`thread_start at thread.rs:80
    frame #19: 0x00007fbe22b466db libpthread.so.0`start_thread + 219
    frame #20: 0x00007fbe2265788f libc.so.6`clone + 63
  thread #12, name = 'peer_connect', stop reason = signal SIGSTOP
    frame #0: 0x00007fbe22b5082e libpthread.so.0`__libc_recv + 94
    frame #1: 0x0000562f5481491a grin`read [inlined] recv_with_flags at net.rs:231
    frame #2: 0x0000562f54814907 grin`read [inlined] read at net.rs:240
    frame #3: 0x0000562f54814907 grin`read [inlined] read at net.rs:256
    frame #4: 0x0000562f54814907 grin`read at tcp.rs:570
    frame #5: 0x0000562f546f7a82 grin`grin_util::read_write::read_exact::h53347feb87392601 + 450
    frame #6: 0x0000562f545323c0 grin`grin_p2p::msg::read_header::hdc27fcbe4d45f3fb + 160
    frame #7: 0x0000562f5453451a grin`grin_p2p::msg::read_message::he4dbdba8240e0ff8 + 42
    frame #8: 0x0000562f5454c308 grin`grin_p2p::handshake::Handshake::initiate::hf4ac3acee476a1f2 + 1080
    frame #9: 0x0000562f5455859b grin`grin_p2p::peer::Peer::connect::hb9b0f6a482fd4b69 + 363
    frame #10: 0x0000562f54546005 grin`grin_p2p::serv::Server::connect::h76a620650b5afbf5 + 1877
    frame #11: 0x0000562f5424804e grin`std::sys_common::backtrace::__rust_begin_short_backtrace::h5407bb19743255a4 + 94
    frame #12: 0x0000562f5428003e grin`std::panicking::try::do_call::ha61a66e54fa31ba9 (.llvm.498615535989353943) + 46
    frame #13: 0x0000562f5482388a grin`__rust_maybe_catch_panic at lib.rs:87
    frame #14: 0x0000562f5427def3 grin`std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::h2b1e68bc911d4fe5 + 147
    frame #15: 0x0000562f5480b31f grin`call_once<(),FnBox<()>> at boxed.rs:702
    frame #16: 0x0000562f54822c20 grin`thread_start [inlined] call_once<(),alloc::boxed::Box<FnBox<()>>> at boxed.rs:702
    frame #17: 0x0000562f54822c14 grin`thread_start [inlined] start_thread at thread.rs:14
    frame #18: 0x0000562f54822b9a grin`thread_start at thread.rs:80
    frame #19: 0x00007fbe22b466db libpthread.so.0`start_thread + 219
    frame #20: 0x00007fbe2265788f libc.so.6`clone + 63```

</p>
</details>

will analysis later.

@hashmap
Copy link
Contributor

hashmap commented Jun 23, 2019

I was able to reproduce it but after running my node for a few hours, which is not great for debugging. Moreover in logs it looks like et least one peer doesn't stop, but in your thread listing and on my machine I don't see any peer threads. However it seems that we still accept connections, it may be the reason of a deadlock.

@hashmap hashmap self-assigned this Jun 23, 2019
@hashmap hashmap added the bug label Jun 23, 2019
@garyyu
Copy link
Contributor Author

garyyu commented Jun 24, 2019

Again, saw it at another server.

hashmap added a commit to cyclefortytwo/grin that referenced this issue Jun 25, 2019
Currently p2p.stop() stops and wait for all peers to exit, that's
basically all we need. However we also run a TCP listener in this thread
which is blocked on `accept` most of the time. We do an attempt to stop
it but it would work only if we get an incoming connection during the
shutdown, which is a week guarantee.

This fix remove joining to p2p-server thread, it stops all peers and
makes an attempt to stop the listener.

Fixes [mimblewimble#2906]
yeastplume pushed a commit that referenced this issue Jun 27, 2019
Currently p2p.stop() stops and wait for all peers to exit, that's
basically all we need. However we also run a TCP listener in this thread
which is blocked on `accept` most of the time. We do an attempt to stop
it but it would work only if we get an incoming connection during the
shutdown, which is a week guarantee.

This fix remove joining to p2p-server thread, it stops all peers and
makes an attempt to stop the listener.

Fixes [#2906]
yeastplume added a commit that referenced this issue Jun 27, 2019
* create 2.0.0 branch

* fix humansize version

* update grin.yml version

* PoW HardFork (#2866)

* allow version 2 blocks for next 6 months

* add cuckarood.rs with working tests

* switch cuckaroo to cuckarood at right heights

* reorder to reduce conditions

* remove _ prefix on used args; fix typo

* Make Valid Header Version dependant on ChainType

* Rustfmt

* Add tests, uncomment header v2

* Rustfmt

* Add FLOONET_FIRST_HARD_FORK height and simplify logic

* assume floonet stays closer to avg 60s block time

* move floonet hf forward by half a day

* update version in new block when previous no longer valid

* my next commit:-)

* micro optimization

* Support new Bulletproof rewind scheme (#2848)

* Update keychain with new rewind scheme

* Refactor: proof builder trait

* Update tests, cleanup

* rustfmt

* Move conversion of SwitchCommitmentType

* Add proof build trait to tx builders

* Cache hashes in proof builders

* Proof builder tests

* Add ViewKey struct

* Fix some warnings

* Zeroize proof builder secrets on drop

* Modify mine_block to use wallet V2 API (#2892)

* update mine_block to use V2 wallet API

* rustfmt

* Add version endpoint to node API, rename pool/push (#2897)

* add node version API, tweak pool/push parameter

* rustfmt

* Upate version api call (#2899)

* Update version number for next (potential) release

* zeroize: Upgrade to v0.9 (#2914)

* zeroize: Upgrade to v0.9

* missed Cargo.lock

* [PENDING APPROVAL] put phase outs of C32 and beyond on hold (#2714)

* put phase outs of C32 and beyond on hold

* update tests for phaseouts on hold

* Don't wait for p2p-server thread (#2917)

Currently p2p.stop() stops and wait for all peers to exit, that's
basically all we need. However we also run a TCP listener in this thread
which is blocked on `accept` most of the time. We do an attempt to stop
it but it would work only if we get an incoming connection during the
shutdown, which is a week guarantee.

This fix remove joining to p2p-server thread, it stops all peers and
makes an attempt to stop the listener.

Fixes [#2906]

* rustfmt
@hashmap
Copy link
Contributor

hashmap commented Jul 3, 2019

Fixed in #2917

@hashmap hashmap closed this as completed Jul 3, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants