Stratum server crashed on miner reconnect

[bladedoyle]

In this case the "Miner" is the pools stratum proxy.

20190118 15:07:40.911 ERROR grin_servers::mining::stratumserver - (Server ID: 0) Share at height 4027, edge_bits 29, nonce 9303825972958685207, job_id 0 submitted too late
20190118 15:07:40.918 WARN grin_servers::mining::stratumserver - (Server ID: 197) Error in connection with stratum client: Connection reset by peer (os error 104)
20190118 15:07:40.918 WARN grin_servers::mining::stratumserver - (Server ID: 0) Dropping worker: 197
20190118 15:07:40.920 ERROR grin_servers::mining::stratumserver - (Server ID: 0) Share at height 4027, edge_bits 29, nonce 4897090650194261194, job_id 0 submitted too late
20190118 15:07:40.921 ERROR grin_servers::mining::stratumserver - (Server ID: 0) Share at height 4026, edge_bits 29, nonce 7273822134867265031, job_id 0 submitted too late
20190118 15:07:40.928 WARN grin_servers::mining::stratumserver - (Server ID: 0) New connection: 10.8.1.167:49066
20190118 15:07:41.012 DEBUG grin_chain::txhashset::txhashset - txhashset: rewind_to_pos: header 7153, output 19456, kernel 12669
20190118 15:07:41.015 DEBUG grin_chain::txhashset::txhashset - txhashset: merkle_proof: output: Commitment(09f91f79f7395251e31e65f200c619909f8c0b5fafcca4d0cd7b8344c1c60d58d5)
20190118 15:07:41.016 DEBUG grin_core::core::pmmr::pmmr - merkle_proof  19455, last_pos 19456
20190118 15:07:41.034 DEBUG grin_chain::txhashset::txhashset - Rewind to header 000013d959ff at 3582
20190118 15:07:41.113 ERROR grin_util::logger -
thread 'stratum_server' panicked at 'called `Option::unwrap()` on a `None` value': libcore/option.rs:355stack backtrace:
   0:     0x55b14eae156d - backtrace::backtrace::trace::h33a8de1a5de4f598
   1:     0x55b14eae0782 - <backtrace::capture::Backtrace as core::default::Default>::default::h13bf884e561085be
   2:     0x55b14eae07f9 - backtrace::capture::Backtrace::new::h810796244ae4b8f2
   3:     0x55b14ea6bff0 - grin_util::logger::send_panic_to_log::{{closure}}::h9acd2464f1e109ee
   4:     0x55b14ebb26b6 - std::panicking::rust_panic_with_hook::h71214e7ce0f7ac01
                        at libstd/panicking.rs:480
   5:     0x55b14ebb2201 - std::panicking::continue_panic_fmt::ha8b8442f4ea9bcac
                        at libstd/panicking.rs:390
   6:     0x55b14ebb20e5 - rust_begin_unwind
                        at libstd/panicking.rs:325
   7:     0x55b14ebfc15c - core::panicking::panic_fmt::h0c93626b89c38af6
                        at libcore/panicking.rs:77
   8:     0x55b14ebfc08b - core::panicking::panic::h158c6e3285d6c3fb
                        at libcore/panicking.rs:52
   9:     0x55b14e53e7d9 - grin_servers::mining::stratumserver::StratumServer::handle_rpc_requests::h53bc65bb65935c5a
  10:     0x55b14e5414ea - grin_servers::mining::stratumserver::StratumServer::run_loop::h4b76d8893f9e851a
  11:     0x55b14e533038 - std::sys_common::backtrace::__rust_begin_short_backtrace::hda6b21d3e6545d86
  12:     0x55b14e509dc8 - std::panicking::try::do_call::h5614c5c1734363da
  13:     0x55b14ebc49d9 - __rust_maybe_catch_panic
                        at libpanic_unwind/lib.rs:102
  14:     0x55b14e513797 - <F as alloc::boxed::FnBox<A>>::call_box::h7f51e328bb8a0e16
  15:     0x55b14ebaec3d - <alloc::boxed::Box<(dyn alloc::boxed::FnBox<A, Output=R> + 'a)> as core::ops::function::FnOnce<A>>::call_once::he3acfc8f1203786f
                        at liballoc/boxed.rs:682
                         - std::sys_common::thread::start_thread::h5213f803a61d7811
                        at libstd/sys_common/thread.rs:24
  16:     0x55b14eb99cf5 - std::sys::unix::thread::Thread::new::thread_start::he89121f566d2a8c7
                        at libstd/sys/unix/thread.rs:90
  17:     0x7f20788c2493 - start_thread
  18:     0x7f20783edace - __clone
  19:                0x0 - <unknown>


20190118 15:07:41.138 DEBUG grin_chain::txhashset::txhashset - txhashset: rewind_to_pos: header 7155, output 19457, kernel 12671
20190118 15:07:41.138 DEBUG grin_chain::txhashset::txhashset - txhashset: merkle_proof: output: Commitment(08caa16e027b6033db7efe4133a2341a7da651bb7e444669e9f404f69259369656)
20190118 15:07:41.138 DEBUG grin_core::core::pmmr::pmmr - merkle_proof  19457, last_pos 19457

[rlinxy]

I got the same problem. Might fixed by changing the unwrap() to unwrap_or().
But I have seldom experience in Rust, so don't know whether it is a good idea or not.
#2414

[bladedoyle]

Another one this evening. I was restarting a few of my stratum servers (which connect to the grin-core stratum):

20190120 05:34:36.609 WARN grin_servers::mining::stratumserver - (Server ID: 0) Dropping worker: 394
20190120 05:34:36.609 WARN grin_servers::mining::stratumserver - (Server ID: 0) New connection: 10.8.0.130:46272
20190120 05:34:36.799 ERROR grin_util::logger -
thread 'stratum_server' panicked at 'called `Option::unwrap()` on a `None` value': libcore/option.rs:355stack backtrace:
   0:     0x55d1fb06e56d - backtrace::backtrace::trace::h33a8de1a5de4f598
   1:     0x55d1fb06d782 - <backtrace::capture::Backtrace as core::default::Default>::default::h13bf884e561085be
   2:     0x55d1fb06d7f9 - backtrace::capture::Backtrace::new::h810796244ae4b8f2
   3:     0x55d1faff8ff0 - grin_util::logger::send_panic_to_log::{{closure}}::h9acd2464f1e109ee
   4:     0x55d1fb13f6b6 - std::panicking::rust_panic_with_hook::h71214e7ce0f7ac01
                        at libstd/panicking.rs:480
   5:     0x55d1fb13f201 - std::panicking::continue_panic_fmt::ha8b8442f4ea9bcac
                        at libstd/panicking.rs:390
   6:     0x55d1fb13f0e5 - rust_begin_unwind
                        at libstd/panicking.rs:325
   7:     0x55d1fb18915c - core::panicking::panic_fmt::h0c93626b89c38af6
                        at libcore/panicking.rs:77
   8:     0x55d1fb18908b - core::panicking::panic::h158c6e3285d6c3fb
                        at libcore/panicking.rs:52
   9:     0x55d1faacb7d9 - grin_servers::mining::stratumserver::StratumServer::handle_rpc_requests::h53bc65bb65935c5a
  10:     0x55d1faace4ea - grin_servers::mining::stratumserver::StratumServer::run_loop::h4b76d8893f9e851a
  11:     0x55d1faac0038 - std::sys_common::backtrace::__rust_begin_short_backtrace::hda6b21d3e6545d86
  12:     0x55d1faa96dc8 - std::panicking::try::do_call::h5614c5c1734363da
  13:     0x55d1fb1519d9 - __rust_maybe_catch_panic
                        at libpanic_unwind/lib.rs:102
  14:     0x55d1faaa0797 - <F as alloc::boxed::FnBox<A>>::call_box::h7f51e328bb8a0e16
  15:     0x55d1fb13bc3d - <alloc::boxed::Box<(dyn alloc::boxed::FnBox<A, Output=R> + 'a)> as core::ops::function::FnOnce<A>>::call_once::he3acfc8f1203786f
                        at liballoc/boxed.rs:682
                         - std::sys_common::thread::start_thread::h5213f803a61d7811
                        at libstd/sys_common/thread.rs:24
  16:     0x55d1fb126cf5 - std::sys::unix::thread::Thread::new::thread_start::he89121f566d2a8c7
                        at libstd/sys/unix/thread.rs:90
  17:     0x7f74dbdfe493 - start_thread
  18:     0x7f74db929ace - __clone
  19:                0x0 - <unknown>

[bladedoyle]

Again.

The problem is that the grin-core stratum server thread crashes.....but the rest of the grin server stays up. So its hard to tell what happened.
what should happen is either:

1. the grin server restarts the failed thread
   or
2. the grin server exists with error and can be auto-restarted

20190121 16:56:03.557 WARN grin_servers::mining::stratumserver - (Server ID: 251) Error in connection with stratum client: Connection reset by peer (os error 104)
20190121 16:56:03.557 WARN grin_servers::mining::stratumserver - (Server ID: 0) Dropping worker: 251
20190121 16:56:03.558 WARN grin_servers::mining::stratumserver - (Server ID: 0) New connection: 10.8.0.133:58276
20190121 16:56:04.654 ERROR grin_util::logger -
thread 'stratum_server' panicked at 'called `Option::unwrap()` on a `None` value': libcore/option.rs:355stack backtrace:
   0:     0x5616681d856d - backtrace::backtrace::trace::h33a8de1a5de4f598
   1:     0x5616681d7782 - <backtrace::capture::Backtrace as core::default::Default>::default::h13bf884e561085be
   2:     0x5616681d77f9 - backtrace::capture::Backtrace::new::h810796244ae4b8f2
   3:     0x561668162ff0 - grin_util::logger::send_panic_to_log::{{closure}}::h9acd2464f1e109ee
   4:     0x5616682a96b6 - std::panicking::rust_panic_with_hook::h71214e7ce0f7ac01
                        at libstd/panicking.rs:480
   5:     0x5616682a9201 - std::panicking::continue_panic_fmt::ha8b8442f4ea9bcac
                        at libstd/panicking.rs:390
   6:     0x5616682a90e5 - rust_begin_unwind
                        at libstd/panicking.rs:325
   7:     0x5616682f315c - core::panicking::panic_fmt::h0c93626b89c38af6
                        at libcore/panicking.rs:77
   8:     0x5616682f308b - core::panicking::panic::h158c6e3285d6c3fb
                        at libcore/panicking.rs:52
   9:     0x561667c357d9 - grin_servers::mining::stratumserver::StratumServer::handle_rpc_requests::h53bc65bb65935c5a
  10:     0x561667c384ea - grin_servers::mining::stratumserver::StratumServer::run_loop::h4b76d8893f9e851a
  11:     0x561667c2a038 - std::sys_common::backtrace::__rust_begin_short_backtrace::hda6b21d3e6545d86
  12:     0x561667c00dc8 - std::panicking::try::do_call::h5614c5c1734363da
  13:     0x5616682bb9d9 - __rust_maybe_catch_panic
                        at libpanic_unwind/lib.rs:102
  14:     0x561667c0a797 - <F as alloc::boxed::FnBox<A>>::call_box::h7f51e328bb8a0e16
  15:     0x5616682a5c3d - <alloc::boxed::Box<(dyn alloc::boxed::FnBox<A, Output=R> + 'a)> as core::ops::function::FnOnce<A>>::call_once::he3acfc8f1203786f
                        at liballoc/boxed.rs:682
                         - std::sys_common::thread::start_thread::h5213f803a61d7811
                        at libstd/sys_common/thread.rs:24
  16:     0x561668290cf5 - std::sys::unix::thread::Thread::new::thread_start::he89121f566d2a8c7
                        at libstd/sys/unix/thread.rs:90
  17:     0x7fafd8792493 - start_thread
  18:     0x7fafd82bdace - __clone
  19:                0x0 - <unknown>


20190121 16:56:08.136 DEBUG grin_servers::grin::seed - monitor_peers: on 0.0.0.0:3414, 17 connected (15 most_work). all 8809 = 62 healthy + 0 banned + 8747 defunct
20190121 16:56:11.626 DEBUG grin_p2p::protocol - handle_payload: received tx kernel: 3d6dfb21e014, msg_len: 32

[rlinxy]

@ignopeverell
I just compiled #2453 again. The rig still can not connect to the stratum. I am using hsp miner.
On the grin console, it shows like the miner keep trying to connect, and the 'Connected' field always be set to 'false'.