Panic when connecting to peer

[alecalve]

This shows up in logs when running 8fc489a, it doesn't panic the whole server, only one thread apparently.

20190115 17:11:36.876 ERROR grin_util::logger - 
thread 'peer_connect' panicked at 'called `Result::unwrap()` on an `Err` value: Error { inner: 

Store Error: chain head }': libcore/result.rs:1009stack backtrace:
   0:     0x55ca67ef5d5d - backtrace::backtrace::trace::hd74837959dc31a2c
   1:     0x55ca67ef4f72 - <backtrace::capture::Backtrace as core::default::Default>::default::hfbe03539066da14f
   2:     0x55ca67ef4fe9 - backtrace::capture::Backtrace::new::hd9d47426559d8b68
   3:     0x55ca67e86250 - grin_util::logger::send_panic_to_log::{{closure}}::h9dfd05e2f10c03c7
   4:     0x55ca67fc5a36 - std::panicking::rust_panic_with_hook::hde420d6fd4455550
                        at libstd/panicking.rs:480
   5:     0x55ca67fc5581 - std::panicking::continue_panic_fmt::h8f394f3c578bcc76
                        at libstd/panicking.rs:390
   6:     0x55ca67fc5465 - rust_begin_unwind
                        at libstd/panicking.rs:325
   7:     0x55ca6800f4dc - core::panicking::panic_fmt::hca5dc4e8b320bc56
                        at libcore/panicking.rs:77
   8:     0x55ca678fca49 - core::result::unwrap_failed::h6c1c2c813d6804bc
   9:     0x55ca67915d28 - <grin_servers::common::adapters::NetToChainAdapter as grin_p2p::types::ChainAdapter>::total_difficulty::ha4744a9a4508bbbf
  10:     0x55ca67d6df1e - grin_p2p::serv::Server::connect::h8bd6469ca4b7705c
  11:     0x55ca6794ae06 - std::sys_common::backtrace::__rust_begin_short_backtrace::hc7cf452d4e4f9188
  12:     0x55ca67921e6d - std::panicking::try::do_call::hb8c8549034e610b9
  13:     0x55ca67fd7ad9 - __rust_maybe_catch_panic
                        at libpanic_unwind/lib.rs:102
  14:     0x55ca6792b07e - <F as alloc::boxed::FnBox<A>>::call_box::h266cfea02ce1ef91
  15:     0x55ca67fc1fbd - <alloc::boxed::Box<(dyn alloc::boxed::FnBox<A, Output=R> + 'a)> as core::ops::function::FnOnce<A>>::call_once::h9d902c911a417e39
                        at liballoc/boxed.rs:682
                         - std::sys_common::thread::start_thread::h44127e03e78ca137
                        at libstd/sys_common/thread.rs:24
  16:     0x55ca67fad335 - std::sys::unix::thread::Thread::new::thread_start::h8f17b97f2223146c
                        at libstd/sys/unix/thread.rs:90
  17:     0x7f54acd6a6da - start_thread
  18:     0x7f54ac87b88e - __clone
  19:                0x0 - <unknown>

[Byte1122]

I had this also. Maybe good to know workaround is to delete the chaindata.

[ignopeverell]

This would occur if the chain had a database (lmdb) error when fetching its current head as the node is connecting to a new peer. I can't reproduce this easily, so a few questions:

1. Did a restart help?
2. Did this happen right at startup?
3. Any other particular condition? (server was interrupted abruptly last time it ran, full disk, etc)

[hashmap]

I guess we hit the limit of LMDB readers, default is 126 seems to be low if the node has 150+ peers.

[alecalve]

It might be linked to the number of peers, my node has 164 at the moment (peer_max_count=250). I'll change it down to 120 to see if it removes the error.

[alecalve]

peer_min_preferred_count was set to 150 too

[hashmap]

TIL LMDB ties a reader to a thread by default, so each peer consumes one reader for its lifetime. It's strange we can get more than ~ 120 peers, I've disabled this behavior, testing. Another option is to increase max number of readers, perhaps we need both.