Skip to content

Interrupting grin during chain validation results in corrupted data #2770

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
0xmichalis opened this issue Apr 21, 2019 · 7 comments
Closed

Interrupting grin during chain validation results in corrupted data #2770

0xmichalis opened this issue Apr 21, 2019 · 7 comments

Comments

@0xmichalis
Copy link
Contributor

0xmichalis commented Apr 21, 2019 •
edited
Loading

I have been able to reproduce #1620 deterministically in v1.1.0.beta-1 (running on Fedora 28).

Steps to reproduce:

  1. start a new node (may be able to reproduce simply with old data too)
  2. let it sync, download, and validate the first batch of blocks
  3. let it reach the point it starts validating the last blocks ("Preparing for chain validation 3/4" or something along those lines shows up in the TUI)
  4. Quit
  5. Restart - grin starts crashlooping - the only way to restart again is to cleanup the lmdb data

My latest try:

20190421 18:41:21.886 INFO grin - This is Grin version 1.1.0-beta.1 (git v1.1.0-win-beta.1-1-g9e8210c), built for x86_64-unknown-linux-gnu by rustc 1.33.0.
20190421 18:41:21.886 DEBUG grin - Built with profile "debug", features "".
20190421 18:41:21.889 WARN grin::cmd::server - Starting GRIN in UI mode...
20190421 18:41:21.978 INFO grin_servers::grin::server - Starting server, genesis block: edc758c1370d
20190421 18:41:22.009 DEBUG grin_store::lmdb - DB Mapsize for /home/mkargaki/.grin/floo/chain_data/lmdb is 134217728
20190421 18:41:22.039 DEBUG grin_chain::txhashset::txhashset - Rewind to header 114900e8e7a6 at 124592
20190421 18:41:22.042 DEBUG grin_chain::txhashset::txhashset - txhashset: rewind_to_pos: header 249176, output 367839, kernel 299221
20190421 18:41:22.043 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: IOErr("failed to fill whole buffer", UnexpectedEof)
20190421 18:41:22.043 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: IOErr("failed to fill whole buffer", UnexpectedEof)
20190421 18:41:22.043 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: IOErr("failed to fill whole buffer", UnexpectedEof)
20190421 18:41:22.043 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: IOErr("failed to fill whole buffer", UnexpectedEof)
20190421 18:41:22.043 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: IOErr("failed to fill whole buffer", UnexpectedEof)
20190421 18:41:22.044 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: IOErr("failed to fill whole buffer", UnexpectedEof)
20190421 18:41:22.044 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: IOErr("failed to fill whole buffer", UnexpectedEof)
20190421 18:41:22.044 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: IOErr("failed to fill whole buffer", UnexpectedEof)
20190421 18:41:22.044 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: IOErr("failed to fill whole buffer", UnexpectedEof)
20190421 18:41:22.044 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: IOErr("failed to fill whole buffer", UnexpectedEof)
20190421 18:41:22.044 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: IOErr("failed to fill whole buffer", UnexpectedEof)
20190421 18:41:22.569 ERROR grin_util::logger - 
thread 'main' panicked at 'no root, invalid tree': src/libcore/option.rs:1038stack backtrace:
   0: grin_util::logger::send_panic_to_log::{{closure}}::h4d1be965e4d42cc4 (0x55dd3f8ead7b)
             at util/src/logger.rs:240
   1: std::panicking::rust_panic_with_hook::hfd44f6005c88b9cd (0x55dd3fdb6b49)
   2: std::panicking::continue_panic_fmt::h49294ee7e45fe937 (0x55dd3fdb6601)
   3: rust_begin_unwind (0x55dd3fdb64e5)
   4: core::panicking::panic_fmt::h50061a1f3a2a7f40 (0x55dd3fdc7dac)
   5: core::option::expect_failed::ha7629b64ee08267d (0x55dd3fdc7e22)
   6: <core::option::Option<T>>::expect::h8fb29199247764e4 (0x55dd3f498ba6)
             at /builddir/build/BUILD/rustc-1.33.0-src/src/libcore/option.rs:312
   7: <grin_core::core::pmmr::pmmr::PMMR<'a, T, B>>::root::h466edd982bd33f32 (0x55dd3f55e009)
             at /home/mkargaki/projects/grin/core/src/core/pmmr/pmmr.rs:144
   8: grin_chain::txhashset::txhashset::Extension::roots::h81c694c3458e92a3 (0x55dd3f4bdfaa)
             at chain/src/txhashset/txhashset.rs:1093
   9: grin_chain::txhashset::txhashset::Extension::validate_roots::hfdce72456b25ad9d (0x55dd3f4be102)
             at chain/src/txhashset/txhashset.rs:1121
  10: grin_chain::chain::setup_head::{{closure}}::h336d1a28aa77a124 (0x55dd3f52ee1a)
             at chain/src/chain.rs:1324
  11: grin_chain::txhashset::txhashset::extending::h4491120da278e77e (0x55dd3f51c7ba)
             at chain/src/txhashset/txhashset.rs:439
  12: grin_chain::chain::setup_head::hd00f84ab6d70e25d (0x55dd3f550451)
             at chain/src/chain.rs:1322
  13: grin_chain::chain::Chain::init::h8dcd9a99b08144c4 (0x55dd3f53bd18)
             at chain/src/chain.rs:185
  14: grin_servers::grin::server::Server::new::hee84b6e5e8ca08db (0x55dd3eb3b6c0)
             at servers/src/grin/server.rs:181
  15: grin_servers::grin::server::Server::start::h50dfa6fccd0f55e4 (0x55dd3e7144aa)
             at /home/mkargaki/projects/grin/servers/src/grin/server.rs:83
  16: grin::cmd::server::start_server_tui::h5c31050d0e9e98dd (0x55dd3e741695)
             at src/bin/cmd/server.rs:48
  17: grin::cmd::server::start_server::h135158a038f4b9dc (0x55dd3e7412b6)
             at src/bin/cmd/server.rs:33
  18: grin::cmd::server::server_command::h5dd1ce1ff68ccb35 (0x55dd3e741f74)
             at src/bin/cmd/server.rs:145
  19: grin::real_main::h935e376770d64802 (0x55dd3e73b9f2)
             at src/bin/grin.rs:158
  20: grin::main::h3e5a47108688770f (0x55dd3e73aa95)
             at src/bin/grin.rs:67
  21: std::rt::lang_start::{{closure}}::hbfe76648c32774f3 (0x55dd3e7f011f)
             at /builddir/build/BUILD/rustc-1.33.0-src/src/libstd/rt.rs:64
  22: std::panicking::try::do_call::ha2b2f50005aee932 (0x55dd3fdb6482)
  23: __rust_maybe_catch_panic (0x55dd3fdb98a9)
  24: std::rt::lang_start_internal::h5b4ed6f311caad20 (0x55dd3fdac02f)
  25: std::rt::lang_start::h2f1f809f1b014057 (0x55dd3e7f00f8)
             at /builddir/build/BUILD/rustc-1.33.0-src/src/libstd/rt.rs:64
  26: main (0x55dd3e73bf1c)
  27: __libc_start_main (0x7f96f975711a)
  28: _start (0x55dd3e6c1d99)
  29: <unknown> (0x0)
@antiochp @ignopeverell @garyyu
@0xmichalis
Copy link
Contributor Author

This is a log on a release node just before the interrupt:

20190421 20:14:14.519 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.529 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.539 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.549 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.560 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.570 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.580 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.590 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.600 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.610 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.620 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.630 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.640 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.651 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.661 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.671 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.681 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.691 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.701 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.711 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.721 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.732 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.742 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.752 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.762 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.772 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.782 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.792 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.802 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.813 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.818 TRACE grin_p2p::conn - Received message header, type Ping, len 16.
20190421 20:14:14.823 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.833 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.843 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.853 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.863 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.873 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.883 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.893 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.904 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.914 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.924 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.934 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.944 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.954 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.964 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.974 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.985 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:14.995 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.005 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.015 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.025 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.035 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.045 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.055 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.066 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.076 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.086 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.096 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.106 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.116 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.126 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.136 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.147 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.157 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.167 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.177 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.187 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.197 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.207 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.218 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.228 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.238 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.248 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.258 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.268 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.278 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.288 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.298 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.309 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.319 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.329 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.339 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.349 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.359 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.369 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.379 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.390 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.400 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.410 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.420 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.430 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.440 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.450 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.460 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.471 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.481 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.491 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.501 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.511 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.521 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.531 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.541 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.551 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.562 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.572 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.582 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.592 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.602 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.612 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.619 DEBUG grin_servers::grin::seed - monitor_peers: on 0.0.0.0:13414, 19 connected (14 most_work). all 160 = 20 healthy + 8 banned + 132 defunct
20190421 20:14:15.622 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.632 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.643 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.653 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.656 TRACE grin_p2p::conn - Received message header, type Pong, len 16.
20190421 20:14:15.662 TRACE grin_p2p::conn - Received message header, type Pong, len 16.
20190421 20:14:15.663 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.664 TRACE grin_p2p::conn - Received message header, type Pong, len 16.
20190421 20:14:15.669 TRACE grin_p2p::conn - Received message header, type Pong, len 16.
20190421 20:14:15.669 TRACE grin_p2p::conn - Received message header, type Pong, len 16.
20190421 20:14:15.669 TRACE grin_p2p::conn - Received message header, type Pong, len 16.
20190421 20:14:15.673 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.683 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.693 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.703 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.714 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.724 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.734 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.741 TRACE grin_p2p::conn - Received message header, type Pong, len 16.
20190421 20:14:15.741 TRACE grin_p2p::conn - Received message header, type Pong, len 16.
20190421 20:14:15.741 TRACE grin_p2p::conn - Received message header, type Pong, len 16.
20190421 20:14:15.741 TRACE grin_p2p::conn - Received message header, type Pong, len 16.
20190421 20:14:15.741 TRACE grin_p2p::conn - Received message header, type Pong, len 16.
20190421 20:14:15.744 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.754 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.764 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.774 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.776 TRACE grin_p2p::conn - Received message header, type Pong, len 16.
20190421 20:14:15.777 TRACE grin_p2p::conn - Received message header, type Pong, len 16.
20190421 20:14:15.784 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.793 TRACE grin_p2p::conn - Received message header, type Pong, len 16.
20190421 20:14:15.795 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.795 TRACE grin_p2p::conn - Received message header, type Pong, len 16.
20190421 20:14:15.805 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.815 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.825 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.835 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.845 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.855 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.865 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.875 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.886 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.896 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.899 TRACE grin_p2p::conn - Received message header, type Pong, len 16.
20190421 20:14:15.906 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.912 TRACE grin_p2p::conn - Received message header, type Pong, len 16.
20190421 20:14:15.916 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.921 TRACE grin_p2p::conn - Received message header, type Pong, len 16.
20190421 20:14:15.926 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.936 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.946 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.956 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.966 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.977 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.987 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:15.997 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.007 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.017 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.018 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: IOErr("failed to fill whole buffer", UnexpectedEof)
20190421 20:14:16.018 ERROR grin_chain::chain - txhashset_write: corrupted storage? oldest_height is 0 when check_txhashset_needed. state sync is needed
20190421 20:14:16.018 DEBUG grin_chain::txhashset::txhashset - zip_write on path: "/tmp"
20190421 20:14:16.023 TRACE grin_p2p::conn - Received message header, type Ping, len 16.
20190421 20:14:16.027 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.037 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.047 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.057 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.068 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.078 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.088 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.098 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.108 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.118 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.128 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.139 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.149 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.159 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.169 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.179 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.189 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.200 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.210 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.220 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.230 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.240 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.250 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.260 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.271 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.281 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.291 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.301 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.311 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.321 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.331 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.341 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.352 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.362 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.372 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.382 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.392 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.402 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.412 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.423 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.432 DEBUG grin_store::leaf_set - leaf_set: copying rewound file /tmp/txhashset/output/pmmr_leaf.bin.2d385384ac3d to /tmp/txhashset/output/pmmr_leaf.bin
20190421 20:14:16.433 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.443 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.453 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.463 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.473 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.483 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.493 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.503 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.514 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.524 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.534 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.544 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.554 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.564 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.574 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.585 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.595 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.605 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.615 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.625 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.634 DEBUG grin_store::prune_list - prune_list: bitmap 27115 pos (29780 bytes), pruned_cache 103363 pos (33680 bytes), shift_cache 27115, leaf_shift_cache 27115
20190421 20:14:16.634 DEBUG grin_store::leaf_set - leaf_set: copying rewound file /tmp/txhashset/rangeproof/pmmr_leaf.bin.2d385384ac3d to /tmp/txhashset/rangeproof/pmmr_leaf.bin
20190421 20:14:16.635 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.645 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.655 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.665 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.676 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.686 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.696 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.706 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.716 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.726 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.736 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.746 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.756 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.767 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.777 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.787 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.797 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.807 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.817 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.827 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.837 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.837 DEBUG grin_store::prune_list - prune_list: bitmap 27115 pos (29780 bytes), pruned_cache 103363 pos (33680 bytes), shift_cache 27115, leaf_shift_cache 27115
20190421 20:14:16.838 TRACE grin_store::lmdb - DB map size: 134217728
20190421 20:14:16.838 TRACE grin_store::lmdb - Space used: 97497088
20190421 20:14:16.838 TRACE grin_store::lmdb - Space remaining: 36720640
20190421 20:14:16.838 TRACE grin_store::lmdb - Percent used: 0.7264  Percent threshold: 0.9000
20190421 20:14:16.838 TRACE grin_store::lmdb - Resize threshold not met (percent-based)
20190421 20:14:16.838 TRACE grin_chain::txhashset::txhashset - Starting new txhashset header extension.
20190421 20:14:16.838 DEBUG grin_chain::txhashset::txhashset - About to rebuild header extension from edc758c1370d to 2d385384ac3d.
20190421 20:14:16.847 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.858 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.868 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.878 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.888 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.898 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.908 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.918 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.929 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.939 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.949 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.959 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.969 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.979 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.989 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:16.999 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.010 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.020 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.030 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.040 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.050 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.060 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.070 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.080 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.091 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.101 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.111 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.121 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.131 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.141 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.151 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.161 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.172 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.172 TRACE grin_p2p::conn - Received message header, type Ping, len 16.
20190421 20:14:17.182 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.192 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.202 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.212 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.222 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.232 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.243 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.253 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.263 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.273 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.283 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.293 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.303 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.313 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.323 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.334 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.344 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.354 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.364 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.374 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.384 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.394 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.404 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.414 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.425 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.435 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.445 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.455 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.465 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.475 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.485 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.496 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.506 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.516 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.526 TRACE grin_servers::grin::sync::state_sync - state_sync: head.height: 124710, tail.height: 124710. header_head.height: 127591, highest_height: 127593
20190421 20:14:17.532 DEBUG grin_p2p::conn - Connection close with 34.73.70.248:51948 initiated by us
20190421 20:14:17.532 DEBUG grin_p2p::conn - Connection close with 34.207.101.126:3414 initiated by us
20190421 20:14:17.532 DEBUG grin_p2p::conn - Connection close with 185.243.113.40:49854 initiated by us
20190421 20:14:17.532 DEBUG grin_p2p::conn - Connection close with 35.188.35.25:56916 initiated by us
20190421 20:14:17.532 DEBUG grin_p2p::conn - Connection close with 39.107.119.132:49540 initiated by us
20190421 20:14:17.532 DEBUG grin_p2p::conn - Connection close with 46.101.201.216:52654 initiated by us
20190421 20:14:17.532 DEBUG grin_p2p::conn - Connection close with 35.230.71.57:52064 initiated by us
20190421 20:14:17.532 DEBUG grin_p2p::conn - Connection close with 34.73.229.124:57374 initiated by us
20190421 20:14:17.533 DEBUG grin_p2p::conn - Connection close with 34.80.179.152:49370 initiated by us
20190421 20:14:17.533 DEBUG grin_p2p::conn - Connection close with 3.85.55.157:3414 initiated by us
20190421 20:14:17.533 DEBUG grin_p2p::conn - Connection close with 148.251.154.104:33348 initiated by us
20190421 20:14:17.533 DEBUG grin_p2p::conn - Connection close with 34.73.242.225:37084 initiated by us
20190421 20:14:17.533 DEBUG grin_p2p::conn - Connection close with 35.247.29.132:58520 initiated by us
20190421 20:14:17.533 DEBUG grin_p2p::conn - Connection close with 34.246.35.64:51848 initiated by us
20190421 20:14:17.533 DEBUG grin_p2p::conn - Connection close with 46.4.91.48:13414 initiated by us
20190421 20:14:17.533 DEBUG grin_p2p::conn - Connection close with 13.66.156.253:39048 initiated by us
20190421 20:14:17.533 DEBUG grin_p2p::conn - Connection close with 35.157.67.48:13414 initiated by us
20190421 20:14:17.533 DEBUG grin_p2p::conn - Connection close with 198.245.50.26:13414 initiated by us
20190421 20:14:17.536 WARN grin_servers::grin::sync::syncer - sync: no peers available, disabling sync
20190421 20:14:17.536 DEBUG grin_servers::common::types - sync_state: sync_status: TxHashsetSetup -> NoSync
20190421 20:14:17.589 WARN grin::cmd::server - Shutting down...
20190421 20:14:18.589 WARN grin::cmd::server - Shutdown complete.
I wasn't running a debug node because it was taking ages to sync but can try once again with it if it's helpful.

@garyyu
Copy link
Contributor

garyyu commented Apr 23, 2019 •
edited
Loading

I also saw several times of this problem, our current shutdown (on kill normally) operation is definitely not safe, this should be a major issue for a production environment since kill is the only way to take effect one single config change / or to upgrade a Grin server.

will check the shutdown process.

@garyyu
Copy link
Contributor

garyyu commented Apr 23, 2019

I see it again on one of my seed servers:

20190423 09:07:54.193 INFO grin_servers::common::adapters - Received 32 block headers from 220.133.14.38:3414
20190423 09:07:54.193 DEBUG grin_chain::pipe - pipe: sync_block_headers: 32 headers from 00000021b994 at 136056
20190423 09:07:54.194 DEBUG grin_chain::txhashset::txhashset - Rewind header extension to 000001144e27 at 136055
20190423 09:07:54.194 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: Custom { kind: Other, error: IOErr("failed to fill whole buffer", UnexpectedEof) }
20190423 09:07:54.194 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: Custom { kind: Other, error: IOErr("failed to fill whole buffer", UnexpectedEof) }
20190423 09:07:54.194 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: Custom { kind: Other, error: IOErr("failed to fill whole buffer", UnexpectedEof) }
20190423 09:07:54.194 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: Custom { kind: Other, error: IOErr("failed to fill whole buffer", UnexpectedEof) }
20190423 09:07:54.194 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: Custom { kind: Other, error: IOErr("failed to fill whole buffer", UnexpectedEof) }
20190423 09:07:54.195 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: Custom { kind: Other, error: IOErr("failed to fill whole buffer", UnexpectedEof) }
20190423 09:07:54.195 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: Custom { kind: Other, error: IOErr("failed to fill whole buffer", UnexpectedEof) }
20190423 09:07:54.195 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: Custom { kind: Other, error: IOErr("failed to fill whole buffer", UnexpectedEof) }
20190423 09:07:54.195 ERROR grin_util::logger - 
thread 'peer' panicked at 'no root, invalid tree': src/libcore/option.rs:1038stack backtrace:
   0: grin_util::logger::send_panic_to_log::{{closure}}::hb1332dc7066eba48 (0x564c1d592270)
   1: std::panicking::rust_panic_with_hook::h744417edfe714d72 (0x564c1d6bb2e9)
             at src/libstd/panicking.rs:482
   2: std::panicking::continue_panic_fmt::h3557b3c3fa21b47b (0x564c1d6bad91)
             at src/libstd/panicking.rs:385
   3: rust_begin_unwind (0x564c1d6bac75)
             at src/libstd/panicking.rs:312
   4: core::panicking::panic_fmt::h74ee8034b317ceed (0x564c1d6d582c)
             at src/libcore/panicking.rs:85
   5: core::option::expect_failed::hbc312d76c0b547f0 (0x564c1d6d58a2)
             at src/libcore/option.rs:1038
   6: <grin_core::core::pmmr::pmmr::PMMR<'a, T, B>>::root::he84f0c2b43b19b07 (0x564c1d486035)
   7: grin_chain::txhashset::txhashset::sync_extending::h0ae6db619809b9d3 (0x564c1d42558f)
   8: grin_chain::pipe::sync_block_headers::h7a21a02151bf3d8b (0x564c1d452821)
   9: grin_chain::chain::Chain::sync_block_headers::h9e365341fad87206 (0x564c1d46d7f6)
  10: <grin_servers::common::adapters::NetToChainAdapter as grin_p2p::types::ChainAdapter>::headers_received::hf557479a51249eb1 (0x564c1d0dacfb)
  11: <grin_p2p::peers::Peers as grin_p2p::types::ChainAdapter>::headers_received::h822981cf5d2731a6 (0x564c1d3d787f)
  12: <grin_p2p::peer::TrackingAdapter as grin_p2p::types::ChainAdapter>::headers_received::h7c69aea3d60c6907 (0x564c1d3f8ba4)
  13: <grin_p2p::protocol::Protocol as grin_p2p::conn::MessageHandler>::consume::h737624cd2fc999f3 (0x564c1d4169c1)
  14: std::sys_common::backtrace::__rust_begin_short_backtrace::hb6d01aceadcd98f7 (0x564c1d3ec9db)
  15: std::panicking::try::do_call::h99a53672a7dd3555 (0x564c1d3e11b3)
  16: __rust_maybe_catch_panic (0x564c1d6c1419)
             at src/libpanic_unwind/lib.rs:92
  17: <F as alloc::boxed::FnBox<A>>::call_box::h5516f4a71d6729fd (0x564c1d408797)
  18: <alloc::boxed::Box<(dyn alloc::boxed::FnBox<A, Output=R> + 'a)> as core::ops::function::FnOnce<A>>::call_once::h2263113ac6678036 (0x564c1d6c085d)
             at /rustc/2aa4c46cfdd726e97360c2734835aa3515e8c858/src/liballoc/boxed.rs:744
      std::sys_common::thread::start_thread::h7f66efdea8634396
             at src/libstd/sys_common/thread.rs:14
      std::sys::unix::thread::Thread::new::thread_start::h6fb76bcf544b6181
             at src/libstd/sys/unix/thread.rs:81
  19: start_thread (0x7f7b281dc0a3)
  20: clone (0x7f7b27cfb87c)
  21: <unknown> (0x0)

@garyyu
Copy link
Contributor

garyyu commented Apr 23, 2019

Look like now it's a very high probability to get this problem, I'm upgrading all my 4 seed servers but 3 of them have this Corrupted storage, could not read an entry from data file and stuck there :(

@michaeldzjap michaeldzjap mentioned this issue Apr 25, 2019
Closed
@hashmap
Copy link
Contributor

hashmap commented Jul 3, 2019

@Kargakis should be fixed by #2812

@0xmichalis
Copy link
Contributor Author

I'm still seeing this in v2.0.0 - now by following a slightly different way:

I'm trying to sync from scratch but it's always get stuck a couple of blocks before the hard fork, and when I restart grin via systemd it crashes with the same error as this issue:

20190719 19:00:09.955 INFO grin - This is Grin version 2.0.0 (git v2.0.0), built for x86_64-unknown-linux-gnu by rustc 1.35.0 (3c235d560 2019-05-20).
20190719 19:00:09.955 DEBUG grin - Built with profile "release", features "".
20190719 19:00:09.955 WARN grin::cmd::server - Starting GRIN w/o UI...
20190719 19:00:09.956 INFO grin_servers::grin::server - Starting server, genesis block: 40adad0aec27
20190719 19:00:09.956 DEBUG grin_store::lmdb - DB Mapsize for /home/mkargaki/.grin/main/chain_data/lmdb is 268435456
20190719 19:00:09.956 DEBUG grin_store::leaf_set - bitmap 1 pos (18 bytes)
20190719 19:00:09.956 DEBUG grin_store::leaf_set - bitmap 1 pos (18 bytes)
20190719 19:00:09.958 DEBUG grin_chain::txhashset::txhashset - Rewind to header 000007010df1 at 262009
20190719 19:00:09.958 DEBUG grin_chain::txhashset::txhashset - txhashset: rewind_to_pos: header 524004, output 3033627, kernel 1580476
20190719 19:00:09.958 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: Custom { kind: Other, error: IOErr("failed to fill whole buffer", UnexpectedEof) }
20190719 19:00:09.958 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: Custom { kind: Other, error: IOErr("failed to fill whole buffer", UnexpectedEof) }
20190719 19:00:09.958 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: Custom { kind: Other, error: IOErr("failed to fill whole buffer", UnexpectedEof) }
20190719 19:00:09.958 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: Custom { kind: Other, error: IOErr("failed to fill whole buffer", UnexpectedEof) }
20190719 19:00:09.958 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: Custom { kind: Other, error: IOErr("failed to fill whole buffer", UnexpectedEof) }
20190719 19:00:09.958 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: Custom { kind: Other, error: IOErr("failed to fill whole buffer", UnexpectedEof) }
20190719 19:00:09.958 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: Custom { kind: Other, error: IOErr("failed to fill whole buffer", UnexpectedEof) }
20190719 19:00:09.958 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: Custom { kind: Other, error: IOErr("failed to fill whole buffer", UnexpectedEof) }
20190719 19:00:09.958 ERROR grin_store::types - Corrupted storage, could not read an entry from data file: Custom { kind: Other, error: IOErr("failed to fill whole buffer", UnexpectedEof) }
20190719 19:00:09.989 ERROR grin_util::logger - 
thread 'main' panicked at 'no root, invalid tree': src/libcore/option.rs:1034stack backtrace:
   0: grin_util::logger::send_panic_to_log::{{closure}}::h4951788bd28f5e9e (0x5600d1da21b1)
   1: rust_panic_with_hook (0x5600d1ede8d9)
             at src/libstd/panicking.rs:478
   2: continue_panic_fmt (0x5600d1ede372)
             at src/libstd/panicking.rs:381
   3: rust_begin_unwind (0x5600d1ede256)
   4: panic_fmt (0x5600d1ef826d)
             at src/libcore/panicking.rs:85
   5: expect_failed (0x5600d1ef82d7)
             at src/libcore/option.rs:1034
   6: grin_core::core::pmmr::pmmr::PMMR<T,B>::root::h4866a56b5e4b04e6 (0x5600d1c8cf16)
   7: grin_chain::txhashset::txhashset::Extension::validate_roots::h3f345796559f21cb (0x5600d1c2f66d)
   8: grin_chain::txhashset::txhashset::extending::hdf8c85246d9359ec (0x5600d1c21a92)
   9: grin_chain::chain::Chain::init::h597a7f031003e327 (0x5600d1c6c9a0)
  10: grin_servers::grin::server::Server::new::hde56540092b7030a (0x5600d1911e75)
  11: grin_servers::grin::server::Server::start::h632caa31385d0395 (0x5600d1707144)
  12: grin::cmd::server::start_server::h24519281179aa67c (0x5600d16fae21)
  13: grin::cmd::server::server_command::h5127c9bfa3ce033a (0x5600d16fb863)
  14: grin::real_main::h0d5a305d062ab86b (0x5600d17317c2)
  15: grin::main::h9882fdd29d1b551f (0x5600d17303a6)
  16: std::rt::lang_start::{{closure}}::hfc5dddc69489901b (0x5600d17081d3)
  17: {{closure}} (0x5600d1ede1f3)
             at src/libstd/rt.rs:49
      do_call<closure,i32>
             at src/libstd/panicking.rs:293
  18: __rust_maybe_catch_panic (0x5600d1ee415a)
             at src/libpanic_unwind/lib.rs:87
  19: try<i32,closure> (0x5600d1edecfd)
             at src/libstd/panicking.rs:272
      catch_unwind<closure,i32>
             at src/libstd/panic.rs:388
      lang_start_internal
             at src/libstd/rt.rs:48
  20: main (0x5600d1734a52)
  21: __libc_start_main (0x7fee8ac93f33)
  22: _start (0x5600d16e7639)
  23: <unknown> (0x0)

@quentinlesceller
Copy link
Member

Closing. Feel free to reopen if this is still the case in grin v3.0.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@hashmap @garyyu @0xmichalis @quentinlesceller