Synced node stops making progress

[hashmap]

After running a few days a node (commit ea1d14d) stuck at some point with error:

20190821 17:08:07.836 DEBUG grin_chain::txhashset::txhashset - txhashset: rewind_to_pos: header 613517, output 3485577, kernel 1846143
20190821 17:08:07.837 DEBUG grin_chain::txhashset::txhashset - Error returned, discarding txhashset extension: TxHashSetErr: missing left sibling in tree, should not have been pruned
 Cause: Unknown
 Backtrace:
20190821 17:08:07.837 INFO grin_chain::chain - Rejected block 000000c81f28 at 306764: Error { inner:

TxHashSetErr: missing left sibling in tree, should not have been pruned }
20190821 17:08:07.837 DEBUG grin_servers::common::adapters - process_block: block 000000c81f28 refused by chain: Other Error: Error { inner:

TxHashSetErr: missing left sibling in tree, should not have been pruned }
20190821 17:08:07.837 DEBUG grin_servers::grin::sync::body_sync - block_sync: 306763/312628 requesting blocks [000000c81f28] from 3 peers

After restart it was able to continue sync, so stored data was not corrupted

[jyap808]

Built from HEAD and saw the same issues where it didn't complete a full sync:

20190828 00:22:56.163 INFO grin_servers::common::types - DandelionEpoch: next_epoch: is_stem: true (90%), relay: Some(PeerAddr(V4(47.244.
116.112:3414)))
20190828 00:23:40.328 INFO grin_chain::chain - Rejected block 000002b38f1a at 318265: Error { inner: 

Old Block }
20190828 00:23:40.328 INFO grin_chain::chain - Rejected block 0000019c59cd at 318271: Error { inner: 

Old Block }
20190828 00:23:40.328 INFO grin_chain::chain - Rejected block 0000010cdbe8 at 318277: Error { inner: 

Old Block }
20190828 00:23:40.328 INFO grin_chain::chain - Rejected block 0000024cbd5a at 318283: Error { inner: 

Old Block }
20190828 00:23:40.328 INFO grin_chain::chain - Rejected block 000000eb214a at 318289: Error { inner: 

Old Block }
20190828 00:23:40.328 INFO grin_chain::chain - Rejected block 0000021a6852 at 318295: Error { inner: 

Old Block }

Now testing and doing a clean build and sync from the parent commit to the referenced OP commit:

git checkout 705fcbb1a62a1cfddb1e8b4fc9c30bab4a388818

[jyap808]

Sync from commit 705fcbb doesn't get past block 321789:

20190828 18:34:00.728 INFO grin_servers::common::adapters - Received 24 block headers from 18.179.2.142:3414
20190828 18:34:13.866 INFO grin_chain::chain - Rejected block 00000e661ced at 321789: Error { inner: 

Invalid Root }
20190828 18:34:13.870 INFO grin_servers::common::adapters - Received 24 block headers from 39.104.49.1:13414
20190828 18:34:26.765 INFO grin_chain::chain - Rejected block 00000e661ced at 321789: Error { inner: 

Invalid Root }
20190828 18:34:39.724 INFO grin_chain::chain - Rejected block 00000e661ced at 321789: Error { inner: 

Invalid Root }
20190828 18:34:52.672 INFO grin_chain::chain - Rejected block 00000e661ced at 321789: Error { inner: 

Invalid Root }
20190828 18:35:05.480 INFO grin_chain::chain - Rejected block 00000e661ced at 321789: Error { inner: 

[antiochp]

Hi @jyap808 can you checkout latest master and see if your node can unstick itself?

We think this is resolved with #3005

Thanks!

[jyap808]

I ran a full sync with Grin HEAD and it worked fine, no issues. Patch LGTM

[jyap808]

Looks like there's still an issue @antiochp. Node is stuck again.

20190901 00:51:06.258 INFO grin_chain::chain - Rejected block 000000dbce95 at 327936: Error { inner: 

Invalid Root }
20190901 00:51:17.311 INFO grin_chain::chain - Rejected block 000000dbce95 at 327936: Error { inner: 

Invalid Root }
20190901 00:51:28.413 INFO grin_chain::chain - Rejected block 000000dbce95 at 327936: Error { inner: 

Invalid Root }
20190901 00:51:39.478 INFO grin_chain::chain - Rejected block 000000dbce95 at 327936: Error { inner: 

Invalid Root }
20190901 00:51:50.527 INFO grin_chain::chain - Rejected block 000000dbce95 at 327936: Error { inner: 

Invalid Root }
20190901 00:52:01.679 INFO grin_chain::chain - Rejected block 000000dbce95 at 327936: Error { inner: 

Invalid Root }
20190901 00:52:12.739 INFO grin_chain::chain - Rejected block 000000dbce95 at 327936: Error { inner: 

Invalid Root }
20190901 00:52:23.767 INFO grin_chain::chain - Rejected block 000000dbce95 at 327936: Error { inner: 

Invalid Root }
20190901 00:52:34.820 INFO grin_chain::chain - Rejected block 000000dbce95 at 327936: Error { inner: 

Invalid Root }
20190901 00:52:45.906 INFO grin_chain::chain - Rejected block 000000dbce95 at 327936: Error { inner: 

Invalid Root }

Stuck on the fork:
https://grinexplorer.net/block/327935

"There was a fork in the blockchain resulting in several competing blocks at this height"

[jyap808]

This can probably be closed. New issue covered in #3030

[garyyu]

@jyap808 I think issue 3030 is not the case of this issue which has an obvious error indicator TxHashSetErr: missing left sibling in tree.

(and your Invalid Root case should already been fixed in latest master branch, perhaps by PR 3017)

(I don't use the # to avoid linking these unrelated issues together)

[yeastplume]

Looks like this can likely be closed?

[quentinlesceller]

Closing.