consensus keeps shutting down on monotonically increase error #936
Description
Describe the bug
When I restarted my docker node today, the following message occurred. I've restarted multiple times to see if it would somehow get "past" it.
INFO [10-15|10:10:29]
/build/snow/engine/common/bootstrapper.go#272: Bootstrapping started syncing with 1 vertices in the accepted frontier
DEBUG[10-15|10:10:29]/build/snow/engine/snowman/bootstrap/bootstrapper.go#142: Starting bootstrapping with 1 pending blocks and 1 from the accepted frontier
DEBUG[10-15|10:10:29]/build/snow/networking/router/handler.go#171: Finished handling message: Accepted
DEBUG[10-15|10:10:30] /build/health/service.go#98: starting to run router
DEBUG[10-15|10:10:30] /build/health/service.go#122: "router" returned healthy with: {"message":{"longestRunningRequest":"318.993739ms","outstandingRequests":1},"timestamp":"2021-10-15T10:10:30.081941564Z","duration":6421,"contiguousFailures":0,"timeOfFirstFailure":null}
DEBUG[10-15|10:10:30] /build/health/service.go#98: starting to run network
DEBUG[10-15|10:10:30] /build/health/service.go#122: "network" returned healthy with: {"message":{"connectedPeers":5,"sendFailRate":0.04655418565679642,"timeSinceLastMsgReceived":"1.082256599s","timeSinceLastMsgSent":"1.082256599s"},"timestamp":"2021-10-15T10:10:30.082176805Z","duration":22492,"contiguousFailures":0,"timeOfFirstFailure":null}
DEBUG[10-15|10:10:30] /build/health/service.go#98: starting to run isBootstrapped
DEBUG[10-15|10:10:30] /build/health/service.go#124: "isBootstrapped" returned unhealthy with: {"error":{"message":"primary subnet has not finished bootstrapping 'P', 'X', 'C'"},"timestamp":"2021-10-15T10:10:30.082163035Z","duration":10841,"contiguousFailures":2,"timeOfFirstFailure":"2021-10-15T10:10:20.081663985Z"}
DEBUG[10-15|10:10:30]/build/snow/networking/router/handler.go#140: Forwarding message to consensus: (MultiPut, NodeID: NodeID-DAtCoXfLT6Y83dgJ7FmQg8eR53hz37J79, RequestID: 4, Received: 1634292630, NumContainers: 2000)
INFO [10-15|10:10:31]/build/snow/engine/snowman/bootstrap/bootstrapper.go#347: bootstrapping fetched 4 blocks. Executing state transitions...
DEBUG[10-15|10:10:31]/build/snow/engine/common/queue/jobs.go#117: Executing: vyJVZL3tZQ7FBg9xGNqs44uzZ9AxBBsQxXCdhRvSYb2Xpep1u
ERROR[10-15|10:10:31]/build/snow/engine/snowman/bootstrap/block_job.go#86: block vyJVZL3tZQ7FBg9xGNqs44uzZ9AxBBsQxXCdhRvSYb2Xpep1u failed verification during bootstrapping due to time must monotonically increase
DEBUG[10-15|10:10:31]/build/snow/networking/router/handler.go#171: Finished handling message: MultiPut
FATAL[10-15|10:10:31]/build/snow/networking/router/handler.go#125: chain shutting down due to error "failed to execute job vyJVZL3tZQ7FBg9xGNqs44uzZ9AxBBsQxXCdhRvSYb2Xpep1u due to failed to verify block in bootstrapping: time must monotonically increase" while processing message: (MultiPut, NodeID: NodeID-DAtCoXfLT6Y83dgJ7FmQg8eR53hz37J79, RequestID: 4, Received: 1634292630, NumContainers: 2000)
INFO [10-15|10:10:31]/build/snow/engine/snowman/transitive.go#146: shutting down consensus engine
INFO [10-15|10:10:31] /build/node/node.go#1115: shutting down node with exit code 1
INFO [10-15|10:10:31] /build/ipcs/chainipc.go#101: shutting down chain IPCs
INFO [10-15|10:10:31] /build/chains/manager.go#802: shutting down chain manager
INFO [10-15|10:10:31] /build/snow/networking/router/chain_router.go#181: shutting down chain router
INFO [10-15|10:10:31] /build/network/network.go#1055: shutting down network
To Reproduce
Simply start my container with
docker run -ti -p 9650:9650 -p 9651:9651 --name avalanchego -v /...path.../.avalanchego avalanchego:master /avalanchego/build/avalanchego --network-id=mainnet --staking-enabled=false --public-ip=........ --log-level=debug --http-host=
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
Operating System
Distributor ID: Ubuntu
Description: Ubuntu 20.04.3 LTS
Release: 20.04
Codename: focal
By submitting this issue I agree to the Terms and Conditions of the Developer Accelerator Program.