v1.7.1 Unable to start,error message "failed to execute job TYXPtsxxxxxxxxyx due to failed to verify block in bootstrapping: proposed timestamp",How to repair

[BOBYou]

Describe the bug

  INFO [12-04|13:57:14] <P Chain> snow/engine/snowman/transitive.go#62: initializing consensus engine
  INFO [12-04|13:57:14] <P Chain> snow/engine/common/bootstrapper.go#82: Starting bootstrap...
  INFO [12-04|13:57:14] api/server/server.go#192: adding route /ext/bc/111111xxxxxYY
  INFO [12-04|13:57:14] api/server/server.go#114: HTTP API server listening on ":9650"
  INFO [12-04|13:57:21] <P Chain> snow/engine/common/bootstrapper.go#272: Bootstrapping started syncing with 1 vertices in the accepted frontier
  INFO [12-04|13:57:23] <P Chain> snow/engine/snowman/bootstrap/bootstrapper.go#360: bootstrapping fetched 277 blocks. Executing state transitions...
  ERROR[12-04|13:57:23] <P Chain> snow/engine/snowman/bootstrap/block_job.go#86: block TYXPtsxxxxxxxxxyx failed verification during bootstrapping due to proposed timestamp (2021-12-04 02:51:43 +0000 UTC) later than next staker change time (2021-12-04 02:15:38 +0000 UTC)
  FATAL[12-04|13:57:23] <P Chain> snow/networking/router/handler.go#144: chain shutting down due to error "failed to execute job TYXPtsxxxxxxxxyx due to failed to verify block in bootstrapping: proposed timestamp (2021-12-04 02:51:43 +0000 UTC) later than next staker change time (2021-12-04 02:15:38 +0000 UTC)" while processing message: (Op: multi_put, NodeID: NodeID-A8jypuxxxxxxxxxxxdiG, RequestID: 3, NumContainers: 1718)
  INFO [12-04|13:57:23] <P Chain> snow/engine/snowman/transitive.go#154: shutting down consensus engine
  INFO [12-04|13:57:23] node/node.go#1153: shutting down node with exit code 1
  INFO [12-04|13:57:23] ipcs/chainipc.go#101: shutting down chain IPCs
  INFO [12-04|13:57:23] chains/manager.go#828: shutting down chain manager
  INFO [12-04|13:57:23] snow/networking/router/chain_router.go#260: shutting down chain router
  INFO [12-04|13:57:23] network/network.go#656: shutting down network
  INFO [12-04|13:57:23] node/node.go#1177: cleaning up plugin subprocesses
  INFO [12-04|13:57:23] node/node.go#1180: finished node shutdown
  INFO [12-04|13:57:23] nat/nat.go#152: Unmapped all ports

To Reproduce
After the upgrade（v1.6 to v1.7.1）,avalanchego automatically restarts frequently.

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Operating System

LSB Version: core-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch
Distributor ID: Ubuntu
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal

./avalanchego --version
avalanche/1.7.1 [database=v1.4.5, commit=8cae4efe4d36f1ce6691c6cce0ad54477908d705]

Additional context
Add any other context about the problem here.

By submitting this issue I agree to the Terms and Conditions of the Developer Accelerator Program.

[3eph1r0th]

I can confirm this issue occurs on multiple validator nodes. Need to fix this ASAP.

[Rejinderi]

I am having this issue as well. Can't start my node.

[JonasDe]

Also seeing this issue. :/

Edit:

It happens consistently, so my node is effectively down for now. Is there any way to recover? I tried downgrading to 1.7.0 but the issue persists.

[patrick-ogrady]

We are actively working on debugging this issue. I'll keep this thread posted with our progress.

[Toinounet21]

Hi, I have this here, is there a link?

failed to execute job xxxxxxxxxxxx due to failed to verify block in bootstrapping: rpc error: code = Unavailable desc = transport is closing"

[BOBYou]

We are actively working on debugging this issue. I'll keep this thread posted with our progress.

v1.7.2 ，The question remains.
Do you want to completely resynchronize the data?
It's too much data, I don't want to do this.
Hope to solve it as soon as possible.

[3eph1r0th]

New issue after restart on multiple nodes:

INFO [12-19|07:03:30] api/server/server.go#114: HTTP API server listening on "0.0.0.0:9650"
INFO [12-19|07:03:42] <P Chain> snow/engine/common/bootstrapper.go#272: Bootstrapping started syncing with 1 vertices in the accepted frontier
INFO [12-19|07:03:43] <P Chain> snow/engine/snowman/bootstrap/bootstrapper.go#360: bootstrapping fetched 0 blocks. Executing state transitions...
ERROR[12-19|07:03:43] <P Chain> snow/engine/snowman/bootstrap/block_job.go#86: block aJiZkUvawP4FuYgFW8gdxtFxzNaPvByV72fgydXhrXCT4RTjG failed verification during bootstrapping due to failed semanticVerifySpend: failed to verify transfer: utxo amount and input amount should be same but are 6363206809 and 6363206720
FATAL[12-19|07:03:43] <P Chain> snow/networking/router/handler.go#144: chain shutting down due to error "failed to execute job aJiZkUvawP4FuYgFW8gdxtFxzNaPvByV72fgydXhrXCT4RTjG due to failed to verify block in bootstrapping: failed semanticVerifySpend: failed to verify transfer: utxo amount and input amount should be same but are 6363206809 and 6363206720" while processing message: (Op: multi_put, NodeID: NodeID-HiFv1DpKXkAAfJ1NHWVqQoojjznibZXHP, RequestID: 4, NumContainers: 2000)
INFO [12-19|07:03:43] <P Chain> snow/engine/snowman/transitive.go#154: shutting down consensus engine
INFO [12-19|07:03:43] node/node.go#1176: shutting down node with exit code 1
INFO [12-19|07:03:43] ipcs/chainipc.go#101: shutting down chain IPCs
INFO [12-19|07:03:43] chains/manager.go#844: shutting down chain manager
INFO [12-19|07:03:43] snow/networking/router/chain_router.go#260: shutting down chain router
INFO [12-19|07:03:43] network/network.go#704: shutting down network
INFO [12-19|07:03:43] node/node.go#1217: cleaning up plugin subprocesses
INFO [12-19|07:03:43] node/node.go#1220: finished node shutdown
INFO [12-19|07:03:43] nat/nat.go#152: Unmapped all ports

[danlaine]

If you are experiencing the above error, you need to bootstrap from an empty database while running >= v1.7.2 for your node to become healthy.

[DarkGhost7]

If you are experiencing the above error, you need to bootstrap from an empty database while running >= v1.7.2 for your node to become healthy.

Wow this sucks is there any more info about why? Currently experiencing the same problem was working fine when I first upgraded my node to 1.7.3, but after re-staking on the node it gives this error.

[lcgogo]

same issue after upgrade to 1.7.3

INFO [01-04|07:53:31] <P Chain> snow/engine/snowman/bootstrap/bootstrapper.go#326: fetched 15000 of 20257 blocks
INFO [01-04|07:53:39] health/service.go#130: "P" became healthy with: {"message":{"consensus":{},"vm":{"percentConnected":0.9644155448927572}},"timestamp":"2022-01-04T07:53:39.554365751Z","duration":142184,"contiguousFailures":0,"timeOfFirstFailure":null}
INFO [01-04|07:53:42] <P Chain> snow/engine/snowman/bootstrap/bootstrapper.go#326: fetched 20000 of 20257 blocks
INFO [01-04|07:53:43] <P Chain> snow/engine/snowman/bootstrap/bootstrapper.go#361: bootstrapping fetched 20257 blocks. Executing state transitions...
ERROR[01-04|07:53:47] <P Chain> snow/engine/snowman/bootstrap/block_job.go#86: block 2fwYFnYMMM5fEL3pqDkvm6vBpV4uiCprAE5TnBtER6si8dk5zr failed verification during bootstrapping due to failed semanticVerifySpend: failed to verify transfer: utxo amount and input amount should be same but are 6894112618 and 6894112582
FATAL[01-04|07:53:47] <P Chain> snow/networking/router/handler.go#145: chain shutting down due to error "failed to execute job 2fwYFnYMMM5fEL3pqDkvm6vBpV4uiCprAE5TnBtER6si8dk5zr due to failed to verify block in bootstrapping: failed semanticVerifySpend: failed to verify transfer: utxo amount and input amount should be same but are 6894112618 and 6894112582" while processing message: (Op: multi_put, NodeID: NodeID-Dw7tuwxpAmcpvVGp9JzaHAR3REPoJ8f2R, RequestID: 106, NumContainers: 39)
INFO [01-04|07:53:47] <P Chain> snow/engine/snowman/transitive.go#154: shutting down consensus engine
INFO [01-04|07:53:47] node/node.go#1164: shutting down node with exit code 1
INFO [01-04|07:53:47] ipcs/chainipc.go#101: shutting down chain IPCs
INFO [01-04|07:53:47] chains/manager.go#828: shutting down chain manager
INFO [01-04|07:53:47] snow/networking/router/chain_router.go#262: shutting down chain router
INFO [01-04|07:53:47] network/network.go#655: shutting down network
INFO [01-04|07:53:47] node/node.go#1205: cleaning up plugin subprocesses
INFO [01-04|07:53:47] node/node.go#1208: finished node shutdown
INFO [01-04|07:53:47] nat/nat.go#152: Unmapped all ports