Slow Sync After Restart The Avalanchego

[lightmelv]

My Avalanchego has stopped sync at 22 April and I restarted it.
I have waited for 2 days and it hasn't catched up with the latest block.
Are there any ways to make the sync faster?

Logs

DEBUG[04-24|02:47:52.329] <C Chain> github.com/ava-labs/coreth/plugin/evm/block.go:145: Accepting block e09cc1d0d044538acb56c8ed7141cb2d55e34e85d8348c3a64ac8e44487cda5a (2hvRPo8BZzRhm4KRcdUez6yUkuxmXoZBjo9Cu6Q6UxTzRLgyPJ) at height 29093499 
DEBUG[04-24|02:47:52.329] <C Chain> github.com/ava-labs/coreth/trie/database.go:506: Dereferenced trie from memory database nodes=0         size=0.00B     time="3.79µs"      gcnodes=78        gcsize=21.50KiB   gctime=6.944513ms    livenodes=24      livesize=4.53KiB
DEBUG[04-24|02:47:52.334] <C Chain> github.com/ava-labs/coreth/trie/database.go:506: Dereferenced trie from memory database nodes=723       size=383.58KiB time=5.395137ms    gcnodes=1,636,414 gcsize=756.43MiB  gctime=11.297413151s livenodes=383,561 livesize=104.92MiB
DEBUG[04-24|02:47:52.348] <C Chain> github.com/ava-labs/coreth/trie/database.go:676: Persisted nodes from memory database nodes=1970      size=498.22KiB time=12.938385ms   flushnodes=1,048,429 flushsize=236.26MiB  flushtime=8.033057069s  livenodes=381,591 livesize=104.43MiB
DEBUG[04-24|02:47:52.349] <C Chain> github.com/ava-labs/coreth/core/state/snapshot/generate.go:204: Journalled generator progress progress=done
DEBUG[04-24|02:47:52.349] <C Chain> github.com/ava-labs/coreth/core/state/snapshot/snapshot.go:701: Journalled disk layer root=3c6b88..406cc1 complete=true
DEBUG[04-24|02:47:52.349] <C Chain> github.com/ava-labs/coreth/core/state/snapshot/snapshot.go:484: Flattened snapshot tree blockHash=e09cc1..7cda5a root=3c6b88..406cc1 size=1         elapsed=1.281ms
DEBUG[04-24|02:47:56.254] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:1364: Inserted new block number=29,093,500 hash=1fd5df..d09b48 parentHash=e09cc1..7cda5a uncles=0 txs=2   gas=7,499,286 elapsed=3.854s      root=50a475..1f2b7c baseFeePerGas=35,055,674,917  blockGasCost=200,000
DEBUG[04-24|02:47:56.254] <C Chain> github.com/ava-labs/coreth/core/tx_pool.go:1408: Reinjecting stale transactions count=0
DEBUG[04-24|02:47:56.255] <C Chain> github.com/ava-labs/coreth/plugin/evm/block.go:145: Accepting block 1fd5df7e239ac8426ca815c45a5bfadef665f75d06bdd01e4711722902d09b48 (F2CEMYEohpcsqb6gvvGwfaWRYbEPKxRbyVUzzdheL8fpJvAb1) at height 29093500 
DEBUG[04-24|02:47:56.255] <C Chain> github.com/ava-labs/coreth/trie/database.go:506: Dereferenced trie from memory database nodes=0         size=0.00B     time="3.788µs"     gcnodes=78        gcsize=21.50KiB   gctime=6.946735ms    livenodes=24      livesize=4.53KiB
DEBUG[04-24|02:47:56.257] <C Chain> github.com/ava-labs/coreth/trie/database.go:506: Dereferenced trie from memory database nodes=182       size=77.93KiB  time=1.209486ms    gcnodes=1,636,596 gcsize=756.51MiB  gctime=11.298620854s livenodes=383,085 livesize=104.96MiB
DEBUG[04-24|02:47:56.277] <C Chain> github.com/ava-labs/coreth/trie/database.go:676: Persisted nodes from memory database nodes=2862      size=725.68KiB time=20.323679ms   flushnodes=1,051,291 flushsize=236.97MiB  flushtime=8.053378597s  livenodes=380,223 livesize=104.25MiB
DEBUG[04-24|02:47:56.278] <C Chain> github.com/ava-labs/coreth/core/state/snapshot/generate.go:204: Journalled generator progress progress=done
DEBUG[04-24|02:47:56.278] <C Chain> github.com/ava-labs/coreth/core/state/snapshot/snapshot.go:701: Journalled disk layer root=50a475..1f2b7c complete=true
DEBUG[04-24|02:47:56.278] <C Chain> github.com/ava-labs/coreth/core/state/snapshot/snapshot.go:484: Flattened snapshot tree blockHash=1fd5df..d09b48 root=50a475..1f2b7c size=1         elapsed=1.249ms

Operating System
Ubuntu 20.04.4 LTS

Avalanchego Version
avalanche/1.10.0

[moraesjeremias]

@lightmelv what's the result of /ext/health on your node?
One on my AVAX nodes simply stopped syncing C and P chains, while X is doing fine!

[lightmelv]

@moraesjeremias It's the same like you, the C and P chains stop syncing

{
  "jsonrpc": "2.0",
  "result": {
    "checks": {
      "C": {
        "message": {
          "consensus": {
            "longestRunningBlock": "1h39m46.513476444s",
            "outstandingBlocks": 10
          },
          "vm": null
        },
        "error": "snowman consensus is not healthy reason: block processing time 1h39m46.513476444s > 30s",
        "timestamp": "2023-04-25T00:56:41.183990493Z",
        "duration": 14687,
        "contiguousFailures": 199,
        "timeOfFirstFailure": "2023-04-24T23:17:41.183876428Z"
      },
      "P": {
        "message": {
          "consensus": {
            "longestRunningBlock": "44m35.175653436s",
            "outstandingBlocks": 12
          },
          "vm": {
            "primary-percentConnected": 0.9868969496493554
          }
        },
        "error": "snowman consensus is not healthy reason: block processing time 44m35.175653436s > 30s",
        "timestamp": "2023-04-25T00:56:41.18472994Z",
        "duration": 753733,
        "contiguousFailures": 89,
        "timeOfFirstFailure": "2023-04-25T00:12:41.184390292Z"
      },
      "X": {
        "message": {
          "consensus": {
            "longestRunningVertex": "0s",
            "outstandingVertices": 0,
            "snowstorm": {
              "longestRunningTransaction": "0s",
              "outstandingTransactions": 0
            }
          },
          "vm": null
        },
        "timestamp": "2023-04-25T00:56:41.183973743Z",
        "duration": 16528
      },
      "bootstrapped": {
        "message": [],
        "timestamp": "2023-04-25T00:56:41.183974562Z",
        "duration": 5761
      },
      "database": {
        "timestamp": "2023-04-25T00:56:41.183983341Z",
        "duration": 979
      },
      "diskspace": {
        "message": {
          "availableDiskBytes": 690363117568
        },
        "timestamp": "2023-04-25T00:56:41.183972076Z",
        "duration": 5602
      },
      "network": {
        "message": {
          "connectedPeers": 1165,
          "sendFailRate": 0,
          "timeSinceLastMsgReceived": "183.977951ms",
          "timeSinceLastMsgSent": "183.977951ms"
        },
        "timestamp": "2023-04-25T00:56:41.183981396Z",
        "duration": 5850
      },
      "router": {
        "message": {
          "longestRunningRequest": "139.241651ms",
          "outstandingRequests": 2
        },
        "timestamp": "2023-04-25T00:56:41.184008519Z",
        "duration": 12075
      }
    },
    "healthy": false
  },
  "id": 1
}

Do you know how to fix it?

I have used 8 CPU, 16 GB RAM and SSD storage

[scottsisco]

I am here to say I am having the same problem as @moraesjeremias and @lightmelv. When looking at /ext/health C-Chain and P-Chain are locking up and X-chain is working fine.

[StephenButtolph]

If anyone seeing this behavior has any debug logs - that's really the first place I'd be interested in looking. So, if this is repeatable for anyone, running with --log-level=debug should be interesting. (A full zip of the logs directory would be great).

[Liamlu28]

After I upgraded to version 1.10.0, the outgoing blocks did slow down because we added a monitor that will check once a second if the blocks are in sync with the main network, but recently some syncs have been taking a long time to process, I'll send out the logs later.

[lightmelv]

@scottsisco try to upgrade your RAM to 32 GB, I have do this and it synced

[StephenButtolph]

I'd also be interested to know how people are running avalanchego that are experiencing this issue.

Are people using:

• Self compiled binaries
• Binaries from the release (installer script)
• Deb files from the apt repo
• Docker image

[lightmelv]

@StephenButtolph I use Self compiled binaries