The sync child process broken and not restart

[yanguoyu]

What happened

The synced block number stops increasing, after restarting the Neuron it will reach the latest block number quickly.

Here is the log

[2023-10-29 20:55:21.011] [error] Sync:ChildProcess: node:internal/process/promises:288
            triggerUncaughtException(err, true /* fromPromise */);
            ^

[UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "AxiosError: timeout of 30000ms exceeded".] {
  code: 'ERR_UNHANDLED_REJECTION'
}

Node.js v18.14.0

Reason

From the log, we know that the reason is sync child process stopped with an uncatched exception so it can not restart again.

[devchenyan]

This error occurs in Neuron v0.111.0 and earlier because of an axios request timeout error in @nervosnetwork/ckb-sdk-core.
Neuron v0.111.1 replaced @nervosnetwork/ckb-sdk-core with @ckb-lumos/rpc ,and the @ckb-lumos/rpc removed axios.
I have tried several times to turn on Neuron after hibernating the computer, and it can restart sync child process normally.
So this issue can be considered resolved for the time being.

[yanguoyu]

It seems will happen at version 0.112.0.
Here is the user's log, the user's Neuron version is 0.112.0

[2023-12-11T21:18:00.671Z] [info]  Database:	saved synced block #6228681
[2023-12-11T21:26:49.102Z] [error] Sync:ChildProcess: 
node:internal/process/promises:288
            triggerUncaughtException(err, true /* fromPromise */);
            ^

[2023-12-11T21:26:49.105Z] [error] Sync:ChildProcess: AxiosError: connect ENOBUFS 127.0.0.1:8114 - Local (undefined:undefined)
    at internalConnect (node:net:1066:16)
    at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18)
    at node:net:1250:9
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11) {
  port: 8114,
  address: '127.0.0.1',
  syscall: 'connect',
  code: 'ENOBUFS',
  errno: -4060,
  config: {
    transitional: {
      silentJSONParsing: true,
      forcedJSONParsing: true,
      clarifyTimeoutError: false
    },
    adapter: [Function: httpAdapter],
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 30000,
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: -1,
    env: {
      FormData: [Function: FormData] {
        LINE_BREAK: '\r\n',
        DEFAULT_CONTENT_TYPE: 'application/octet-stream'
      }
    },
    validateStatus: [Function: validateStatus],
    headers: {
      Accept: 'application/json, text/plain, */*',
      'Content-Type': 'application/json',
      'User-Agent': 'axios/0.27.2',
      'Content-Length': 342
    },
    method: 'post',
    data: '{"id":9971,"method":"get_cells","params":[{"script":{"code_hash":"0xe4d4ecc6e5f9a059bf2f7a82cca292083aebc0c421566a52484fe2ec51a9fb0c","hash_type":"type","args":"0x665aaf111f26fd850fd3a97d17c1d603179cac31"},"script_type":"lock","filter":{"block_range":["0x5f0ac9","0x5f0aca"]},"script_search_mode":"prefix"},"asc","0x64",null],"jsonrpc":"2.0"}',
    url: 'http://127.0.0.1:8114'
  },
  request: <ref *2> Writable {
    _writableState: WritableState {
      objectMode: false,
      highWaterMark: 16384,
      finalCalled: false,
      needDrain: false,
      ending: false,
      ended: false,
      finished: false,
      destroyed: false,
      decodeStrings: true,
      defaultEncoding: 'utf8',
      length: 0,
      writing: false,
      corked: 0,
      sync: true,
      bufferProcessing: false,
      onwrite: [Function: bound onwrite],
      writecb: null,
      writelen: 0,
      afterWriteTickInfo: null,
      buffered: [],
      bufferedIndex: 0,
      allBuffers: true,
      allNoop: true,
      pendingcb: 0,
      constructed: true,
      prefinished: false,
      errorEmitted: false,
      emitClose: true,
      autoDestroy: true,
      errored: null,
      closed: false,
      closeEmitted: false,
      [Symbol(kOnFinished)]: []
    },
    _events: [Object: null prototype] {
      response: [Function: handleResponse],
      error: [Function: handleRequestError],
      socket: [ [Function: handleRequestSocket], [Function: destroyOnTimeout] ]
    },
    _eventsCount: 3,
    _maxListeners: undefined,
    _options: {
      maxRedirects: 21,
      maxBodyLength: 10485760,
      protocol: 'http:',
      path: '/',
      method: 'POST',
      headers: {
        Accept: 'application/json, text/plain, */*',
        'Content-Type': 'application/json',
        'User-Agent': 'axios/0.27.2',
        'Content-Length': 342
      },
      agent: undefined,
      agents: { http: undefined, https: undefined },
      auth: undefined,
      hostname: '127.0.0.1',
      port: '8114',
      nativeProtocols: {
        'http:': {
          _connectionListener: [Function: connectionListener],
          METHODS: [
            'ACL',         'BIND',       'CHECKOUT',
            'CONNECT',     'COPY',       'DELETE',
            'GET',         'HEAD',       'LINK',
            'LOCK',        'M-SEARCH',   'MERGE',
            'MKACTIVITY',  'MKCALENDAR', 'MKCOL',
            'MOVE',        'NOTIFY',     'OPTIONS',
            'PATCH',       'POST',       'PROPFIND',
            'PROPPATCH',   'PURGE',      'PUT',
            'REBIND',      'REPORT',     'SEARCH',
            'SOURCE',      'SUBSCRIBE',  'TRACE',
            'UNBIND',      'UNLINK',     'UNLOCK',
            'UNSUBSCRIBE'
          ],
          STATUS_CODES: {
            '100': 'Continue',
            '101': 'Switching Protocols',
            '102': 'Processing',
            '103': 'Early Hints',
            '200': 'OK',
            '201': 'Created',
            '202': 'Accepted',
            '203': 'Non-Authoritative Information',
            '204': 'No Content',
            '205': 'Reset Content',
            '206': 'Partial Content',
            '207': 'Multi-Status',
            '208': 'Already Reported',
            '226': 'IM Used',
            '300': 'Multiple Choices',
            '301': 'Moved Permanently',
            '302': 'Found',
            '303': 'See Other',
            '304': 'Not Modified',
            '305': 'Use Proxy',
            '307': 'Temporary Redirect',
            '308': 'Permanent Redirect',
            '400': 'Bad Request',
            '401': 'Unauthorized',
            '402': 'Payment Required',
            '403': 'Forbidden',
            '404': 'Not Found',
            '405': 'Method Not Allowed',
            '406': 'Not Acceptable',
            '407': 'Proxy Authentication Required',
            '408': 'Request Timeout',
            '409': 'Conflict',
            '410': 'Gone',
            '411': 'Length Required',
            '412': 'Precondition Failed',
            '413': 'Payload Too Large',
            '414': 'URI Too Long',
            '415': 'Unsupported Media Type',
            '416': 'Range Not Satisfiable',
            '417': 'Expectation Failed',
            '418': "I'm a Teapot",
            '421': 'Misdirected Request',
            '422': 'Unprocessable Entity',
            '423': 'Locked',
            '424': 'Failed Dependency',
            '425': 'Too Early',
            '426': 'Upgrade Required',
            '428': 'Precondition Required',
            '429': 'Too Many Requests',
            '431': 'Request Header Fields Too Large',
            '451': 'Unavailable For Legal Reasons',
            '500': 'Internal Server Error',
            '501': 'Not Implemented',
            '502': 'Bad Gateway',
            '503': 'Service Unavailable',
            '504': 'Gateway Timeout',
            '505': 'HTTP Version Not Supported',
            '506': 'Variant Also Negotiates',
            '507': 'Insufficient Storage',
            '508': 'Loop Detected',
            '509': 'Bandwidth Limit Exceeded',
            '510': 'Not Extended',
            '511': 'Network Authentication Required'
          },
          Agent: [Function: Agent] { defaultMaxSockets: Infinity },
          ClientRequest: [Function: ClientRequest],
          IncomingMessage: [Function: IncomingMessage],
          OutgoingMessage: [Function: OutgoingMessage],
          Server: [Function: Server],
          ServerResponse: [Function: ServerResponse],
          createServer: [Function: createServer],
          validateHeaderName: [Function: __node_internal_],
          validateHeaderValue: [Function: __node_internal_],
          get: [Function: get],
          request: [Function: request],
          setMaxIdleHTTPParsers: [Function: setMaxIdleHTTPParsers],
          maxHeaderSize: [Getter],
          globalAgent: [Getter/Setter]
        },
        'https:': {
          Agent: [Function: Agent],
          globalAgent: Agent {
            _events: [Object: null prototype],
            _eventsCount: 2,
            _maxListeners: undefined,
            defaultPort: 443,
            protocol: 'https:',
            options: [Object: null prototype],
            requests: [Object: null prototype] {},
            sockets: [Object: null prototype] {},
            freeSockets: [Object: null prototype] {},
            keepAliveMsecs: 1000,
            keepAlive: false,
            maxSockets: Infinity,
            maxFreeSockets: 256,
            scheduling: 'lifo',
            maxTotalSockets: Infinity,
            totalSocketCount: 0,
            maxCachedSessions: 100,
            _sessionCache: [Object],
            [Symbol(kCapture)]: false
          },
          Server: [Function: Server],
          createServer: [Function: createServer],
          get: [Function: get],
          request: [Function: request]
        }
      },
      pathname: '/'
    },
    _ended: false,
    _ending: true,
    _redirectCount: 0,
    _redirects: [],
    _requestBodyLength: 342,
    _requestBodyBuffers: [
      {
        data: Buffer(342) [Uint8Array] [
          123,  34, 105, 100,  34,  58,  57,  57,  55,  49,  44,  34,
          109, 101, 116, 104, 111, 100,  34,  58,  34, 103, 101, 116,
           95,  99, 101, 108, 108, 115,  34,  44,  34, 112,  97, 114,
           97, 109, 115,  34,  58,  91, 123,  34, 115,  99, 114, 105,
          112, 116,  34,  58, 123,  34,  99, 111, 100, 101,  95, 104,
           97, 115, 104,  34,  58,  34,  48, 120, 101,  52, 100,  52,
          101,  99,  99,  54, 101,  53, 102,  57,  97,  48,  53,  57,
           98, 102,  50, 102,  55,  97,  56,  50,  99,  99,  97,  50,
           57,  50,  48,  56,
          ... 242 more items
        ],
        encoding: undefined
      }
    ],
    _onNativeResponse: [Function (anonymous)],
    _currentRequest: <ref *1> ClientRequest {
      _events: [Object: null prototype] {
        response: [Function: bound onceWrapper] {
          listener: [Function (anonymous)]
        },
        abort: [Function (anonymous)],
        aborted: [Function (anonymous)],
        connect: [Function (anonymous)],
        error: [Function (anonymous)],
        socket: [Function (anonymous)],
        timeout: [Function (anonymous)]
      },
      _eventsCount: 7,
      _maxListeners: undefined,
      outputData: [
        {
          data: 'POST / HTTP/1.1\r\n' +
            'Accept: application/json, text/plain, */*\r\n' +
            'Content-Type: application/json\r\n' +
            'User-Agent: axios/0.27.2\r\n' +
            'Content-Length: 342\r\n' +
            'Host: 127.0.0.1:8114\r\n' +
            'Connection: close\r\n' +
            '\r\n',
          encoding: 'latin1',
          callback: null
        },
        {
          data: Buffer(342) [Uint8Array] [
            123,  34, 105, 100,  34,  58,  57,  57,  55,  49,  44,  34,
            109, 101, 116, 104, 111, 100,  34,  58,  34, 103, 101, 116,
             95,  99, 101, 108, 108, 115,  34,  44,  34, 112,  97, 114,
             97, 109, 115,  34,  58,  91, 123,  34, 115,  99, 114, 105,
            112, 116,  34,  58, 123,  34,  99, 111, 100, 101,  95, 104,
             97, 115, 104,  34,  58,  34,  48, 120, 101,  52, 100,  52,
            101,  99,  99,  54, 101,  53, 102,  57,  97,  48,  53,  57,
             98, 102,  50, 102,  55,  97,  56,  50,  99,  99,  97,  50,
             57,  50,  48,  56,
            ... 242 more items
          ],
          encoding: undefined,
          callback: [Function (anonymous)]
        }
      ],
      outputSize: 524,
      writable: true,
      destroyed: false,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 342,
      _hasBody: true,
      _trailer: '',
      finished: false,
      _headerSent: true,
      _closed: false,
      socket: Socket {
        connecting: false,
        _hadError: true,
        _parent: null,
        _host: null,
        _closeAfterHandlingError: false,
        _readableState: ReadableState {
          objectMode: false,
          highWaterMark: 16384,
          buffer: BufferList { head: null, tail: null, length: 0 },
          length: 0,
          pipes: [],
          flowing: true,
          ended: false,
          endEmitted: false,
          reading: false,
          constructed: true,
          sync: true,
          needReadable: true,
          emittedReadable: false,
          readableListening: false,
          resumeScheduled: false,
          errorEmitted: true,
          emitClose: false,
          autoDestroy: true,
          destroyed: true,
          errored: Error: connect ENOBUFS 127.0.0.1:8114 - Local (undefined:undefined)
              at internalConnect (node:net:1066:16)
              at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18)
              at node:net:1250:9
              at process.processTicksAndRejections (node:internal/process/task_queues:77:11) {
            errno: -4060,
            code: 'ENOBUFS',
            syscall: 'connect',
            address: '127.0.0.1',
            port: 8114
          },
          closed: true,
          closeEmitted: true,
          defaultEncoding: 'utf8',
          awaitDrainWriters: null,
          multiAwaitDrain: false,
          readingMore: false,
          dataEmitted: false,
          decoder: null,
          encoding: null,
          [Symbol(kPaused)]: false
        },
        _events: [Object: null prototype] {
          end: [Function: onReadableStreamEnd],
          connect: [Function: bound onceWrapper] {
            listener: [Function (anonymous)]
          },
          free: [Function: onFree],
          close: [ [Function: onClose], [Function: socketCloseListener] ],
          timeout: [ [Function: onTimeout], [Function: destroy] ],
          agentRemove: [Function: onRemove],
          error: [Function: socketErrorListener],
          drain: [Function: ondrain]
        },
        _eventsCount: 8,
        _maxListeners: undefined,
        _writableState: WritableState {
          objectMode: false,
          highWaterMark: 16384,
          finalCalled: false,
          needDrain: false,
          ending: false,
          ended: false,
          finished: false,
          destroyed: true,
          decodeStrings: false,
          defaultEncoding: 'utf8',
          length: 0,
          writing: false,
          corked: 0,
          sync: true,
          bufferProcessing: false,
          onwrite: [Function: bound onwrite],
          writecb: null,
          writelen: 0,
          afterWriteTickInfo: null,
          buffered: [],
          bufferedIndex: 0,
          allBuffers: true,
          allNoop: true,
          pendingcb: 0,
          constructed: true,
          prefinished: false,
          errorEmitted: true,
          emitClose: false,
          autoDestroy: true,
          errored: Error: connect ENOBUFS 127.0.0.1:8114 - Local (undefined:undefined)
              at internalConnect (node:net:1066:16)
              at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18)
              at node:net:1250:9
              at process.processTicksAndRejections (node:internal/process/task_queues:77:11) {
            errno: -4060,
            code: 'ENOBUFS',
            syscall: 'connect',
            address: '127.0.0.1',
            port: 8114
          },
          closed: true,
          closeEmitted: true,
          [Symbol(kOnFinished)]: []
        },
        allowHalfOpen: false,
        _sockname: null,
        _pendingData: null,
        _pendingEncoding: '',
        server: null,
        _server: null,
        parser: null,
        _httpMessage: [Circular *1],
        [Symbol(async_id_symbol)]: 349076,
        [Symbol(kHandle)]: null,
        [Symbol(lastWriteQueueSize)]: 0,
        [Symbol(timeout)]: null,
        [Symbol(kBuffer)]: null,
        [Symbol(kBufferCb)]: null,
        [Symbol(kBufferGen)]: null,
        [Symbol(kCapture)]: false,
        [Symbol(kSetNoDelay)]: true,
        [Symbol(kSetKeepAlive)]: true,
        [Symbol(kSetKeepAliveInitialDelay)]: 60,
        [Symbol(kBytesRead)]: 0,
        [Symbol(kBytesWritten)]: 0
      },
      _header: 'POST / HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: application/json\r\n' +
        'User-Agent: axios/0.27.2\r\n' +
        'Content-Length: 342\r\n' +
        'Host: 127.0.0.1:8114\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: Agent {
        _events: [Object: null prototype] {
          free: [Function (anonymous)],
          newListener: [Function: maybeEnableKeylog]
        },
        _eventsCount: 2,
        _maxListeners: undefined,
        defaultPort: 80,
        protocol: 'http:',
        options: [Object: null prototype] { noDelay: true, path: null },
        requests: [Object: null prototype] {},
        sockets: [Object: null prototype] {
          '127.0.0.1:8114:': [ [Socket], [Socket] ]
        },
        freeSockets: [Object: null prototype] {},
        keepAliveMsecs: 1000,
        keepAlive: false,
        maxSockets: Infinity,
        maxFreeSockets: 256,
        scheduling: 'lifo',
        maxTotalSockets: Infinity,
        totalSocketCount: 2,
        [Symbol(kCapture)]: false
      },
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/',
      _ended: false,
      res: null,
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: '127.0.0.1',
      protocol: 'http:',
      _redirectable: [Circular *2],
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kEndCalled)]: false,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype] {
        accept: [ 'Accept', 'application/json, text/plain, */*' ],
        'content-type': [ 'Content-Type', 'application/json' ],
        'user-agent': [ 'User-Agent', 'axios/0.27.2' ],
        'content-length': [ 'Content-Length', 342 ],
        host: [ 'Host', '127.0.0.1:8114' ]
      },
      [Symbol(errored)]: null,
      [Symbol(kUniqueHeaders)]: null
    },
    _currentUrl: 'http://127.0.0.1:8114/',
    _timeout: null,
    [Symbol(kCapture)]: false
  }
}

Node.js v18.14.0

[devchenyan]

fixed: nervosnetwork/neuron#3025