read ECONNRESET at TLSWrap.onStreamRead #35824

master-genius · 2020-10-27T06:54:52Z

Version: v14.14.0
Platform: Linux
Subsystem:Ubuntu 20.04

What steps will reproduce the bug?

I create http2 server and benchmark use ab:

ab -n 5000 -c 50 'https://localhost:1234/?name=123'

after this, the server is crashed.

events.js:292
      throw er; // Unhandled 'error' event
      ^

Error: read ECONNRESET
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:209:20)
Emitted 'error' event on TLSSocket instance at:
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  errno: -104,
  code: 'ECONNRESET',
  syscall: 'read'
}

I had try to handle all 'error' event (socket, http2Stream, http2Session), but all are ineffective.

The text was updated successfully, but these errors were encountered:

mmomtchev · 2020-10-27T09:30:26Z

Can you post your code, the default server doesn't do that. One of the causes can be blocking the event loop.

master-genius · 2020-10-27T11:55:46Z

Can you post your code, the default server doesn't do that. One of the causes can be blocking the event loop.

In order to avoid the influence of the framework, http2 module is used for testing.

'use strict'

const http2 = require('http2')
const fs = require('fs')

process.chdir(__dirname)

let serv = http2.createSecureServer({
  cert : fs.readFileSync('./rsa/localhost.cert'),
  key : fs.readFileSync('./rsa/localhost.key')
})

serv.on('stream', (stream, headers) => {

  stream.on('frameError', (type,code,id)=>{})

  stream.on('error', (err) => {stream.destroy()})

  stream.on('aborted', () => {
    if (!stream.destroyed) {
      stream.destroy()
    }
  })

  stream.on('data', chunk => {

  })

  stream.on('end', () => {
    stream.end('ok')
  })

})

serv.on('tlsClientError', (err, tls) => {
  console.error(err)
  tls.destroy()
})

serv.setTimeout(5000, (sess) => {
  sess.destroy()
})

serv.on('connection', (sock) => {
  sock.setTimeout(2000, () => {
    if (!sock.destroyed) {
      sock.destroy()
    }
  })
})

serv.listen(1235)

test command:

ab  -n 3500  -c 50  'https://localhost:1235/'

mmomtchev · 2020-10-27T15:02:49Z

I confirm that there is indeed a bug in the TLS code, both in 14.x and master

Add an error handler to the underlying TLSWrap socket mostly for the occasional ECONNRESET that a Linux kernel might throw under heavy load Send a tlsClientError event in this case instead of an uncatcheable socket error Fixes: nodejs#35824

mmomtchev · 2020-10-27T15:25:39Z

@master-genius I replaced the uncatchable error with a tlsClientError event

mmomtchev · 2020-10-27T19:42:43Z

@master-genius, @lpinca is right, your error can (and should be) handled in the user code:

serv.on('secureConnection', (sock) => {
    sock.on('error', (err) => {
        console.log(err);
    });
});

master-genius · 2020-10-27T23:18:16Z

@mmomtchev ok, thanks.

Kukunin · 2020-11-21T11:47:27Z

I had the same issue with my mediasoup server. Earlier, it randomly crashed with the same exception:

      throw er; // Unhandled 'error' event
      ^

Error: read ECONNRESET
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:209:20)
Emitted 'error' event on TLSSocket instance at:
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  errno: -104,
  code: 'ECONNRESET',
  syscall: 'read'
}

after applying the 'error' handler

  server.on('request', (req, res) => {
    req.socket.on('error', () => {});
    res.end('Unavailable');
  });

  server.on('secureConnection', (socket) => {
    socket.on('error', () => {});
  });

I've started receiving another crash:

events.js:292
      throw er; // Unhandled 'error' event
      ^

Error: write ECONNRESET
    at WriteWrap.onWriteComplete [as oncomplete] (internal/stream_base_commons.js:94:16)
    at handleWriteReq (internal/stream_base_commons.js:65:21)
    at writeGeneric (internal/stream_base_commons.js:145:15)
    at TLSSocket.Socket._writeGeneric (net.js:785:11)
    at TLSSocket.Socket._write (net.js:797:8)
    at writeOrBuffer (_stream_writable.js:352:12)
    at TLSSocket.Writable.write (_stream_writable.js:303:10)
    at WebSocketRequest.accept (/home/newvid/node_modules/websocket/lib/WebSocketRequest.js:450:21)
    at /home/newvid/node_modules/protoo-server/lib/transports/WebSocketServer.js:110:33
    at Object.task (/home/newvid/server.js:221:25)
Emitted 'error' event on TLSSocket instance at:
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  errno: -104,
  code: 'ECONNRESET',
  syscall: 'write'
}

Considering that THERE IS an error handler and that this bug is relatively fresh (25 days ago), it might indicate that there is one more related bug. That's why I'm reporting this right now.

The protoo-server package is a simple wrapper around websocket server, and it has the following (https://github.com/versatica/protoo/blob/master/server/lib/transports/WebSocketServer.js#L60)[line] to address the same issue, but it doesn't help either.

I use node v14.15.0 on Ubuntu 16.04 LTS. Going to install node v15 and see, whether the bug is still there.

mmomtchev · 2020-11-22T01:24:16Z

@Kukunin This is different than the recent issue #36180
The tricky part with ECONNRESET is that it is usually reported by the other end, not the one that it is at fault
If your server says ECONNRESET this means your client responded with a RST message
What is your client? Is it Node? Can you capture a network trace of the session?

Kukunin · 2020-11-22T13:26:16Z

My client is a browser. My server is an SFU for WebRTC using mediasoup, it handles a lot of clients simultaneously.

The problem is not ECONNRESET itself, I know what it is. The problem is that the whole server process crashes with Unhandled 'error' event despite the handlers ARE there.

I've upgraded to Node v15.2.1, it hasn't been crashing for 26 hours already. Maybe it's something that was fixed already.

I captured network traffic with tcpdump and I can see a lot of RST flags and only few of them (happens once-twice per day) actually crashes the app. I wasn't able to reproduce this locally with a dummy client (on Python), that sends RST flags, maybe because locally I have node v15.0.x.

mmomtchev · 2020-11-22T15:55:13Z

The TLS phase is also important: if you go through this discussion, my PR and @lpinca's comment, you will see that the TLS error handler is in Node during the handshake then once the user data starts flowing, it is expected that the user code handles it.
That part of the code has some key differences between 14.x and 15.x, so it is possible that this was fixed.
Try to identify if this RST comes during the handshake or during the data transfer.
You can also try running with NODE_DEBUG set to tls
Either there is an error handler that is missing (some of them like that one on secureConnection are not very intuitive), either someone forgot about an 'error' in Node. Your error is on the write call, so it is definitely not something very common. This means that the remote end closed the socket before the normal end of the TLS connection.

mmomtchev · 2020-11-22T15:59:02Z

And instead of simply ignoring the 'error', try also to add a console.error() to your handler both inside the handler and in the initialization to make sure it is called.

Kukunin · 2020-11-24T12:52:45Z

Thank you for your answers.

I can confirm, that upgrading to node 15.2.1 has fixed the problem - it hasn't been happening for the last 3 days.

jaiswarvipin · 2022-02-28T18:54:18Z

Hi @mmomtchev,
Currently i am using the Node.js v17.6.0 and getting the same TLS -104 error and because of this hosted kubernets POD getting crashed.

Client is, Another NodeJs server using "http" lib to make the POST call.

Error we got
`node:events:505
throw er; // Unhandled 'error' event
^

Error: read ECONNRESET
at TLSWrap.onStreamRead (node:internal/stream_base_commons:217:20)
Emitted 'error' event on ClientRequest instance at:
at TLSSocket.socketErrorListener (node:_http_client:442:9)
at TLSSocket.emit (node:events:527:28)
at emitErrorNT (node:internal/streams/destroy:164:8)
at emitErrorCloseNT (node:internal/streams/destroy:129:3)
at processTicksAndRejections (node:internal/process/task_queues:83:21) {
errno: -104,
code: 'ECONNRESET',
syscall: 'read'
}

Node.js v17.6.0`

and we had, all expected cases

`httpServer.createServer((request, response) => {
try{
//response.setHeader('Content-Type', 'text/plain');
response.setHeader('Content-Type', 'text/plain');
/* if boadcaster is not assigned then do needful */

	/* console.log("Start") */

	response.setTimeout(parseInt(process.env.DEFAULT_API_REQUEST_TIME_OUT), function (_pConnectionTimeOut) {
		/* Send response */
		sendResponse(response, responseObj.HTTP_STATUS.REQUESTTIMEOUT, responseObj.ERROR_CODES.SERVICE_TIME_OUT, responseObj.ERROR.TYPE.REQUESTTIMEOUT, responseObj.MESSAGES.DEFAULT.REQUESTTIMEOUT);
		request.ab
	});

	let strQueryString = urlObj.parse(request.url, true).query;
	let strEndPoint = urlObj.parse(request.url, true).pathname;

	let body = "";
	request.on('error', (err) => {
		console.log("Request error")
		console.log(err)
		request.destroy();
		//response.write(err);
		response.end();
	}).on('socket',function (socket) {
		socket.on('timeout', function() {
			request.destroy();
			/* Send response */
			sendResponse(response, responseObj.HTTP_STATUS.REQUESTTIMEOUT, responseObj.ERROR_CODES.SERVICE_TIME_OUT, responseObj.ERROR.TYPE.REQUESTTIMEOUT, responseObj.MESSAGES.DEFAULT.REQUESTTIMEOUT);
		});
	}).on('data', (chunk) => {
		body += chunk.toString()
	}).on('end', () => {
		/* Validating and upstreating the requst */
		processHTTPRequest(strEndPoint, request, response, body, (pAPIResponse) => {
		/* Writing the Resposne text */
		//response.write(pAPIResponse);
		//response.end();
		//echo(pAPIResponse);
		});
	});
}catch(exception){
	console.log("===================HTTP OBJECT ERROR===============")
	console.log(exception);
	console.log("===================HTTP OBJECT ERROR===============")
}

/* on client conention error /
}).on("clientError",function(err,socket){
console.log("Client Connection Error ");
console.log(err);
console.log("Client Connection Error");
/ On client connection error /
//socket.end(JSON.parse({"error": responseObj.HTTP_STATUS.BAD_REQUEST,"error_description": responseObj.MESSAGES.DEFAULT.BAD_REQUEST_BODY,"error_code": responseObj.HTTP_STATUS.BAD_REQUEST,"statuscode": responseObj.HTTP_STATUS.BAD_REQUEST}));
}).on("secureConnection",function(socket){
console.log("secure Connection Error ");
/ On client connection error */
//socket.end(JSON.parse({"error": responseObj.HTTP_STATUS.BAD_REQUEST,"error_description": responseObj.MESSAGES.DEFAULT.BAD_REQUEST_BODY,"error_code": responseObj.HTTP_STATUS.BAD_REQUEST,"statuscode": responseObj.HTTP_STATUS.BAD_REQUEST}));
}).listen(process.env.HTTP_ENCY_PORT);`

this occurred, when, generally when traffic is reaching to the 4000+ TPS.

Kindly help.

Thanks & Regards
Jaiswar Vipin Kumar R.

mmomtchev · 2022-02-28T21:58:09Z

@jaiswarvipin, this is Sparta

jaiswarvipin · 2022-03-05T17:47:29Z

I am sorry, did understand, cloud you pls elaborate and help

jaiswarvipin · 2022-03-05T20:35:35Z

#42154

fandi70 · 2022-04-18T17:28:41Z

in server.js
setInterval(() => {
require('./app_node/routes/web')(app, sessionMap, startDEVICE)
}, 30000);

gopinaresh · 2022-06-08T21:18:18Z

Hi team,
Need help on this issue. My application was deployed in GKE (Google Cloud Project) and seeing the below error

read ECONNRESET
at TLSWrap.onStreamRead (internal/stream_base_commons.js:209:20)

I am using the https call module and this happens only when there is connecting from external vendor api which is not internal network and I don't see this issue when there is internal applications with in the network.

Currently I am using the Node version in my docker file as14.15.0. So based on the above comments I see this is an bug in Node V14. I just want to check is this fix on Node V16 LTS version

mmomtchev · 2022-06-09T12:16:31Z

@gopinash, I will explain you the problem with this issue.
The problem is that an ex-employer of mine organised a false rape accusation against me that he was able to cover up on not less than 6 different judicial procedures in the French system - and he has been paying people to post dicks and viagra ever since - for about 10 years now. So I did found the problem with that issue, but what bothers me is that every time I submit a PR, I get more dicks, viagra and whatever - because of his firm belief that I will take this shit to be able to work in Node. I won't. I have already contacted the OpenJS foundation once, I will probably do so again, this time getting all the companies involved in this project aboard, and if this continues, I will surely post my PRs somewhere else explaining the whole situation. Because some people simply refuse to accept that some things in life are off limits and that we live in a civilised society.

crazybits · 2022-10-23T01:45:18Z

similar issue here, any hint help to solve the issue would be appreciated?
os: MacOS
node: v18.9.0
ts-node: v9.0.0

AxiosError: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20) {
  syscall: 'read', 
  code: 'ECONNRESET',
  errno: -54,
  config: {
    transitional: {
      silentJSONParsing: true,
      forcedJSONParsing: true,
      clarifyTimeoutError: false
    },
    adapter: [Function: httpAdapter],
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 0,
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: -1,
    env: { FormData: [Function] },
    validateStatus: [Function: validateStatus],
    headers: { Accept: 'application/json', 'User-Agent': 'axios/0.27.2' },
    url: 'http://127.0.0.1:8080/v1/',
    method: 'get',
    withCredentials: true,
    cancelToken: CancelToken { promise: [Promise], _listeners: [] },
    data: undefined
  },

crazybits · 2022-10-23T03:29:16Z

looks the issue is due to the concurrent issue, when i limit the count of promises to 10 in await Promise.all(promises);, the issue fixed

youngtaos · 2023-01-18T19:02:48Z

What steps will reproduce the bug?

[dev:start] Error: read ECONNRESET
[dev:start] at TLSWrap.onStreamRead (node:internal/stream_base_commons:217:20) {
[dev:start] errno: -4077,
[dev:start] code: 'ECONNRESET',
[dev:start] syscall: 'read',
[dev:start] response: undefined
[dev:start] }

ckmirafss · 2024-01-29T14:10:52Z

I'm also encountering errors with Next.js 14.

PoojaDurgad added the http2 Issues or PRs related to the http2 subsystem. label Oct 27, 2020

mmomtchev mentioned this issue Oct 27, 2020

tls: handle ECONNRESET on the underlying socket #35829

Closed

3 tasks

master-genius closed this as completed Oct 27, 2020

mmomtchev mentioned this issue Nov 5, 2020

http: client inconsistent behavior with content length header #35958

Closed

timotheeguerin mentioned this issue Oct 4, 2021

Retry network errors Azure/autorest#4322

Open

ktalebian mentioned this issue Oct 13, 2021

[BUG] can deploy twilio/flex-plugin-builder#610

Closed

jlecomte mentioned this issue Jan 15, 2022

Error: read ECONNRESET at TLSWrap.onStreamRead grpc/grpc-node#2026

Closed

murgatroid99 mentioned this issue Jan 18, 2022

grpc-js: Add secureConnection error handling in server grpc/grpc-node#2027

Merged

aeschli mentioned this issue Oct 7, 2022

Web Tests are constantly failing for me under MacOS in CI pipeline microsoft/vscode-test-web#43

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

read ECONNRESET at TLSWrap.onStreamRead #35824

read ECONNRESET at TLSWrap.onStreamRead #35824

master-genius commented Oct 27, 2020

mmomtchev commented Oct 27, 2020

master-genius commented Oct 27, 2020

mmomtchev commented Oct 27, 2020

mmomtchev commented Oct 27, 2020

mmomtchev commented Oct 27, 2020

master-genius commented Oct 27, 2020

Kukunin commented Nov 21, 2020

mmomtchev commented Nov 22, 2020

Kukunin commented Nov 22, 2020 •

edited

Loading

mmomtchev commented Nov 22, 2020

mmomtchev commented Nov 22, 2020

Kukunin commented Nov 24, 2020

jaiswarvipin commented Feb 28, 2022 •

edited

Loading

mmomtchev commented Feb 28, 2022

jaiswarvipin commented Mar 5, 2022

jaiswarvipin commented Mar 5, 2022

fandi70 commented Apr 18, 2022

gopinaresh commented Jun 8, 2022

mmomtchev commented Jun 9, 2022

crazybits commented Oct 23, 2022 •

edited

Loading

crazybits commented Oct 23, 2022

youngtaos commented Jan 18, 2023

ckmirafss commented Jan 29, 2024

read ECONNRESET at TLSWrap.onStreamRead #35824

read ECONNRESET at TLSWrap.onStreamRead #35824

Comments

master-genius commented Oct 27, 2020

What steps will reproduce the bug?

mmomtchev commented Oct 27, 2020

master-genius commented Oct 27, 2020

mmomtchev commented Oct 27, 2020

mmomtchev commented Oct 27, 2020

mmomtchev commented Oct 27, 2020

master-genius commented Oct 27, 2020

Kukunin commented Nov 21, 2020

mmomtchev commented Nov 22, 2020

Kukunin commented Nov 22, 2020 • edited Loading

mmomtchev commented Nov 22, 2020

mmomtchev commented Nov 22, 2020

Kukunin commented Nov 24, 2020

jaiswarvipin commented Feb 28, 2022 • edited Loading

mmomtchev commented Feb 28, 2022

jaiswarvipin commented Mar 5, 2022

jaiswarvipin commented Mar 5, 2022

fandi70 commented Apr 18, 2022

gopinaresh commented Jun 8, 2022

mmomtchev commented Jun 9, 2022

crazybits commented Oct 23, 2022 • edited Loading

crazybits commented Oct 23, 2022

youngtaos commented Jan 18, 2023

ckmirafss commented Jan 29, 2024

Kukunin commented Nov 22, 2020 •

edited

Loading

jaiswarvipin commented Feb 28, 2022 •

edited

Loading

crazybits commented Oct 23, 2022 •

edited

Loading