Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ECONNRESET #128

Open
kevinmartin opened this issue Nov 6, 2014 · 27 comments
Open

ECONNRESET #128

kevinmartin opened this issue Nov 6, 2014 · 27 comments

Comments

@kevinmartin
Copy link

I get these errors sometimes. Randomly.

    Error: read ECONNRESET 
        at exports._errnoException (util.js:742:11) 
        at TCP.onread (net.js:535:26)

and

    Error: write ECONNRESET 
        at exports._errnoException (util.js:742:11) 
        at Object.afterWrite (net.js:749:14) 

I read somewhere that this also happens on node-mysql, but that it could be fixed by using pool. I began using pool, but it still happens.

Suggestions?

@sidorares
Copy link
Owner

Can you post very basic example of how are you using pool? Pool code in node-mysql2 is exactly the same as in node-mysql, but couple of month behind (I pull it manually from time to time)

@kevinmartin
Copy link
Author

Yes, of course:

'use strict';

var pool, execute,
    mysql           = require('mysql2'),
    thunkify        = require('thunkify');

pool = mysql.createPool({
    connectionLimit: 5,
    database: process.env.DB_DATABASE,
    user: process.env.DB_USERNAME,
    password: process.env.DB_PASSWORD,
    host: process.env.DB_HOST
    //ssl: 'Amazon RDS'
});

pool.on('error', function (err) {
    console.error(err);
});

execute = thunkify(pool.execute.bind(pool));

module.exports = {
    pool: pool,
    query: function *() {
        var results = yield execute.apply(null, arguments);
        return results[0];
    },
    ...
};

A sample of how one might use the above code:

var db = require('./db.js');

yield db.query('SELECT 1;');

@kevinmartin
Copy link
Author

System:

OS: Ubuntu 14.04
Host: Heroku
Stack: Celadon-14
Node Version: 0.11.14
Framework: Koa
Database Host: Amazon RDS
MySQL Version: 5.6.21

@sidorares
Copy link
Owner

Thanks for examples. So I guess expected behaviour is error parameter in the callback (propagated up as exception via co):

var db = require('./db.js');
var result;
try {
  // we should be either here
  result = yield db.query('SELECT 1;');
} catch(e) {
  // or here regardless on connection
  console.log(e);
}

This might be a bug similar to mysqljs/mysql#941 - I'll try to update to latest pool code from node-mysql

@kevinmartin
Copy link
Author

Yes, I have a try/catch on all of my queries. It doesn't get caught.

I guess it might be related to mysqljs/mysql#941

@dougwilson
Copy link
Collaborator

The node-mysql issue you are referring to wasn't an issue of an error being thrown and not being catchable--it was an issue where when there was an error under certain conditions, just nothing happened and the error was swallowed.

@kevinmartin
Copy link
Author

In that issue, the error was given back through the generic callback, just not emitted (unless I read it too fast).

After reviewing some of the code in this package (also pretty quickly), it seems that Pool#execute uses the generic callback style, which should be caught and thrown by co

@dougwilson
Copy link
Collaborator

In that issue, the error was given back through the generic callback, just not emitted (unless I read it too fast).

Yes, the error was given through the callback, but if you didn't supply the callback (because you wanted to use events), then the error would never appear anywhere. That was what the issue was about :)

@kevinmartin
Copy link
Author

Hi,

Just wanted to update and say that I still get this issue, I'm not sure why though. I'm not able to replicate. Here is a longer log from Heroku/Papertrail:

Nov 26 08:20:38 ****-production app/web.2:    Error: write ECONNRESET 
Nov 26 08:20:38 ****-production app/web.2:        at exports._errnoException (util.js:742:11) 
Nov 26 08:20:38 ****-production app/web.2:        at Object.afterWrite (net.js:749:14) 
Nov 26 08:20:58 ****-production app/web.2:    Error: write ECONNRESET 
Nov 26 08:20:58 ****-production app/web.2:        at exports._errnoException (util.js:742:11) 
Nov 26 08:20:58 ****-production app/web.2:        at Object.afterWrite (net.js:749:14) 
Nov 26 08:23:23 ****-production app/web.1:    Error: EBADF, read 
Nov 26 08:23:23 ****-production app/web.1:        at Error (native) 
Nov 26 08:26:11 ****-production app/web.1:    Error: write ECONNRESET 
Nov 26 08:26:12 ****-production app/web.1:    Error: write ECONNRESET 
Nov 26 08:26:12 ****-production app/web.1:        at exports._errnoException (util.js:742:11) 
Nov 26 08:26:12 ****-production app/web.1:        at Object.afterWrite (net.js:749:14) 
Nov 26 08:26:12 ****-production app/web.1:        at exports._errnoException (util.js:742:11) 
Nov 26 08:26:12 ****-production app/web.1:        at Object.afterWrite (net.js:749:14) 
Nov 26 08:31:26 ****-production app/web.1:    Error: write ECONNRESET 
Nov 26 08:31:26 ****-production app/web.1:        at exports._errnoException (util.js:742:11) 
Nov 26 08:31:26 ****-production app/web.1:        at Object.afterWrite (net.js:749:14) 
Nov 26 08:31:42 ****-production app/web.1:    Error: write ECONNRESET 
Nov 26 08:31:42 ****-production app/web.1:        at exports._errnoException (util.js:742:11) 
Nov 26 08:31:42 ****-production app/web.1:        at Object.afterWrite (net.js:749:14) 
Nov 26 08:33:12 ****-production app/web.2:    Error: write EPIPE 
Nov 26 08:33:12 ****-production app/web.2:        at exports._errnoException (util.js:742:11) 
Nov 26 08:33:12 ****-production app/web.2:        at Object.afterWrite (net.js:749:14) 
Nov 26 08:34:52 ****-production app/web.2:    Error: write ECONNRESET 
Nov 26 08:34:52 ****-production app/web.2:        at exports._errnoException (util.js:742:11) 
Nov 26 08:34:52 ****-production app/web.2:        at Object.afterWrite (net.js:749:14) 
Nov 26 08:35:00 ****-production app/web.2:    Error: write ECONNRESET 
Nov 26 08:35:00 ****-production app/web.2:        at exports._errnoException (util.js:742:11) 
Nov 26 08:35:00 ****-production app/web.2:        at Object.afterWrite (net.js:749:14) 
Nov 26 08:37:32 ****-production app/web.1:    Error: read ECONNRESET 
Nov 26 08:37:32 ****-production app/web.1:        at exports._errnoException (util.js:742:11) 
Nov 26 08:37:32 ****-production app/web.1:        at TCP.onread (net.js:535:26) 
Nov 26 08:37:41 ****-production app/web.2:    Error: write ECONNRESET 
Nov 26 08:37:41 ****-production app/web.2:        at exports._errnoException (util.js:742:11) 
Nov 26 08:37:41 ****-production app/web.2:        at Object.afterWrite (net.js:749:14) 
Nov 26 08:37:42 ****-production app/web.2:    Error: write ECONNRESET 
Nov 26 08:37:43 ****-production app/web.2:        at exports._errnoException (util.js:742:11) 
Nov 26 08:37:43 ****-production app/web.2:        at Object.afterWrite (net.js:749:14) 
Nov 26 08:38:52 ****-production app/web.2:    Error: write ECONNRESET 
Nov 26 08:38:52 ****-production app/web.2:        at exports._errnoException (util.js:742:11) 
Nov 26 08:38:52 ****-production app/web.2:        at Object.afterWrite (net.js:749:14) 
Nov 26 08:39:19 ****-production app/web.2:    Error: write ECONNRESET 
Nov 26 08:39:19 ****-production app/web.2:        at exports._errnoException (util.js:742:11) 
Nov 26 08:39:19 ****-production app/web.2:        at Object.afterWrite (net.js:749:14) 
Nov 26 08:43:34 ****-production app/web.2:    Error: read ECONNRESET 
Nov 26 08:43:34 ****-production app/web.2:        at exports._errnoException (util.js:742:11) 
Nov 26 08:43:34 ****-production app/web.2:        at TCP.onread (net.js:535:26) 
Nov 26 08:44:56 ****-production app/web.2:    Error: write ECONNRESET 
Nov 26 08:44:56 ****-production app/web.2:        at exports._errnoException (util.js:742:11) 
Nov 26 08:44:56 ****-production app/web.2:        at Object.afterWrite (net.js:749:14) 
Nov 26 08:44:58 ****-production app/web.1:    Error: write ECONNRESET 
Nov 26 08:44:58 ****-production app/web.1:        at exports._errnoException (util.js:742:11) 
Nov 26 08:44:58 ****-production app/web.1:        at Object.afterWrite (net.js:749:14) 
Nov 26 08:45:19 ****-production app/web.1:    Error: read ECONNRESET 
Nov 26 08:45:19 ****-production app/web.1:        at exports._errnoException (util.js:742:11) 
Nov 26 08:45:19 ****-production app/web.1:        at TCP.onread (net.js:535:26) 
Nov 26 08:48:49 ****-production app/web.2:    Error: read ECONNRESET 
Nov 26 08:48:49 ****-production app/web.2:        at exports._errnoException (util.js:742:11) 
Nov 26 08:48:49 ****-production app/web.2:        at TCP.onread (net.js:535:26) 
Nov 26 08:51:54 ****-production app/web.1:    Error: read ECONNRESET 
Nov 26 08:51:54 ****-production app/web.1:        at exports._errnoException (util.js:742:11) 
Nov 26 08:51:54 ****-production app/web.1:        at TCP.onread (net.js:535:26) 
Nov 26 08:52:09 ****-production app/web.2:    Error: read ECONNRESET 
Nov 26 08:52:09 ****-production app/web.2:        at exports._errnoException (util.js:742:11) 
Nov 26 08:52:09 ****-production app/web.2:        at TCP.onread (net.js:535:26) 
Nov 26 08:52:24 ****-production app/web.1:    Error: write ECONNRESET 
Nov 26 08:52:24 ****-production app/web.1:        at exports._errnoException (util.js:742:11) 
Nov 26 08:52:24 ****-production app/web.1:        at Object.afterWrite (net.js:749:14) 
Nov 26 08:53:16 ****-production app/web.2:    Error: write ECONNRESET 
Nov 26 08:53:16 ****-production app/web.2:        at exports._errnoException (util.js:742:11) 
Nov 26 08:53:16 ****-production app/web.2:        at Object.afterWrite (net.js:749:14) 
Nov 26 08:55:16 ****-production app/web.2:    Error: write EPIPE 
Nov 26 08:55:16 ****-production app/web.2:        at exports._errnoException (util.js:742:11) 
Nov 26 08:55:16 ****-production app/web.2:        at Object.afterWrite (net.js:749:14) 

As you can see, I get ECONNRESET (onread/afterwrite), EPIPE, and EBADF

Any suggestions?

@jsbinette
Copy link

I get this error too but only when I use koa-proxy which depends on co-request. When I hit the post directly on the (VPN) server, I don't have the problem.

Error: read ECONNRESET
      at exports._errnoException (util.js:742:11)
      at TCP.onread (net.js:535:26)

The code I'm using is very simple

router.post('/api/Produit/:method',  proxy({
  host: "http://192.168.XXX.X"
}))

@sidorares
Copy link
Owner

@jsbinette - is that new connection can't be established to mysql server or existing drops? Do you have local mysql server localhost:3306 ? Which version of node-mysql2 are you using?

@jsbinette
Copy link

@sidorares Sorry turns out it doesn't apply. In my case, koa-better-body interfered somehow with the post when it went thru the proxy.

@farzher
Copy link

farzher commented Jan 10, 2019

getting this sometimes randomly. simple test server, nothing complex happening on it.

{ Error: read ECONNRESET
    at PromisePool.query (./node_modules/mysql2/promise.js:323:22)
    at Object.select (./lib/db.js:36:44)
    at <anonymous>
  message: 'read ECONNRESET',
  code: 'ECONNRESET',
  errno: 'ECONNRESET',
  sqlState: undefined,
  sqlMessage: undefined }

@sidorares
Copy link
Owner

@farzher could be anything - server idle timeout, network issues etc

@farzher
Copy link

farzher commented Jan 10, 2019

@sidorares you don't seem surprised that this causes random errors on production servers.
shouldn't the library do some kind of error catching retries before throwing an error?
or is everyone supposed to write custom wrapper code to do that themselves? (i missed the memo)
or is computer software just supposed to have some errors?

@sidorares
Copy link
Owner

@farzher this is a low level driver and retries out of the scope. It's not clear from your report if you can catch and process error in your code or it's something that your code can't handle ( if latter, it definitely need to be fixed )

The "best practise" usually is to never use individual connections directly and use connection pool ( even if one connection is enough from perf point of view ). But even with pool you absolutely must be able to handle errors from production code and make a decision what to do ( retry / log / report to user )

@jsbinette
Copy link

jsbinette commented Jan 10, 2019 via email

@sidorares
Copy link
Owner

@jsbinette you can always use pool instead of single connection for that

connection here is "immutable" in that sense. If it dies you create new one instead of ressurrecting internal state (re-connecting tcp / re-doing hanshake etc )

@farzher
Copy link

farzher commented Jan 11, 2019

this is a low level driver and retries out of the scope

@sidorares that's fair. maybe make mention that in the readme so lazy people like me are reminded to do proper error handling.

i'm using a pool, you can see the error is at PromisePool.query mysql2/promise.js:323

@sidorares
Copy link
Owner

lazy people like me are reminded to do proper error handling.

error handling is always required with any library. Built in retries just make it less frequent/visible ( you still have to handle case when number of attempts is over limit )

In theory we might have it easier for pool users to have automatic retries, for example have a function you pass to config that can decide "retry or not" based on failed query.

@farzher
Copy link

farzher commented Jan 11, 2019

i guess i didn't mean error handling, error.. retrying? idk it's not a concept i thought about.

if db.query() gives an error, i assume the database is broken or something and handle that as gracefully as i can. what i didn't realize is that sometimes those errors are fixable simply by retrying!

so you're telling me i'm supposed to look at the error message you return, figure out which ones are retryable, and retry? that's the part i'm lazy about, i assumed you'd take care of that because why should i have to.

@sidorares
Copy link
Owner

The biggest problem is the driver doesn't know what's save to retyr ( aka "idempotent" ). Say you do "please delete row with oldest updatedAt" and you get network error. At the moment you see error you don't even know if row was already deleted, so recovery from error is completely app logic specific

@farzher
Copy link

farzher commented Jan 11, 2019

makes sense. i was just running a select though, you could always auto retry that.

you're saying most of the time if the connection is messed up before you try to execute the query, you'll auto retry to get a working connection? but this ECONNRESET error i got happened at an unlucky time where you weren't sure if it executed?

@sidorares
Copy link
Owner

driver sets fatal flag on error, if it's not set it means you can still use connection and make another query, if set - you need to recycle it and create new one ( pool does this for you ) -

err.fatal = true;

@farzher
Copy link

farzher commented Jan 14, 2019

for anyone getting this error randomly. i guess you're supposed to write this wrapper around pool.query (or whatever call you're using). stops the problem for me.

function query_with_retries(sql, params, retries_left=1) {
  try {
    return pool.query(sql, params)
  } catch(err) {
    if(retries_left >= 1 && err.code === 'ECONNRESET') {
      console.error({msg:'query_with_retries retrying!', retries_left, err})
      return query_with_retries(sql, params, retries_left-1)
    } else {
      throw err
    }
  }
}

@benbotto
Copy link
Contributor

benbotto commented Dec 11, 2019

I'm sorry to wake the dead here, but I'm struggling to fix this in my code as well. It has caught me by surprise after deploying to production, which isn't the ideal time to encounter the error.

MySQL has a wait_timeout which is 8 hours by default. If no queries run against a connection in that 8-hour time frame, MySQL evidently terminates the connection. Is my understanding correct in that this library's connection pool does not handle those terminations and blindly attempts to reuse the terminated connections? Is it not possible for the library to establishing new connections when the server closes idle ones? I see farzher's code snippet (and thanks for that!), but is there a more keep-alive type solution that can be used as a workaround? I've seen some people using a SELECT 1-on-an-interval approach, but I don't understand how that will keep all of the connections in a pool alive.

I see this in the MySQL documentation, but it doesn't seem to be the case for mysql2:

With Pool, disconnected connections will be removed from the pool freeing up space for a new connection to be created on the next getConnection call.

Thanks for the support.

@benbotto benbotto mentioned this issue Dec 13, 2019
@ytejas1701
Copy link

Very late to the discussion, but @benbotto could you please tell me how you solved this issue finally? I wasn't able to find any proper solutions on the internet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants