Error in dns.js "Cannot call method 'indexOf' of undefined"

[aviynw]

I'm getting the following error in my application.

Uncaught TypeError: Cannot call method 'indexOf' of undefined
FROM
Object.onanswer [as oncomplete] (dns.js:121:51)

My application is continuously making a lot of requests to two different servers, this error happens on average like once every several hours, but is inconsistent and I can go a day without it happening. I'm running the 64-bit version of node 10.33 on LInux (Ubuntu 14.04).

I've had this error crop up every now and then for a while (I think I first encountered it a couple of months ago) but is used to happen like once every few days, I'm not sure what has changed recently to make it happen more frequently now

[misterdjules]

The way that dns.lookup handles the result from cares.getaddrinfo looks a bit flaky. Basically, it assumes that if addresses is truthy, then adresses is not empty.

However, this is not necessarily true. We could fix it by handling this case in the onanswer callback: misterdjules@268fb2b, or by setting argv[0] to Null() if results is empty here: https://github.com/joyent/node/blob/v0.10/src/cares_wrap.cc#L870.

@tjfontaine What do you think?

@aviynw Even if the solutions mentioned above fix your issue, It would be great to have more information about which dns lookup triggers this issue.

One way to do that would be to run node with --abort-on-uncaught-exception and allow the system to produce core files. Then, when the uncaught exception is triggered, a core file would be generated and we could inspect the state of the program to get a better understanding of what was going on. I haven't used core files generated on Linux with mdb on SmartOS yet, and when trying to do so for the first time today I wasn't able to get the same amount of information that I get when the core file is generated on SmartOS. So I'll need to dig a bit more into that before I can actually help you. I'll keep you posted.

You could also gather more information on your side to narrow down what dns lookup causes this issue to arise (by logging the dns lookup that happened before the error is thrown). We could then maybe write some code that reproduces it more reliably.

[tjfontaine]

I would rather handle in the js logic than special case this scenario in C++ (that is not to say there won't be changes necessary on both sides)

But more importantly as @misterdjules points out I am interested to know more about what is going on here because I wouldn't expect process._errno to necessarily have useful state at this point.

It may be as you point out that uv_inet_ntop has failed, if that's the case we need to carry that information forward.

Though what should happen in the case that we have multiple (or at least two) results, and only the second one is malformed and we error in that case. Is it more appropriate to throw away all results, only the malformed results, or only return values up to the malformed results.

In any event being able to reproduce this case is important.

For a linux environment please make sure you can get a core file by setting ulimit -c unlimited before running the node process, and then make sure you run your node process like node --abort-on-uncaught-exception myscript.js, the order does matter, the argument must come before the filename.

[aviynw]

Thanks for the responses. I haven't run into the problem since, but I haven't been letting my program run for more than several hours and I've been making less requests than I was previously. I'm still developing my application, so its constantly changing. Though I'll hopefully be moving to production soon which means letting it run for a long time and making lots of requests like I was previously, so maybe the error will popup again, and I'll run node with --abort-on-uncaught-exception in case it does.

However, I'm having another issue on my Linux machine where requests to one server are slowing down requests to another server. I'm not having this issue on my Joyent box so I'll be running it on there until I get this other issue sorted out, and I am unsure if the dns error will present itself on the Joyent machine as well.

Also, when I reported this error my program was frequently crashing with FATAL ERROR: JS Allocation failed - process out of memory due to a memory leak of mine. Is it possible that running out of memory would sometimes cause the dns error instead of the "out of memory" error? (I wasn't actually running out of memory on my machine, but just hitting some sort of v8 limit).

Anyways,
I'll keep you posted.

-Avi

[tjfontaine]

You can run on any Linux and we can still do the work, it doesn't need to be a smartos instance.

It may very well be related to out of memory, it would be interesting to test that theory.

Either way if you are getting those errors running with ulimit set to unlimited will let us find out better just what is going on.

[misterdjules]

@aviynw Any update regarding this issue?

[aviynw]

I haven't run into this issue since so I think it is most likely related to my "FATAL ERROR: JS Allocation failed - process out of memory" which I'm not getting anymore now that I've fixed my leaks. But, my code has changed in other areas as well, so its hard to know for sure.

[goldfire]

We had this issue twice in production today and we didn't come anywhere close to running out of memory, so I'm inclined to think that isn't related. Unfortunately I was literally in the middle of nowhere when this happened and wasn't able to see what was going on live, but so far looking through the logs nothing looks out of the ordinary.

What is odd is that we've been running 0.10.36 for about 3 weeks on Ubuntu 14.04 with the same level of requests, and this is the first time this has happened. In fact, nothing in our stack has changed during this time.

[johntmyers]

This is happening for me now too. On 0.10.33. Cannot reproduce because it happens very randomly.

[johntmyers]

@goldfire is this still a problem for you?

[misterdjules]

@goldfire @johntmyers Is it possible for you to run node with --abort-on-uncaught-exception so that it generates a core file when an error is thrown and not caught? That would allow us to investigate the state of the program at the time the exception was thrown. If you start doing that, you'll want to make sure first that your environment allows the OS to generate a core file for your node programs first.

I'll be more than happy to help you with this if you need.

[goldfire]

We haven't had this happen since I last posted. Still running the same version of node, so not sure what that says.

[johntmyers]

Hi, yes, I can do that. However we did find one bug that was causing the
dns.js crash on a mongoose connection -- but that is something we are
fixing. Once we have that bug fixed I'll see if we still have an issue.

On Mon, Mar 23, 2015 at 12:35 PM, Julien Gilli [email protected]
wrote:

@goldfire https://github.com/goldfire @johntmyers
https://github.com/johntmyers Is it possible for you to run node with
--abort-on-uncaught-exception so that it generates a core file when an
error is thrown and not caught? That would allow us to investigate the
state of the program at the time the exception was thrown. If you start
doing that, you'll want to make sure first that your environment allows the
OS to generate a core file for your node programs first.

I'll be more than happy to help you with this if you need.

—
Reply to this email directly or view it on GitHub
#8698 (comment).

[SyRenity]

Happens in our case periodically, when we create a lot of parallel promises (with Q) working with remote resources.

Any chance it's a multi-threading issue?

[whoeverest]

I get this error too, when running the script below:

https://gist.github.com/whoeverest/2e1d3771b4d489cd932c

Similar to what happens to @SyRenity the problem happens when there are a lot of requests going on in parallel.

EDIT: This happens on Node 0.12.2 but not on 0.10.36

[misterdjules]

Adding the confirmed-bug label, because this code throws an error when it's actually dealing with an operational error, not a programmer error.

[chrisbnt]

(continuing from #25327)

Got the same crash again, but some useful information too. At the same time the crash occurred, the same process was repeatedly hitting two handled errors which were logged:

Error: connect EMFILE
Error: getaddrinfo EMFILE

I'm guessing this is related to addresses[0] being null or undefined in dns.js

[chrisbnt]

This has happened a few more times in production, I have not yet confirmed, but circumstantially it does appear that running out of available sockets (eg. max open files hitting it's limit) triggers the crash.

[repunck]

running ubuntu 14.04, node v0.10.38, meanjs stack, it happens regularly and we've not beeing able to find out why... we will try the --abort-on-uncaught-exception thing

[pepkin88]

I just had this error and 15 minutes earlier I had EMFILE errors for connects, so similar to @barn3y's case.

Node v0.12.0, Ubuntu 14.04

[rhengles]

I also run into this issue today. My app runs flawlessly in Windows, but in Linux it threw this error.

Node 0.10.38, Amazon Linux AMI 2015.03

[geoff448]

Ran into this issue today and we solved by upping the soft and hard ulimit value of ubuntu from 4096 to 524288