CreateReadStream doesn't close the file

[petersirka]

I found a big problem in streams. I performed multiple tests on Ubuntu 14.04 LTS and OSX too and I see multiple opened files through lsof -p PID. I use autoClose: true (in default) and I destroy the stream manually with this module https://github.com/stream-utils/destroy.

This problem is not recorded for every request, but only if I performed multiple requests at the same time. My website falls down.

How can I prevent this problem? Is there a bug? How to close opened files? Extending ulimit is not a solution.

Thanks for any advice.

PS: node v0.12.0

[dayuoba]

if i use code like :

  var fs = require('fs');                                                                                                                                                                                    
  var http = require('http');

  var server = http.Server(function(req, res) {
     var content = fs.createReadStream('power.log');
     content.pipe(res);
  });

   server.listen(9999);

and when i use wrk for press testing.i got the same problem as you got.

./wrk -c 200 -d 30 -t 12 -v http://localhost:9999

but,when i change code to:

var fs = require('fs');
var http = require('http');

var server = http.Server(function(req, res) {
  var content = fs.createReadStream('power.log');
  content.pipe(res);
  content.on('end', function() {
   content.close();
  });
});

server.listen(9999);

it did not happen. but i am not sure whether i do it on correct way.

[jasnell]

@petersirka can you give a test case that demonstrates what you're doing when you see the problem? @dayuoba's response is useful but it would be easier to see what you did in particular to bring out the issue.

[petersirka]

@jasnell I have created a video for io.js, but the problem is same in node.js:
https://www.dropbox.com/s/3avchxknfc2nrxj/nodejs-iojs-stream-problem.mov

Example:
https://www.dropbox.com/s/vt5yz7o2w5g5vpj/nodejs-stream-problem.zip

Thanks.

[dazhazit]

Can I bump this?
I'm fs.createReadStream (ing) a file to a crud server then deleting it using fs.unlink. Windows works fine with no issues. Linux is giving me ETXTBSY with Ubuntu 14.04 and node v0.10.28. The node version is a little older but I can't change it due to it being part of a larger corporate project with a frozen node version. Is there a work around? (And yes I have tried the suggestions above and yes I've also tried the autoClose flag and manually closing the FD).
I'm at my wits bloody end with this, it's killing my unit tests on linux and preventing me from releasing a bugfix.
Sorry I can't release code example due to NDA.

[therightstuff]

nodejs/node#1180 (comment)

could this be related?

[danielgindi]

I've encountered this too.
Seems like when there's pressure - node.js waits with closing the files, and then too many open files accumulate before it had the chance to close them.
So it looks to me like a mechanism that tries to handle pressure in a "lazy" way, but it's too lazy.

[mcollina]

@danielgindi can you open up a new issue on https://github.com/nodejs/node, with an example on how to reproduce?

[mcollina]

I think this problem can really be solved by using http://npm.im/pump instead of pipe, and in fact we are bringing it into core nodejs/node#13506.

I'm closing this, but feel free to open a new issue.

[danielgindi]

@mcollina It would be a rather complicated example. It does not happen in the simple cases that were demonstrated in this issue. It actually happens when there are more users of the same fd, or even the same file with a different fd, and under extreme pressure.
I'll try to write something that will cause this.

[mcollina]

@danielgindi thanks! Be sure you are using a supported version of node.