Subprocess on windows unable to get stdout data

[airportyh]

Simply creating a subprocess and reading data from its stdout fails on Windows. However, strangely, I was only able to reproduce this bug when running a test under mocha - not sure what mocha's doing that's so special. Here is the gist that demonstrates the problem

https://gist.github.com/3365454

I am running Windows 7 inside VMWare.

[TooTallNate]

I think your main problem is that you're calling exec (which buffers data until the end) instead of spawn (which is what it looks like you intend to be calling. Also, side note, the "end" callback doesn't take any arguments (err and data).

I'm not sure why you'd be seeing differences between OS X and Windows though. Could be something about the way that mocha is implemented.

[airportyh]

I just changed it to use spawn, but the exact same thing happened. Here's my new spawn.js

https://gist.github.com/3366250

[piscisaureus]

@airportyh Can you also log the stderr data? p.stderr.on('data', function() { etc.

[airportyh]

@piscisaureus handling stderr actually uncovered a bug in my code on windows - that it couldn't find the mocha executable: thanks for the catch. I've fixed that in the new version.

https://gist.github.com/3372204

[TooTallNate]

So does your script work now? Can this be closed?

[airportyh]

@piscisaureus No, the problem persists. Same result. The next thing I can do is dig into what Mocha is doing to try to isolate.

[piscisaureus]

airporthyr: so does mocha actually run or not? Because in that case, spawn() is working fine :-)

[airportyh]

Spawn worked, but the stdout data event(s) didn't emit. See the results on Windows in the gist.

[piscisaureus]

One more suggestion: can you try to remove the p.on('exit', function() { listener?

[airportyh]

Yeah, taking that out still doesn't let it receive any data events. It ends up hanging there with no data displayed.

[airportyh]

Drilled down to the Mocha source and was able to get closer. It looks like calling process.exit() has the potential to muffle stdout messages to parent process on Windows. Here's my new test that demonstrates - now no longer depends on Mocha. I've also verified that you get the same results regardless of using spawn() or exec().

https://gist.github.com/3386599

[mistaecko]

This is still a major issue at least on my system (Windows 8). Calling process.exit on a spawned process will truncate the output, supposedly because stdout and stderr are not flushed before exiting.

The 'grunt' project (a task runner) uses this code snippet to guarantee that pipes are flushed before exiting, and I successfully patched other projects e.g. typescript and mocha that way and it solved the problem for me.

exports.exit = function exit(exitCode) {
  if (process.stdout._pendingWriteReqs || process.stderr._pendingWriteReqs) {
    process.nextTick(function() {
      exit(exitCode);
    });
  } else {
    process.exit(exitCode);
  }
};

see https://github.com/gruntjs/grunt/blob/master/lib/util/exit.js

To me it looks very much as if this should be handled at a lower level (in node), in order to improve reliability and stability of node as a platform on Windows.

[matthiasg]

+1 mistaecko

[Mithgol]

After some recent API changes, as far as I see, the callback of process.nextTick is called before any input/output operations.

So in the above code setImmediate should probably be used instead of process.nextTick.

What do you think?

[mistaecko]

setImmediate didn't work for me, but waiting for the pipes to 'drain' did work with node 0.8 and 0.10:

function exit(code) {
    var draining = 0;
    var onDrain = function(){
        if (!draining--) process.exit(code);
    };
    if (process.stdout.bufferSize) {
        draining++;
        process.stdout.once('drain', onDrain);
    }
    if (process.stderr.bufferSize) {
        draining++;
        process.stderr.once('drain', onDrain);
    }
    if(!draining)
        process.exit(code);
}

see: mochajs/mocha#333 (comment)

[mistaecko]

Btw, the main ticket for this issue is #3584.

[PrimaryFeather]

+1 mistaecko