child_process.spawn() has undocumented 200kb buffer limit

[gladys123]

The child_process.spawn() function will pause a script's execution if it writes more than 200kb of stdout that isn't captured. I discovered this when my spawned process would suspiciously pause after exactly a certain amount of output.

Adding a listener for stdout that does nothing fixed it and allowed the script to run:

myProcess.stdout.on('data', function (data) {});

This is not documented in the docs. Is this intentional (and hence should be documented) or a bug?

[bnoordhuis]

That depends on with what options you spawned the child process. With { stdio: 'pipe' } (the default), the child's stdout and stderr file descriptors are pipes and those have limited (and platform-specific) capacity. If you don't plan on consuming the output from the child, create it with { stdio: 'ignore' }.

It's possible the documentation could be clearer (pull requests welcome) but it's working as intended, it implements flow control the same way foo | bar does in the shell.

[Fishrock123]

It's possible the documentation could be clearer

Probably, I've found that option confusing in the past.

[thebergamo]

I got this problem in 2015 and write a post on medium, maybe I can help to document this :)

[thebergamo]

@bnoordhuis I see the limit buffer for this cases are stored in max_buffer_string as I can see here:

node/src/spawn_sync.cc

Lines 766 to 770 in ed3d8b1

	Local<Value> js_max_buffer = js_options->Get(env()->max_buffer_string());
	if (IsSet(js_max_buffer)) {
	if (!js_max_buffer->IsUint32())
	return UV_EINVAL;
	max_buffer_ = js_max_buffer->Uint32Value();

and the error comes here:

node/src/spawn_sync.cc

Lines 596 to 598 in ed3d8b1

	if (max_buffer_ > 0 && buffered_output_size_ > max_buffer_) {
	SetError(UV_ENOBUFS);
	Kill();

I think for better documentation will be nice, if I can provide a value of the max bytes we can write in the std/out/err by default.

[tinybug]

it's really a problem, it will cause the process hang-up if you do not handle the stdout or stderr. I think the document should be clearer, or it will be really weird for someone who haven't handled the stdout or stderr.

[thebergamo]

To fix the docs just need to know whats the real limit as I say in my last comment :(

[cjihrig]

I think (but I could be wrong), that @bnoordhuis comment "those have limited (and platform-specific) capacity" means that the limits you're looking for come from the operating system, and not values defined in Node/libuv.

[thebergamo]

Yeah, I see this comment @cjihrig, but looking in the code I'm little confused about this. :/ Will try found this information in some place :/

[Trott]

@nodejs/documentation

[kopax]

I am using spawn in CI and for some reason it doesn't redirect to stdout directly while it does on my host.

This result to the following message:

[ERROR] stderr maxBuffer exceeded

is there a way to increase maxBuffer or to force npx to work the stdout ?