child_process.spawn() has undocumented 200kb buffer limit

The child_process.spawn() function will pause a script's execution if it writes more than 200kb of stdout that isn't captured. I discovered this when my spawned process would suspiciously pause after exactly a certain amount of output.

Adding a listener for stdout that does nothing fixed it and allowed the script to run:

myProcess.stdout.on('data', function (data) {});

This is not documented in the docs. Is this intentional (and hence should be documented) or a bug?

That depends on with what options you spawned the child process. With { stdio: 'pipe' } (the default), the child's stdout and stderr file descriptors are pipes and those have limited (and platform-specific) capacity. If you don't plan on consuming the output from the child, create it with { stdio: 'ignore' }.

It's possible the documentation could be clearer (pull requests welcome) but it's working as intended, it implements flow control the same way foo | bar does in the shell.

It's possible the documentation could be clearer

Probably, I've found that option confusing in the past.

I got this problem in 2015 and write a post on medium, maybe I can help to document this :)

@bnoordhuis

@bnoordhuis I see the limit buffer for this cases are stored in max_buffer_string as I can see here:

node/src/spawn_sync.cc

Lines 766 to 770 in ed3d8b1

    
           Local<Value> js_max_buffer = js_options->Get(env()->max_buffer_string()); 
        
           if (IsSet(js_max_buffer)) { 
        
             if (!js_max_buffer->IsUint32()) 
        
               return UV_EINVAL; 
        
             max_buffer_ = js_max_buffer->Uint32Value();

and the error comes here:

node/src/spawn_sync.cc

Lines 596 to 598 in ed3d8b1

    
           if (max_buffer_ > 0 && buffered_output_size_ > max_buffer_) { 
        
             SetError(UV_ENOBUFS); 
        
             Kill();

I think for better documentation will be nice, if I can provide a value of the max bytes we can write in the std/out/err by default.

it's really a problem, it will cause the process hang-up if you do not handle the stdout or stderr. I think the document should be clearer, or it will be really weird for someone who haven't handled the stdout or stderr.

To fix the docs just need to know whats the real limit as I say in my last comment :(

@bnoordhuis

I think (but I could be wrong), that @bnoordhuis comment "those have limited (and platform-specific) capacity" means that the limits you're looking for come from the operating system, and not values defined in Node/libuv.

@cjihrig

Yeah, I see this comment @cjihrig, but looking in the code I'm little confused about this. :/ Will try found this information in some place :/

@nodejs/documentation

Update child_process.md

doc: update child_process.md

Do not use temporary files for adoc generation, closes #54

I am using spawn in CI and for some reason it doesn't redirect to stdout directly while it does on my host.

This result to the following message:

[ERROR] stderr maxBuffer exceeded

is there a way to increase maxBuffer or to force npx to work the stdout ?

doc: update child_process.md

Navigation Menu

Repository navigation

child_process.spawn() has undocumented 200kb buffer limit #4236

2 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	Local<Value> js_max_buffer = js_options->Get(env()->max_buffer_string());
	if (IsSet(js_max_buffer)) {
	if (!js_max_buffer->IsUint32())
	return UV_EINVAL;
	max_buffer_ = js_max_buffer->Uint32Value();

	if (max_buffer_ > 0 && buffered_output_size_ > max_buffer_) {
	SetError(UV_ENOBUFS);
	Kill();

child_process.spawn() has undocumented 200kb buffer limit #4236

Description

Activity

bnoordhuis commented on Dec 11, 2015

Fishrock123 commented on Dec 11, 2015

thebergamo commented on Jun 29, 2016

thebergamo commented on Jun 29, 2016

tinybug commented on Jul 12, 2016

thebergamo commented on Jul 12, 2016

cjihrig commented on Jul 12, 2016

thebergamo commented on Jul 13, 2016

2 remaining items

Trott commented on Jul 15, 2017

kopax commented on Apr 20, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions