Repository navigation
Handing spawn() stdio as socket wont duplex properly #15714
Description
- Version: At least v6.9.1 and v8.5.0
- Platform: Darwin Kernel Version 14.5.0: Wed Jul 29 02:26:53 PDT 2015; x86_64
I'm trying to implement a sort of PassThrough-style stream that can be used as mock stdio for ChildProcess.spawn(). PassThrough itself isn't appropriate because it doesn't have a file descriptor. So I'm doing this by creating a socket that speaks to an echo server. What I'm finding is that the server receives the data and successfully pipes it, but the passthrough socket never seems to receive it.
I've poked around the node internals a bit to figure-out why, but I don't have much to share at this time. It looks like spawn() may create another socket that's readable and not writable, and uses the socket I passed as its handle [here]. But my socket remains readable and writable as I would expect, so I still can't explain the behavior!
Possibly related to #9413. Here's some code to reproduce the issue and play with.
'use strict';
const ChildProcess = require('child_process');
const Net = require('net');
const makePassThrough = (cb) => {
const srv = Net.createServer((sock) => {
sock.on('data', (data) => console.log('server sock saw data:', data.toString()));
sock.once('close', () => console.log('server sock closed'));
sock.once('end', () => console.log('server sock ended'));
sock.once('error', (err) => console.log('server sock errored', err));
sock.once('timeout', () => console.log('server sock timed-out'));
sock.pipe(sock);
});
srv.once('close', () => console.log('server closed'));
srv.once('error', cb);
srv.listen(() => {
const sock = new Net.Socket();
sock.once('close', () => console.log('client sock closed'));
sock.once('end', () => console.log('client sock ended'));
sock.once('error', (err) => console.log('client sock errored', err));
sock.once('timeout', () => console.log('client sock timed-out'));
sock.once('error', cb);
sock.connect(srv.address().port, () => cb(null, sock));
});
};
makePassThrough((err, stdout) => {
if (err) {
throw err;
}
stdout.on('data', (data) => console.log('subproc stdout saw data:', data.toString()));
const subproc = ChildProcess.spawn('echo', ['is anybody out there?'], {
stdio: ['ignore', stdout, 'ignore']
});
subproc.on('close', process.exit);
});
/* Program outputs the following,
$ node index.js
server sock saw data: is anybody out there?
# Does not exit
*/For my purposes I'm all set– I can use stdio: 'pipe' and pipe the process's stdio thru actual passthrough streams. This is still a curiosity, though!
Activity
gireeshpunathil commentedon Apr 8, 2018
@devinivy - is this still outstanding? From your description, I understood upto the point you were saying that the child data reaches upto the socket in the server. Then what? where else it should flow as per your expectation?
devinivy commentedon Apr 8, 2018
I just tried this on node v8.11.1 and got the same results. After coming back to this issue after some time, I can see how it's a little confusing—sorry for that. I will try to be a little clearer.
I was trying to author a passthrough-style stream that can be used with
spawn(). Because the stream must have an underlying file descriptor in order to be used withspawn(), I decided to try to create a passthrough stream usingNetby implementing an "echo server" (hencesock.pipe(sock)). In other wordsstdoutis implemented as the client of an echo server: anything that it writes should be written back to it.As such, I expected the output of the program to be,
Instead I only get the first line, which is very hard for me to explain to myself. Especially because if I write directly to
stdout(rather than throughspawn()), it works correctly as a passthrough:Does this provide some more useful context?
gireeshpunathil commentedon Apr 8, 2018
thanks. one or two iterations and we should be good to understand each other!
so: no data being available through:
is this the problem?
devinivy commentedon Apr 8, 2018
Yes, exactly! I would expect that line to log
subproc stdout saw data: is anybody out there?in the program's execution. I would additionally expect the subprocess to close, and the program to exit.gireeshpunathil commentedon Apr 8, 2018
Here is the relevant sections in the strace dump:
I guess the summary of the puzzle is, where is fd 9 pointing to?
[-]Handing spawn() stdio a socket wont duplex properly[/-][+]Handing spawn() stdio as socket wont duplex properly[/+]gireeshpunathil commentedon Apr 9, 2018
I installed 2 handlers: one on the socket in question and one on subproc.stdout, the socket that acts as the parent end of the IPC pipe thus:
and the trace revealed this:
bind socket 0x2cdc5bb49dc9 with data handler dh1:
emit 'data' on socket 0x2cdc5bb4a501. No handler for this socket, so no handler invoked, the stream is read again, and chunk is ignored.
bind socket 0x10312b449dc9 with data handler dh2:
emit 'data' on socket 0x10312b44a501 that is still different from the one for which we istalled dh2, but the handler seem to have invoked as if the data handler is added on that socket!
Something beyond my understanding, so /cc @nodejs/streams to see what is going on: the precreated socket is not used for IPC is understood, but how do we link it with the active socket it is the key requriement here.
mcollina commentedon Apr 9, 2018
I still do not understand what is the issue and what are you trying to achieve.
gireeshpunathil commentedon Apr 9, 2018
so looks like pre-rcreated stream looses its association with the underlying socket and / or the event handlers. The
datais written to the stream, but no user callback was invoked.mcollina commentedon Apr 9, 2018
I'm still lost why we should support such a thing. It seems very fragile, and very prone to breakage.
I think the spawned process hold a reference to the handle/fd, and when that gets collected that is freed. Please verify this.
This is normal. If you call
write()on a stream, there is no guarantee that the callback is called. If the streams error, the callbacks are all dropped.devinivy commentedon Apr 9, 2018
I attempted this for testing purposes. I was authoring a CLI tool and wanted to mock stdin/stdout for my tests. One of my CLI commands uses
spawn(), so I was trying to pass my mock stdio (which werePassThroughstreams) directly to the subprocess. This didn't work becausePassThroughstreams naturally aren't backed by a file descriptor, so I tried to write a passthrough stream that is backed by a file descriptor. This was one approach I tried that didn't work, but I couldn't understand why it didn't work.Is this to say that the program's execution makes sense specifically if the stream errored?
Is the caveat I missed simply that you can't pass a
stdinas a writable orstdout/stderras readables underspawn()'sstdiooption? If there's a practical/known limitation there, I'd be happy to make a PR to the documentation to address it.mcollina commentedon Apr 9, 2018
I agree that we should document that this is not really feasible.
HCelante commentedon Apr 13, 2021
Hi, @BridgeAR a question, is this issue still open? I would like to start collaborating on this project.
JPMoles commentedon Jun 30, 2023
Hi, I'm looking to contribute to this project. Is documentation still needed for this issue, considering that the previous pull request was closed?
Ed-roro commentedon Oct 8, 2024
I do not know if this issue is no longer a thing or is necessary. But i made a PR to update the documentation to reflect the discovery found in this thread and sent it out.
Please review and request changes if needed 😊🤟🏿
doc: add note about stdio streams in child_process
doc: add note about stdio streams in child_process
doc: add note about stdio streams in child_process