Repository navigation
net.connection.onread handler: may crash on socket end #31823
Description
- Version: reproduced on v12.15.0 and v13.8.0
- Platform: Linux debian 4.19.0-5-amd64 deps: update openssl to 1.0.1j #1 SMP Debian 4.19.37-5+deb10u2 (2019-08-08) x86_64 GNU/Linux
- Subsystem: net
When using an onread handler for net connections (issue #25436), it seems that the following assertion error may occur when the connection is closed:
node[14962]: ../src/stream_base.cc:497:virtual void node::CustomBufferJSListener::OnStreamRead(ssize_t, const uv_buf_t&): Assertion `(buf.base) == (buffer_.base)' failed.
1: 0x9dab80 node::Abort() [node]
2: 0x9dac07 [node]
3: 0xa860b2 node::CustomBufferJSListener::OnStreamRead(long, uv_buf_t const&) [node]
4: 0xa90716 node::LibuvStreamWrap::OnUvRead(long, uv_buf_t const*) [node]
5: 0x12e416f [node]
6: 0x12ea228 [node]
7: 0x12d85ab uv_run [node]
8: 0xa1d120 node::NodeMainInstance::Run() [node]
9: 0x9acb78 node::Start(int, char**) [node]
10: 0x7f72e736e09b __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
11: 0x94a115 [node]
Aborted
What steps will reproduce the bug?
This test requires a test server - I used the following code to run a test server on localhost:9999:
require('net')
.createServer(conn => conn.write('test'))
.listen({host: 'localhost', port: 9999});
The actual test script (which connects to localhost:9999):
const net = require('net');
const opts = {
host: 'localhost',
port: 9999,
onread: {
buffer: Buffer.allocUnsafe(4096),
callback: (size, buf) => {
console.log('Data received');
}
}
};
let count = 0;
function doTest() {
console.log('=== Attempt ' + (++count) + ' ===')
const socket = net.connect(opts, () => console.log('Connected'));
socket.once('end', () => console.log('End received'));
socket.once('close', () => {
doTest(); // if we got here, everything worked, so just repeat the test until it fails
});
socket.end();
}
doTest();
How often does it reproduce? Is there a required condition?
It doesn't happen that frequently - it usually takes around 4-30 attempts, using the test script above, for a crash to happen.
It seems that it is necessary for the server to send a message to cause the issue, even though the crash appears to be on connection end. The console output suggests that the crash occurs after the data is received, but before the end event is received.
I haven't looked too much into the cause, but it doesn't seem to occur if the server and client are in the same script.
What is the expected behavior?
Test script should endlessly loop without crashing (if it doesn't crash within a few hundred thousand cycles, I'd say it's successful).
Output looks like:
=== Attempt 1 ===
Connected
Data received
End received
=== Attempt 2 ===
Connected
Data received
End received
=== Attempt 3 ===
Connected
Data received
End received
...
What do you see instead?
=== Attempt 25 ===
Connected
Data received
End received
=== Attempt 26 ===
Connected
Data received
node[14962]: ../src/stream_base.cc:497:virtual void node::CustomBufferJSListener::OnStreamRead(ssize_t, const uv_buf_t&): Assertion `(buf.base) == (buffer_.base)' failed.
1: 0x9dab80 node::Abort() [node]
2: 0x9dac07 [node]
3: 0xa860b2 node::CustomBufferJSListener::OnStreamRead(long, uv_buf_t const&) [node]
4: 0xa90716 node::LibuvStreamWrap::OnUvRead(long, uv_buf_t const*) [node]
5: 0x12e416f [node]
6: 0x12ea228 [node]
7: 0x12d85ab uv_run [node]
8: 0xa1d120 node::NodeMainInstance::Run() [node]
9: 0x9acb78 node::Start(int, char**) [node]
10: 0x7f72e736e09b __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
11: 0x94a115 [node]
Aborted
Update: I've tried this on a few other systems. Haven't been able to reproduce on Node v13.8.0 on Windows x64. On another Linux x64 machine, it seems to have taken hundreds of cycles to reproduce.
Activity
addaleax commentedon Feb 18, 2020
@animetosho Do you think you could get a core dump for these? Or are you able to build modified versions of Node.js yourself? It would be great to know what the values of
nread,buf.baseandbuffer_.baseare in this case, i.e. applyingand then running seeing what the output before the failure is would be very helpful.
(Alternatively, that information can probably be gathered from a core dump.)
Fwiw, I can’t produce on x64 Linux, so … @nodejs/platform-macos
animetosho commentedon Feb 18, 2020
Thanks for the response.
I tried running it in GDB, but it looks like it can't identify those parameters. Doesn't look like the official binary packages include debug symbols (which would be useful to have...).
Unfortunately I don't have access to any fast machines for a while, so compiling Node takes forever here, but I can try.
If it's of any use, I've uploaded a core dump here. This is run with the official Node v13.8.0 binary so you should be able to load it with that.
animetosho commentedon Feb 21, 2020
So I managed to get a compile working, though I question the accuracy of this build as it segfaults a lot. For a failure case, I get the following output:
net: fix crash if POLLHUP is received
net: fix crash if POLLHUP is received
net: fix crash if POLLHUP is received
carlhoerberg commentedon Dec 21, 2021
Just wanted to mention that I got this in Node 16.13.1 too. Seems hard to reproduce, only happens in our production environment with hundreds of different TLS connections..
Reverting to
.on("data")doesn't seem to have any performance penalty that we can measure so that has been our solution.animetosho commentedon Dec 21, 2021
Just here to add that, whilst I'm unable to reproduce it myself, I've had someone else also report that it's still happening (issue referenced directly above your comment). Have no clue what the cause is, as it doesn't sound like a common case, but it seems not reliable enough to rely on.
porsager commentedon Oct 27, 2022
Experiencing the same issue here. Would have been nice to leverage the benefits of onread, but with an error like this it doesn't seem safe to use. Perhaps this issue could be reopened, or should I open another one?
net: fix crash if POLLHUP is received
net: fix crash if POLLHUP is received