read ECONNRESET at TLSWrap.onStreamRead

Version: v14.14.0
Platform: Linux
Subsystem:Ubuntu 20.04

What steps will reproduce the bug?

I create http2 server and benchmark use ab:

ab -n 5000 -c 50 'https://localhost:1234/?name=123'

after this, the server is crashed.

events.js:292
      throw er; // Unhandled 'error' event
      ^

Error: read ECONNRESET
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:209:20)
Emitted 'error' event on TLSSocket instance at:
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  errno: -104,
  code: 'ECONNRESET',
  syscall: 'read'
}

I had try to handle all 'error' event (socket, http2Stream, http2Session), but all are ineffective.

Can you post your code, the default server doesn't do that. One of the causes can be blocking the event loop.

Can you post your code, the default server doesn't do that. One of the causes can be blocking the event loop.

In order to avoid the influence of the framework, http2 module is used for testing.

'use strict'

const http2 = require('http2')
const fs = require('fs')

process.chdir(__dirname)

let serv = http2.createSecureServer({
  cert : fs.readFileSync('./rsa/localhost.cert'),
  key : fs.readFileSync('./rsa/localhost.key')
})

serv.on('stream', (stream, headers) => {

  stream.on('frameError', (type,code,id)=>{})

  stream.on('error', (err) => {stream.destroy()})

  stream.on('aborted', () => {
    if (!stream.destroyed) {
      stream.destroy()
    }
  })

  stream.on('data', chunk => {

  })

  stream.on('end', () => {
    stream.end('ok')
  })

})

serv.on('tlsClientError', (err, tls) => {
  console.error(err)
  tls.destroy()
})

serv.setTimeout(5000, (sess) => {
  sess.destroy()
})

serv.on('connection', (sock) => {
  sock.setTimeout(2000, () => {
    if (!sock.destroyed) {
      sock.destroy()
    }
  })
})

serv.listen(1235)

test command:

ab  -n 3500  -c 50  'https://localhost:1235/'

I confirm that there is indeed a bug in the TLS code, both in 14.x and master

tls: handle ECONNRESET on the underlying socket

@master-genius

@master-genius I replaced the uncatchable error with a tlsClientError event

@master-genius

@master-genius, @lpinca is right, your error can (and should be) handled in the user code:

serv.on('secureConnection', (sock) => {
    sock.on('error', (err) => {
        console.log(err);
    });
});

@mmomtchev

@mmomtchev ok, thanks.

I had the same issue with my mediasoup server. Earlier, it randomly crashed with the same exception:

      throw er; // Unhandled 'error' event
      ^

Error: read ECONNRESET
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:209:20)
Emitted 'error' event on TLSSocket instance at:
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  errno: -104,
  code: 'ECONNRESET',
  syscall: 'read'
}

after applying the 'error' handler

  server.on('request', (req, res) => {
    req.socket.on('error', () => {});
    res.end('Unavailable');
  });

  server.on('secureConnection', (socket) => {
    socket.on('error', () => {});
  });

I've started receiving another crash:

events.js:292
      throw er; // Unhandled 'error' event
      ^

Error: write ECONNRESET
    at WriteWrap.onWriteComplete [as oncomplete] (internal/stream_base_commons.js:94:16)
    at handleWriteReq (internal/stream_base_commons.js:65:21)
    at writeGeneric (internal/stream_base_commons.js:145:15)
    at TLSSocket.Socket._writeGeneric (net.js:785:11)
    at TLSSocket.Socket._write (net.js:797:8)
    at writeOrBuffer (_stream_writable.js:352:12)
    at TLSSocket.Writable.write (_stream_writable.js:303:10)
    at WebSocketRequest.accept (/home/newvid/node_modules/websocket/lib/WebSocketRequest.js:450:21)
    at /home/newvid/node_modules/protoo-server/lib/transports/WebSocketServer.js:110:33
    at Object.task (/home/newvid/server.js:221:25)
Emitted 'error' event on TLSSocket instance at:
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  errno: -104,
  code: 'ECONNRESET',
  syscall: 'write'
}

Considering that THERE IS an error handler and that this bug is relatively fresh (25 days ago), it might indicate that there is one more related bug. That's why I'm reporting this right now.

The protoo-server package is a simple wrapper around websocket server, and it has the following (https://github.com/versatica/protoo/blob/master/server/lib/transports/WebSocketServer.js#L60)[line] to address the same issue, but it doesn't help either.

I use node v14.15.0 on Ubuntu 16.04 LTS. Going to install node v15 and see, whether the bug is still there.

@Kukunin

@Kukunin This is different than the recent issue #36180
The tricky part with ECONNRESET is that it is usually reported by the other end, not the one that it is at fault
If your server says ECONNRESET this means your client responded with a RST message
What is your client? Is it Node? Can you capture a network trace of the session?

My client is a browser. My server is an SFU for WebRTC using mediasoup, it handles a lot of clients simultaneously.

The problem is not ECONNRESET itself, I know what it is. The problem is that the whole server process crashes with Unhandled 'error' event despite the handlers ARE there.

I've upgraded to Node v15.2.1, it hasn't been crashing for 26 hours already. Maybe it's something that was fixed already.

I captured network traffic with tcpdump and I can see a lot of RST flags and only few of them (happens once-twice per day) actually crashes the app. I wasn't able to reproduce this locally with a dummy client (on Python), that sends RST flags, maybe because locally I have node v15.0.x.

@lpinca

The TLS phase is also important: if you go through this discussion, my PR and @lpinca's comment, you will see that the TLS error handler is in Node during the handshake then once the user data starts flowing, it is expected that the user code handles it.
That part of the code has some key differences between 14.x and 15.x, so it is possible that this was fixed.
Try to identify if this RST comes during the handshake or during the data transfer.
You can also try running with NODE_DEBUG set to tls
Either there is an error handler that is missing (some of them like that one on secureConnection are not very intuitive), either someone forgot about an 'error' in Node. Your error is on the write call, so it is definitely not something very common. This means that the remote end closed the socket before the normal end of the TLS connection.

And instead of simply ignoring the 'error', try also to add a console.error() to your handler both inside the handler and in the initialization to make sure it is called.

Thank you for your answers.

I can confirm, that upgrading to node 15.2.1 has fixed the problem - it hasn't been happening for the last 3 days.

@mmomtchev

Hi @mmomtchev,
Currently i am using the Node.js v17.6.0 and getting the same TLS -104 error and because of this hosted kubernets POD getting crashed.

Client is, Another NodeJs server using "http" lib to make the POST call.

Error we got
`node:events:505
throw er; // Unhandled 'error' event
^

Error: read ECONNRESET
at TLSWrap.onStreamRead (node:internal/stream_base_commons:217:20)
Emitted 'error' event on ClientRequest instance at:
at TLSSocket.socketErrorListener (node:_http_client:442:9)
at TLSSocket.emit (node:events:527:28)
at emitErrorNT (node:internal/streams/destroy:164:8)
at emitErrorCloseNT (node:internal/streams/destroy:129:3)
at processTicksAndRejections (node:internal/process/task_queues:83:21) {
errno: -104,
code: 'ECONNRESET',
syscall: 'read'
}

Node.js v17.6.0`

and we had, all expected cases

`httpServer.createServer((request, response) => {
try{
//response.setHeader('Content-Type', 'text/plain');
response.setHeader('Content-Type', 'text/plain');
/* if boadcaster is not assigned then do needful */

	/* console.log("Start") */

	response.setTimeout(parseInt(process.env.DEFAULT_API_REQUEST_TIME_OUT), function (_pConnectionTimeOut) {
		/* Send response */
		sendResponse(response, responseObj.HTTP_STATUS.REQUESTTIMEOUT, responseObj.ERROR_CODES.SERVICE_TIME_OUT, responseObj.ERROR.TYPE.REQUESTTIMEOUT, responseObj.MESSAGES.DEFAULT.REQUESTTIMEOUT);
		request.ab
	});

	let strQueryString = urlObj.parse(request.url, true).query;
	let strEndPoint = urlObj.parse(request.url, true).pathname;

	let body = "";
	request.on('error', (err) => {
		console.log("Request error")
		console.log(err)
		request.destroy();
		//response.write(err);
		response.end();
	}).on('socket',function (socket) {
		socket.on('timeout', function() {
			request.destroy();
			/* Send response */
			sendResponse(response, responseObj.HTTP_STATUS.REQUESTTIMEOUT, responseObj.ERROR_CODES.SERVICE_TIME_OUT, responseObj.ERROR.TYPE.REQUESTTIMEOUT, responseObj.MESSAGES.DEFAULT.REQUESTTIMEOUT);
		});
	}).on('data', (chunk) => {
		body += chunk.toString()
	}).on('end', () => {
		/* Validating and upstreating the requst */
		processHTTPRequest(strEndPoint, request, response, body, (pAPIResponse) => {
		/* Writing the Resposne text */
		//response.write(pAPIResponse);
		//response.end();
		//echo(pAPIResponse);
		});
	});
}catch(exception){
	console.log("===================HTTP OBJECT ERROR===============")
	console.log(exception);
	console.log("===================HTTP OBJECT ERROR===============")
}

/* on client conention error /
}).on("clientError",function(err,socket){
console.log("Client Connection Error ");
console.log(err);
console.log("Client Connection Error");
/ On client connection error /
//socket.end(JSON.parse({"error": responseObj.HTTP_STATUS.BAD_REQUEST,"error_description": responseObj.MESSAGES.DEFAULT.BAD_REQUEST_BODY,"error_code": responseObj.HTTP_STATUS.BAD_REQUEST,"statuscode": responseObj.HTTP_STATUS.BAD_REQUEST}));
}).on("secureConnection",function(socket){
console.log("secure Connection Error ");
/ On client connection error */
//socket.end(JSON.parse({"error": responseObj.HTTP_STATUS.BAD_REQUEST,"error_description": responseObj.MESSAGES.DEFAULT.BAD_REQUEST_BODY,"error_code": responseObj.HTTP_STATUS.BAD_REQUEST,"statuscode": responseObj.HTTP_STATUS.BAD_REQUEST}));
}).listen(process.env.HTTP_ENCY_PORT);`

this occurred, when, generally when traffic is reaching to the 4000+ TPS.

Kindly help.

Thanks & Regards
Jaiswar Vipin Kumar R.

@jaiswarvipin

@jaiswarvipin, this is Sparta

I am sorry, did understand, cloud you pls elaborate and help

#42154

in server.js
setInterval(() => {
require('./app_node/routes/web')(app, sessionMap, startDEVICE)
}, 30000);

Hi team,
Need help on this issue. My application was deployed in GKE (Google Cloud Project) and seeing the below error

read ECONNRESET
at TLSWrap.onStreamRead (internal/stream_base_commons.js:209:20)

I am using the https call module and this happens only when there is connecting from external vendor api which is not internal network and I don't see this issue when there is internal applications with in the network.

Currently I am using the Node version in my docker file as14.15.0. So based on the above comments I see this is an bug in Node V14. I just want to check is this fix on Node V16 LTS version

@gopinash, I will explain you the problem with this issue.
The problem is that an ex-employer of mine organised a false rape accusation against me that he was able to cover up on not less than 6 different judicial procedures in the French system - and he has been paying people to post dicks and viagra ever since - for about 10 years now. So I did found the problem with that issue, but what bothers me is that every time I submit a PR, I get more dicks, viagra and whatever - because of his firm belief that I will take this shit to be able to work in Node. I won't. I have already contacted the OpenJS foundation once, I will probably do so again, this time getting all the companies involved in this project aboard, and if this continues, I will surely post my PRs somewhere else explaining the whole situation. Because some people simply refuse to accept that some things in life are off limits and that we live in a civilised society.

similar issue here, any hint help to solve the issue would be appreciated?
os: MacOS
node: v18.9.0
ts-node: v9.0.0

AxiosError: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20) {
  syscall: 'read', 
  code: 'ECONNRESET',
  errno: -54,
  config: {
    transitional: {
      silentJSONParsing: true,
      forcedJSONParsing: true,
      clarifyTimeoutError: false
    },
    adapter: [Function: httpAdapter],
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 0,
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: -1,
    env: { FormData: [Function] },
    validateStatus: [Function: validateStatus],
    headers: { Accept: 'application/json', 'User-Agent': 'axios/0.27.2' },
    url: 'http://127.0.0.1:8080/v1/',
    method: 'get',
    withCredentials: true,
    cancelToken: CancelToken { promise: [Promise], _listeners: [] },
    data: undefined
  },

looks the issue is due to the concurrent issue, when i limit the count of promises to 10 in await Promise.all(promises);, the issue fixed

What steps will reproduce the bug?

[dev:start] Error: read ECONNRESET
[dev:start] at TLSWrap.onStreamRead (node:internal/stream_base_commons:217:20) {
[dev:start] errno: -4077,
[dev:start] code: 'ECONNRESET',
[dev:start] syscall: 'read',
[dev:start] response: undefined
[dev:start] }

I'm also encountering errors with Next.js 14.

Navigation Menu

Repository navigation

read ECONNRESET at TLSWrap.onStreamRead #35824

What steps will reproduce the bug?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

read ECONNRESET at TLSWrap.onStreamRead #35824

Description

What steps will reproduce the bug?

Activity

mmomtchev commented on Oct 27, 2020

master-genius commented on Oct 27, 2020

mmomtchev commented on Oct 27, 2020

mmomtchev commented on Oct 27, 2020

mmomtchev commented on Oct 27, 2020

master-genius commented on Oct 27, 2020

Kukunin commented on Nov 21, 2020

mmomtchev commented on Nov 22, 2020

Kukunin commented on Nov 22, 2020

mmomtchev commented on Nov 22, 2020

mmomtchev commented on Nov 22, 2020

Kukunin commented on Nov 24, 2020

jaiswarvipin commented on Feb 28, 2022

mmomtchev commented on Feb 28, 2022

jaiswarvipin commented on Mar 5, 2022

jaiswarvipin commented on Mar 5, 2022

fandi70 commented on Apr 18, 2022

gopinaresh commented on Jun 8, 2022

mmomtchev commented on Jun 9, 2022

crazybits commented on Oct 23, 2022

crazybits commented on Oct 23, 2022

youngtaos commented on Jan 18, 2023

ckmirafss commented on Jan 29, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions