Skip to content

segfault in TLS #1696

Closed
Closed
segfault in TLS#1696
Labels
tlsIssues and PRs related to the tls subsystem.

Description

@EricTheOne
EricTheOne
opened on May 13, 2015

Reproduces 100% of the time within a minute of running the server. Server opens multiple TLS client connections (as a client).

I'll gladly help with testing a fix or sharing some of the backtrace data, but cannot share server code or specific scenario.

Please see nodejs/node-v0.x-archive#8780, I'm not 100% sure but I think it may be the same issue. I'm unable to reproduce it in node 0.12.2 (server is running fine, the segfault doesn't happen).

Backtrace (commit 7693705):

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000069eed0 in SSL_read ()
(gdb) backtrace
#0  0x000000000069eed0 in SSL_read ()
#1  0x0000000000d40a39 in node::TLSWrap::ClearOut() ()
#2  0x0000000000d4117e in node::TLSWrap::OnReadImpl(long, uv_buf_t const*, uv_handle_type, void*) ()
#3  0x0000000000ce109b in node::JSStream::ReadBuffer(v8::FunctionCallbackInfo<v8::Value> const&) ()
#4  0x0000000000850002 in v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) ()
#5  0x000000000087721b in v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) ()
#6  0x00002307b4e060bb in ?? ()
#7  0x00003c26fce690f9 in ?? ()
#8  0x00002307b4e06001 in ?? ()
#9  0x00007ffe6b7e47f0 in ?? ()
#10 0x00007ffe6b7e4850 in ?? ()
#11 0x00002307b4e99739 in ?? ()
#12 0x00002bb54efbee61 in ?? ()
#13 0x00003c26fce6b669 in ?? ()
...

dmesg:

segfault at 30 ip 000000000069eed0 sp 00007ffe6b7e41a8 error 4 in iojs[400000+d0e000]

Although nodejs/node-v0.x-archive#8780 contains a suggested fix, I don't see that it was applied in node.js 0.12.2, so either something else fixed it or the situation is different.

Naively applying the fix suggested in 8780 on io.js 2.0.1 (18d457b) doesn't seem to help. Instead I'm failing on an assertion.

"Fix" (tls_wrap.cc):

if (wrap->enc_out_) {
    NodeBIO::FromBIO(wrap->enc_out_)->Read(nullptr, wrap->write_size_);
    // Ensure that the progress will be made and `InvokeQueued` will be called.
    wrap->ClearIn();
}

Backtrace with fix applied:

Program terminated with signal SIGABRT, Aborted.
#0  0x00007fc17f768cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56  ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) backtrace
#0  0x00007fc17f768cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007fc17f76c0d8 in __GI_abort () at abort.c:89
#2  0x00007fc17f761b86 in __assert_fail_base (fmt=0x7fc17f8b2830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0xf7d1f9 "(bio->ptr) != (nullptr)", file=file@entry=0xf7d1e0 "../src/node_crypto_bio.h", line=line@entry=77, 
    function=function@entry=0xf7e080 <_ZZN4node7NodeBIO7FromBIOEP6bio_stE19__PRETTY_FUNCTION__> "static node::NodeBIO* node::NodeBIO::FromBIO(BIO*)") at assert.c:92
#3  0x00007fc17f761c32 in __GI___assert_fail (assertion=0xf7d1f9 "(bio->ptr) != (nullptr)", file=0xf7d1e0 "../src/node_crypto_bio.h", line=77, 
    function=0xf7e080 <_ZZN4node7NodeBIO7FromBIOEP6bio_stE19__PRETTY_FUNCTION__> "static node::NodeBIO* node::NodeBIO::FromBIO(BIO*)") at assert.c:101
#4  0x0000000000673989 in node::NodeBIO::FromBIO(bio_st*) [clone .isra.22] [clone .part.23] ()
#5  0x0000000000d409bf in node::TLSWrap::EncOutCb(node::WriteWrap*, int) ()
#6  0x0000000000850002 in v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) ()
#7  0x000000000087721b in v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) ()
#8  0x00003993517060bb in ?? ()
#9  0x00007fff1da78730 in ?? ()
#10 0x0000399351706001 in ?? ()
#11 0x00007fff1da78710 in ?? ()
#12 0x00007fff1da78778 in ?? ()
#13 0x0000399351a8292c in ?? ()
#14 0x00001507279beb79 in ?? ()
#15 0x0000000000000000 in ?? ()

Activity

  1. EricTheOne commented on May 13, 2015

    @EricTheOne
    EricTheOne
    on May 13, 2015
    Author

    @indutny

  2. mscdex
    added
    tlsIssues and PRs related to the tls subsystem.
    on May 13, 2015
  3. indutny
    added a commit that references this issue on May 14, 2015

    tls: emit errors on close whilst async action

    e329020

  4. indutny commented on May 14, 2015

    @indutny
    indutny
    on May 14, 2015
    Member

    May I ask you to give a try to the following patch #1702 ?

  5. EricTheOne commented on May 14, 2015

    @EricTheOne
    EricTheOne
    on May 14, 2015
    Author

    @indutny gladly and thanks for the quick response. Applied the patch on io.js master (4e2f999), still getting segfault:

    Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000069eed0 in SSL_read ()
(gdb) backtrace
#0  0x000000000069eed0 in SSL_read ()
#1  0x0000000000d40ae9 in node::TLSWrap::ClearOut() ()
#2  0x0000000000d4122e in node::TLSWrap::OnReadImpl(long, uv_buf_t const*, uv_handle_type, void*) ()
#3  0x0000000000ce109b in node::JSStream::ReadBuffer(v8::FunctionCallbackInfo<v8::Value> const&) ()
#4  0x0000000000850002 in v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) ()
#5  0x000000000087721b in v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) ()
#6  0x00000abcd52060bb in ?? ()
#7  0x00000abcd5206001 in ?? ()
#8  0x00007fffb979c580 in ?? ()
#9  0x00007fffb979c5d0 in ?? ()
#10 0x00000abcd55a6f1a in ?? ()
#11 0x000030f334efeea9 in ?? ()
#12 0x000015aab64b2489 in ?? ()
#13 0x00001bd0488c0841 in ?? ()
#14 0x000030f334efeea9 in ?? ()
#15 0x00000b493ed87e89 in ?? ()
#16 0x00001bd04889a081 in ?? ()
#17 0x00007fffb979c6a8 in ?? ()
#18 0x00000abcd5296468 in ?? ()
#19 0x000015aab64b2489 in ?? ()
#20 0x00001bd0488addf1 in ?? ()
#21 0x00000abcd54c8824 in ?? ()
#22 0x00000abcd54c8824 in ?? ()
#23 0x00003d23e40be3c1 in ?? ()
#24 0x00002f4a2ebf7a29 in ?? ()
#25 0x00000a9d72d66921 in ?? ()
#26 0x000015aab64b2489 in ?? ()
#27 0x000011d9d60bb679 in ?? ()
#28 0x00000a9d72dab0a9 in ?? ()
#29 0x00000a9d72d8dcf1 in ?? ()
#30 0x00007fff00000001 in ?? ()
#31 0x00000abc00000002 in ?? ()
#32 0x00007fffb979c700 in ?? ()
#33 0x000011d9d60bb631 in ?? ()
#34 0x00000a9d72d04141 in ?? ()
#35 0x00000a9d72d04161 in ?? ()
#36 0x00000b493ed87e89 in ?? ()
#37 0x00000a9d72d04101 in ?? ()
#38 0x00001bd0488c06f9 in ?? ()
#39 0x00000a9d72d04161 in ?? ()
#40 0x00000a9d72dc9181 in ?? ()
#41 0x000011d9d607ccf9 in ?? ()
#42 0x00000a9d72dc9181 in ?? ()
#43 0x000011d9d607ccf9 in ?? ()
#44 0x00007fffb979c6e0 in ?? ()
#45 0x00000abcd5214b35 in ?? ()
#46 0x000012917282dcf1 in ?? ()
#47 0x00001bd0488addf1 in ?? ()
#48 0x0000000200000000 in ?? ()
#49 0x00000a9d72dc9181 in ?? ()
#50 0x0000000a00000000 in ?? ()
#51 0x00007fffb979c778 in ?? ()
#52 0x00000abcd5621a9f in ?? ()
#53 0x000015aab64b2489 in ?? ()
#54 0x000012917282dcf1 in ?? ()
#55 0x00001bd0488addf1 in ?? ()
#56 0x00000a9d72d04101 in ?? ()
#57 0x00007fffb979c758 in ?? ()
#58 0x000030f334e1c389 in ?? ()
#59 0x00007fffb979c800 in ?? ()
#60 0x0000000000ac2c48 in v8::internal::Isolate::RunMicrotasks() ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

  6. EricTheOne commented on May 14, 2015

    @EricTheOne
    EricTheOne
    on May 14, 2015
    Author

    By the way, as I was testing the patch npm seemed to fail installing four different modules.
    I'm running configure && make && make install on the io.js repo, anything I'm missing to get npm to work?

    > heapdump@0.3.5 install /.../node_modules/heapdump
> node-gyp rebuild

gyp WARN install got an error, rolling back install
gyp ERR! configure error 
gyp ERR! stack Error: 404 status code downloading tarball
gyp ERR! stack     at Request.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/install.js:246:14)
gyp ERR! stack     at emitOne (events.js:82:20)
gyp ERR! stack     at Request.emit (events.js:169:7)
gyp ERR! stack     at Request.onRequestResponse (/usr/local/lib/node_modules/npm/node_modules/request/request.js:1156:10)
gyp ERR! stack     at emitOne (events.js:77:13)
gyp ERR! stack     at ClientRequest.emit (events.js:169:7)
gyp ERR! stack     at HTTPParser.parserOnIncomingClient (_http_client.js:414:21)
gyp ERR! stack     at HTTPParser.parserOnHeadersComplete (_http_common.js:88:23)
gyp ERR! stack     at TLSSocket.socketOnData (_http_client.js:304:20)
gyp ERR! stack     at emitOne (events.js:77:13)
gyp ERR! System Linux 3.xx.x-xx-generic
gyp ERR! command "/usr/local/bin/iojs" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /.../node_modules/heapdump
gyp ERR! node -v v2.0.2
gyp ERR! node-gyp -v v1.0.3
gyp ERR! not ok
  7. indutny
    added a commit that references this issue on May 18, 2015

    tls: prevent use-after-free

    79c1ba1

  8. EricTheOne commented on May 26, 2015

    @EricTheOne
    EricTheOne
    on May 26, 2015
    Author

    @indutny I've tested the changes on 2.1.0 as well, still segfaults:

    #0  0x000000000069f060 in SSL_read ()
#1  0x0000000000d31499 in node::TLSWrap::ClearOut() ()
#2  0x0000000000d31bce in node::TLSWrap::OnReadImpl(long, uv_buf_t const*, uv_handle_type, void*) ()
#3  0x0000000000cd285b in node::JSStream::ReadBuffer(v8::FunctionCallbackInfo<v8::Value> const&) ()
#4  0x000000000084d342 in v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) ()
#5  0x000000000087457b in v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) ()

    Any way I can help with more info?

  9. indutny commented on May 26, 2015

    @indutny
    indutny
    on May 26, 2015
    Member

    @EricTheOne I commented about it on other issue. I really need a test case that reproduces the problem. It doesn't look like I fully understand the problem atm.

  10. indutny
    added 7 commits that reference this issue on May 29, 2015

    tls: emit errors on close whilst async action

    593fd56

    tls: prevent use-after-free

    df95595

    tls: emit errors on close whilst async action

    c359ada

    tls: prevent use-after-free

    c0f4ddf

    tls: emit errors on close whilst async action

    5795e83

    tls: prevent use-after-free

    75930bb

  11. indutny commented on Jun 6, 2015

    @indutny
    indutny
    on Jun 6, 2015
    Member

    @EricTheOne PTAL at #1910, I think it should fix the issue for you ;)

  12. indutny
    added 2 commits that reference this issue on Jun 30, 2015

    _stream_wrap: prevent use after free in TLS

    c2aea93

    _stream_wrap: prevent use after free in TLS

    9180140

  13. Fishrock123 commented on Jul 7, 2015

    @Fishrock123
    Fishrock123
    on Jul 7, 2015
    Contributor

    Ala #1910 (comment), @EricTheOne did that patch fix this issue? :)

  14. mscdex
    added a commit that references this issue on Jul 9, 2015

    _stream_wrap: prevent use after free in TLS

    3a80d9c

  15. Fishrock123 commented on Jul 24, 2015

    @Fishrock123
    Fishrock123
    on Jul 24, 2015
    Contributor

    I'm going to assume it did. Please re-open if it didn't.

  16. Fishrock123
    closed this as completedon Jul 24, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    tlsIssues and PRs related to the tls subsystem.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions