Skip to content

flaky: sequential/test-cli-syntax-* #24403

Closed
Closed
flaky: sequential/test-cli-syntax-*#24403
Labels
cliIssues and PRs related to the Node.js command-line interface.flaky-testIssues and PRs involving tests that fail intermittently in CI.testIssues and PRs related to Node.js core tests and test infrastructure.

Description

@refack
refack
opened on Nov 16, 2018
  • Version: master
  • Platform: Windows 2008R2
  • Subsystem: cli
  • test file - parallel/test-cli-syntax
  • CI job binary-windows/21615/COMPILED_BY=vs2017,RUNNER=win2008r2
  • CI worker test-rackspace-win2008r2-x64-5
  • output: 
     11:40:44 not ok 71 parallel/test-cli-syntax
 11:40:44   ---
 11:40:44   duration_ms: 1.229
 11:40:44   severity: fail
 11:40:44   exitcode: 1
 11:40:44   stack: |-
 11:40:44     assert.js:86
 11:40:44       throw new AssertionError(obj);
 11:40:44       ^
 11:40:44     
 11:40:44     AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
 11:40:44     + actual - expected
 11:40:44     
 11:40:44     + 3221225477
 11:40:44     - 1
 11:40:44         at common.mustCall (c:\workspace\node-test-binary-windows\test\parallel\test-cli-syntax.js:59:14)
 11:40:44         at c:\workspace\node-test-binary-windows\test\common\index.js:346:15
 11:40:44         at ChildProcess.exithandler (child_process.js:301:5)
 11:40:44         at ChildProcess.emit (events.js:182:13)
 11:40:44         at maybeClose (internal/child_process.js:977:16)
 11:40:44         at Socket.stream.socket.on (internal/child_process.js:395:11)
 11:40:44         at Socket.emit (events.js:182:13)
 11:40:44         at Pipe._handle.close (net.js:608:12)
 11:40:44   ...\

Activity

  1. refack
    added
    windowsIssues and PRs related to the Windows platform.
    testIssues and PRs related to Node.js core tests and test infrastructure.
    cliIssues and PRs related to the Node.js command-line interface.
    flaky-testIssues and PRs involving tests that fail intermittently in CI.
    on Nov 16, 2018

  2. mhdawson commented on Nov 20, 2018

    @mhdawson
    mhdawson
    on Nov 20, 2018
    Member

    Also failed on linux here: https://ci.nodejs.org/job/node-test-commit-linux-containered/8755/nodes=ubuntu1604_sharedlibs_openssl110_x64/consoleFull

    21:14:55 not ok 192 parallel/test-cli-syntax
21:14:55   ---
21:14:55   duration_ms: 2.556
21:14:55   severity: fail
21:14:55   exitcode: 1
21:14:55   stack: |-
21:14:55     assert.js:753
21:14:55         throw newErr;
21:14:55         ^
21:14:55     
21:14:55     AssertionError [ERR_ASSERTION]: ifError got unwanted exception: Command failed: /home/iojs/build/workspace/node-test-commit-linux-containered/out/Release/node --check /home/iojs/build/workspace/node-test-commit-linux-containered/test/fixtures/syntax/illegal_if_not_wrapped.js
21:14:55     Segmentation fault (core dumped)
21:14:55     
21:14:55         at ChildProcess.exithandler (child_process.js:294:12)
21:14:55         at ChildProcess.emit (events.js:189:13)
21:14:55         at maybeClose (internal/child_process.js:977:16)
21:14:55         at Socket.stream.socket.on (internal/child_process.js:395:11)
21:14:55         at Socket.emit (events.js:189:13)
21:14:55         at Pipe._handle.close (net.js:608:12)
21:14:55   ...

    Worker: https://ci.nodejs.org/computer/test-joyent-ubuntu1604_sharedlibs_container-x64-4/
    Config: ubuntu1604_sharedlibs_openssl110_x64

  3. Trott commented on Nov 23, 2018

    @Trott
    Trott
    on Nov 23, 2018
    Member

    @mhdawson Core dumped? Yikes! Hopefully build-related and not related to the test? @nodejs/build

  4. refack commented on Nov 23, 2018

    @refack
    refack
    on Nov 23, 2018
    ContributorAuthor

    Core dumped? Yikes!

    IIUC the core dump was for the tested spawned child process, not the testing process.

    node/test/parallel/test-cli-syntax.js

    Lines 35 to 40 in d45e303

    const cmd = [node, ..._args].join(' ');
    exec(cmd, common.mustCall((err, stdout, stderr) => {
    assert.ifError(err);
    assert.strictEqual(stdout, '');
    assert.strictEqual(stderr, '');
    }));

  5. Trott
    added a commit that references this issue on Dec 1, 2018

    test: show stdout and stderr in test-cli-syntax when it fails

    3fb627b

  6. addaleax commented on Dec 4, 2018

    @addaleax
    addaleax
    on Dec 4, 2018
    Member

    Managed to get a core dump locally; interesting parts of the thread stack traces:

    Thread 7 (Thread 0x7fd3e1708740 (LWP 32401)):
#0  _int_free (have_lock=0, p=0x557d4eedc500, av=0x7fd3e05a3c40 <main_arena>) at malloc.c:4167
#1  __GI___libc_free (mem=0x557d4eedc510) at malloc.c:3124
#2  0x0000557d4d144471 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*) ()
#3  0x0000557d4d1445e8 in node::native_module::NativeModuleLoader::~NativeModuleLoader() ()
#4  0x00007fd3e01fb041 in __run_exit_handlers (status=0, listp=0x7fd3e05a3718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108
#5  0x00007fd3e01fb13a in __GI_exit (status=<optimized out>) at exit.c:139
#6  0x0000557d4d113131 in node::Environment::Exit(int) ()
#7  0x0000557d4d1388d1 in node::Exit(v8::FunctionCallbackInfo<v8::Value> const&) ()
#8  0x0000557d4d3b1237 in v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) ()
#9  0x0000557d4d3b2558 in v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) ()
#10 0x00001c51e784fb5d in ?? ()
#11 0x00001c51e7884b30 in ?? ()
#12 0x00001c51e784fac1 in ?? ()
#13 0x00007ffef03e0990 in ?? ()
#14 0x0000000000000006 in ?? ()

[...]

Thread 1 (Thread 0x7fd3df9b6700 (LWP 32404)):
#0  0x0000557d4ef3a400 in ?? ()
#1  0x0000557d4d6eb6dc in v8::internal::ArrayBufferCollector::FreeingTask::RunInternal() ()
#2  0x0000557d4d1cbc45 in node::(anonymous namespace)::PlatformWorkerThread(void*) ()
#3  0x00007fd3e05b06db in start_thread (arg=0x7fd3df9b6700) at pthread_create.c:463
#4  0x00007fd3e02d988f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

    I feel like I’ve been seeing a lot more issues based on race conditions with C++ destructors being run from __run_exit_handlers(), but before all threads are torn down… I’m not sure what would have caused that change. Should we maybe be calling _exit() instead? Is there a way to first stop all other threads?

    /cc @joyeecheung

  7. addaleax
    removed
    windowsIssues and PRs related to the Windows platform.
    on Dec 4, 2018
  8. addaleax
    added a commit that references this issue on Dec 4, 2018

    src: dispose of V8 platform in `process.exit()`

    b3c539a

  9. joyeecheung commented on Dec 4, 2018

    @joyeecheung
    joyeecheung
    on Dec 4, 2018
    Member

    About the specific stack trace, was the destructor being called when one of the strings on the native module loader was already destroyed? I think the std::strings on the loader should all be local to it, so it's kind of difficult to how would the order be reversed..

    On the race condition in general, is the current expectation to destroy all the per-process globals only after all the threads are torn down? I wonder if using shared_ptr for these globals would help, but that probably doesn't actually solve the root cause..

  10. addaleax commented on Dec 4, 2018

    @addaleax
    addaleax
    on Dec 4, 2018
    Member

    Potential fix is in #24828 :)

  11. addaleax commented on Dec 4, 2018

    @addaleax
    addaleax
    on Dec 4, 2018
    Member

    And no, I don’t think this is about the specific stack trace – it’s probably some destructor that was called earlier that destroyed some resource on which the V8 platform worker thread relied, or something similar, I would say.

  12. Fishrock123 commented on Dec 4, 2018

    @Fishrock123
    Fishrock123
    on Dec 4, 2018
    Contributor

    Another failure:

    assert.js:86
  throw new AssertionError(obj);
  ^

AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:

139 !== 1

    at common.mustCall (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-cli-syntax.js:97:14)
    at /home/iojs/build/workspace/node-test-commit-linux-containered/test/common/index.js:346:15
    at ChildProcess.exithandler (child_process.js:301:5)
    at ChildProcess.emit (events.js:189:13)
    at maybeClose (internal/child_process.js:977:16)
    at Socket.stream.socket.on (internal/child_process.js:395:11)
    at Socket.emit (events.js:189:13)
    at Pipe._handle.close (net.js:612:12)

  13. 36 remaining items

  14. gireeshpunathil commented on Feb 5, 2019

    @gireeshpunathil
    gireeshpunathil
    on Feb 5, 2019
    Member

    this should be good to close now; ncu data did now show this up.
    cc @refack @Trott

  15. Trott commented on Feb 5, 2019

    @Trott
    Trott
    on Feb 5, 2019
    Member

    Agreed: Not seeing it in ncu-ci data. Closing. Obviously, re-open if I'm wrong.

  16. Trott
    closed this as completedon Feb 5, 2019
  17. BethGriggs
    added 6 commits that reference this issue on Feb 12, 2019

    test: show stdout and stderr in test-cli-syntax when it fails

    46e37ad

    test: move test-cli-syntax to sequential

    4e51e3d

    test: mark test-cli-syntax as flaky/unreliable

    2869b78

    src: dispose of V8 platform in `process.exit()`

    3c3f350

    test: move test-cli-syntax to sequential

    09e092e

    test: mark test-cli-syntax as flaky/unreliable

    6c5eb1f
  18. rvagg
    added 3 commits that reference this issue on Feb 28, 2019

    test: show stdout and stderr in test-cli-syntax when it fails

    850a1af

    test: move test-cli-syntax to sequential

    fedb5d8

    test: mark test-cli-syntax as flaky/unreliable

    eafe870
  19. BethGriggs
    added 2 commits that reference this issue on Mar 27, 2019

    src: dispose of V8 platform in `process.exit()`

    25ca9f1

    src: dispose of V8 platform in `process.exit()`

    f4be176
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    cliIssues and PRs related to the Node.js command-line interface.flaky-testIssues and PRs involving tests that fail intermittently in CI.testIssues and PRs related to Node.js core tests and test infrastructure.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions