Skip to content

test: investigate async-hooks/test-signalwrap #14070

Closed
Closed
test: investigate async-hooks/test-signalwrap#14070
Assignees
Trott
Labels
async_hooksIssues and PRs related to the async hooks subsystem.flaky-testIssues and PRs involving tests that fail intermittently in CI.testIssues and PRs related to Node.js core tests and test infrastructure.

Description

@refack
refack
opened on Jul 4, 2017
  • Version: master
  • Platform: aix61-ppc64
  • Subsystem: test,async_hooks
not ok 39 async-hooks/test-signalwrap
  ---
  duration_ms: 0.240
  severity: fail
  stack: |-
    Mismatched onsigusr2 function calls. Expected exactly 2, actual 0.
        at Object.exports.mustCall (/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/common/index.js:484:10)
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/async-hooks/test-signalwrap.js:14:30)
        at Module._compile (module.js:569:30)
        at Object.Module._extensions..js (module.js:580:10)
        at Module.load (module.js:503:32)
        at tryModuleLoad (module.js:466:12)
        at Function.Module._load (module.js:458:3)
        at Function.Module.runMain (module.js:605:10)
        at startup (bootstrap_node.js:158:16)
  ...

https://ci.nodejs.org/job/node-test-commit-aix/6994/nodes=aix61-ppc64/console

/cc @nodejs/async_hooks @nodejs/platform-aix

Activity

  1. refack
    added
    aixIssues and PRs related to the AIX platform.
    async_hooksIssues and PRs related to the async hooks subsystem.
    flaky-testIssues and PRs involving tests that fail intermittently in CI.
    testIssues and PRs related to Node.js core tests and test infrastructure.
    on Jul 4, 2017

  2. gireeshpunathil commented on Jul 4, 2017

    @gireeshpunathil
    gireeshpunathil
    on Jul 4, 2017
    Member

    Running a thousand times locally to see if it fails.

  3. refack commented on Jul 4, 2017

    @refack
    refack
    on Jul 4, 2017
    ContributorAuthor

    Stress CI (the whole test/async-hooks/ suite [which runs in parallel]): https://ci.nodejs.org/job/node-stress-single-test/1314/nodes=aix61-ppc64/

  4. refack commented on Jul 5, 2017

    @refack
    refack
    on Jul 5, 2017
    ContributorAuthor

    Stress CI (the whole test/async-hooks/ suite [which runs in parallel]): https://ci.nodejs.org/job/node-stress-single-test/1314/nodes=aix61-ppc64/

    Stress returned no results 🤷‍♂️

  5. Trott commented on Jul 5, 2017

    @Trott
    Trott
    on Jul 5, 2017
    Member

    Stress returned no results

    You had previously landed something that has the async-hooks tests running parallel, right? If so, maybe try the stress test again under load by passing something like -j 96 --repeat 960? Just an idea, not a particularly informed guess or anything as I haven't looked at the test. It's kinda my standard first thing to try when I can't make a test that's failing on CI fail otherwise.

  6. refack commented on Jul 5, 2017

    @refack
    refack
    on Jul 5, 2017
    ContributorAuthor

    You had previously landed something that has the async-hooks tests running parallel, right? If so, maybe try the stress test again under load by passing something like -j 96 --repeat 960? Just an idea, not a particularly informed guess or anything as I haven't looked at the test. It's kinda my standard first thing to try when I can't make a test that's failing on CI fail otherwise.

    Worth a shot - https://ci.nodejs.org/job/node-stress-single-test/1315/nodes=aix61-ppc64/
    Why 96?

  7. Trott commented on Jul 5, 2017

    @Trott
    Trott
    on Jul 5, 2017
    Member

    Why 96?

    I think there's a default shell setting on macOS that will stop things at 100, so I usually go for just under that as a first pass. And 96 instead of 97, 98, or 99 because it's kinda sorta almost a power of 2 (64 + 32), and we all prefer powers of 2, don't we?

  8. refack commented on Jul 5, 2017

    @refack
    refack
    on Jul 5, 2017
    ContributorAuthor

    @Trott you are an evil genius 😉 this gave us a plethora of weirdness... No fails in async-hooks/test-signalwrap though.

    A harness burp:

    ok 869 async-hooks/test-callback-error
  ---
  duration_ms: 4.485
  ...
Exception in thread Thread-23:
Traceback (most recent call last):
  File "/opt/freeware/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/opt/freeware/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "tools/test.py", line 140, in RunSingle
    output = case.Run()
  File "/home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64/tools/test.py", line 537, in Run
    "TEST_THREAD_ID": "%d" % self.thread_id
  File "/home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64/tools/test.py", line 519, in RunCommand
    env)
  File "/home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64/tools/test.py", line 752, in Execute
    pty_out = pty_out
  File "/home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64/tools/test.py", line 649, in RunProcess
    **rest
  File "/opt/freeware/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/opt/freeware/lib/python2.7/subprocess.py", line 1231, in _execute_child
    self.pid = os.fork()
OSError: [Errno 11] Resource temporarily unavailable

ok 870 async-hooks/test-callback-error

    A fail in callback-error (here we go again 🤦‍♂️ )

    not ok 609 async-hooks/test-callback-error
  ---
  duration_ms: 4.97
  severity: fail
  stack: |-
    start case 1
    assert.js:586
    assert.ifError = function ifError(err) { if (err) throw err; };
                                                      ^
    
    Error: spawnSync /home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64/out/Release/node EAGAIN
        at exports._errnoException (util.js:1020:11)
        at Object.spawnSync (internal/child_process.js:918:20)
        at spawnSync (child_process.js:548:24)
        at Object.<anonymous> (/home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64/test/async-hooks/test-callback-error.js:51:17)
        at Module._compile (module.js:569:30)
        at Object.Module._extensions..js (module.js:580:10)
        at Module.load (module.js:503:32)
        at tryModuleLoad (module.js:466:12)
        at Function.Module._load (module.js:458:3)
        at Function.Module.runMain (module.js:605:10)
  ...

    A different callback-error error:

    not ok 41 async-hooks/test-callback-error
  ---
  duration_ms: 7.592
  severity: fail
  stack: |-
    start case 1
    end case 1: 726.624ms
    start case 2
    end case 2: 428.448ms
    start case 3
    /home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64/test/async-hooks/test-callback-error.js:84
      child.stdout.on('data', (data) => {
                  ^
    
    TypeError: Cannot read property 'on' of undefined
        at Object.<anonymous> (/home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64/test/async-hooks/test-callback-error.js:84:15)
        at Module._compile (module.js:569:30)
        at Object.Module._extensions..js (module.js:580:10)
        at Module.load (module.js:503:32)
        at tryModuleLoad (module.js:466:12)
        at Function.Module._load (module.js:458:3)
        at Function.Module.runMain (module.js:605:10)
        at startup (bootstrap_node.js:158:16)
        at bootstrap_node.js:575:3
  ...

  9. BridgeAR commented on Sep 28, 2017

    @BridgeAR
    BridgeAR
    on Sep 28, 2017
    Member

    @refack if I understand your comment correct it seems like the original error just had a hiccup and other errors came up as being flaky instead?

  10. refack commented on Sep 28, 2017

    @refack
    refack
    on Sep 28, 2017
    ContributorAuthor

    @BridgeAR It seems like stressing AIX gives many varied failures. But AFAICT the issue in the OP is still unresolved, but also unreproducible.

    Personally I haven't seen any of those since. I'll run another stress test, and if nothing comes up consistently I think this can be closed.

    https://ci.nodejs.org/job/node-stress-single-test/1424/nodes=aix61-ppc64/

  11. refack commented on Sep 29, 2017

    @refack
    refack
    on Sep 29, 2017
    ContributorAuthor

    Well we got one of the original:

    not ok 24 async-hooks/test-signalwrap
  ---
  duration_ms: 7.424
  severity: fail
  stack: |-
    Mismatched onsigusr2Again function calls. Expected exactly 1, actual 0.
        at Object.exports.mustCall (/home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64/test/common/index.js:485:10)
        at process.onsigusr2 (/home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64/test/async-hooks/test-signalwrap.js:48:34)
        at process.<anonymous> (/home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64/test/common/index.js:517:15)
        at emitNone (events.js:105:13)
        at process.emit (events.js:207:7)
        at Signal.wrap.onsignal (internal/process.js:198:44)
  ...

    (1 out of 100 while testing with -j 96 async-hooks)

  12. joyeecheung commented on Oct 31, 2017

    @joyeecheung
    joyeecheung
    on Oct 31, 2017
    Member

    Showed up in the 8.9.0 LTS proposal #16630 , this time on CentOS

    https://ci.nodejs.org/job/node-test-commit-linux/13690/nodes=centos7-64/consoleFull

    not ok 45 async-hooks/test-signalwrap
  ---
  duration_ms: 0.156
  severity: fail
  stack: |-
    Mismatched onsigusr2Again function calls. Expected exactly 1, actual 0.
        at Object.exports.mustCall (/home/iojs/build/workspace/node-test-commit-linux/nodes/centos7-64/test/common/index.js:501:10)
        at process.onsigusr2 (/home/iojs/build/workspace/node-test-commit-linux/nodes/centos7-64/test/async-hooks/test-signalwrap.js:48:34)
        at process.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux/nodes/centos7-64/test/common/index.js:533:15)
        at emitNone (events.js:106:13)
        at process.emit (events.js:208:7)
        at Signal.wrap.onsignal (internal/process.js:208:44)
  ...

  13. 4 remaining items

  14. Trott commented on Dec 22, 2017

    @Trott
    Trott
    on Dec 22, 2017
    Member

    I think I know what's probably causing this....but first let me see if I can reproduce it...

    Stress test with -j 96 --repeat 192 test/async-hooks/test-signalwrap.js on alpine34: https://ci.nodejs.org/job/node-stress-single-test/1571/nodes=alpine34-container-x64/

  15. Trott commented on Dec 22, 2017

    @Trott
    Trott
    on Dec 22, 2017
    Member

    Also, with -j 1 --repeat 192 test/async-hooks/test-signalwrap.js: https://ci.nodejs.org/job/node-stress-single-test/1572/nodes=alpine34-container-x64/

  16. Trott commented on Dec 22, 2017

    @Trott
    Trott
    on Dec 22, 2017
    Member

    Oh, thank goodness, the first one (-j 96) managed to reproduce the issue:

    not ok 25 async-hooks/test-signalwrap
  ---
  duration_ms: 11.527
  severity: fail
  stack: |-
    Mismatched onsigusr2 function calls. Expected exactly 2, actual 1.
        at Object.exports.mustCall (/home/iojs/build/workspace/node-stress-single-test/nodes/alpine34-container-x64/test/common/index.js:496:10)
        at Object.<anonymous> (/home/iojs/build/workspace/node-stress-single-test/nodes/alpine34-container-x64/test/async-hooks/test-signalwrap.js:14:30)
        at Module._compile (module.js:660:30)
        at Object.Module._extensions..js (module.js:671:10)
        at Module.load (module.js:577:32)
        at tryModuleLoad (module.js:517:12)
        at Function.Module._load (module.js:509:3)
        at Function.Module.runMain (module.js:701:10)
        at startup (bootstrap_node.js:194:16)
  ...
  17. Trott
    added a commit that references this issue on Dec 22, 2017

    test: fix unreliable async-hooks/test-signalwrap

    a332b74

  18. Trott commented on Dec 22, 2017

    @Trott
    Trott
    on Dec 22, 2017
    Member

    Proposed fix: #17827

  19. Trott
    added a commit that references this issue on Dec 25, 2017

    test: fix unreliable async-hooks/test-signalwrap

    457ceeb
  20. Trott
    closed this as completedin 457ceebon Dec 25, 2017
  21. MylesBorins
    added 3 commits that reference this issue on Jan 8, 2018

    test: fix unreliable async-hooks/test-signalwrap

    e297f2e

    test: fix unreliable async-hooks/test-signalwrap

    2522a28

    test: fix unreliable async-hooks/test-signalwrap

    ee4cbac
  22. MylesBorins
    added a commit that references this issue on May 22, 2018

    test: fix unreliable async-hooks/test-signalwrap

    3aefc25
  23. MylesBorins
    added a commit that references this issue on Jun 14, 2018

    test: fix unreliable async-hooks/test-signalwrap

    38e4dd1
  24. rvagg
    added a commit that references this issue on Aug 16, 2018

    test: fix unreliable async-hooks/test-signalwrap

    7d263ff
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

async_hooksIssues and PRs related to the async hooks subsystem.flaky-testIssues and PRs involving tests that fail intermittently in CI.testIssues and PRs related to Node.js core tests and test infrastructure.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions