Investigate flaky test-watch-mode (no. 3) #44898

Activity

1. tniessen

   added

   flaky-testIssues and PRs involving tests that fail intermittently in CI.Issues and PRs involving tests that fail intermittently in CI.

   linuxIssues and PRs related to the Linux platform.Issues and PRs related to the Linux platform.

   on Oct 5, 2022
2. tniessen commented on Oct 7, 2022

   

   tniessen

   on Oct 7, 2022

   MemberAuthor

   More actions

   Similar output:

   16:12:03 not ok 3793 sequential/test-watch-mode
   16:12:06   ---
   16:12:06   duration_ms: 3.677
   16:12:06   severity: fail
   16:12:06   exitcode: 1
   16:12:06   stack: |-
   16:12:06     TAP version 13
   16:12:06     # Subtest: watch mode
   16:12:06         # Subtest: should watch changes to a file - event loop ended
   16:12:06         ok 1 - should watch changes to a file - event loop ended
   16:12:06           ---
   16:12:06           duration_ms: 1657.422205
   16:12:06           ...
   16:12:06         # Subtest: should watch changes to a failing file
   16:12:06         ok 2 - should watch changes to a failing file
   16:12:06           ---
   16:12:06           duration_ms: 2144.675239
   16:12:06           ...
   16:12:06         # Subtest: should not watch when running an non-existing file
   16:12:06         ok 3 - should not watch when running an non-existing file
   16:12:06           ---
   16:12:06           duration_ms: 848.075766
   16:12:06           ...
   16:12:06         # Subtest: should watch when running an non-existing file - when specified under --watch-path
   16:12:06         ok 4 - should watch when running an non-existing file - when specified under --watch-path # SKIP
   16:12:06           ---
   16:12:06           duration_ms: 281.828198
   16:12:06           ...
   16:12:06         # Subtest: should watch changes to a file - event loop blocked
   16:12:06         ok 5 - should watch changes to a file - event loop blocked
   16:12:06           ---
   16:12:06           duration_ms: 1094.820609
   16:12:06           ...
   16:12:06         # Subtest: should watch changes to dependencies - cjs
   16:12:06         ok 6 - should watch changes to dependencies - cjs
   16:12:06           ---
   16:12:06           duration_ms: 2049.295256
   16:12:06           ...
   16:12:06         # Subtest: should watch changes to dependencies - esm
   16:12:06         ok 7 - should watch changes to dependencies - esm
   16:12:06           ---
   16:12:06           duration_ms: 2216.116407
   16:12:06           ...
   16:12:06         # Subtest: should restart multiple times
   16:12:06         ok 8 - should restart multiple times
   16:12:06           ---
   16:12:06           duration_ms: 3098.411375
   16:12:06           ...
   16:12:06         # Subtest: should pass arguments to file
   16:12:06         ok 9 - should pass arguments to file
   16:12:06           ---
   16:12:06           duration_ms: 1908.309201
   16:12:06           ...
   16:12:06         # Subtest: should not load --require modules in main process
   16:12:06         ok 10 - should not load --require modules in main process
   16:12:06           ---
   16:12:06           duration_ms: 1360.515837
   16:12:06           ...
   16:12:06         # Subtest: should not load --import modules in main process
   16:12:06         not ok 11 - should not load --import modules in main process
   16:12:06           ---
   16:12:06           duration_ms: 2044.114142
   16:12:06           failureType: 'testCodeFailure'
   16:12:06           error: |-
   16:12:06             Expected values to be strictly deep-equal:
   16:12:06             + actual - expected
   16:12:06             
   16:12:06               [
   16:12:06             +   "Completed running '/usr/home/iojs/node-tmp/.tmp.3792/3.js'",
   16:12:06             +   "Restarting '/usr/home/iojs/node-tmp/.tmp.3792/3.js'"
   16:12:06             -   "Restarting '/usr/home/iojs/node-tmp/.tmp.3792/3.js'",
   16:12:06             -   "Completed running '/usr/home/iojs/node-tmp/.tmp.3792/3.js'"
   16:12:06               ]
   16:12:06           code: 'ERR_ASSERTION'
   16:12:06           stack: |-
   16:12:06             assertRestartedCorrectly (file:///usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd12-x64/test/sequential/test-watch-mode.mjs:74:10)
   16:12:06             Object.<anonymous> (file:///usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd12-x64/test/sequential/test-watch-mode.mjs:218:5)
   16:12:06             process.processTicksAndRejections (node:internal/process/task_queues:95:5)
   16:12:06             async ItTest.run (node:internal/test_runner/test:512:9)
   16:12:06             async Promise.all (index 10)
   16:12:06             async Suite.run (node:internal/test_runner/test:739:7)
   16:12:06           ...
   16:12:06         1..11
   16:12:06     not ok 1 - watch mode
   16:12:06       ---
   16:12:06       duration_ms: 3336.446593
   16:12:06       failureType: 'subtestsFailed'
   16:12:06       error: '1 subtest failed'
   16:12:06       code: 'ERR_TEST_FAILURE'
   16:12:06       ...
   16:12:06     1..1
   16:12:06     # tests 1
   16:12:06     # pass 0
   16:12:06     # fail 1
   16:12:06     # cancelled 0
   16:12:06     # skipped 0
   16:12:06     # todo 0
   16:12:06     # duration_ms 3344.389888
   16:12:06   ...
   

   @MoLow Looks like the order within some arrays is non-deterministic. Not sure if that's a bug in the implementation or in the test.
3. tniessen

   added

   freebsdIssues and PRs related to the FreeBSD platform.Issues and PRs related to the FreeBSD platform.

   on Oct 7, 2022
4. MoLow commented on Oct 10, 2022

   

   MoLow

   on Oct 10, 2022

   Member

   More actions

   On vacation, will have to wait unless someone else wants to give it a look
5. p-b-o commented on Oct 12, 2022

   

   p-b-o

   on Oct 12, 2022

   Contributor

   More actions

   This issue was observed on Windows on Arm platform too.

   @tniessen @MoLow @richardlau Would you accept to mark those tests as flaky, waiting for a proper fix?
6. kapouer commented on Oct 15, 2022

   

   kapouer

   on Oct 15, 2022

   Contributor

   More actions

   That test is failing in a variety of ways on our debian builds too (https://buildd.debian.org/status/package.php?p=nodejs&suite=experimental)
7. p-b-o commented on Oct 18, 2022

   

   p-b-o

   on Oct 18, 2022

   Contributor

   More actions

   Created a PR to mark those test-watch-mode as flaky, so it can unlock the current situation.
8. kapouer commented on Oct 18, 2022

   

   kapouer

   on Oct 18, 2022

   Contributor

   More actions

   IMO they're more than flaky, several times they just hang for a long time.
9. p-b-o commented on Oct 18, 2022

   

   p-b-o

   on Oct 18, 2022

   Contributor

   More actions

   Is there a timeout of stuff like this we can set for nodejs tests? Or deletion is the only alternative?
10. kapouer commented on Oct 18, 2022

    

    kapouer

    on Oct 18, 2022

    Contributor

    More actions

    In theory there is already a timeout for tests, but in the case of that test it didn't work (as usual, take what I say with precautions because it's with the debian build of nodejs) and the whole test suite just hang.
11. p-b-o commented on Oct 18, 2022

    

    p-b-o

    on Oct 18, 2022

    Contributor

    More actions

    @richardlau @MoLow Should we delete those tests for now, as they seem to create some difficult problems? I know it's not ideal, but hanging a testsuite and report random failures is really a huge difficulty.
12. MoLow commented on Oct 18, 2022

    

    MoLow

    on Oct 18, 2022

    Member

    More actions

    the timeouts happen in the test-watch-mode-inspect tests, and flakyness in test-watch-mode,
    I am planning to take a deeper look but I am with a temporary computer for a while - I think for the time it is ok to mark as flaky without deleting
13. MoLow

    added

    watch-modeIssues and PRs related to watch mode.Issues and PRs related to watch mode.

    on Oct 18, 2022
14. tniessen commented on Nov 19, 2022

    

    tniessen

    on Nov 19, 2022

    MemberAuthor

    More actions

    Another occurrence on macOS:

    === release test-watch-mode-files_watcher ===
    Path: parallel/test-watch-mode-files_watcher
    Error: --- stdout ---
    TAP version 13
    # Subtest: watch mode file watcher
        # Subtest: should watch changed files
        ok 1 - should watch changed files
          ---
          duration_ms: 20.78446
          ...
        # Subtest: should throttle changes
        ok 2 - should throttle changes
          ---
          duration_ms: 230.579784
          ...
        # Subtest: should ignore files in watched directory if they are not filtered
        ok 3 - should ignore files in watched directory if they are not filtered
          ---
          duration_ms: 1004.011198
          ...
        # Subtest: should allow clearing filters
        ok 4 - should allow clearing filters
          ---
          duration_ms: 1219.288977
          ...
    Command: out/Release/node --expose-internals /Users/runner/work/node/node/test/parallel/test-watch-mode-files_watcher.mjs
    --- TIMEOUT ---
    
    ===
    === 1 tests failed
    ===
    
15. 60 remaining items

    Load more
16. aduh95

    added a commit that references this issue on Jun 18, 2026

    test: fix flaky test-watch-mode-inspect timeout

    ...

    d9f4e8e
17. marco-ippolito

    added a commit that references this issue on Jul 29, 2026

    test: fix flaky test-watch-mode-inspect timeout

    ...

    546f967
18. kotlarmilos

    added a commit that references this issue on Aug 12, 2026

    Merge upstream Node.js v24.18.1 into dotnet/main (#375)

    ...

    4dc0c19
19. christianaurichzm

    added a commit that references this issue on Aug 28, 2026

    test: deflake test-watch-mode-restart-esm-loading-error

    ...

    c48847c
20. nodejs-github-bot

    added a commit that references this issue on Aug 31, 2026

    test: deflake test-watch-mode-restart-esm-loading-error

    ...

    bfaefce
21. aduh95

    added a commit that references this issue on Sep 3, 2026

    test: deflake test-watch-mode-restart-esm-loading-error

    ...

    8916332
22. R31K4G3

    added a commit that references this issue on Sep 9, 2026

    test: deflake test-watch-mode-restart-esm-loading-error

    ...

    32281ce
23. aduh95

    added 2 commits that reference this issue on Sep 9, 2026

    test: deflake test-watch-mode-restart-esm-loading-error

    ...

    e725152

    test: deflake test-watch-mode-restart-esm-loading-error

    ...

    500b98c