Skip to content

Using custom loaders results in a load loop / OOM #47566

Closed
#47620
Closed
Using custom loaders results in a load loop / OOM#47566
#47620
Labels
confirmed-bugIssues and PRs for confirmed bugs.loadersIssues and PRs related to ES module loaders.

Description

@merceyz
merceyz
opened on Apr 14, 2023

Version

v20.0.0-nightly20230414c94be4125b

Platform

Linux server 5.15.0-69-generic #76-Ubuntu SMP Fri Mar 17 17:19:29 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Subsystem

No response

What steps will reproduce the bug?

touch loader.js
echo "setInterval(() => {}, 1000);" > test.cjs
node --loader ./loader.js test.cjs

How often does it reproduce? Is there a required condition?

No response

What is the expected behavior? Why is that the expected behavior?

Node.js starts up without loading the provided loader over and over again.

What do you see instead?

Node.js spams the console with the following warning while the memory and CPU usage climbs until it's stopped / OOMs.

(node:5652) ExperimentalWarning: Custom ESM Loaders is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)

Additional information

Possibly relevant #44710

Activity

  1. merceyz
    changed the title [-]Combining `--require` and `--loader` results in a load loop[/-] [+]Combining `--require` and `--loader` results in a load loop / OOM[/+] on Apr 14, 2023

  2. aduh95 commented on Apr 15, 2023

    @aduh95
    aduh95
    on Apr 15, 2023
    Contributor

    @nodejs/loaders

  3. VoltrexKeyva
    added
    loadersIssues and PRs related to ES module loaders.
    on Apr 15, 2023

  4. aduh95 commented on Apr 17, 2023

    @aduh95
    aduh95
    on Apr 17, 2023
    Contributor

    I'm able to reproduce with the suggested code snippet, but it looks like there's more to it than just combining --require and --loader, here's what I get when running the following at the root of the node repo:

    $ out/Release/node --no-warnings --require ./test/fixtures/printA.js --import ./test/fixtures/printB.js --loader ./test/fixtures/empty.js ./test/fixtures/printC.js
A
A
B
C

    It looks like the reported bug only happens on REPL. Worth noting that we also another bug that makes the --required script run twice instead of once.

  5. aduh95
    added
    replIssues and PRs related to the REPL subsystem.
    on Apr 17, 2023

  6. merceyz commented on Apr 17, 2023

    @merceyz
    merceyz
    on Apr 17, 2023
    MemberAuthor

    It looks like the reported bug only happens on REPL

    I can also reproduce the issue without the REPL:

    touch require.cjs
touch loader.mjs
echo "setInterval(() => {}, 1000);" > test.cjs
node --require ./require.cjs --loader ./loader.mjs test.cjs

  7. aduh95 commented on Apr 17, 2023

    @aduh95
    aduh95
    on Apr 17, 2023
    Contributor

    I don't think it has much to do with --require:

    $ out/Release/node --loader ./test/fixtures/empty.js
[…never ending output of experimental warning…]
  8. merceyz
    changed the title [-]Combining `--require` and `--loader` results in a load loop / OOM[/-] [+]Using custom loaders results in a load loop / OOM[/+] on Apr 17, 2023

  9. merceyz commented on Apr 17, 2023

    @merceyz
    merceyz
    on Apr 17, 2023
    MemberAuthor

    Indeed, I've updated the issue to reflect that 👍

    Worth noting that we also another bug that makes the --required script run twice instead of once

    That's expected behaviour in my opinion, once in the main process and once in the Worker for the loaders.

  10. JakobJingleheimer commented on Apr 17, 2023

    @JakobJingleheimer
    JakobJingleheimer
    on Apr 17, 2023
    Member

    I can't repro at all (admittedly on macOS ARM, but I don't see how this could be OS-related):

    At main's current HEAD (e050cec), the OP's

    snippet 
    touch loader.mjs
echo "setInterval(() => {}, 1000);" > test.cjs
node --loader ./loader.mjs test.cjs

    runs endlessly (as expected) and memory stays constant (at 20.2 MB, after start-up), with only 1 experimental warning.

    I get the same behaviour on v20.0.0-proposal's current HEAD (bb533f7), but with slightly less memory consumption (20.1 MB after start-up).

    The code that controls whether the warning is emitted is quite simple and handles multiple instances of the module loader:

    node/lib/internal/modules/esm/loader.js

    Lines 405 to 420 in b342a1b

    let emittedExperimentalWarning = false;
    /**
    * A loader instance is used as the main entry point for loading ES modules. Currently, this is a singleton; there is
    * only one used for loading the main module and everything in its dependency graph, though separate instances of this
    * class might be instantiated as part of bootstrap for other purposes.
    * @param {boolean} useCustomLoadersIfPresent If the user has provided loaders via the --loader flag, use them.
    * @returns {DefaultModuleLoader | CustomizedModuleLoader}
    */
    function createModuleLoader(useCustomLoadersIfPresent = true) {
    if (useCustomLoadersIfPresent) {
    const userLoaderPaths = getOptionValue('--experimental-loader');
    if (userLoaderPaths.length > 0) {
    if (!emittedExperimentalWarning) {
    emitExperimentalWarning('Custom ESM Loaders');
    emittedExperimentalWarning = true;
    }

    (emittedExperimentalWarning lives at the top level of esm/loader.js)

    Am I missing something?

  11. aduh95 commented on Apr 17, 2023

    @aduh95
    aduh95
    on Apr 17, 2023
    Contributor

    I think this is happening because the loader is going to try to spawn a worker thread from within the worker thread, which is also going to do the same thing, and that's how we end up with Mr Meeseeks situation until the system is out of memory or the process terminates. I'm working on a fix.

  12. aduh95
    added
    confirmed-bugIssues and PRs for confirmed bugs.
    and removed
    replIssues and PRs related to the REPL subsystem.
    on Apr 17, 2023

  13. bmeck commented on Apr 17, 2023

    @bmeck
    bmeck
    on Apr 17, 2023 via email
    Member

  14. kirrg001 commented on Apr 19, 2023

    @kirrg001
    kirrg001
    on Apr 19, 2023

    Experiencing the same. The loader is loaded over and over again as soon as I import specific libraries.

    node --experimental-loader ./esm-loader.mjs index.mjs

    esm-loader.mjs

    // works
import { get } from 'node:https';

// both do not work and restart the loader in a loop
import CoffeeScript from 'coffeescript';
import mysql from 'mysql';

    (node:9286) ExperimentalWarning: Custom ESM Loaders is an experimental feature and might change at any time
    (Use node --trace-warnings ... to show where the warning was created)
    (node:9286) ExperimentalWarning: Custom ESM Loaders is an experimental feature and might change at any time
    (Use node --trace-warnings ... to show where the warning was created)
    (node:9286) ExperimentalWarning: Custom ESM Loaders is an experimental feature and might change at any time
    (Use node --trace-warnings ... to show where the warning was created)
    (node:9286) ExperimentalWarning: Custom ESM Loaders is an experimental feature and might change at any time
    (Use node --trace-warnings ... to show where the warning was created)

    Works fine with Node v18

  15. 12 remaining items

  16. targos
    added a commit that references this issue on May 2, 2023

    loader: use default loader as cascaded loader in the in loader worker

    91e4a7c

  17. TomasHubelbauer commented on May 4, 2023

    @TomasHubelbauer
    TomasHubelbauer
    on May 4, 2023

    FYI 20.1.0 is out and the PR for this issue, #47620, is included in it.

  18. aheissenberger commented on May 5, 2023

    @aheissenberger
    aheissenberger
    on May 5, 2023

    I tested with node 20.1.0 and the problem still exists without any changes:
    cyco130/vavite#39 (comment)

  19. TomasHubelbauer commented on May 5, 2023

    @TomasHubelbauer
    TomasHubelbauer
    on May 5, 2023

    Just in case this is useful to anyone - in my case it actually did fix the issue… But it also revealed another one which I think is related: #47880.

  20. kirrg001 commented on May 8, 2023

    @kirrg001
    kirrg001
    on May 8, 2023

    Thank you. 20.1.0 fixed the issue for us.

  21. targos
    added a commit that references this issue on Nov 11, 2023

    loader: use default loader as cascaded loader in the in loader worker

    ee9cef2
  22. targos
    added a commit that references this issue on Nov 23, 2023

    loader: use default loader as cascaded loader in the in loader worker

    b77a6b2
  23. sercher
    added 2 commits that reference this issue on Apr 25, 2024

    loader: use default loader as cascaded loader in the in

    cb8981f

    loader: use default loader as cascaded loader in the in

    c07d36d
  24. BridgeAR
    added a commit that references this issue on Aug 4, 2026

    fix(loader): avoid Node 20.0 loader recursion

    79dc7cf
  25. pabloerhard
    added a commit that references this issue on Aug 11, 2026

    fix(loader): avoid Node 20.0 loader recursion (#9632)

    77cd463
  26. dd-octo-sts
    added 2 commits that reference this issue on Aug 12, 2026

    fix(loader): avoid Node 20.0 loader recursion (#9632)

    88e775e

    fix(loader): avoid Node 20.0 loader recursion (#9632)

    24891fd
  27. pabloerhard
    added 2 commits that reference this issue on Aug 17, 2026

    fix(loader): avoid Node 20.0 loader recursion (#9632)

    5bd990b

    fix(loader): avoid Node 20.0 loader recursion (#9632)

    d2090c5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    confirmed-bugIssues and PRs for confirmed bugs.loadersIssues and PRs related to ES module loaders.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions