Skip to content

Large virtual memory (13-20 GB) when running Peertube without --jitless #45557

Closed as not planned
Closed as not planned
Large virtual memory (13-20 GB) when running Peertube without --jitless#45557
Labels
questionIssues asking questions about Node.js.

Description

@pbeyssac
pbeyssac
opened on Nov 21, 2022

Version

v18.7.0 (also tested on node14)

Platform

FreeBSD tube 12.3-RELEASE FreeBSD 12.3-RELEASE GENERIC amd64 (also tested on Linux)

Subsystem

No idea (but no idea how to check for this)

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

Same behaviour at every run. Seems easy to reproduce.

What is the expected behavior?

13-20 GB seems to be a lot more than necessary.

Starting with --jitless seems to shrink the memory size significantly to ~1 GB, which looks more reasonable:

image

Perhaps there's a specific Javascript module with specific code which triggers excessive memory use in the JIT compiler, but I have no idea how to check for this.

What do you see instead?

nodejs grows up to 13 GB - 20 GB memory (depending on node version/OS -- same observations on node14/FreeBSD, Linux, and on other Peertube instances)

image

Additional information

Posted here on the Peertube forum, where I was advised to come here and ask for help.

https://framacolibri.org/t/high-memory-use-for-node-process-v4-3-0-v4-3-1/16155/7

Activity

  1. tniessen commented on Nov 21, 2022

    @tniessen
    tniessen
    on Nov 21, 2022
    Member

    @pbeyssac Could you provide the output of cat /proc/74144/status please, replacing 74144 with the actual PID of the node process when its memory usage appears to be high?

  2. pbeyssac commented on Nov 21, 2022

    @pbeyssac
    pbeyssac
    on Nov 21, 2022
    Author

    @pbeyssac Could you provide the output of cat /proc/74144/status please, replacing 74144 with the actual PID of the node process when its memory usage appears to be high?

    Thanks for your reply.

    Here goes:

    % cat /proc/94670/status
node 94670 67081 94670 67081 pts/5 ctty 1669029587,527265 4,154345 0,557096 kqread 1001 1001 1001,1001 ioc-tube

    Note that it is FreeBSD's /proc, which is different from the Linux output, so here's the relevant bit in the man page:

         status  The process status.  This file is read-only and returns a single
             line containing multiple space-separated fields as follows:

             •   command name
             •   process id
             •   parent process id
             •   process group id
             •   session id
             •   device name of the controlling terminal, or a minus sign
                 (“-”) if there is no controlling terminal.
             •   a list of process flags: ctty if there is a controlling
                 terminal, sldr if the process is a session leader, noflags if
                 neither of the other two flags are set.
             •   the process start time in seconds and microseconds, comma
                 separated.
             •   the user time in seconds and microseconds, comma separated.
             •   the system time in seconds and microseconds, comma separated.
             •   the wait channel message
             •   the process credentials consisting of the effective user id
                 and the list of groups (whose first member is the effective
                 group id) all comma separated.
             •   the hostname of the jail in which the process runs, or ‘-’ to
                 indicate that the process is not running within a jail.

    Please find attached the memory map of the process as returned by /proc/${pid}/map.
    map.txt

    Another remark (I'm not sure this is relevant, but just in case): the process I started with --jitless died on an error which looks like a Peertube error, but which never happens when running with JIT. Perhaps a race condition due to the code running at a different speed.

    [...:443] 2022-11-21 12:06:17.761 error: uncaughtException: Cannot read properties of undefined (reading 'timeoutType')
TypeError: Cannot read properties of undefined (reading 'timeoutType')
    at _resume (/var/www/peertube/versions/peertube-v4.3.1/node_modules/undici/lib/client.js:1167:29)
    at resume (/var/www/peertube/versions/peertube-v4.3.1/node_modules/undici/lib/client.js:1131:3)
    at connect (/var/www/peertube/versions/peertube-v4.3.1/node_modules/undici/lib/client.js:1116:3) {
  "error": {
    "stack": "TypeError: Cannot read properties of undefined (reading 'timeoutType')\n    at _resume (/var/www/peertube/versions/peertube-v4.3.1/node_modules/undici/lib/client.js:1167:29)\n    at resume (/var/www/peertube/versions/peertube-v4.3.1/node_modules/undici/lib/client.js:1131:3)\n    at connect (/var/www/peertube/versions/peertube-v4.3.1/node_modules/undici/lib/client.js:1116:3)",
    "message": "Cannot read properties of undefined (reading 'timeoutType')"
  },
(...)

  3. tniessen commented on Nov 21, 2022

    @tniessen
    tniessen
    on Nov 21, 2022
    Member
    % cat /proc/94670/status
node 94670 67081 94670 67081 pts/5 ctty 1669029587,527265 4,154345 0,557096 kqread 1001 1001 1001,1001 ioc-tube

    Note that it is FreeBSD's /proc

    This looks like /stat, not /status. /stat does not contain the accumulated memory info.

  4. pbeyssac commented on Nov 21, 2022

    @pbeyssac
    pbeyssac
    on Nov 21, 2022
    Author

    This looks like /stat, not /status. /stat does not contain the accumulated memory info.

    I confirm it is really /proc/${pid}/status on FreeBSD (see the "cat", directly copied from my terminal). FreeBSD /proc differs from Linux and has no /proc/*/stat anyway.

    Here /rlimit just in case (but these are static limits, not usage):

    % cat /proc/94670/rlimit 
cpu -1 -1
fsize -1 -1
data 34359738368 34359738368
stack 536870912 536870912
core -1 -1
rss -1 -1
memlock 65536 65536
nproc 63680 63680
nofile 1882368 1882368
sbsize -1 -1
vmem -1 -1
npts -1 -1
swap -1 -1
kqueues -1 -1
umtx -1 -1

    man page excerpt:

         rlimit  This is a read-only file containing the process current and
             maximum limits.  Each line is of the format rlimit current max,
             with -1 indicating infinity.

    Excerpt from ps:

    USER       PID  %CPU %MEM      VSZ     RSS TT  STAT  STARTED        TIME COMMAND
peertube 94670   0,0  0,3 13254356  203024  5  S+J   12:19       0:12,45 node: peertube (node)

  5. tniessen commented on Nov 21, 2022

    @tniessen
    tniessen
    on Nov 21, 2022
    Member

    Sorry, I was looking at the man page for CentOS, not FreeBSD.

    The RSS looks reasonable. Maybe the JIT compiler creates a bunch of large memory mappings thus blowing up the reserved address space in virtual memory, but that's usually not a problem. (For example, V8 might decide to reserve a 4GiB address range for every single WebAssembly instance for performance reasons.)

  6. bnoordhuis commented on Nov 22, 2022

    @bnoordhuis
    bnoordhuis
    on Nov 22, 2022
    Member

    Maybe the JIT compiler creates a bunch of large memory mappings

    That's what happens. I'm going to guess peertube loads wasm files (undici does, of that I'm sure) and V8 makes large reservations for wasm modules, basically XL guard pages. That's virtual memory though, not RSS, so it's harmless.

  7. tniessen
    added
    questionIssues asking questions about Node.js.
    on Nov 22, 2022
  8. tniessen
    changed the title [-]Extremely high node18/node14 memory use (13-20 GB) when running Peertube, fixed by running with --jitless[/-] [+]Large virtual memory (13-20 GB) when running Peertube without --jitless[/+] on Nov 22, 2022
  9. bnoordhuis
    closed this as not plannedon Nov 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionIssues asking questions about Node.js.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions