Skip to content

rustc hangs after ICEing due to memory limit #110771

Closed
#110975
Closed
rustc hangs after ICEing due to memory limit#110771
#110975
Labels
C-bugCategory: This is a bug.P-mediumMedium priorityT-compilerRelevant to the compiler team, which will review and decide on the PR/issue.regression-from-stable-to-nightlyPerformance or correctness regression from stable to nightly.regression-untriagedUntriaged performance or correctness regression.

Description

@matthiaskrgr
matthiaskrgr
opened on Apr 24, 2023

I use prlimit to limit rustcs memory usage and max runtime per thread (like a light sandboxing basically) and I have a giant
files.par_iter().for_each(|file| rustc_flags.iter(|flag| exec( prlimit_run rustc flag file.rs))).collect::vec<ICE>(); loop

I run prlimit --noheadings --as=3076000000 --cpu=30 rustc file flags, so a rustc process is limited to roughly 3 gb of ram and 30 seconds of cpu time.

I noticed that after 39cf520 #109507 (I bisected this), my rayon loop would sometimes get stuck randomly.
It was very weird because there was no cpu load, just as if someone had temporarily suspended one of the rayon threads and now we'd wait for it/them to finish indefinitely.

I can reproduce the problem tests/ui/limits/huge-struct.rs for example:

prlimit --noheadings --as=3076000000 --cpu=30 /home/matthias/.rustup/toolchains/master/bin/rustc /tmp/IM/huge-struct.rs -Zdump-mir-dir=/tmp/icemaker_global_tempdir.lyoD6fyhTfAh/rustc_testrunner_tmpdir.VNlnLWzycl69 -Zno-codegen -Zunstable-options -Zvalidate-mir -Zverify-llvm-ir=yes -Zincremental-verify-ich=yes -Zmir-opt-level=0 -Zmir-opt-level=1 -Zmir-opt-level=2 -Zmir-opt-level=3 -Zmir-opt-level=5 -Zunsound-mir-opts -Zdump-mir=all --emit=mir -Zprint-mono-items=eager -Zpolymorphize=on -Zalways-encode-mir -Zdrop-tracking -Zdrop-tracking-mir=yes -Zverbose -Zextra-const-ub-checks --edition=2018 -Ztranslate-lang=en_US -Zprint-type-sizes -Zmaximal-hir-to-mir-coverage -Zstrict-init-checks=yes '-Zcrate-attr=feature(return_type_notation)' '-Zcrate-attr=feature(async_fn_in_trait)' '-Zcrate-attr=feature(impl_trait_in_assoc_type)' '-Zcrate-attr=feature(transmute_generic_consts)' '-Zcrate-attr=feature(fn_ptr_trait)'

This causes rustc to be killed by prlimit because it would hit the memory limit, sometimes its backtrace is much shorter than usual

thread 'rustc' panicked at 'memory allocation of 671088640 bytes failed', library/alloc/src/alloc.rs:412:17
stack backtrace:
   0:     0x7fb16cf68f33 - std::backtrace_rs::backtrace::libunwind::trace::h93382db32298e592
                               at /rustc/64bcb326516ef7490db46de88b87a4c0990097fe/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x7fb16cf68f33 - std::backtrace_rs::backtrace::trace_unsynchronized::h077b1367e10a1417
                               at /rustc/64bcb326516ef7490db46de88b87a4c0990097fe/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x7fb16cf68f33 - std::sys_common::backtrace::_print_fmt::h68f17e98ca35b7bf
                               at /rustc/64bcb326516ef7490db46de88b87a4c0990097fe/library/std/src/sys_common/backtrace.rs:65:5
   3:     0x7fb16cf68f33 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h341120b670d06b15
                               at /rustc/64bcb326516ef7490db46de88b87a4c0990097fe/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x7fb16cfc9d4f - core::fmt::write::ha614952dcf5c10f0
                               at /rustc/64bcb326516ef7490db46de88b87a4c0990097fe/library/core/src/fmt/mod.rs:1247:17
   5:     0x7fb16cf5bf61 - std::io::Write::write_fmt::h10934ba4b215c50c
                               at /rustc/64bcb326516ef7490db46de88b87a4c0990097fe/library/std/src/io/mod.rs:1712:15
   6:     0x7fb16cf68d45 - std::sys_common::backtrace::_print::heb0370ce5b64b518
                               at /rustc/64bcb326516ef7490db46de88b87a4c0990097fe/library/std/src/sys_common/backtrace.rs:47:5
   7:     0x7fb16cf68d45 - std::sys_common::backtrace::print::h98331c1359b10e7b
                               at /rustc/64bcb326516ef7490db46de88b87a4c0990097fe/library/std/src/sys_common/backtrace.rs:34:9
   8:     0x7fb16cf6b84f - std::panicking::default_hook::{{closure}}::h5b08409ff035083a
   9:     0x7fb16cf6b507 - std::panicking::default_hook::hfc3c525b7b9bf324
                               at /rustc/64bcb326516ef7490db46de88b87a4c0990097fe/library/std/src/panicking.rs:293:9
thread 'rustc' panicked at 'memory allocation of 6291456 bytes failed', library/alloc/src/alloc.rs:412:17

and my console would not be freed, so there is still something running.
You may need to run this a couple of times to hit the hang.

Activity

  1. matthiaskrgr
    added
    regression-from-stable-to-nightlyPerformance or correctness regression from stable to nightly.
    C-bugCategory: This is a bug.
    regression-untriagedUntriaged performance or correctness regression.
    on Apr 24, 2023
  2. rustbot
    added
    I-prioritizeIssue needs a team member to assess the impact. Will be replaced by P-{low,medium,high,critical}
    on Apr 24, 2023

  3. matthiaskrgr commented on Apr 24, 2023

    @matthiaskrgr
    matthiaskrgr
    on Apr 24, 2023
    MemberAuthor

    cc @Amanieu

  4. the8472 commented on Apr 24, 2023

    @the8472
    the8472
    on Apr 24, 2023
    Member

    Maybe related, maybe not. I just ran into a hang in backtraces when working on something else:

    (gdb) bt
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007fafc1163de5 in std::sys::unix::futex::futex_wait () at library/std/src/sys/unix/futex.rs:62
#2  0x00007fafc115c3a0 in std::sys::unix::locks::futex_mutex::Mutex::lock_contended () at library/std/src/sys/unix/locks/futex_mutex.rs:56
#3  0x00007fafc1161986 in std::sys::unix::locks::futex_mutex::Mutex::lock () at library/std/src/sys/unix/locks/futex_mutex.rs:28
#4  std::sync::mutex::Mutex::lock<()> () at library/std/src/sync/mutex.rs:273
#5  0x00007fafc1174d65 in std::sys_common::backtrace::lock () at library/std/src/sys_common/backtrace.rs:17
#6  std::sys_common::backtrace::print () at library/std/src/sys_common/backtrace.rs:33
#7  0x00007fafc114a8f6 in std::panicking::default_hook::{closure#1} () at library/std/src/panicking.rs:274
#8  0x00007fafc114a52f in std::panicking::default_hook () at library/std/src/panicking.rs:293
#9  0x00007fafc4cbe7ff in rustc_driver_impl::DEFAULT_HOOK::{closure#0}::{closure#0} () at compiler/rustc_driver_impl/src/lib.rs:1208
#10 core::ops::function::FnOnce::call_once<rustc_driver_impl::DEFAULT_HOOK::{closure#0}::{closure_env#0}, (&core::panic::panic_info::PanicInfo)> () at library/core/src/ops/function.rs:250
#11 core::ops::function::FnOnce::call_once<rustc_driver_impl::DEFAULT_HOOK::{closure#0}::{closure_env#0}, (&core::panic::panic_info::PanicInfo)> () at library/core/src/ops/function.rs:250
#12 0x00007fafc114b0d3 in std::panicking::rust_panic_with_hook () at library/std/src/panicking.rs:704
#13 0x00007fafc117558f in std::panicking::begin_panic_handler::{closure#0} () at library/std/src/panicking.rs:586
#14 0x00007fafc1175086 in std::sys_common::backtrace::__rust_end_short_backtrace<std::panicking::begin_panic_handler::{closure_env#0}, !> () at library/std/src/sys_common/backtrace.rs:150
#15 0x00007fafc114aa92 in std::panicking::begin_panic_handler () at library/std/src/panicking.rs:584
#16 0x00007fafc11c3843 in core::panicking::panic_fmt () at library/core/src/panicking.rs:67
#17 0x00007fafc11c38dd in core::panicking::panic () at library/core/src/panicking.rs:117
#18 0x00007fafc1152dd9 in alloc::vec::Vec::extend_desugared<std::backtrace_rs::symbolize::gimli::elf::ParsedSym, alloc::alloc::Global, core::iter::adapters::map::Map<core::iter::adapters::filter::Filter<core::iter::adapters::filter::Filter<core::slice::iter::Iter<object::elf::Sym64<object::endian::LittleEndian>>, std::backtrace_rs::symbolize::gimli::elf::{impl#1}::parse::{closure_env#0}>, std::backtrace_rs::symbolize::gimli::elf::{impl#1}::parse::{closure_env#1}>, std::backtrace_rs::symbolize::gimli::elf::{impl#1}::parse::{closure_env#2}>> () at library/alloc/src/vec/mod.rs:2827
#19 alloc::vec::spec_extend::{impl#0}::spec_extend<std::backtrace_rs::symbolize::gimli::elf::ParsedSym, core::iter::adapters::map::Map<core::iter::adapters::filter::Filter<core::iter::adapters::filter::Filter<core::slice::iter::Iter<object::elf::Sym64<object::endian::LittleEndian>>, std::backtrace_rs::symbolize::gimli::elf::{impl#1}::parse::{closure_env#0}>, std::backtrace_rs::symbolize::gimli::elf::{impl#1}::parse::{closure_env#1}>, std::backtrace_rs::symbolize::gimli::elf::{impl#1}::parse::{closure_env#2}>, alloc::alloc::Global> () at library/alloc/src/vec/spec_extend.rs:17

    In my case this was due to a logic bug I introduced, but the same place could panic due to allocation failure.

  5. Amanieu commented on Apr 25, 2023

    @Amanieu
    Amanieu
    on Apr 25, 2023
    Member

    I can't reproduce this on my machine.

    Looking at the backtrace from @the8472, this seems to be happening because a second panic occurred while executing a panic hook, which led to a deadlock on an internal lock used by the backtrace mechanism. I think the proper solution here is to skip executing the panic hook and immediately abort if a panic occurred while executing a panic hook.

  6. matthiaskrgr
    added a commit that references this issue on Apr 25, 2023

    Rollup merge of rust-lang#110782 - matthiaskrgr:revert_panic_oom, r=A…

    2d72abc

  7. apiraino commented on Apr 25, 2023

    @apiraino
    apiraino
    on Apr 25, 2023
    Contributor

    WG-prioritization assigning priority (Zulip discussion).

    @rustbot label -I-prioritize +P-medium +T-compiler

  8. rustbot
    added
    P-mediumMedium priority
    T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.
    and removed
    I-prioritizeIssue needs a team member to assess the impact. Will be replaced by P-{low,medium,high,critical}
    on Apr 25, 2023
  9. matthiaskrgr
    added a commit that references this issue on May 15, 2023

    Rollup merge of rust-lang#110975 - Amanieu:panic_count, r=joshtriplett

    23f7f7e
  10. Dylan-DPC
    added a commit that references this issue on May 16, 2023

    Rollup merge of rust-lang#110975 - Amanieu:panic_count, r=joshtriplett

    d990e43
  11. Noratrieb
    added a commit that references this issue on May 16, 2023

    Rollup merge of rust-lang#110975 - Amanieu:panic_count, r=joshtriplett

    4197be9
  12. bors
    added a commit that references this issue on May 27, 2023

    Auto merge of rust-lang#110975 - Amanieu:panic_count, r=joshtriplett

    f91b634
  13. bors
    closed this as completedin f91b634on May 27, 2023
  14. thomcc
    added a commit that references this issue on Jul 18, 2023

    Rollup merge of #110782 - matthiaskrgr:revert_panic_oom, r=Amanieu

    4cd1211
  15. RalfJung
    added a commit that references this issue on Apr 20, 2024

    Rollup merge of #110782 - matthiaskrgr:revert_panic_oom, r=Amanieu

    e9e3ea1
  16. RalfJung
    added a commit that references this issue on Apr 27, 2024

    Rollup merge of #110782 - matthiaskrgr:revert_panic_oom, r=Amanieu

    af9f00d
  17. christian-schilling
    added 2 commits that reference this issue on Jan 27, 2026

    Rollup merge of #110782 - matthiaskrgr:revert_panic_oom, r=Amanieu

    26d45c1

    Rollup merge of #110782 - matthiaskrgr:revert_panic_oom, r=Amanieu

    28faba0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-bugCategory: This is a bug.P-mediumMedium priorityT-compilerRelevant to the compiler team, which will review and decide on the PR/issue.regression-from-stable-to-nightlyPerformance or correctness regression from stable to nightly.regression-untriagedUntriaged performance or correctness regression.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions