Massive compiler perf regression

First reported on Reddit here. See that post for more details and examples.

This crate has gone from a build time of 17 seconds with the compiler consuming 168MB of RAM at peak to 320 secs and 1670MB. Apparently this regression occurred somewhere between jan 15-18, and is in the current beta. The author hypothesizes that it may have to do with complex usage of traits.

Quoting a comment from Reddit (see here) so it doesn't get lost:

Paging /u/arielby and /u/nikomatsakis, they always know what's going on in the type checker

EDIT: Does the Jan 18 nightly contain the obligation forest (#30533), by any chance?

@nikomatsakis

cc @nikomatsakis @arielb1

+1 on this (for reference output of cargo rustc -- -Z time-passes). When talking about usage of complex traits (example, once again, here), it was always been slow, but on an acceptible rate. The author of library even goes about that in a README.md:

Be aware, due to an ongoing issue with rustc, the compile time of your code will exponentially increase with the complexity of your parsers. In practice I've found things get bad after about 10 combinations or so. You can get around this by boxing a parser...

The issue he mentioned is #22204. This (current) may be considered duplicate of that, though, again, the problem was not of the current magnitude.

The author considers boxing to help:

This "flattens" the type signature of the parser into a trait object, which will improve compile-time at the cost of runtime performance due to dynamic dispath.

But, once again, it doesn't help much now.

Ah, as well, this may be considered as a regression from stable to beta:
Code compiles rather quickly on stable 1.6, but slowly on beta 1.7. I just went further to find exact nightlies, which differ in compilation times.

Note that the obligation forest PR was merged on January 16, so it did get merged in the window in which the regression happened.

Seems pretty likely to have to do with the obligation forest and also with the change in caching behavior that accompanied it. I can investigate some.

Is the syntax_syntex issue related? It was always bad but now it's definitely worse.

I can certainly confirm that this is traits-related; at least, the crate in question spends about 85% of its compilation time in middle::traits. I'm looking more in depth now.

@petevine

@petevine unlikely to be related to that, but I don't know

Seems pretty clear that the caching of sized results is no longer kicking in like it once did. This probably affects the script crate from servo too.

e.g., we repeatedly prove that T: Sized for the following types T. Here, the first number is the number of times we have proved it in my current logs. Note that compilation did not yet complete. :)

  79297 <*const alloc::rc::RcBox<fn() -> Box<peruse::parsers::Parser<I=[grammar_lexer::Token], O=grammar::Expr>> {parser::program::expression}>
  81467 <core::marker::PhantomData<alloc::rc::RcBox<fn() -> Box<peruse::parsers::Parser<I=[grammar_lexer::Token], O=grammar::Expr>> {parser::program::expression}>>
  81467 <core::nonzero::NonZero<*const alloc::rc::RcBox<fn() -> Box<peruse::parsers::Parser<I=[grammar_lexer::Token], O=grammar::Expr>> {parser::program::expression}>>
  82807 <core::ptr::Shared<alloc::rc::RcBox<fn() -> Box<peruse::parsers::Parser<I=[grammar_lexer::Token], O=grammar::Expr>> {parser::program::expression}>>
  82984 <peruse::parsers::RecursiveParser<[grammar_lexer::Token], grammar::Expr, fn() -> Box<peruse::parsers::Parser<I=[grammar_lexer::Token], O=grammar::Expr>> {parser::program::expression}>
  82999 <alloc::rc::Rc<fn() -> Box<peruse::parsers::Parser<I=[grammar_lexer::Token], O=grammar::Expr>> {parser::program::expression}>
 132273 <fn() -> Box<peruse::parsers::Parser<I=[grammar_lexer::Token], O=grammar::Expr>> {parser::program::expression}

I'm willing to re-open the issue on the basis that there is still ground to get back, even if the worst part is solved.

triage: P-medium

However, I'm bumping priority to medium.

@nrc

Needs to be retested and a test case added to perf.rlo cc @nrc @Mark-Simulacrum.

Will add test case to perf.rlo later, but currently time -v yields:

$ /usr/bin/time -v cargo build
   Compiling regex-syntax v0.2.2
   Compiling libc v0.2.5
   Compiling memchr v0.1.7
   Compiling aho-corasick v0.4.0
   Compiling regex v0.1.48
   Compiling peruse v0.3.0 (https://github.com/DanSimon/peruse.git#dbfc0054)
   Compiling parser v0.1.0 (file:///home/mark/Edit/rust-compilation-time-ram-regression)
    Finished debug [unoptimized + debuginfo] target(s) in 9.63 secs
    Command being timed: "cargo build"
    User time (seconds): 10.44
    System time (seconds): 0.31
    Percent of CPU this job got: 108%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:09.91
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 150980
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 0
    Minor (reclaiming a frame) page faults: 122458
    Voluntary context switches: 101
    Involuntary context switches: 42
    Swaps: 0
    File system inputs: 0
    File system outputs: 66112
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0

So I think this issue can be considered fixed. Can someone clarify exactly what part of the compiler was being stressed? I can add this crate to perf.rlo no problem, but perhaps something better can be designed?

@Mark-Simulacrum

I've reported this originally, and now I experience the same values as @Mark-Simulacrum. I've tested it with several nightlies over the year, and it seems it was getting better and better (from original 30 secs to 20, 15, 12 and finally 10 as of today). Feels great. Thanks to the rust team for the effort put on compiling speed optimizations!

This exact issue with the example crate may be closed as solved, however the original drop in performance was caused by the introduction of Obligation Forest as explained above by @nikomatsakis.
The latest thing I know, it was reverted on beta in #31851. Hope someone can clarify it better and further, although I'm afraid the track can be lost.

Edit: Actually, the crate itself takes only 5 seconds to build, which is awesome.

Closing in favor of rust-lang-deprecated/rustc-perf-collector#2 which tracks adding a test for this.

Rollup merge of rust-lang#53025 - ljedrz:debug_asserts_limited, r=varkor

Navigation Menu

Repository navigation

Massive compiler perf regression #31157

15 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Massive compiler perf regression #31157

Description

Activity

dirk commented on Jan 24, 2016

aturon commented on Jan 24, 2016

White-Oak commented on Jan 24, 2016

White-Oak commented on Jan 24, 2016

gereeter commented on Jan 24, 2016

nikomatsakis commented on Jan 24, 2016

MagaTailor commented on Jan 24, 2016

nikomatsakis commented on Jan 25, 2016

nikomatsakis commented on Jan 25, 2016

nikomatsakis commented on Jan 25, 2016

nikomatsakis commented on Jan 25, 2016

15 remaining items

nikomatsakis commented on Feb 18, 2016

nikomatsakis commented on Feb 18, 2016

nikomatsakis commented on Feb 18, 2016

brson commented on Oct 20, 2016

Mark-Simulacrum commented on Oct 20, 2016

White-Oak commented on Oct 21, 2016

Mark-Simulacrum commented on Jun 20, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions