Skip to content

Do not store LLVM bitcode in RLIBs by default #66961

New issue
New issue
Closed
Closed
Do not store LLVM bitcode in RLIBs by default#66961
Labels
C-enhancementCategory: An issue proposing an enhancement or a PR with one.I-compiletimeIssue: Problems and improvements with respect to compile times.I-heavyIssue: Problems and improvements with respect to binary size of generated code.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.

Description

@michaelwoerister
michaelwoerister
opened on Dec 2, 2019

@nnethercote asked me to write up a more detailed plan for getting rid of LLVM bitcode in RLIBs, so here goes:

Current situation

  • We store LLVM bitcode in every RLIB and Rust dylib so that rustc can
    perform cross-crate LTO.
  • This has quite a bit of a cost in terms of compile times and file sizes (See here).
  • Cross-crate LTO is a niche use case and the current approach makes that costlier
    than necessary too (because then machine code is duplicated)
  • When doing xLTO the shortcomings of the current implementation become
    especially obvious because then we have two basically identical LLVM bitcode
    files for each module (modulo compression).
  • Embedding bitcode also causes the incremental compilation cache to be much
    bigger because we have to keep object files and bitcode files in the cache.

Proposed Solution

The proposed solution is to follow Clang's model: When compiling for cross-crate
LTO, no machine code is generated and the .o file is actually LLVM bitcode.
There are special "fat" object files that contain regular machine code and
additionally, in a special section, uncompressed LLVM bitcode. These fat objects
would mainly be used for the standard library.

The consequences of this approach are:

  • Code compiled by the user (i.e. everything except libstd) would only be either
    machine code or LLVM bitcode.
  • When compiling an RLIB rustc needs to know if it is intended to be used for
    LTO or not. Thus Cargo needs to invoke rustc differently.
  • We save quite a bit of space and time in the common case.
  • Because the "fat" object files are a standard LLVM thing, libstd can partake
    in LTO steps performed by the LLVM linker plugin.

Open questions

  • Do we need to keep things backwards compatible by defaulting to "fat" object files? If the new default is to not store LLVM bitcode in RLIBs, then non-cargo build systems would start to silently not do LTO.
    • My preference would be to make no-bitcode the new default, but make the compiler emit a warning if wants to do LTO but encounters rlibs without bitcode.
  • Should rustc be able to transparently handle bitcode-only RLIBs in the non-LTO case?
    • My preference would be yes. Possibly also add a warning that that is unexpected?
  • LLVM's WASM implementation doesn't seem to support "fat" object files. I don't know if this is a fundamental restriction. I guess WASM projects are actually more likely to use LTO than regular projects. If WASM doesn't support custom sections, then we could also keep the current setup of having LLVM bitcode in a separate file in the RLIB archive. Maybe @alexcrichton or @fitzgen know more about WASM object files?

Steps to get there

That actually depends on how we resolve the open questions above.

cc #66598

Activity

  1. jonas-schievink
    added
    C-enhancementCategory: An issue proposing an enhancement or a PR with one.
    I-compiletimeIssue: Problems and improvements with respect to compile times.
    I-heavyIssue: Problems and improvements with respect to binary size of generated code.
    T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.
    on Dec 2, 2019

  2. alexcrichton commented on Dec 2, 2019

    @alexcrichton
    alexcrichton
    on Dec 2, 2019
    Member

    Thanks for writing this up @michaelwoerister! This all sounds spot on to me, and my thoughts on the open questions would be:

    Do we need to keep things backwards compatible by defaulting to "fat" object files?

    I think we have a lot of liberties here to do whatever change we want. It's basically an implementation detail that compressed bitcode is in each rlib, so we basically need to ensure that use cases today continue to work tomorrow. Once we do that I think it's reasonably for us to both switch the defaults and to completely remove all support for compressed llvm bitcode.

    One unfortunate consequence of this, though, will likely be that we need to default to emitting "fat" object files. For example today you can do:

    $ rust foo.rs --crate-type lib
$ rustc bar.rs -C lto --extern foo=libfoo.rlib

    and that "just works". If we were to omit bitcode entirely it wouldn't work as expected (would either not actually do LTO or would generate an error). So I think that the backwards compatible way to implement this change would be:

    • By default, rustc generates "fat" object files (more on wasm later)
    • The current -C linker-plugin-lto flag triggers object files in rlibs to be bitcode. No machine code is generated.
    • A new -C llvm-bc-in-object=false flag would be added to rustc. This would disable embedding LLVM bitcode in object files, aka not creating "fat" object files but rather just machine code ones.
    • Cargo would switch to passing llvm-bc-in-object=false by default, unless it thinks a dependency is going to be used for LTO. It could even get more clever and skip things like proc-macro LTO.
      • If Cargo is performing LTO, this is actually an interesting question. In this case we do not want to perfom codegen in each crate (like we do today). Instead we want the behavior of linker-plugin-lto, but we're not actually performing linker-plugin-lto. I would need to investigate, but ideally rustc would literally pass linker-plugin-lto to all dependencies, meaning rlibs only have *.o files that are actually LLVM bitcode. If linker-plugin-lto on dependencies triggers other sorts of logic though we may need to add a dedicated -C object-is-llvm-bitcode flag (or something like that).
    • The compiler's LTO logic will need an update. If rustc itself is performing LTO it looks for *.o files isntead of *.bc.z files. This logic update would probably check crate metadata to see whether it's an llvm bitcode file or a fat object file.

    Given all that I think we can preserve everyone's use case today, while also switching all users to faster compilations by default. If you're not performing LTO, you never generate bitcode files. If you're performing LTO, you never generate machine code in intermediate rlibs.

    Should rustc be able to transparently handle bitcode-only RLIBs in the non-LTO case?

    I would personally weakly say "no" to be more strict, but I don't feel stringly either way. I feel like though there's not much reason to support this and it's easier to implement an error rather than codegen'ing more object files on the fly.

    If WASM doesn't support custom sections, then we could also keep the current setup of having LLVM bitcode in a separate file in the RLIB archive.

    I do think it's pretty critical to keep LTO working for wasm. This is actually somewhat unfortunate though. In the meantime I think we'll either have to (a) work with LLVM to implement this because wasm does have custom sections, or (b) we'll need to keep logic for including *.bc files in an rlib and a wasm-specific hook which says "go look for *.bc instead of looking at *.o" if it's a "fat" object file.

    For LTO-ready rlibs on wasm though the *.o files would actually be *.bc, it's only the "fat object file" rlibs, like libstd, which would have this extra *.bc file.

  3. RalfJung commented on Mar 25, 2020

    @RalfJung
    RalfJung
    on Mar 25, 2020
    Member

    @michaelwoerister is there a link missing here?

    This has quite a bit of a cost in terms of compile times and file sizes (See here).

  4. michaelwoerister commented on Mar 25, 2020

    @michaelwoerister
    michaelwoerister
    on Mar 25, 2020
    MemberAuthor

    @nnethercote's original PR (linked above) has some quite compelling numbers: #66598 (comment)

  5. Dylan-DPC-zz
    added 2 commits that reference this issue on Mar 26, 2020

    Rollup merge of rust-lang#70384 - nnethercote:refactor-object-file-ha…

    c916a78

    Rollup merge of rust-lang#70384 - nnethercote:refactor-object-file-ha…

    b15423e

  6. nnethercote commented on Apr 2, 2020

    @nnethercote
    nnethercote
    on Apr 2, 2020
    Contributor

    #70458 is for the rustc side of things. rust-lang/cargo#8066 is for the Cargo side of things.

  7. nnethercote commented on Apr 21, 2020

    @nnethercote
    nnethercote
    on Apr 21, 2020
    Contributor

    #70458 turned out to be over-ambitious, and has been closed. #71323 is the more restrained alternative.

  8. nnethercote commented on Apr 22, 2020

    @nnethercote
    nnethercote
    on Apr 22, 2020
    Contributor

    #71323 has landed.

  9. nnethercote
    closed this as completedon Apr 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-enhancementCategory: An issue proposing an enhancement or a PR with one.I-compiletimeIssue: Problems and improvements with respect to compile times.I-heavyIssue: Problems and improvements with respect to binary size of generated code.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions