Skip to content

Reduce memory usage by using global Arc-based interning - #8284

Merged
bors[bot] merged 5 commits into
rust-lang:masterfrom
jonas-schievink:global-intern
Apr 2, 2021
Merged

bors[bot] merged 5 commits into
rust-lang:masterfrom
jonas-schievink:global-intern

Conversation

@jonas-schievink

Copy link
Copy Markdown
Contributor

This saves around 50 mb when running analysis-stats on r-a itself. Not a lot, but this infra can be easily reused to intern more stuff.

Comment thread crates/hir_def/src/intern.rs Resolved
@jonas-schievink

Copy link
Copy Markdown
Contributor Author

Added interning of more types. Memory impact looks roughly like this:

in crates/rust-analyzer, run
$ cargo run --features jemalloc --release --bin rust-analyzer -- -v analysis-stats ../.. --parallel --memory-usage

base (master)

Total:               14.66s, 81ginstr, 1196mb
   182mb FileItemTreeQuery
    79mb BodyQuery


intern TypeRef+TraitRef

Total:               14.45s, 81ginstr, 1143mb
   144mb FileItemTreeQuery
    79mb BodyQuery


intern ModPath in Attr

Total:               14.72s, 81ginstr, 1117mb
   128mb FileItemTreeQuery
    79mb BodyQuery


also intern ModPath in item_tree::MacroCall

Total:               14.10s, 81ginstr, 1110mb
   119mb FileItemTreeQuery
    79mb BodyQuery


intern ModPath in Path

Total:               14.09s, 81ginstr, 1073mb
   105mb FileItemTreeQuery
    70mb TraitSolveQuery (purge)
    66mb ImplDatumQuery
    65mb BodyQuery

@jonas-schievink jonas-schievink changed the title Global TypeRef/TraitRef interning Reduce memory usage by using global Arc-based interning Apr 1, 2021
@flodiebold

Copy link
Copy Markdown
Member

Nice, we should try that for types as well!

@jonas-schievink

Copy link
Copy Markdown
Contributor Author

bors r+

bors Bot added a commit that referenced this pull request Apr 1, 2021
8284: Reduce memory usage by using global `Arc`-based interning r=jonas-schievink a=jonas-schievink

This saves around 50 mb when running `analysis-stats` on r-a itself. Not a lot, but this infra can be easily reused to intern more stuff.

Co-authored-by: Jonas Schievink <jonasschievink@gmail.com>
@bors

bors Bot commented Apr 1, 2021

Copy link
Copy Markdown
Contributor

Timed out.

@jonas-schievink

Copy link
Copy Markdown
Contributor Author

warning: spurious network error (10 tries remaining): [6] Couldn't resolve host name (Could not resolve host: crates.io)

bors retry

bors Bot added a commit that referenced this pull request Apr 1, 2021
8284: Reduce memory usage by using global `Arc`-based interning r=jonas-schievink a=jonas-schievink

This saves around 50 mb when running `analysis-stats` on r-a itself. Not a lot, but this infra can be easily reused to intern more stuff.

Co-authored-by: Jonas Schievink <jonasschievink@gmail.com>
@bors

bors Bot commented Apr 1, 2021

Copy link
Copy Markdown
Contributor

Build failed:

Comment thread crates/hir_def/src/intern.rs Resolved
Comment thread crates/hir_def/src/intern.rs Resolved
@matklad

matklad commented Apr 2, 2021

Copy link
Copy Markdown
Contributor

I wonder if we might want do something like this, but without interning, for basically everything?

I still have a picture in my head where all things of the same type are co-allocated in the same table, instead of being owned by parents.

@jonas-schievink

Copy link
Copy Markdown
Contributor Author

bors r+

@bors

bors Bot commented Apr 2, 2021

Copy link
Copy Markdown
Contributor

@bors
bors Bot merged commit f4d5698 into rust-lang:master Apr 2, 2021
@jonas-schievink
jonas-schievink deleted the global-intern branch April 2, 2021 16:51
@lnicola

lnicola commented Apr 2, 2021 •

Copy link
Copy Markdown
Member

changelog fix

@lnicola

lnicola commented Apr 2, 2021 •

Copy link
Copy Markdown
Member

This put quite a dent in the memory usage:

image

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants