Repository navigation
import performance: large codebases that use a lot of attrs classes start taking a long time to load #575
Description
I need to do some more rigorous analysis of this, so my level of confidence here is "strong hunch", but: using attrs extensively, across a large codebase, can contribute non-trivially to Python's scourge of slow load times.
Many things in Python are slow: calling functions, importing modules, and so on. But one of the slowest things is reading and parsing Python code. This is why Python compiles and caches .pyc files - it's a substantial optimization to startup time.
Using attrs, unfortunately, partially undoes this optimization, because many of the methods in attr._make:
- generate a bunch of source code
compile()it, theneval()the bytecode they just compiled.
Now, they do this for a good reason. Transparency and debug-ability are great, and more than once being able to "see through" the attrs validator stack while debugging has been legitimately useful. So I wouldn't want to propose a technique that makes a substantially different tradeoff. Not to mention that friendliness to PyPy's JIT is achieved via this mechanism, and definitely wouldn't want to trade that off. But could we possibly piggyback on Python's native caching technique here, and cache both the .pyc output and the source code, in the __pycache__ directory or thereabouts?
Activity
hynek commentedon Sep 19, 2019
I know I’m starting to sound like a broken record but this seems like something we/you can start experimenting once we have pluggable method makers. You could basically wrap an existing one and do you shenanigans?
glyph commentedon Sep 19, 2019
I'm not sure why I'd want pluggability here — I just want it to be always fast. Pluggability would help with alternate runtimes, I guess? Like brython or something?
hynek commentedon Sep 19, 2019
Pluggability allows for experimenting that may or may not lead to main line inclusion. I’m always big on allowing for incremental improvement and the whole shitshow around
import attrsand...well...pluggable method makers are a constant, very painful reminder of that.tejasvi commentedon Oct 11, 2021
I am working on an application sensitive to startup time and attrs is responsible for >60% slowdown (used by markdown-it-py). Since the processing is cpu bound (due to
compile() -> exec()) threaded imports don't help here.hynek commentedon Oct 12, 2021
I gave markdown-it-py a brief look and it appears that it defines a grand total of four attrs classes. I find it hard to believe that the creation of four classes causes this much overhead. I've also tried importing it and it looks instantaneous.
Are you sure we're not talking about runtime cost – aka you're parsing a Markdown file on startup or something?
A5rocks commentedon May 17, 2025
I was looking at this in relation to trio's import time and the number of attrs classes is a better predictor of self-time than the number of (non-comment, non-blank) lines! (I got the numbers via
python -X importtime -c "import trio, trio.testing"on Windows, and this is after applying python-trio/trio#3263)(unfortunately, I don't trust myself with a profiler enough to profile import time to double check it really is attrs! Maybe it's just that files with more classes have more evaluation at import time (because class scopes get evaluated then)?)
I hope this shows that attrs performance is somewhat relevant!
hynek commentedon May 27, 2025
We're aware, but there's only so much we can do without starting to write C code.
The recent 25.2.0 improved class-creation time by 30-50%, but I'm not sure if we can do that again. I think @Tinche had some ideas, but it'll probably not get as good as this.