Repository navigation
Mimalloc differences from upstream #113141
Description
This issue tracks divergence in our copy of mimalloc from upstream http://github.com/microsoft/mimalloc. The purpose is to help with upstreaming fixes and also so that our modifications are not lost as we pull in new mimalloc versions.
base version: v2.1.2
Simple bug fixes
- gh-90815: Fix mimalloc build on WASI #111524 (fixed upstream)
- gh-111544: Fix mimalloc build on AIX #111593
- gh-111906: Remove
unused functionwarnings during mimalloc build on FREEBSD #111907 - gh-90815: Fix mimalloc atomic.h on Windows arm64 #111527
- gh-112027: Don't print mimalloc warning after mmap() call #113372
- gh-117547: Fix mimalloc compile error on OpenBSD #117548
- gh-117755: Fix mimalloc for huge allocation on s390x #117809
- gh-116984: Make mimalloc header includes relative to the current file #118808
- gh-123022: Fix crash with
Py_Initializein background thread #123052 (May be specific to CPython's usage) - gh-123826: Fix Unused Static Warnings in Objects/mimalloc/prim/unix/prim.c #123827 (fixed upstream)
Mimalloc changes to support GC
- gh-112532: Use separate mimalloc heaps for GC objects #113263
- gh-112532: Isolate abandoned segments by interpreter #113717
- gh-112532: Tag mimalloc heaps and pages #113742
- gh-112532: Improve mimalloc page visiting #114133
- gh-112529: Use _PyThread_Id() in mimalloc (free-threaded build) #115488
Activity
corona10 commentedon Dec 18, 2023
Hmm, how about generating patchfiles and storing them somewhere in the repository for records? :)
I also need feedback about the solution from @vstinner since people in Red Hat is the best expert for this area.
corona10 commentedon Dec 18, 2023
And which commit is our mimalloc baseline code?
I can spend time for implementing a tool that generating diff files and comparing them on the CI level if it looks okay :)
corona10 commentedon Dec 18, 2023
cc @DinoV
vstinner commentedon Dec 18, 2023
That sounds painful to maintain. I would prefer to have a process to upstream changes first (mimalloc), and then apply them downstream (Python).
Here we wanted to move quickly to add mimalloc and fix all portability issues. But we should be more "upstream first" for following changes.
corona10 commentedon Dec 18, 2023
Ah okay if they accept our changes, it looks reasonable.
colesbury commentedon Dec 18, 2023
I created this issue in anticipation of further mimalloc changes. There will be changes to support GC in the
--disable-gilbuilds and a few for the optimistic dict accesses. We are going to need to continue to move quickly. An upstream first model is not going to work yet.I don't think it will be too hard to pull in new mimalloc versions. I've done it a few times in the nogil forks: diff vs. base to get patch file, patch new version, fix merge conflicts. We can write up the workflow, but I think automation is overkill at this point.
I had a nice chat with Daan (mimalloc author) on Friday and we got a chance to discuss some (but not all) of the mimalloc changes to support GC. I think we'll be able to get changes merged upstream, but for non-trivial changes I think it's helpful to understand and be able to explain how APIs are used in CPython. Even if time were not an issue, I would not want to be in a situation where we propose something upstream and then it turns out we needed something slightly different.
ericsnowcurrently commentedon Jan 4, 2024
CC @daanx
Link *Mimalloc differences from upstream* python#113141
make mimalloc includes relative to the current file, upstream of pyth…
daanx commentedon May 19, 2024
I have started upstreaming mimalloc specific modifications into mimalloc to make it easier to downstream future releases of mimalloc. The ideal goal would be to enable Python to pull in mimalloc with no further modifications -- let's see how far we can get.
For now, I am documenting here which commits fix particular issues together with the version of mimalloc that will have these commits. (last update: 2024-06-03)
Planned for v2.1.8
Mimalloc changes to support GC
gh-112532: Isolate abandoned segments by interpreter #113717 (v2.1.8, various upstream commits, some are: microsoft/mimalloc@d9aa19a (sub-processes), microsoft/mimalloc@8f87455 (visit arena abandoned per sub process), microsoft/mimalloc@855e3b2 (all abandoned block visiting), and a concurrency "leak" fix microsoft/mimalloc@96b69d7)
This now works with the new bitmap based abandoned segments which scales better.
gh-112532: Improve mimalloc page visiting #114133 (v2.1.8, upstream commit microsoft/mimalloc@f7fe5bf)
(note: would like to check correctness of
fast_divideand the range possible ofbsize)Mimalloc changes for lock-free reads (using QSBR)
upstreamed in v2.1.7:
Simple bug fixes
unused functionwarnings during mimalloc build on FREEBSD #111907 (upstream commit microsoft/mimalloc@87c4012, v2.1.7)Mimalloc changes to support GC
gh-112532: Use separate mimalloc heaps for GC objects #113263 (upstream commit microsoft/mimalloc@710d613, v2.1.7) note: renamed
_mi_heap_init_exto_mi_heap_init(and the original heap init tomi_thread_heap_init).gh-112532: Tag mimalloc heaps and pages #113742 (upstream commit microsoft/mimalloc@0c4041f, v2.1.7) note: added an error condition if page with a certain custom tag cannot be reclaimed in a heap with the same tag; is EINVAL strong enough? maybe make it EFAULT to abort in such case?. It should not happen though if used properly.
(edit: I made it EFAULT to catch any bugs early)
[Make the garbage collector thread-safe in
--disable-gilbuilds #112529: Use _PyThread_Id() in mimalloc (free-threaded build) gh-112529: Use _PyThread_Id() in mimalloc (free-threaded build) #115488](gh-112529: Use _PyThread_Id() in mimalloc (free-threaded build) #115488, v2.1.7) (upstream commit microsoft/mimalloc@66052f1)cdce8p commentedon Jul 8, 2024
For reference, just opened #121487 to address a deprecation warning with
ATOMIC_VAR_INITwhich was already fixed upstream inv1.8.4.Squashed 'depends/mimalloc/' changes from 91ba1f374..7a2a41134
Squashed 'depends/mimalloc/' changes from 91ba1f374..b0958d81f
Squashed 'depends/mimalloc/' changes from 91ba1f374..dfa50c37d
Squashed 'depends/mimalloc/' changes from 91ba1f374..dfa50c37d
Remove unused function warnings during mimalloc build on CYGWIN
pythongh-148461: use arc4random_buf() on CYGWIN
gh-148461: Use arc4random_buf() in mimalloc on Cygwin (#148462)
StanFromIreland commentedon Jun 25, 2026
We have four CodeQL warnings for bundled mimalloc:
I wrote a PR upstream (microsoft/mimalloc#1321) to fix two and note the other two.
prevent syscalls on Android, upstream of python/cpython#158530 (and p…
UWP compilation, upstream of python/cpython#152477 (and python/cpytho…