Repository navigation
from_pretrained with safetensors mmap crashes on Windows for large multi-shard checkpoints (copy-on-write commit charge exhausts memory) #48285
Description
System Info
transformersversion: 5.4.0 (also reproduced onmain/ 5.16.0.dev0)- Platform: Windows 11 Pro, build 10.0.26200
- Python version: 3.10.11
- PyTorch version: 2.14.0a0 (XPU)
- safetensors version: 0.8.0
Who can help?
@Cyrilvallez (model loading / from_pretrained)
@SunMarc (quantization / Big Model Inference)
Information
- The official example scripts
- My own modified scripts
Tasks
- An officially supported task in the
examplesfolder (such as GLUE/SQuAD, ...) - My own task or dataset (give details below)
Reproduction
Summary
Loading deepseek-ai/DeepSeek-R1-Distill-Qwen-14B (27.51 GB across 4 safetensors shards) via from_pretrained crashes silently on a 32 GB Windows machine with exit code 0xC0000005 (access violation). The model needs only ~10 GB once quantized to int4.
The root cause is that from_pretrained opens all safetensors shards simultaneously via safe_open(..., backend="mmap") and keeps them open for the entire load (modeling_utils.py, lines 4459–4473):
for file in checkpoint_files:
if load_config.disable_mmap or _is_on_hf_mount(file):
with open(file, "rb") as _fh:
merged_state_dict.update(_safe_load_bytes(_fh.read()))
continue
is_mps = load_config.device_map is not None and any(
(d.type if isinstance(d, torch.device) else d) == "mps"
for d in load_config.device_map.values()
)
backend, device = ("pread", "mps") if is_mps else ("mmap", "cpu")
file_pointer = safe_open(file, framework="pt", device=device, backend=backend)
all_pointer.add(file_pointer)
for k in file_pointer.keys():
merged_state_dict[k] = file_pointer.get_slice(k) # don't materialize yetOn Linux, copy-on-write mmap costs essentially nothing — the kernel overcommits and pages are only allocated on write. Since the loader only reads, Linux pays zero.
On Windows, every copy-on-write mapping is charged against the system commit limit (RAM + pagefile) at map time, for every page that could conceivably be written. The 27.5 GB checkpoint = ~7.2 million pages, all reserved upfront. This is pure overhead since the mappings are only ever read, but Windows enforces it regardless.
Why it crashes
With an integrated GPU (which shares system RAM), the GPU's ~9.1 GB reservation and the 28.26 GB checkpoint mapping reservation compete for the same 31.5 GB pool. Free RAM drops from 26.6 GB to 0.23 GB, the next allocation fails, and the process dies silently — no Python exception, no traceback.
Key observations
- The 24 GB private commit jump happens before the progress bar reaches 1% — it's the mappings being established, not weights being read.
privatecommit then stays flat while free RAM drains to nothing — the reservation was made upfront.- transformers v4 loaded shards one at a time (peak = largest shard: 8.12 GB). v5 opens all shards at once (peak = sum of all shards: 27.51 GB).
- MPS already uses
preadbackend (line 4468), but Windows still getsmmap.
Note on disable_mmap
disable_mmap=True exists but reads the entire file into memory via open(file, "rb").read(), which is even worse for large checkpoints. The pread backend in safetensors would be the correct middle ground — it reads only the requested slices without the commit charge overhead.
Reproducer
import threading
import time
import psutil
import torch
from torchao.quantization import Int4WeightOnlyConfig
from transformers import AutoModelForCausalLM, TorchAoConfig
MODEL = "deepseek-ai/DeepSeek-R1-Distill-Qwen-14B" # 27.5 GB on disk, 4 shards
proc, stop = psutil.Process(), threading.Event()
def sample():
peak, last = 0.0, 0.0
while not stop.wait(0.1):
priv = proc.memory_info().private / 1024**3
peak = max(peak, priv)
now = time.perf_counter()
if now - last >= 1.0:
last = now
print(
f" private={priv:6.2f} GB peak={peak:6.2f} "
f"free RAM={psutil.virtual_memory().available / 1024**3:6.2f} GB",
flush=True,
)
threading.Thread(target=sample, daemon=True).start()
torch.xpu.init()
AutoModelForCausalLM.from_pretrained(
MODEL,
dtype=torch.float16,
device_map="xpu",
quantization_config=TorchAoConfig(
Int4WeightOnlyConfig(group_size=128, int4_packing_format="plain_int32")
),
)
stop.set()
print("loaded without crashing")Output on transformers 5.4.0 (crashes):
private= 1.81 GB peak= 1.81 free RAM= 25.80 GB # interpreter + torch
private= 10.93 GB peak= 10.93 free RAM= 19.76 GB # torch.xpu.init()
private= 35.24 GB peak= 35.24 free RAM= 16.67 GB # 4 shards mapped
Loading weights: 0% # nothing read yet
private= 31.09 GB peak= 35.24 free RAM= 5.27 GB # 32%
private= 31.11 GB peak= 35.24 free RAM= 0.64 GB # 53%
private= 31.09 GB peak= 35.24 free RAM= 0.28 GB # 55%
Loading weights: 58%
EXITCODE=-1073741819 # 0xC0000005
Output on transformers 4.57.6 (succeeds):
Load completes successfully, peaking at 18.74 GB.
Expected behavior
from_pretrained should successfully load multi-shard safetensors checkpoints on Windows without exhausting system memory through mapping reservations.
Suggested fix
Use the pread backend on Windows, similar to how MPS is already handled. A minimal change at line 4468:
is_mps = load_config.device_map is not None and any(
(d.type if isinstance(d, torch.device) else d) == "mps"
for d in load_config.device_map.values()
)
is_windows = sys.platform == "win32"
if is_mps:
backend, device = "pread", "mps"
elif is_windows:
backend, device = "pread", "cpu"
else:
backend, device = "mmap", "cpu"Alternatively, expose a backend parameter in from_pretrained so users can choose explicitly.
Activity
SunMarc commentedon Aug 25, 2026
Thanks for the report @eryk-roch ! Can you open a PR with the fix so that we can test ? cc @McPatate @matthewdouglas for safetensors / windows.
eryk-roch commentedon Aug 25, 2026
Analysis: mmap commit charge on Windows
We hit this on our side as well while testing XPU workloads on Windows with 32 GB RAM. Did some digging into why it works on Linux but not Windows.
The key difference is how the two OSes handle copy-on-write file mappings. Linux overcommits. Pages are only charged when actually written to, and since
_load_pretrained_modelonly reads, the cost is effectively zero. Windows charges the full mapping againstRAM + pagefileatCreateFileMappingtime regardless of access pattern. With 4 shards totaling ~27.5 GB mapped simultaneously, the commit reservation alone eats all available memory before a single tensor is read.The current code at (
modeling_utils.py, lines 4492):has no Windows-aware path. MPS already uses
preadfor similar reasons (mmap is problematic there too).What would fix it
Option A:
preadon Windows (quick fix)Same pattern as the existing MPS path.
preadbackend in safetensors already works on Windows (usesseek_readunder the hood). We expecting slower loading compared to mmap on NVMe, but in practice the GPU transfer dominates total time so it barely shows up end-to-end.Downside is we lose mmap benefits (readahead, page cache) on Windows entirely. For most real-world use cases this doesn't matter much since you're loading once and running inference, but it's worth noting.
Option B: per-shard mmap with sequential close
Instead of opening all shards at once, process one shard at a time: open → materialize its tensors into the model → close → next shard. Peak commit = largest shard (~8 GB) instead of sum of all shards (~27.5 GB).
This keeps mmap performance but needs changes to the
_load_pretrained_modelflow. Right now it builds a flatmerged_state_dictfrom all shards before passing everything toconvert_and_load_state_dict_in_model. The sequential approach would need the loading loop to be aware of shard boundaries. Also need to think about what happens with the thread pool (4 workers). Currently it can pull tensors from any shard in parallel, with per-shard loading you'd lose cross-shard parallelism.Sum up
Option A is a straightforward fix and stops the crash. I have got a patch ready if you want a PR. Option B is cleaner long term and would actually benefit all platforms (lower peak memory during loading in general), but it's more invasive to implement.
I am happy to contribute either. What's the preferred direction?
@SunMarc @McPatate @matthewdouglas
matthewdouglas commentedon Aug 25, 2026
I was able to reproduce this on a Windows machine with a 24GB NVIDIA GPU, 16GB system RAM, using the same model. I used Python 3.10, PyTorch 2.14 RC, transformers/safetensors/accelerate all latest stable.
In order to reproduce this I needed to set the pagefile limit down lower than the default. If I go too low, I get an earlier exit with an
OSError. Out of curiosity, @eryk-roch, in your testing, what was your pagefile limit? If it was set at Windows default limits, I have to assume you had a disk size maybe 256GB or smaller?Regardless, I think the change proposed in Option A is reasonable as it's not very invasive and does improve the situation overall. So I would suggest starting from there. It's still going to require more committed memory than Linux, but less so than it does today and maybe one further workaround on that is to use smaller shards.
On your particular hardware, since it is an iGPU, I'm not sure I understand the argument that GPU transfer dominates anything. Even so I wouldn't be so sure it's going to be slower than mmap anyway.
Option B is something I'm not particularly convinced on at this point.
eryk-roch commentedon Aug 26, 2026
@matthewdouglas Thanks for confirming and for the go-ahead on Option A.
Fair point on the iGPU. There’s no real PCIe transfer there, it’s all shared memory, so pread overhead matters more than I implied. It still seemed fine in practice, but I shouldn’t have handwaved it like that.
Regarding the pagefile, I was using the default Windows settings on a 512 GB NVMe system disk, so the managed pagefile was probably around 4–5 GB. I didn’t configure it manually. The numbers just don’t work
eryk-roch commentedon Aug 26, 2026
Created PR - #48341