Repository navigation
7.3: PCI: vmd: Flush DMA writes before handling MSI on Meteor Lake (MTL016) - #14
Open
not-varram wants to merge 1 commit into
Open
not-varram wants to merge 1 commit into
not-varram wants to merge 1 commit into
Conversation
4 tasks done
Per the Intel Core Ultra Processor Specification Update
(ID 792254 of 9/1/2026), the Meteor Lake VMD (8086:7d0b) is affected by an
MSI erratum:
MTL016: MSI from VMD-Owned Device May Pass Memory Write
Problem When the storage subsystem is configured to operate in RAID
0 or 1 mode, a Message Signaled Interrupt (MSI) from an
Intel® Volume Management Device (Intel® VMD) owned device
may interrupt a core before a previous write from the device
is completed.
Workaround None identified. The VMD MSI interrupt-handler should
initially perform a dummy register read to the MSI initiator
device prior to any writes to ensure proper PCIe ordering.
The VMD can signal its MSI before the posted writes carrying a child
device's DMA data have landed in memory. vmd_irq() then demuxes to the
child handler while the child's completion queue is not yet coherent, so
the handler observes no completion and returns without consuming it. The
I/O is only recovered when the block layer timeout expires and polls the
queue:
nvme nvme0: I/O tag 253 (50fd) QID 1 timeout, completion polled
The practical effect is therefore not a lost I/O but a 30 second stall
of the entire storage stack, repeated under any sustained read load.
This was originally diagnosed and fixed by Kai-Heng Feng in September
2024 [1]. That patch used udelay(4); Keith Busch objected that the delay
is merely a side effect of the read, and that flushing the pending
device-to-host writes is what the erratum actually requires. Kai-Heng
agreed to respin with a dummy register read. The thread then stalled on
an open question from Manivannan Sadhasivam [2]: whether the read must
target the child device (the "MSI initiator" named by the erratum)
rather than the VMD, and whether the workaround belongs in the NVMe
driver instead. No revision followed, and the erratum has remained
unmitigated in mainline since.
This implements the flush read Keith asked for, and answers the open
question empirically: the read MUST complete at the initiating child
device. A read of the VMD's own config BAR was tried first and does not
help - it terminates at the VMD and never traverses the downstream
link, so it does not order against the child's posted writes (measured:
timeout rate unchanged). Reading the initiator's config space does
order correctly: per PCIe ordering rules the read completion cannot
pass the device's earlier posted writes, so returning from the read
guarantees the completion queue entry is visible.
The initiator's config address is captured per-IRQ at MSI allocation
time from the requesting device, so the hot path adds one config read
only on affected parts, and only for vectors owned by a child device.
The read is serialized with cfg_lock like all other VMD config access.
No NVMe driver change is needed.
Measured on a Dell Pro Max 14 MC14250 (Meteor Lake, VMD 8086:7d0b,
KIOXIA BG6 512GB, 7.0.0-30-generic). The drive is healthy (46C, 0 media
errors, 2% used) and ASPM is disabled on the link with all L1 substates
off, so neither ASPM nor APST is involved.
Dropping caches and reading 3000 shared libraries, measuring
/proc/pressure/io "full" (every task on the system blocked on I/O),
with nvme_core.io_timeout=5:
unpatched: round 1 wall 37.4s full I/O stall 32.6s timeouts 0
round 2 wall 30.6s full I/O stall 28.7s timeouts 1
round 3 wall 30.3s full I/O stall 27.4s timeouts 1
VMD-BAR read: round 1 wall 44.8s full I/O stall 38.6s timeouts 8
(insufficient) round 2 wall 234.8s full I/O stall 207.9s timeouts 37
round 3 wall 3.6s full I/O stall 0.8s timeouts 0
this patch: round 1 wall 3.6s full I/O stall 1.3s timeouts 0
round 2 wall 3.6s full I/O stall 1.3s timeouts 0
round 3 wall 3.6s full I/O stall 1.4s timeouts 0
Before any workaround, 208 seconds of total-system I/O stall
accumulated in the first 13 minutes of uptime, roughly 27% of wall
clock. Userspace experiences this as GUI applications taking 30-60+
seconds to start while throughput between stalls looks entirely normal
(1.9 GB/s QD1) - which is what makes the fault easy to misattribute to
ASPM or to the drive.
[1] https://lore.kernel.org/20240903025544.286223-1-kai.heng.feng@canonical.com/
[2] https://lore.kernel.org/20240913111142.4cgrmirofhhgrbqm@thinkpad/
Reported-by: Kai-Heng Feng <kai.heng.feng@canonical.com>
Suggested-by: Keith Busch <kbusch@kernel.org>
Assisted-by: LLM
Signed-off-by: Rickey Bartlett <subtexel@gmail.com>
Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@oss.qualcomm.com>
[bhelgaas: include erratum source and text, update URLs]
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Link: https://patch.msgid.link/20260825043220.9047-1-subtexel@gmail.com
(cherry picked from commit b437504)
Assisted-by: LLM
Signed-off-by: Varsh Satti <varsh@varram.me>
not-varram
force-pushed
the
vmd-mtl016-7.3
branch
from
October 12, 2026 19:28
a239ff8 to
4dfe051
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Clean cherry-pick of upstream commit
b43750456515("PCI: vmd: Flush DMA writes before handling MSI on Meteor Lake") from the PCI tree. Authored by Rickey Bartlett; authorship is preserved.Refs CachyOS/linux-cachyos#1076. This is the 7.3 counterpart to #13, which has the full description and before/after numbers.
Summary: Meteor Lake VMD (
8086:7d0b) is affected by Intel erratum MTL016. An MSI can arrive before the NVMe completion reaches memory, causingnvme nvme0: I/O tag ... timeout, completion polledand system-wide stalls of up to 30 s. Upstream fixes it with a dummy config read from the child device before demuxing the IRQ.Status
drivers/pci/controller/vmd.con7.3/fixesis blob241023e, which is exactly the pre-image of the upstream diff.Cc: stabletag, so 7.3.x stable won't pick it up on its own.