Repository navigation
virtio-net: mergeable receive buffers are never offered, so a Linux guest posts 19 buffers per received packet #4035
Description
virtio-net advertises GUEST_TSO4/GUEST_TSO6 and GUEST_USO4/GUEST_USO6 but never
VIRTIO_NET_F_MRG_RXBUF. The feature name exists as an unused field in
NetworkFeaturesBank0 (vm/devices/virtio/virtio_net/src/lib.rs) and appears nowhere else
in the virtio tree.
A Linux guest that negotiates the guest-side segmentation features without mergeable receive
buffers falls back to its big-packets receive layout, which posts MAX_SKB_FRAGS + 2 buffers
for every received packet. I measured 19 on a 6.x guest by logging the payload length of each
receive chain.
That multiplies every per-descriptor cost by 19. It also costs one GuestMemory::subrange per
packet, which allocates an Arc, because a chain that long arrives as an indirect table.
For a sense of scale: removing an unrelated per-packet allocation and a redundant descriptor
read from the same path (#4034) recovered roughly 18 to 23 percent of the drained packet rate
on a saturated receive queue. The 19x amplification is the larger remaining factor there.
Offering the feature is not just setting the bit. The device would have to spread one received
packet across several buffers and report the count in num_buffers, and the BufferAccess
contract in net_backend pre-assigns exactly one buffer per RxId: the backend asks
capacity(id), then writes the whole frame into that buffer, with no way to acquire more part
way through the write. The spec expects the device to take buffers as it needs them while
filling.
One shape that would leave net_backend untouched: hold queued chains in a free list inside
virtio-net's own pool, have capacity() report the aggregate, and let the write consume as
many chains as the frame needs, setting num_buffers to the count. Three constraints come
with it. The device advertises VIRTIO_F_IN_ORDER, so chains would have to be consumed and
completed strictly in available order. fill_ready must not report more RxIds than it can
back. And push_guest_addresses, which the MANA path uses for zero copy, only knows its
segments once the chains are chosen.
I plan to implement this, roughly along the lines above. Raising it first in case you would
rather it took a different shape, or in case there is a reason the feature was left out that I
am missing.
Activity
jstarks commentedon Jul 26, 2026
I don't think we advertise GUEST_TSO yet, just HOST_TSO. Daman Mulye (@damanm24) has been working on this in #3413. I agree that once we do that, we need to advertise MRG_RXBUF as well; the PR is not currently doing that.
bitranox commentedon Jul 26, 2026
Correcting the issue text: you are right that main does not advertise GUEST_TSO.
traits()setsguest_csumonly, so a Linux guest negotiates neither GUEST_TSO nor GUEST_USO and stays in the small-packets receive layout. The 19-buffer chains I measured come from a local patch of ours advertising GUEST_TSO4/6 and GUEST_USO4/6. The issue as filed describes our tree, not main.That makes the pairing you described load bearing, and there is a trap in it worth flagging for #3413. I implemented MRG_RXBUF against the current
BufferAccessmodel to find out what it takes.Buffers cannot be reserved to an
RxIdwhen the slot is advertised. A small packet uses only the first buffer of its reservation, and the rest cannot be handed back: with the feature negotiated the guest folds the nextnum_buffers - 1used entries into the same packet, so returning an unwritten buffer makes it read a virtio-net header out of memory the device never wrote. Those leftovers stay consumed but unpublished, ahead of every laterRxIdin the in-order completion order, so nothing behind them publishes either. The guest's receive ring drains and never refills.It needs two slots in flight to show up, since a single one recycles its own leftovers, and it presents as a guest that cannot complete DHCP on virtio while netvsc and MANA on the same guest are fine. Six buffers posted and two small packets delivered returned exactly one buffer to the guest.
Taking buffers while copying, the way
virtio_net_receive_rcudoes, avoids it: a packet consumes exactly the buffers it fills, from the front of the queue, so they are consecutive in available order and every consumed buffer is published. One cost:push_guest_addressescannot report a buffer for a slot no packet has been written to yet, which makes mergeable incompatible with a backend that posts guest DMA addresses up front. The tap and consomme backends never call it.A second thing that matters for throughput: advertise a receive id only while the unbound capacity can still back every outstanding id with a worst-case frame. Advertising one id per buffer lets a coalesced frame meet an id holding far less, and the dropped frame costs TCP a retransmit. Measured on one guest over virtio, host to guest: without that rule TCP receive fell from 15523 to 6110 Mbit/s; with it TCP is at parity and the UDP burst that motivated the work goes from 2.40 to 34.9 Mbit/s, with the tap drop rate falling from 40.9 percent to about 1 percent.
I can put the branch up if that is useful.