Repository navigation
In bridging mode, forwarding broadcast fragmented packets under specific conditions causes a buffer double-free. #3736
Description
- Version VPP
vpp v26.10-rc0~324-gf9020c106
- Config VPP
2.1 OS version:
cat /etc/os-release
PRETTY_NAME="Ubuntu Questing Quokka (development branch)"
NAME="Ubuntu"
VERSION_ID="25.10"
VERSION="25.10 (Questing Quokka)"
VERSION_CODENAME=questing
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=questing
LOGO=ubuntu-logo
2.2 VPP startup
startup.conf
unix {
nodaemon
log /var/log/vpp.log
full-coredump
cli-listen /run/vpp/cli.sock
cli-no-pager
startup-config /home/zhangqi/C/vpp/bridge_init.conf
}
api-trace { on }
api-segment { gid root }
socksvr { default }
memory {
main-heap-size 1G
}
cpu {
main-core 0
corelist-workers 1
}
dpdk {
uio-driver vfio-pci
dev 0000:03:00.0
log-level debug
no-tx-checksum-offload
}
plugins {
plugin dpdk_plugin.so { enable }
}
2.3 VPP config and network flow
cat /home/zhangqi/C/vpp/bridge_init.conf
Create tap interface id=0 for communication between VPP and Linux kernel
create tap id 0
Add tap0 to bridge-domain 1
set interface l2 bridge tap0 1
Create loopback interface used as BVI gateway
create loopback interface
Bind loop0 as BVI interface of bridge-domain 1
set interface l2 bridge loop0 1 bvi
Bring all layer2 member interfaces up
create bond mode xor load-balance l2
bond add BondEthernet0 GigabitEthernet3/0/0
Bind bond to bridge-domain 1
set interface l2 bridge BondEthernet0 1
set interface state GigabitEthernet3/0/0 up
set interface state BondEthernet0 up
set interface state tap0 up
set interface state loop0 up
Assign gateway ipv4 address to BVI interface loop0
set interface ip address loop0 192.168.158.129/24
ip route add 255.255.255.255/32 via local
ip route add 224.0.0.0/8 via local
┌─────────────────────────────────────────────────┐
│ │
│ Linux Host │
│ │ │
│ v │
│ tap0 │
│ │ │
│ v │
│ ┌─────────────────────────────────────┐ │
│ │ VPP Bridge Domain │ │
│ │ │ │
│ │ ┌──────┐ ┌────────────┐ │ │
│ │ │ tap0 │ │ BVI │ │ │
│ │ │(member) │ (member) │ │ │
│ │ └───┬──┘ └─────┬──────┘ │ │
│ │ v /\ │ │
│ │ └──────┬──────┘ │ │
│ │ v │ │
│ │ ┌──────────┴──────────┐ │ │
│ │ │ GigabitEthernet3/0/0│ │ │
│ │ │ (member) │ │ │
│ │ └──────────┬──────────┘ │ │
│ └───────────────┼─────────────────────┘ │
│ v │
│ External Network │
└─────────────────────────────────────────────────┘
Bridge Members: tap0, GigabitEthernet3/0/0, BVI
Packet Path: tap0 (ingress) -> Bridge Flood/Learn -> GigabitEthernet3/0/0 (egress)
- VPP core dump stack
root@zhangqi-VMware-Virtual-Platform:/home/zhangqi/C/vpp# gdb -c /root/core-4844 ./build-root/install-vpp_debug-native/vpp/bin/vpp
GNU gdb (Ubuntu 16.3-1ubuntu2) 16.3
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
https://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./build-root/install-vpp_debug-native/vpp/bin/vpp...
warning: Can't open file /memfd:selog_ssvm (deleted) during file-backed mapping note processing
warning: Can't open file /memfd:buffers-numa-0 (deleted) during file-backed mapping note processing
warning: Can't open file /memfd:stat segment (deleted) during file-backed mapping note processing
warning: Can't open file anon_inode:[vfio-device] during file-backed mapping note processing
warning: Can't open file /memfd:seg_2-0 (deleted) during file-backed mapping note processing
[New LWP 4846]
[New LWP 4845]
[New LWP 4844]
This GDB supports auto-downloading debuginfo from the following URLs:
https://debuginfod.ubuntu.com
Enable debuginfod for this session? (y or [n]) n
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./build-root/install-vpp_debug-native/vpp/bin/vpp -c ./minimal_startup.conf'.
Program terminated with signal SIGABRT, Aborted.
#0 __GI_abort () at ./stdlib/abort.c:95
buffer double free core-dump stack:
warning: 95 ./stdlib/abort.c: No such file or directory
--Type for more, q to quit, c to continue without paging--
[Current thread is 1 (Thread 0x77dc2f3f56c0 (LWP 4846))]
(gdb) bt
#0 __GI_abort () at ./stdlib/abort.c:95
#1 0x000058ded51c931d in os_exit (code=1) at /home/zhangqi/C/vpp/src/vpp/vnet/main.c:471
#2 0x000077dc767857d4 in unix_signal_handler (signum=6, si=0x77dc2f3f3df0, uc=0x77dc2f3f3cc0)
at /home/zhangqi/C/vpp/src/vlib/unix/main.c:236
#3
#4 __pthread_kill_implementation (threadid=, signo=6, no_tid=0) at ./nptl/pthread_kill.c:44
#5 __pthread_kill_internal (threadid=, signo=6) at ./nptl/pthread_kill.c:89
#6 __GI___pthread_kill (threadid=, signo=signo@entry=6) at ./nptl/pthread_kill.c:100
#7 0x000077dc76245e2e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#8 0x000077dc76228888 in __GI_abort () at ./stdlib/abort.c:77
#9 0x000058ded51c92cb in os_panic () at /home/zhangqi/C/vpp/src/vpp/vnet/main.c:447
#10 0x000077dc766356a9 in debugger () at /home/zhangqi/C/vpp/src/vppinfra/error.c:52
#11 0x000077dc766354ae in _clib_error (how_to_die=2, function_name=0x77dc767c013d "vlib_buffer_validate_alloc_free",
line_number=305, fmt=0x77dc767c015d "%s %U buffer 0x%x") at /home/zhangqi/C/vpp/src/vppinfra/error.c:111
#12 0x000077dc766eea5a in vlib_buffer_validate_alloc_free (vm=0x77dc379e0c40, buffers=0x77dc2f3f4660, n_buffers=32,
expected_state=VLIB_BUFFER_KNOWN_ALLOCATED) at /home/zhangqi/C/vpp/src/vlib/buffer.c:304
#13 0x000077dc352a69d8 in vlib_buffer_pool_put (vm=0x77dc379e0c40, buffer_pool_index=0 '\000', buffers=0x77dc2f3f4660, n_buffers=32)
at /home/zhangqi/C/vpp/src/vlib/buffer_funcs.h:742
#14 0x000077dc352a6305 in dpdk_ops_vpp_enqueue_x86_64_v3 (mp=0xac02c7b00, obj_table=0xac02cb140, n=96)
at /home/zhangqi/C/vpp/src/plugins/dpdk/buffer.c:231
#15 0x000077dc3511fd5c in rte_pktmbuf_free ()
from /home/zhangqi/C/vpp/build-root/install-vpp_debug-native/vpp/lib/x86_64-linux-gnu/vpp_plugins/dpdk_plugin.so
#16 0x000077dc3511df7f in vmxnet3_xmit_pkts ()
from /home/zhangqi/C/vpp/build-root/install-vpp_debug-native/vpp/lib/x86_64-linux-gnu/vpp_plugins/dpdk_plugin.so
#17 0x000077dc352ac8d0 in rte_eth_tx_burst (port_id=0, queue_id=1, tx_pkts=0x77dc37a65c80, nb_pkts=30)
at /opt/vpp/external/x86_64/include/rte_ethdev.h:6738
#18 0x000077dc352aa024 in tx_burst_vector_internal (vm=0x77dc379e0c40, xd=0x77dc37a267c0, mb=0x77dc37a65c80, n_left=30, queue_id=1,
is_shared=0 '\000') at /home/zhangqi/C/vpp/src/plugins/dpdk/device/device.c:164
#19 0x000077dc352a9249 in dpdk_device_class_tx_fn_x86_64_v3 (vm=0x77dc379e0c40, node=0x77dc36477580, f=0x77dc36daac80)
--Type for more, q to quit, c to continue without paging--
at /home/zhangqi/C/vpp/src/plugins/dpdk/device/device.c:456
#20 0x000077dc7672920a in dispatch_node (vm=0x77dc379e0c40, node=0x77dc36477580, type=VLIB_NODE_TYPE_INTERNAL, frame=0x77dc36daac80,
dispatch_reason=VLIB_NODE_DISPATCH_REASON_PENDING_FRAME, last_time_stamp=1123958823067)
at /home/zhangqi/C/vpp/src/vlib/main.c:903
#21 0x000077dc76729eb4 in dispatch_pending_node (vm=0x77dc379e0c40, pending_frame_index=21, last_time_stamp=1123958823067)
at /home/zhangqi/C/vpp/src/vlib/main.c:1059
#22 0x000077dc76725115 in vlib_main_or_worker_loop (vm=0x77dc379e0c40, is_main=0) at /home/zhangqi/C/vpp/src/vlib/main.c:1592
#23 0x000077dc7672bbc5 in vlib_worker_thread_fn (arg=0x77dc38f6e600) at /home/zhangqi/C/vpp/src/vlib/main.c:2046
#24 0x000077dc7675afd5 in vlib_worker_thread_bootstrap_fn (arg=0x77dc38f6e600) at /home/zhangqi/C/vpp/src/vlib/threads.c:499
#25 0x000077dc762a3d64 in start_thread (arg=) at ./nptl/pthread_create.c:448
#26 0x000077dc763373bc in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
(gdb)
-
Reproduction conditions
When the attached mdns.pcap(compressed as mdns.zip) packets (broadcast fragments) are replayed on the Linux tap0 interface, if the physical interface GigabitEthernet3/0/0 is busy transmitting, these two fragmented packets are received by tap0 at the same time. Eventually, when the dispatch node GigabitEthernet3/0/0-output executes later than ip4-local-full-reassembly, a buffer double-free occurs. The node scheduling at this moment is shown below:
Context trace for bi 678815 b 0x100296e7c0, visited 12
ethernet-input (777)
l2-input (764)
l2-learn (762)
l2-flood (766)
l2-output (761)
ip4-input (722)
BondEthernet0-output (809)
ip4-lookup (736)
BondEthernet0-tx (808)
ip4-receive (733)
ip4-local-full-reassembly (716)
GigabitEthernet3/0/0-output (803) -
Root cause
During fragmented packet reassembly, the first fragment is cloned in l2-flood, and the data buffer's ref_count becomes 2. During reassembly in ip4-local-full-reassembly, the next_buffer of the cloned buffer is modified, which ultimately causes this next_buffer to result in a double-free.
Activity