I'm a newbie who is just starting to dabble with the DPDK. My end goal is to send and receive packets between Host A and Host C in the following topology. A more detailed description is given below.
OS: Ubuntu 20.04
Kernel: 5.15.0-73-generic
NICs: Mellanox ConnectX6-Dx (2 port)
portmap (Host A's port 0 - Host B's port 0), (Host B's port 1 - Host C's port 0)
driver: I haven't bind vfio-pci with /usertools/dpdk-devbind.py because MLX5 doesn't need to (source: DPDK's MLX5 document)
------ ------ ------
|Host A|------|Host B|------|Host C|
------ ------ ------
198.18.0.10 198.18.1.10
I've tried multiple DPDK's forwarding applications for Host B with multiple communication applications for Host A and C which are give below
DPDK Applications for Host B: testpmd, l2fwd, l3fwd (lpm, acl), ip-pipeline
communication Applications for Host A and C: testpmd (start tx_first), ping, arping, netcet (UDP), udaddy (UDP, RDMA)
However, the DPDK did not forward the UDP packet (the application of the bold letter failed).
Here are some observations on this.
Host A's ping application
root@Host_A: ping 198.18.1.10
PING 198.18.1.10 (198.18.1.10) 56(84) bytes of data.
64 bytes from 198.18.1.10: icmp_seq=1 ttl=64 time=0.243 ms
64 bytes from 198.18.1.10: icmp_seq=2 ttl=64 time=0.209 ms
64 bytes from 198.18.1.10: icmp_seq=3 ttl=64 time=0.161 ms
64 bytes from 198.18.1.10: icmp_seq=4 ttl=64 time=0.156 ms
64 bytes from 198.18.1.10: icmp_seq=5 ttl=64 time=0.155 ms
--- 198.18.1.10 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4073ms
Host B's forwarding
root@Host_B:./dpdk-testpmd -l 0-3 -n 4 -- -i --rss-udp --rxq=2 --txq=2
EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:101d) device: 0000:01:00.0 (socket -1)
EAL: Probe PCI driver: mlx5_pci (15b3:101d) device: 0000:01:00.1 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
Warning: NUMA should be configured manually by using --port-numa-config and --ring-numa-config parameters along with --numa.
testpmd: create a new mbuf pool <mb_pool_0>: n=171456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 0)
Port 0: XX:XX:XX:XX:XX:XX
Configuring Port 1 (socket 0)
Port 1: XX:XX:XX:XX:XX:XX
Checking link statuses...
Done
testpmd> start
Host B's tcpdump
root@Host_B: tcpdump.4.9.3 -i mlx5_0
17:43:56.712052 IP 198.18.0.10 > 198.18.1.10: ICMP echo request, id 1, seq 1, length 64
17:43:56.712129 IP 198.18.1.10 > 198.18.0.10: ICMP echo reply, id 1, seq 1, length 64
17:43:57.713508 IP 198.18.0.10 > 198.18.1.10: ICMP echo request, id 1, seq 2, length 64
17:43:57.713515 IP 198.18.1.10 > 198.18.0.10: ICMP echo reply, id 1, seq 2, length 64
17:43:58.737455 IP 198.18.0.10 > 198.18.1.10: ICMP echo request, id 1, seq 3, length 64
17:43:58.737527 IP 198.18.1.10 > 198.18.0.10: ICMP echo reply, id 1, seq 3, length 64
17:43:59.761458 IP 198.18.0.10 > 198.18.1.10: ICMP echo request, id 1, seq 4, length 64
17:43:59.761526 IP 198.18.1.10 > 198.18.0.10: ICMP echo reply, id 1, seq 4, length 64
17:44:00.785433 IP 198.18.0.10 > 198.18.1.10: ICMP echo request, id 1, seq 5, length 64
17:44:00.785509 IP 198.18.1.10 > 198.18.0.10: ICMP echo reply, id 1, seq 5, length 64
root@Host_B: tcpdump.4.9.3 -i mlx5_1
17:43:56.712038 IP 198.18.0.10 > 198.18.1.10: ICMP echo request, id 1, seq 1, length 64
17:43:56.712096 IP 198.18.1.10 > 198.18.0.10: ICMP echo reply, id 1, seq 1, length 64
17:43:57.713481 IP 198.18.0.10 > 198.18.1.10: ICMP echo request, id 1, seq 2, length 64
17:43:57.713554 IP 198.18.1.10 > 198.18.0.10: ICMP echo reply, id 1, seq 2, length 64
17:43:58.737446 IP 198.18.0.10 > 198.18.1.10: ICMP echo request, id 1, seq 3, length 64
17:43:58.737491 IP 198.18.1.10 > 198.18.0.10: ICMP echo reply, id 1, seq 3, length 64
17:43:59.761439 IP 198.18.0.10 > 198.18.1.10: ICMP echo request, id 1, seq 4, length 64
17:43:59.761490 IP 198.18.1.10 > 198.18.0.10: ICMP echo reply, id 1, seq 4, length 64
17:44:00.785422 IP 198.18.0.10 > 198.18.1.10: ICMP echo request, id 1, seq 5, length 64
17:44:00.785473 IP 198.18.1.10 > 198.18.0.10: ICMP echo reply, id 1, seq 5, length 64
For ping, you can see that it works fine. I omitted it because it's long, but arping, start tx_first from testpmd also works fine.
However, when you send a UDP packet, the dpdk applications fail to perform any forwarding behavior at all.
Host A's UDP application
root@Host_A:/home/nxc# netcat -v -u -z 198.18.1.10 4791
Host A's tcpdump
root@Host_A: tcpdump.4.9.3 -i mlx5_0
17:55:16.155283 IP Host_A.46439 > 198.18.1.10.4791: UDP, length 1
17:55:16.155366 IP Host_A.46439 > 198.18.1.10.4791: UDP, length 1
17:55:17.155524 IP Host_A.46439 > 198.18.1.10.4791: UDP, length 1
17:55:18.155777 IP Host_A.46439 > 198.18.1.10.4791: UDP, length 1
17:55:19.156045 IP Host_A.46439 > 198.18.1.10.4791: UDP, length 1
Host B's tcpdump
root@Host_B: tcpdump.4.9.3 -i mlx5_0
17:55:16.149324 IP 198.18.0.10.46439 > 198.18.1.10.4791: UDP, length 1
17:55:16.149333 IP 198.18.0.10.46439 > 198.18.1.10.4791: UDP, length 1
17:55:17.149570 IP 198.18.0.10.46439 > 198.18.1.10.4791: UDP, length 1
17:55:18.149824 IP 198.18.0.10.46439 > 198.18.1.10.4791: UDP, length 1
17:55:19.150079 IP 198.18.0.10.46439 > 198.18.1.10.4791: UDP, length 1
root@Host_B: tcpdump.4.9.3 -i mlx5_1
none (packet is undetected)
The same thing happens with other dpdk forwarding applications and UDP applications.
UDP application has no problem since they work fine with directly connected interface such as (Host A - Host B) and (Host B - Host C).
I tried a number of different things to resolve this issue.
- Add Static routes, ARPs for Host A and C
route add -net 198.18.1.10 netmask 255.255.255.255 dev enp1s0f0np0
arp -s 198.18.1.10 XX:XX:XX:XX:XX:XX
Debug DPDK applications and find that
nb_rx = rte_eth_rx_burst(port_id, rxq, pkts_burst, MAX_PKT_BURST);returns 0 (which means nb_rx=0) while communicating UDP. Thus I believe that DPDK drops the UDP packet somewhere or forwarding to other interface.I've added some rss (receiver side scaling) parameter but it doesn't work
Any advice can be huge help to me.