I'm running iperf2 on ubuntu18.04 and trying to test the max bandwidth with one CPU. My topology is client(with CX5 100G nic)->switch(100G)->server(with CX5 100G nic).
Unfortunately, the speed is only 30Gbps. Then I do a loopback test, the speed is 60Gbps and the cpu usage is 100%. What should I do?
Binding to local address 10.0.0.2
Write buffer size: 128 KByte
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.0.0.2 port 42749 connected with 10.0.0.1 port 5001
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 0.00-1.00 sec 3.71 GBytes 31.9 Gbits/sec 30379/0 0 6025K/354 us
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 1.00-2.00 sec 3.59 GBytes 30.8 Gbits/sec 29415/0 0 6684K/271 us
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 2.00-3.00 sec 3.63 GBytes 31.2 Gbits/sec 29712/0 0 7407K/399 us
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 3.00-4.00 sec 3.71 GBytes 31.8 Gbits/sec 30359/0 0 8285K/399 us
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 4.00-5.00 sec 3.73 GBytes 32.0 Gbits/sec 30520/0 0 8285K/328 us
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 5.00-6.00 sec 3.77 GBytes 32.4 Gbits/sec 30911/0 0 8285K/378 us
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 6.00-7.00 sec 3.77 GBytes 32.4 Gbits/sec 30920/0 0 8285K/384 us
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 7.00-8.00 sec 3.74 GBytes 32.1 Gbits/sec 30646/0 0 8285K/325 us
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 8.00-9.00 sec 3.77 GBytes 32.4 Gbits/sec 30917/0 0 8285K/334 us
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 9.00-10.00 sec 3.72 GBytes 32.0 Gbits/sec 30503/0 0 8285K/278 us
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 0.00-10.00 sec 37.1 GBytes 31.9 Gbits/sec 304286/0 0 8285K/278 us
Binding to local address 10.0.0.2
Write buffer size: 128 KByte
TCP window size: 2.50 MByte (default)
------------------------------------------------------------
[ 3] local 10.0.0.2 port 37687 connected with 10.0.0.2 port 5001
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 0.00-1.00 sec 3.85 GBytes 33.1 Gbits/sec 31562/0 0 1087K/41 us
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 1.00-2.00 sec 5.82 GBytes 50.0 Gbits/sec 47638/0 0 1087K/20 us
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 2.00-3.00 sec 7.65 GBytes 65.7 Gbits/sec 62644/0 0 1087K/19 us
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 3.00-4.00 sec 7.46 GBytes 64.1 Gbits/sec 61140/0 0 1342K/19 us
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 4.00-5.00 sec 7.66 GBytes 65.8 Gbits/sec 62716/0 0 1342K/19 us
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 5.00-6.00 sec 7.60 GBytes 65.3 Gbits/sec 62231/0 0 1342K/21 us
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 6.00-7.00 sec 7.67 GBytes 65.9 Gbits/sec 62833/0 0 1342K/17 us
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 7.00-8.00 sec 7.71 GBytes 66.2 Gbits/sec 63134/0 0 1342K/19 us
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 8.00-9.00 sec 7.73 GBytes 66.4 Gbits/sec 63291/0 0 1342K/19 us
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 9.00-10.00 sec 7.73 GBytes 66.4 Gbits/sec 63311/0 0 2110K/44 us
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT
[ 3] 0.00-10.00 sec 70.9 GBytes 60.9 Gbits/sec 580508/0 0 2110K/44 us
For one thing, your loopback test will likely have a much larger packet size than your over-the-network test. That will mean TCP needs to to fewer trips up/down the protocol stack to transfer a given quantity of data.
For another, your loopback test will not include any of the overhead for the driver of your 100G NIC.
Also, one of the many limits to the performance of a TCP connection is:
Throughput <= WindowSize / RoundTripTime
and while the round-trip time on your small 100G setup may be small, they are still longer than loopback, and you are trying for a throughput which is quite large, so the default limits for the TCP window size of your stack may be impeding getting higher throughput.