I've been encountering relatively disappointing performances relating to an in-house made vpn.
| Interface | speed Mb/s |
|---|---|
| enp5s0 | 950 |
| tun0 | 550 |
And I came to believe the culprit (among some others) to be TCP segmentation (see this article).
The thing is, linux networking is not my specialty and I feel a bit out of my depth, I'm not so sure to understand how GSO works, what it needs or even that is indeed what I'm looking for (I hope not to fall prey to a XY problem).
I will be using simpletun by Homer Simpson as a case study.
I'm open for alternative, but I'm mainly asking about the proper way to enable GSO between a TUN interface to offload forward segmentation as late as possible (that I understand to be advantageous as far as performances are concerned).
The question concerns networking on linux. I'm not concerned with interoperability with any other OS for the time being.
Expanding on the problem
documenting
I've been having a hard time finding proper documentation on this feature. For the time being, I've only been able to find
- this function in the tuntap driver regarding ioctl calls
- this function regarding setting offload options
- these notes relating to tuntap options
- the header file if_tun.h
- The official docs on the tuntap driver
What I did
I added the ifr flag IFF_VNET_HDR as explained in some of the sources given
I added one ioctl call, and another to set what I think are the correct flag for offloading forward
- F_CSUM : to disable checksum
- F_TS04, F_TSO6 : to enable tcp segmentation fault
/**************************************************************************
* tun_alloc: allocates or reconnects to a tun/tap device. The caller *
* must reserve enough space in *dev. *
**************************************************************************/
int tun_alloc(char *dev, int flags) {
struct ifreq ifr;
int fd, err;
char *clonedev = "/dev/net/tun";
if( (fd = open(clonedev , O_RDWR)) < 0 ) {
perror("Opening /dev/net/tun");
return fd;
}
memset(&ifr, 0, sizeof(ifr));
// flags = 0x0001 ≡ IFF_TUN
// flag IFF_VNET_HDR necessary ?
ifr.ifr_flags = flags | IFF_NO_PI | IFF_VNET_HDR;
if (*dev) {
strncpy(ifr.ifr_name, dev, IFNAMSIZ);
}
if( (err = ioctl(fd, TUNSETIFF, (void *)&ifr)) < 0 ) {
perror("ioctl(TUNSETIFF)");
close(fd);
return err;
}
// setting up offload
unsigned long offload_flags = TUN_F_CSUM | TUN_F_TSO4 | TUN_F_TSO6;
if ( (err = ioctl(fd, TUNSETOFFLOAD, offload_flags)) < 0 ) {
perror("ioctl(TUNSETOFFLOAD)");
close(fd);
return err;
}
// --------------------------------------------------------------
strcpy(dev, ifr.ifr_name);
return fd;
}
Expected results
I expected the vpn to gain performances because of offloading.
results obtained
It resulted in being slightly slower. results obtained with iperf3
| Interface | speed Mb/s |
|---|---|
| enp5s0 | 942 |
| tun0 | 324 |
This is confusing to me, I could understand a specific error but I do not understand how I could end up with slower results.
Annexes
iperf3
> iperf3 -c $SERVER_IP
Connecting to host 192.168.50.112, port 5201
[ 5] local 192.168.30.175 port 53126 connected to 192.168.50.112 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 114 MBytes 952 Mbits/sec 0 529 KBytes
[ 5] 1.00-2.00 sec 113 MBytes 949 Mbits/sec 0 557 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 939 Mbits/sec 0 617 KBytes
[ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec 0 747 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 783 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 944 Mbits/sec 0 783 KBytes
[ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 0 820 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 944 Mbits/sec 0 909 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 944 Mbits/sec 0 909 KBytes
[ 5] 9.00-10.00 sec 111 MBytes 933 Mbits/sec 0 953 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.09 GBytes 939 Mbits/sec receiver
iperf Done.
> iperf3 -B 10.10.9.1%tun0 -c $SERVER_IP
Connecting to host 192.168.50.112, port 5201
[ 5] local 10.10.9.1 port 52443 connected to 192.168.50.112 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 107 KBytes 879 Kbits/sec 20 2.83 KBytes
[ 5] 1.00-2.00 sec 45.2 KBytes 371 Kbits/sec 10 4.24 KBytes
[ 5] 2.00-3.00 sec 41.0 KBytes 336 Kbits/sec 10 5.66 KBytes
[ 5] 3.00-4.00 sec 39.6 KBytes 324 Kbits/sec 8 4.24 KBytes
[ 5] 4.00-5.00 sec 39.6 KBytes 324 Kbits/sec 10 5.66 KBytes
[ 5] 5.00-6.00 sec 39.6 KBytes 324 Kbits/sec 10 5.66 KBytes
[ 5] 6.00-7.00 sec 79.2 KBytes 649 Kbits/sec 8 7.07 KBytes
[ 5] 7.00-8.00 sec 39.6 KBytes 324 Kbits/sec 10 5.66 KBytes
[ 5] 8.00-9.00 sec 39.6 KBytes 324 Kbits/sec 11 1.41 KBytes
[ 5] 9.00-10.00 sec 39.6 KBytes 324 Kbits/sec 9 2.83 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 510 KBytes 418 Kbits/sec 106 sender
[ 5] 0.00-10.08 sec 431 KBytes 351 Kbits/sec receiver
iperf Done.