IPsec in Linux kernel - how to figure out what's going on

Question

I'm writing an IPsec implementation for a microcontroller and I want to test it using a standard Linux box running Debian Lenny. Both devices should secure the communication between them using IPsec ESP in tunnel mode. The keys are setup manually using setkey. There's no (or at least should be no) user space program involved in processing an IPsec packet. Now I want to see how my created packets are processed by the Linux kernel. To see the raw packets I capture them using tcpdump and analyze them using wireshark.

• What's the best way to obtain debug information about IPsec processing?
• How can I figure out whether the packet is accepted by the kernel?
• How can I view the reason for a packet to be dropped?

Answer 1

You can instrument the XFRM (or perhaps ipv4/esp.c) kernel code to print out debug messages at the right spots.

For example, in net/ipv4/esp.c there exists a function esp_input() which has some error cases, but you'll see most the interesting stuff is in the xfrm/*.c code.

That said, I didn't have a problem interoperating a custom IPSec with Linux. Following the 43xx specs and verifying the packets came out correctly via wireshark seemed to do well. If you're having issues and don't want to instrument the kernel then you can setup iptables rules and count the number of (various type of) packets at each point.

Finally, be sure you've actually added a security policy (SP) as well as a security association (SA) and setup firewall rules properly.