HTTP GET packet sniffer in Scapy

Question

I am trying to code a simple sniffer in Scapy, which only prints HTTP packets with GET method only. Here's the code:

#!/usr/bin/python
from scapy.all import *

def http_header(packet):
        http_packet=str(packet)
        if http_packet.find('GET'):
                print GET_print(packet)
        print packet
def GET_print(packet1):
        print "***************************************GET PACKET****************************************************"
        print packet1

        print "*****************************************************************************************************"


sniff(iface='eth0',prn=http_header)

Here is the output:

*****************************************************************************************************
None
T��Г
     )�pEa��@@���h��#/��t
                             �}LGku���U
oTE��I(��Ͻ�9qi���S��?��
                          XuW�F=���-�k=X:�
***************************************GET PACKET****************************************************
T��Г
     )�pE���@@���h��#/��t
                               ʪLGku����
oTE��I�K��AH�*�e��>�v1#D�(mG5T�o�?��8��喷╭���Ի�"�KT^�'�mB���]�����k>
                                                                                �_x�X�����8V?�Ǽw/�Z�=���N�À��\r�����)+}���l�c�9��j;���h��5�T�9Hۖ/O��)��P
         މY�qf爂�%�_`��6x��5D�I3���O�
t��tpI#�����$IC��E��
                     �G�
J��α���=�]��vһ���b5^|P��DK�)uq�2��ț�w�
                    tB������y=���n�i�r�.D6�kI�a���6iC���c'��0dPqED�4����[�[��hGh̃��~|Y/�>`\6yP  Dq١?T��Mѵ���f�;���Җ��Ǵ  gY���di�_x�8|
eo�p�xW9��=���vŅYe�}�T�ۨɑy�^�C
-�_(�<�{����}�������r
$��J�k-�9����}�Ϡf�27��QKԛ�`�GY�8��Sh���Y@8�E9�Rϔ�&a�/vkф��6�DF`�/9�I�d( ��-��[A
                                                                                     ��)pP��y\ռj]���8�_���vf�b����I7�������+�P<_`
*****************************************************************************************************

What I am expecting is:

GET / HTTP/1.1
    Host: google.com
    User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140722 Firefox/24.0 Iceweasel/24.7.0
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    Accept-Language: en-US,en;q=0.5
    Accept-Encoding: gzip, deflate
    Cookie: PREF=ID=758a20b5fbd4eac9:U=2b2dedf6c84b001f:FF=0:TM=1412150291:LM=1415430021:S=Q-QemmrLqsSsEA9i; NID=67=mRdkPVhtImrOTLi5I1e5JM22J7g26jAcdiDEjj9C5q0H5jj0DWRX27hCM7gLJBeiowW-8omSv-1ycH595SW2InWX2n1JMMNh6b6ZrRsZ9zOCC2a-vstOQnBDSJu6K9LO
    Connection: keep-alive

What can I do to get my expected output?

Answer 1

You need to use the sprintf function of the packet instead of printing the packet itself. You also need to split the string returned from it and join it back together with newline characters, otherwise it spits it out all on one line:

#!/usr/bin/python
from scapy.all import *

def http_header(packet):
        http_packet=str(packet)
        if http_packet.find('GET'):
                return GET_print(packet)

def GET_print(packet1):
    ret = "***************************************GET PACKET****************************************************\n"
    ret += "\n".join(packet1.sprintf("{Raw:%Raw.load%}\n").split(r"\r\n"))
    ret += "*****************************************************************************************************\n"
    return ret

sniff(iface='eth0', prn=http_header, filter="tcp port 80")

I also added a filter for TCP port 80, but this could be removed if you need to.

Example output:

***************************************GET PACKET****************************************************
'GET /projects/scapy/doc/usage.html HTTP/1.1
Host: www.secdev.org
Connection: keep-alive
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/537.36
Referer: https://www.google.co.uk/
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-GB,en;q=0.8,en-US;q=0.6
If-None-Match: "28c84-48498d5654df67640-gzip"
If-Modified-Since: Mon, 19 Apr 2010 15:44:17 GMT

'
*****************************************************************************************************

Pierre points out that you can do away with the http_header function entirely by using the lfilter argument to sniff(). I took the liberty of making the code a little more succinct at the same time:

#!/usr/bin/python
from scapy.all import *

stars = lambda n: "*" * n

def GET_print(packet):
    return "\n".join((
        stars(40) + "GET PACKET" + stars(40),
        "\n".join(packet.sprintf("{Raw:%Raw.load%}").split(r"\r\n")),
        stars(90)))

sniff(
    iface='eth0',
    prn=GET_print,
    lfilter=lambda p: "GET" in str(p),
    filter="tcp port 80")

Answer 2

EDIT:

Please note that Scapy-http is now DEPRECATED and is included in Scapy 2.4.3+. Use import scapy.layers.http or load_layer("http") to enable it.

Answer:

There's a scapy http module that you can install by running pip install scapy-http. Once that is installed, you can import it by running import scapy_http.http. This is separate from your scapy module but adds functionality to scapy so you still need to import scapy as you usually would.

Once imported, change your filter line to

sniff(iface="eth0",
prn=GET_print,
lfilter= lambda x: x.haslayer(scapy_http.http.HTTPRequest))

I removed the filter="tcp and port 80" option because, using the http lfilter will return all HTTP Request queries regardless of port, except SSL for the obvious reason that it cannot be sniffed under the usual circumstances. You may want to keep the filter option for performance reasons.

Answer 3

I had commented on one way to improve it but I decided to whip together a more complete solution. This won't have the asterisk packet breaks but instead just prints the headers as pretty printed dictionary so this may work for you or may not but you can also customize it to suit your needs. Aside from the formatting, this seems like the most efficient means posted on this question so far and you can delegate to a function to add formatting and further deconstruct the dict.

#!/usr/bin/env python2

import argparse
import pprint
import sys

# Suppress scapy warning if no default route for IPv6. This needs to be done before the import from scapy.
import logging
logging.getLogger("scapy.runtime").setLevel(logging.ERROR)


# Try to import sniff from scapy.all and show error w/ install instructions if it cannot be imported.
try:
    from scapy.all import sniff
except ImportError:
    sys.stderr.write("ERROR: You must have scapy installed.\n")
    sys.stderr.write("You can install it by running: sudo pip install -U 'scapy>=2.3,<2.4'")
    exit(1)

# Try to import scapy_http.http and show error w/ install instructions if it cannot be imported.
try:
    import scapy_http.http
except ImportError:
    sys.stderr.write("ERROR: You must have scapy-http installed.\n")
    sys.stderr.write("You can install it by running: sudo pip install -U 'scapy>=1.8'")
    exit(1)


if __name__ == "__main__":
    # Parser command line arguments and make them available.
    parser = argparse.ArgumentParser(
        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
        description="Print HTTP Request headers (must be run as root or with capabilities to sniff).",
    )
    parser.add_argument("--interface", "-i", help="Which interface to sniff on.", default="eth0")
    parser.add_argument("--filter", "-f", help='BPF formatted packet filter.', default="tcp and port 80")
    parser.add_argument("--count", "-c", help="Number of packets to capture. 0 is unlimited.", type=int, default=0)
    args = parser.parse_args()

    # Sniff for the data and print it using lambda instead of writing a function to pretty print.
    # There is no reason not to use a function you write for this but I just wanted to keep the example simply while
    # demoing how to only match HTTP requests and to access the HTTP headers as pre-created dict's instead of
    # parsing the data as a string.
    sniff(iface=args.interface,
          promisc=False,
          filter=args.filter,
          lfilter=lambda x: x.haslayer(scapy_http.http.HTTPRequest),
          prn=lambda pkt: pprint.pprint(pkt.getlayer(scapy_http.http.HTTPRequest).fields, indent=4),
          count=args.count
    )