Iptables netfilter queue nfq_set_verdict_mark (nfq_set_verdict2) doesn't seem to be applying a mark

Question

I'm having some trouble with iptables and a fresh set of eyeballs would probably be able to see where the problem lies.

I'm trying to use the netfilter queue to mark packets (via nfq_set_verdict_mark -- old documentation says to use nfq_set_verdict2 but that function seems to no longer be in the header or library) in the PREROUTING chain of mangle table. The packet then should move on to PREROUTING chain of nat table, where I have rules to read the mark and redirect accordingly. Indeed, this was working in an earlier test but now is not!

Here's the callback function for my simplified test - it simply marks with 0xFFFFFFFF and accepts:

static int cb(struct nfq_q_handle *qh, struct nfgenmsg *nfmsg,
          struct nfq_data *nfa, void *data)
{
    uint32_t id = print_pkt(nfa); // from example code
    printf("marking packet...\n");
    return nfq_set_verdict_mark(qh,id,NF_ACCEPT,0xFFFFFFFF,0,NULL);
}

Here are the iptable rules I'm using:

// Using queue -- not working
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j QUEUE
iptables -t nat -A PREROUTING -p tcp --dport 80 -m mark --mark 0xFFFFFFFF -j REDIRECT --to-ports 9009

I also use conntrack --flush to reset connection status to ensure the packets encounter the nat table. I use 0xFFFFFFFF to prevent possible byte ordering issues. Packets clearly enter queue and are accepted ("marking packet..." is printed, browser can access content on port 80). Redirection, however, does not happen.

If I set mark directly instead of in queue, the redirection works. It's like the mark isn't being added in the queue. It was certainly marking before and I haven't changed library versions or anything.

// Without queue -- redirection happens
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 0xFFFFFFFF
iptables -t nat -A PREROUTING -p tcp --dport 80 -m mark --mark 0xFFFFFFFF -j REDIRECT --to-ports 9009

Here's print_pkt for good measure - not written by me, but taken from netfilter_queue example, has not been changed, and was previously working:

static u_int32_t print_pkt (struct nfq_data *tb)
{
        int id = 0;
        struct nfqnl_msg_packet_hdr *ph;
        struct nfqnl_msg_packet_hw *hwph;
        u_int32_t mark,ifi; 
        int ret;
        char *data;

        ph = nfq_get_msg_packet_hdr(tb);
        if (ph) {
                id = ntohl(ph->packet_id);
                printf("hw_protocol=0x%04x hook=%u id=%u ",
                        ntohs(ph->hw_protocol), ph->hook, id);
        }

        hwph = nfq_get_packet_hw(tb);
        if (hwph) {
                int i, hlen = ntohs(hwph->hw_addrlen);

                printf("hw_src_addr=");
                for (i = 0; i < hlen-1; i++)
                        printf("%02x:", hwph->hw_addr[i]);
                printf("%02x ", hwph->hw_addr[hlen-1]);
        }

        mark = nfq_get_nfmark(tb);
        if (mark)
                printf("mark=%u ", mark);

        ifi = nfq_get_indev(tb);
        if (ifi)
                printf("indev=%u ", ifi);

        ifi = nfq_get_outdev(tb);
        if (ifi)
                printf("outdev=%u ", ifi);
        ifi = nfq_get_physindev(tb);
        if (ifi)
                printf("physindev=%u ", ifi);

        ifi = nfq_get_physoutdev(tb);
        if (ifi)
                printf("physoutdev=%u ", ifi);

        ret = nfq_get_payload(tb, &data);
        if (ret >= 0)
                printf("payload_len=%d ", ret);

        fputc('\n', stdout);

        return id;
}

Also, the main function (also taken from example code and was previously working):

int main(int argc, char **argv)
{
        struct nfq_handle *h;
        struct nfq_q_handle *qh;
        struct nfnl_handle *nh;
        int fd;
        int rv;
        char buf[4096] __attribute__ ((aligned));

        printf("opening library handle\n");
        h = nfq_open();
        if (!h) {
                fprintf(stderr, "error during nfq_open()\n");
                exit(1);
        }

        printf("unbinding existing nf_queue handler for AF_INET (if any)\n");
        if (nfq_unbind_pf(h, AF_INET) < 0) {
                fprintf(stderr, "error during nfq_unbind_pf()\n");
                exit(1);
        }

        printf("binding nfnetlink_queue as nf_queue handler for AF_INET\n");
        if (nfq_bind_pf(h, AF_INET) < 0) {
                fprintf(stderr, "error during nfq_bind_pf()\n");
                exit(1);
        }

        printf("binding this socket to queue '0'\n");
        qh = nfq_create_queue(h,  0, &cb, NULL);
        if (!qh) {
                fprintf(stderr, "error during nfq_create_queue()\n");
                exit(1);
        }

        printf("setting copy_packet mode\n");
        if (nfq_set_mode(qh, NFQNL_COPY_META, 0xffff) < 0) {
                fprintf(stderr, "can't set packet_copy mode\n");
                exit(1);
        }

        fd = nfq_fd(h);

        while ((rv = recv(fd, buf, sizeof(buf), 0)) && rv >= 0) {
                printf("pkt received\n");
                nfq_handle_packet(h, buf, rv);
        }

        printf("unbinding from queue 0\n");
        nfq_destroy_queue(qh);

#ifdef INSANE
        /* normally, applications SHOULD NOT issue this command, since
         * it detaches other programs/sockets from AF_INET, too ! */
        printf("unbinding from AF_INET\n");
        nfq_unbind_pf(h, AF_INET);
#endif

        printf("closing library handle\n");
        nfq_close(h);

        exit(0);
}

Redirection is the expected behaviour in my test code, but it's not happening. What is the missing link? I've spent hours on this problem (that was working as expected prior) to no avail. Help!!

Answer 1

Yes, with the latest libnetfilter_queue, one should use only nfq_set_verdict2 function in the below fashion

 nfq_set_verdict2(handle,
                  packet_id,
                  NF_AXXEPT,
                  nfq_get_nfmark(nfa) | 0xFFFFFFFF,
                  0,
                  NULL);

It worked for me.

Answer 2

old documentation says to use nfq_set_verdict2 but that function seems to no longer be in the header or library

Perhaps you are using an outdated version of libnetfilter_queue hdrs+lib ? Version 1.0.2 (netfilter.org) seems to be the latest release, and has this function.