What's the difference between different KUBE-MARK-MASQ rules in kubernetes chain?

Question

I know that the rule in KUBE-MARK-MASQ chain is a mark rule:

iptables -t nat -nvL KUBE-MARK-MASQ

Chain KUBE-MARK-MASQ (123 references)
 pkts bytes target     prot opt in     out     source               destination
   16   960 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK or 0x4000

It's to mark a packet and then the packets can do SNAT in KUBE-POSTROUTING chain,the source IP can be changed to node's ip.But what confused me is that why there are so many different KUBE-MARK-MASQ rules in k8s chains?For example,in KUBE-SERVICES chain,there are lots of KUBE-MARK-MASQ rules.What are they marking for?The pod..Or else?

Let's see an example:

KUBE-MARK-MASQ  tcp  --  *      *      !10.244.0.0/16        10.96.0.10           /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153

It's a kube-dns's clusterip rule.My pods' CIDR is 10.244.0.0/16.Why the rule's source ip has ! ?If a pod in the node want to send a packet outbound,it shouldn't have !,then it can do SNAT in KUBE-POSTROUTING to change to node's ip,is my understanding wrong?

And there are also other KUBE-MARK-MASQ rule in KUBE-SEP-XXX chain:

KUBE-MARK-MASQ  all  --  *      *       10.244.2.162         0.0.0.0/0            /* default/echo-load-balance: */
DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/echo-load-balance: */ tcp to:10.244.2.162:8080

The pod's ip is 10.244.2.162,and the rule source's ip matches pod's ip.What is it used for?

And in KUBE-FW-XXX chain:

KUBE-MARK-MASQ  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/echo-load-balance: loadbalancer IP */
KUBE-SVC-P24HJGZOUZD6OJJ7  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/echo-load-balance: loadbalancer IP */
KUBE-MARK-DROP  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/echo-load-balance: loadbalancer IP */

Why the source's ip here is 0.0.0.0/0?What is it used for?

Answer 1

The IP address 0.0.0.0/0 means it match any kind of IP, the only thing I'm not quite sure is the rules below:

KUBE-MARK-MASQ  all  --  *      *       10.244.2.162         0.0.0.0/0            /* default/echo-load-balance: */
DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/echo-load-balance: */ tcp to:10.244.2.162:8080

Why do we need to masquerade the packet when the source IP address is itself?

Answer 2

To see all rules iptables-save output is useful.

iptables processing chart might help understand this:

(diagram from Wikipedia)

When you isolate rules for a single service nat rules looks like this:

*nat
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-SEP-232DQYSHL5HNRYWJ -s 10.244.0.7/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-232DQYSHL5HNRYWJ -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.244.0.7:53
-A KUBE-SEP-LPGSDLJ3FDW46N4W -s 10.244.0.5/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-LPGSDLJ3FDW46N4W -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.244.0.5:53
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SVC-TCOU7JCQXEZGVUNU ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns -> 10.244.0.5:53" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-LPGSDLJ3FDW46N4W
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns -> 10.244.0.7:53" -j KUBE-SEP-232DQYSHL5HNRYWJ

-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully

In this case it balances DNS queries between 2 instances with 50% chance to hit one DNS instance:

--mode random --probability 0.5

And yes, it does look a bit complicated. But that's what you get when your building universal solution for all cases.