linux ip routing with multiple uplinks SINGLE interface

Question

trying to setup a Proxmox machine that is running 3 vms. it has 3 public ips but these ips are on a single interface (eth0).

the 3 vms are on a bridge (vmbr0) with an address of 172.16.0.1/24

I have enable ip masquerading and forwarding. but I cannot figure out how to make each of the 3 vms (172.16.0.2, 172.16.0.3, 172.16.0.4) route out through a specific one of the public ips.

I have tried the standard iproute with 3 tables setting the gateways and rules but no matter what rule i set the vms still route out through the primary ip.

trouble is the 3 public ips are on complete seperate networks so they each have a different gateway. I know how to use iproute to do this if each public ip was on a seperate physical interface but this machine has all 3 on a single interface and iproute doesn't seem to like that because if I do ip route add default via 23.92.26.1 dev eth0:2 table node2 and then later list everything it shows via eth0. so aparently iproute doesn't like psuedo interfaces. I don't know a lot about iptables and I'm sure theres a way to do this with pure iptables but haven't found anything. all my google searches come up with iproute tables wich like i said don't seem to work with a signle interface.

Thank you in advance

Answer 1

considering ProxMox is running Debian try adding something like the following to your /etc/network/interfaces for each of the extra links

post-up route add -net <network identifier> netmask <netmask> gw <links gateway>
pre-down route del -net <network identifier> netmask <netmask> gw <links gateway>

and then with iptables if you want 172.16.0.2 to go through the second ip do like the following: (this is called Source NAT or SNAT) the --to-source specifies what ip you want the outgoing connections remapped to.

iptables -t nat -A POSTROUTING -s 172.16.0.2/24 -j SNAT --to-source <ip address you want it to go out of>

if you want all incoming connections on the same ip to go to 172.16.0.2 then you would also add the following (this is called Destination NAT or DNAT)

iptables -t nat -A PREROUTING -d <ip/mask bit> -j DNAT --to-destination 172.16.0.2

Answer 2

Question:

(1)3VM - 172.16.0.2, 172.16.0.3, 172.16.0.4

(2)Gateway - 172.16.0.1/24

(3)3 publicIP: $IP_A(gateway $IP_A_G), $IP_B(gateway $IP_B_G), $IP_C(gateway $IP_C_G)

(4)Your aim is every VM use the different public IP to visit outsite , suck as:

VM1(172.16.0.2) ----> $IP_A
VM2(172.16.0.3) ----> $IP_B
VM3(172.16.0.4) ----> $IP_C

So , I think you can use ip route to do this:

(1)In Promox(172.16.0.1)

echo "200 IPA" >> /etc/iproute2/rt_tables
echo "201 IPB" >> /etc/iproute2/rt_tables
echo "202 IPC" >> /etc/iproute2/rt_tables

Reboot Promox once .

(2)Create router

ip route del default table IPA
ip route add default via $IP_A_G  table IPA
ip route del default table IPB
ip route add default via $IP_B_G  table IPB
ip route del default table IPC
ip route add default via $IP_C_G  table IPC

(3)Add route for each VM

ip rule add from 172.16.0.2 lookup IPA pref 200
ip rule add from 172.16.0.3 lookup IPB pref 201
ip rule add from 172.16.0.4 lookup IPC pref 202
ip route flush cache

DONE