I'm using AWS Firecracker to create two VMs (both have Ubuntu 22.04 OS) on a container that has Ubuntu 22.04 OS. I create a bridge br0 and assign IP address of 192.168.0.7 to it as the gateway. Then I create tun/tap virtual network interfaces for each vm (tap0 and tap1) connected to br0 and assign IP addresses of 192.168.0.170 (VM1) and 192.168.0.171 (VM2).
I can ping gateway when I'm in each VM and my host can ping both virtual machines. Even VMs have access to internet through the bridge network interface in the host.
The problem is that I cannot ping VM2, when I'm inside VM1, unless I remove 192.168.0.0/24 automatically generated routing by the kernel. For example for VM1, I need to remove this:
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.170
By using ip r del 192.168.0.0/24, after that I'm able to ping VM2. Note that the default routing way is through the gateway 192.168.0.7 in both VM1 and VM2.
I thought that both VMs being on the same NIC is enough for them to be able to ping each other and they don't need to go through gateway. Any idea why it happens? Thanks for your comments and thoughts.