Docker swarm ingress - unable to connect through two networks

Question

I tried to run docker swarm over two different networks.

• First network is 10.10.100.x/24
• Second network is 10.10.150.x/24

Both networks can see each other. There are no firewall rules between them to block any traffic. Specifically I tested 7946 TCP and UDP and 4789 UDP. I can confirm, that I can connect from the first network to the second network on both ports and both protocols. And also from the second network to the first network without any issue.

Docker swarm is up and running and used engine is 20.10.11 I can see that all nodes have status=Ready and availability=Active.

Ingress network is default:

and I can see all peers listed there as well.

But when I deploy any service to any node with port -p 20000:80, then I can see this node only from the network where it was deployed.

• If service lands on the first network, it is accessible only through nodes from the first network, not from the second.
• If service lands on the second network, it is accessible only through nodes from the second network, not from the first.

Any thoughts how to fix this?

Thanks

---

update 1:

Tried to run swarm with additional parameter docker swarm init --default-addr-pool 172.100.0.0/16. Result remains the same.

---

update 2: Based on the advice from @BMitch

I verified with sudo tcpdump -nn -s0 -v port 4789 or 7946, that port 7946 works (UDP and TCP). I also verified with the previous tcpdump command and nc -z -v -u 10.10.150.200 4789 (run from the first network), that port 4789 works as well.

Answer 1

In the end - problem was in the NAT. Our second network was behind NAT, which caused this issue. Once we removed NAT, everything worked.

Answer 2

Same issue for me, routing and overlay work great but ingress load balancer only works through the same site endpoints that runs the container.

Oddly I discovered ingress load balancer works cross sites when using nc -l as server socket, making the whole even more obscure to me.

REM: Underlay network is wireguard VPN (L3 point-to-point)