Docker routing into ipvlan from host network

Question

Goal: I like to be able to ping and access the docker clients from my host network. And if possible, I like to have as much as possible configured in my docker-compose.yml.

Remark: ICMP (ping) is just used for simplification. Actually, I like to access ssh on 22 and some other ports. Mapping ports is my current solution, but since I have many docker client container it becomes messy.

  ___________        ___________         ___________ 
 |   host   |        |  docker  |       |  docker  |
 |  client  |        |   host   |       |  client  |   
 | ..16.50  |  <-->  | ..16.10  |       |          |
 |          |        | ..20.1   | <-->  | ..20.5   |
 |          |                           |          |
 |          |  <----- not working ----> |          |
 

Problem: I am able to ping my docker host from docker clients and host clients, but not the docker clients from host clients.

That's my configuration on ubuntu 22.04.

docker host:               192.168.16.10/24
client host network:       192.168.16.50/24
default gw host network:   192.168.161  /24
docker client (container): 192.168.20.5 /24

docker-compose.yml

version: '3'

networks:
  ipvlan20:
    name: ipvlan20
    driver: ipvlan
    driver_opts:
      parent: enp3s0.20
      com.docker.network.bridge.name: br-ipvlan20
      ipvlan-mode: l3
    ipam:
      config:
        - subnet: "192.168.20.0/24"
          gateway: "192.168.20.1"
          
services:
  portainer:
    image: alpine
    hostname: ipvlan20
    container_name: ipvlan20
    restart: always
    command: ["sleep","infinity"]
    
    dns: 192.168.16.1
    networks:
      ipvlan20:
        ipv4_address: 192.168.20.5

On my docker host, I added the following link with the vlan gateway IP.

ip link add myipvlan20 link enp3s0.20 type ipvlan mode l3
ip addr add 192.168.20.1/24 dev myipvlan20
ip link set myipvlan20 up

And on my host client, I added a rout to the docker host for the docker client network.

ip route add 192.168.20.0/24 via 192.168.16.10

I tried also:

Do I have to use macvlan? I tried that, but also unsuccessfully.

Do I have to use l3? I also tried with l2, but unsuccessfully as well.