How can I safely run a GUI application inside a docker container with multiple networks?

Question

I am trying to access GUI of a client side of my application running inside a docker container. So far I have been using xhost + to disable access control, allowing clients to connect from any host, but this is obviously not a safe solution. I tried to solve this problem by exposing a socket and using XAuth to allow my application to use host's display but this requires my client container to be on a host network.

Here is the relevant part of my docker-compose.yml file:

networks:
  backend:
  frontend:
    driver: bridge

services:
  client:
    build:
      context: ./client
      dockerfile: Dockerfile
    container_name: client
    networks:
      frontend:
    restart: always
    stdin_open: true
    tty: true
    command: ${RUN_CLIENT}
    volumes:
      - ./client/:/usr/src/client/
      - /tmp/.X11-unix:/tmp/.X11-unix
      - ~/.Xauthority:/root/.Xauthority
    env_file:
      - ${CLIENT_ENV_FILE}

Client's Dockerfile contains command RUN apt-get install -y x11-apps xauth to install Xauth in the container. DISPLAY environment variable is passed inside of CLIENT_ENV_FILE like this DISPLAY=${DISPLAY}

I have tried adding network-mode: host key but that is incompatible with defining networks. Furthermore I have tried configuring my frontend network like this:

networks:
  backend:
  frontend:
    name: host
    external: true
    driver: bridge

which results in the following error: network-scoped alias is supported only for containers in user defined networks.

I have found several question about this topic which were answered but none of them covered the case of multiple user defined networks.

Is there a way to use XAuth to display GUI of a client running in a container on a user defined network and if so, how?