Why docker stack deploy is not starting all my containers?

Question

I deployed my docker stack:

⇒  docker stack deploy -c docker-compose.yml my_stack
Creating network my_stack_network
Creating service my_stack_redis
Creating service my_stack_wsgi
Creating service my_stack_nodejs
Creating service my_stack_nginx
Creating service my_stack_haproxy
Creating service my_stack_postgres

But when I do docker container ls, it only shows three containers:

~|⇒  docker container ls | grep my_stack                     
212720bfafc3   postgres:11                     "docker-entrypoint.s…"   4 minutes ago   Up 3 minutes   5432/tcp                                     my_stack_postgres.1.9nx7jb21whi61aboe9hmet6m2
3132dd980589   isiq/nginx-brotli:1.21.0        "/docker-entrypoint.…"   4 minutes ago   Up 4 minutes   80/tcp                                       my_stack_nginx.1.isl2c78z6w5ptizurm3a4cnte
62ef3c76fb9e   redis:6.2.4                     "docker-entrypoint.s…"   4 minutes ago   Up 4 minutes   6379/tcp                                     my_stack_redis.1.xnisrd1i6hod6jkm64623cpzj

But docker stack ps lists all of them as Running:

~|⇒  docker stack ps --no-trunc my_stack
ID                          NAME                  IMAGE                                                                                                     NODE      DESIRED STATE   CURRENT STATE           ERROR     PORTS
1fqwlgblhi5q0cdl5cy75ucli   my_stack_haproxy.1    haproxy:2.3.9@sha256:f63aabf39efcd277b04a503d38e59e80224a0c11f47b2568b13b0092698c5a3a                               Running         New 2 minutes ago                 
isl2c78z6w5ptizurm3a4cnte   my_stack_nginx.1      isiq/nginx-brotli:1.21.0@sha256:436cbc0d8cd051e7bdb197d7915fe90fa5a1bdadea6d02272ba117fccf30c936          tadoba    Running         Running 2 minutes ago             
1myvtgl11qqw2xa9cv79uikcs   my_stack_nodejs.1     nodejs:my_stack                                                                                                     Running         New 2 minutes ago                 
9nx7jb21whi61aboe9hmet6m2   my_stack_postgres.1   postgres:11@sha256:5d2aa4a7b5f9bdadeddcf87cf7f90a176737a02a30d917de4ab2e6a329bd2d45                       tadoba    Running         Running 2 minutes ago             
xnisrd1i6hod6jkm64623cpzj   my_stack_redis.1      redis:6.2.4@sha256:6bc98f513258e0c17bd150a7a26f38a8ce3e7d584f0c451cf31df70d461a200a                       tadoba    Running         Running 2 minutes ago             
mzmmb7a3bxjpfkfa3ea5o5w85   my_stack_wsgi.1       wsgi:my_stack                                                                                                       Running         New 2 minutes ago    

Checking logs of containers not listed in docker container ls gives No such container error:

~|⇒  docker logs -f 1myvtgl11qqw2xa9cv79uikcs            
Error: No such container: 1myvtgl11qqw2xa9cv79uikcs

~|⇒  docker logs -f mzmmb7a3bxjpfkfa3ea5o5w85                         
Error: No such container: mzmmb7a3bxjpfkfa3ea5o5w85

~|⇒  docker logs -f 1fqwlgblhi5q0cdl5cy75ucli                         
Error: No such container: 1fqwlgblhi5q0cdl5cy75ucli

What could be the reason? How can I debug this?

Update

It seems that services without dependencies are able to join the network. But services with dependencies on other services are not able to. I am unable to figure out the reason. Here is the gist with output of docker inspect network's output.

PS

I run watch 'docker container ls | grep my_app' in another terminal before running docker stack deploy .... But those three containers never appear in the watch list. Rest of three do appear.

I am running all nodes on the same remote machine connected through ssh. This is the output of docker node ls:

~|⇒  docker node ls                          
ID                            HOSTNAME   STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
z9hovq8ry6qont3m2rbn6upy4 *   tadoba     Ready     Active         Leader           20.10.11

Here is my docker compose file for reference:

version: "3.8"

services:
    postgres:
        image: postgres:11
        volumes:
            - my_app_postgres_volume:/var/lib/postgresql/data
            - type: tmpfs
              target: /dev/shm
              tmpfs:
                size: 536870912 # 512MB
        environment:
            POSTGRES_DB: my_app_db
            POSTGRES_USER: my_app
            POSTGRES_PASSWORD: my_app123
        networks:
            - my_app_network

    redis:
        image: redis:6.2.4
        volumes:
            - my_app_redis_volume:/data
        networks:
            - my_app_network

    wsgi:
        image: wsgi:my_app3_stats
        volumes:
            - /my_app/frontend/static/
            - ./wsgi/my_app:/my_app
            - /my_app/frontend/clientApp/node_modules
            - /etc/timezone:/etc/timezone:ro
            - /etc/localtime:/etc/localtime:ro
        depends_on:
            - postgres
            - redis
        ports:
            - 9090
        environment:
            C_FORCE_ROOT: 'true'
            SERVICE_PORTS: 9090
        networks:
            - my_app_network
        deploy:
            replicas: 1
            update_config:
                parallelism: 1
                delay: 10s
            restart_policy:
                condition: on-failure
                max_attempts: 3
                window: 120s

    nodejs:
        image: nodejs:my_app3_stats
        volumes:
            - ./nodejs/frontend:/frontend
            - /frontend/node_modules
        depends_on:
            - wsgi
        ports:
            - 9998:9999 
        environment:
            BACKEND_API_URL: http://aa.bb.cc.dd:9764/api/ 
        networks:
            - my_app_network

    nginx:
        image: isiq/nginx-brotli:1.21.0
        volumes:
            - ./nginx:/etc/nginx/conf.d:ro
            - ./wsgi/my_app:/my_app:ro
            - my_app_nginx_volume:/var/log/nginx/
            - /etc/timezone:/etc/timezone:ro
            - /etc/localtime:/etc/localtime:ro
        networks:
            - my_app_network
            
    haproxy:
        image: haproxy:2.3.9
        volumes:
            - ./haproxy:/usr/local/etc/haproxy/:ro
            - /var/run/docker.sock:/var/run/docker.sock
            - /etc/timezone:/etc/timezone:ro
            - /etc/localtime:/etc/localtime:ro
        depends_on:
            - wsgi
            - nodejs
            - nginx
        ports:
            - 9764:80
        networks:
            - my_app_network
        deploy:
            placement:
                constraints: [node.role == manager]

volumes:
    my_app_postgres_volume:
    my_app_redis_volume:
    my_app_nginx_volume:
    my_app_pgadmin_volume:

networks:
    my_app_network:
        driver: overlay

Output of docker service ps <service-name> for services not listed under tadoba node:

~/my_app|master-py3⚡ 
⇒  docker service ps my_app_nodejs
ID             NAME                   IMAGE                     NODE      DESIRED STATE   CURRENT STATE            ERROR     PORTS
i04jpykp9ign   my_app_nodejs.1        nodejs:bodhitree3_stats             Running         New about a minute ago             

~/my_app|master-py3⚡ 
⇒  docker service ps my_app_haproxy 
ID             NAME                    IMAGE           NODE      DESIRED STATE   CURRENT STATE            ERROR     PORTS
of4fcsxuq24c   my_app_haproxy.1        haproxy:2.3.9             Running         New about a minute ago             

~/my_app|master-py3⚡ 
⇒  docker service ps my_app_wsgi   
ID             NAME                 IMAGE                   NODE      DESIRED STATE   CURRENT STATE       ERROR     PORTS
yt9nuhule39z   my_app_wsgi.1        wsgi:bodhitree3_stats             Running         New 2 minutes ago      

Answer 1

Firstly all of the services are running which is good. But docker container ls isn't cluster-aware i.e. it is showing the current containers running on this node. From the output docker stack ps --no-trunc my_stack I can see that there is another node labeled tadoba. So if you can log in to the other node you can see the running containers.

You can list the nodes on your cluster by running docker node ls.

If you want you can set up your docker contexts so you can change your docker context which will remove the need to log in and log out of the nodes. You can find more info here.

Answer 2

If your Docker stack is running but no replicas are being created, and you don't see any errors when running the docker service ps <STACK_NAME> command, restarting Docker might help resolve the issue.

You can try the following steps:

Open a terminal or command prompt. Restart Docker using the appropriate command for your operating system:

• On Linux (systemd-based systems like Ubuntu):

  sudo systemctl restart docker
  

• On macOS:

  sudo service docker restart
  

• On Windows (PowerShell):

  Restart-Service docker
  

Wait for Docker to restart and then check if the replicas are created by running the docker service ps <STACK_NAME> command again. Restarting Docker can help resolve various issues related to container management and networking, and it's often a good first step when troubleshooting unexpected behavior in Docker.

If the problem persists after restarting Docker, there might be other factors causing the issue, such as incorrect configuration or resource limitations. In that case, it would be helpful to provide more details about your environment, the specific Docker stack configuration, and any relevant error messages or logs.