How can i pass the secrets to the ngrok.yml file in the container and in the FTP container to an environment?

Question

Goal:

So I'm just building a little project for my school, and i have a little old computer that i'm using as a server now.
My goal is to make the FTP container public using ngrok container so that i can access it using a domain from anywhere.

---

Setup:

• Ubuntu Server
• Docker
• compose.yaml
• ngrok.yml

---

Expected:

So i have created a compose.yaml that should pretty much build the desired service.
Sensitive data, like the user and password, are passed to the containers using secrets.
In the ftp: A shell command creates an environment variable and stores the secret value so the FTP container grants access.
In the ngrok: pretty much follows the same logic, but instead of creating an environment, it changes the value of the AUTH_TOKEN to the actual token in the ngrok.yml file.
Then i should be able to connect from anywhere and transfer files.

---

Current behavior:

When building the service with Docker Compose, I receive a warning from the daemon WARN[0000] The "AUTH_TOKEN" variable is not set. Defaulting to a blank string., but daemon can still finish the build.
However, when I look at the running containers using docker ps, they are always restarting:

CONTAINER ID   IMAGE                            COMMAND                  CREATED         STATUS                          PORTS     NAMES
4f4e56f716cc   ngrok/ngrok:latest                "/nix/store/n98vsmwd…"   6 minutes ago   Restarting (1) 26 seconds ago             app-ngrok-1
ec97742b74fc   delfer/alpine-ftp-server:latest   "/sbin/tini -- /bin/…"   6 minutes ago   Restarting (0) 28 seconds ago             app-ftp-1

Upon inspecting the ftp container, it logs the following:

deluser: can't find alpineftp in /etc/group
Changing password for alpineftp
New password:
Bad password: similar to username
Retype password:
passwd: password for alpineftp changed by root

Similarly, when inspecting ngrok, it logs the following:

ERROR:  unknown shorthand flag: 'c' in -c

---

Questions:

• How can I pass the values using secrets ?
• Is there a better approach to pass secrets ?

---

I'm learning how to use Docker, and that's why I'm building this little project.
I'm not good at bash since my knowledge is limited, but i'm all ears and really looking forward to growing!

Code:

compose.yaml:

version: "3.8"

services:
  ftp:
    image: delfer/alpine-ftp-server:latest
    restart: always
    command:
      - "/bin/sh"
      - "-c"
      - "USERS=$(cat /run/secrets/FTP_USERS); export USERS"

    environment:
      - ADDRESS=some-generated-domain.ngrok-free.app
    volumes:
      - /mnt/storage/public:/ftp
    networks:
      - public_web
    secrets:
      - FTP_USERS
  ngrok:
    image: ngrok/ngrok:latest
    restart: always
    command:
      - "/bin/sh"
      - "-c"
      - "AUTH_TOKEN=$(cat /run/secrets/AUTH_TOKEN); sed -i `s/AUTH_TOKEN/$AUTH_TOKEN/g` /etc/ngrok.yml;"
      - "start"
      - "--all"
      - "--config"
      - "/etc/ngrok.yml"

    volumes:
      - /mnt/storage/conf/ngrok.yml:/etc/ngrok.yml
    depends_on:
      - ftp
    networks:
      - public_web
    ports:
      - 21:21
    secrets:
      - AUTH_TOKEN
networks:
  public_web:

secrets:
  AUTH_TOKEN:
    file: /mnt/storage/.secrets/ngrokToken.txt
  FTP_USERS:
    file: /mnt/storage/.secrets/ftpUsers.txt

ngrok.yml:

authtoken: AUTH_TOKEN
region: eu
tunnels:
  ftp:
    labels:
      - hostname=Transfer_Files
      - service=ftp
    proto: tcp
    addr: 21
    hostname: some-generated-domain.ngrok-free.app

Answer 1

When embedding shell scripts in your compose file (or other YAML documents), you will in almost all cases find it more manageable to take advantage of YAMLs various extended quoting operators. Instead of this:

command:
- "sh"
- "-c"
- "echo this is a test; date > some_file; some_other_command"

Use the block literal quote operator:

command:
- "sh"
- "-c"
- |
  echo this is a test
  date > some_file
  some_other_command"

This is both easier to read and avoids nested quoting problems.

---

The first container in your compose file is a no-op:

ftp:
  image: delfer/alpine-ftp-server:latest
  restart: always
  command:
    - "/bin/sh"
    - "-c"
    - |
      USERS=$(cat /run/secrets/FTP_USERS)
      export USERS

It will start up, run your shell script, and then exit (and then repeat that infinitely because you have set restart: always).

---

Your ngrok container is also invalid; when you run a script with sh -c "some script", only the first argument to -c is the script. You have:

command:
  - "/bin/sh"
  - "-c"
  - "AUTH_TOKEN=$(cat /run/secrets/AUTH_TOKEN); sed -i `s/AUTH_TOKEN/$AUTH_TOKEN/g` /etc/ngrok.yml;"
  - "start"
  - "--all"
  - "--config"
  - "/etc/ngrok.yml"

So all you're actually running is:

AUTH_TOKEN=$(cat /run/secrets/AUTH_TOKEN)
sed -i `s/AUTH_TOKEN/$AUTH_TOKEN/g` /etc/ngrok.yml

Your also using backtics to quote a string which is an error; backtics are used for command substitution. You need to use double quotes (") to quote that expression.

I looks like you mean:

command:
  - "/bin/sh"
  - "-c"
  - |
    AUTH_TOKEN=$(cat /run/secrets/AUTH_TOKEN)
    sed -i "s/AUTH_TOKEN/$AUTH_TOKEN/g" /etc/ngrok.yml
    start --all --config /etc/ngrok.yml

But that won't work for a couple of reasons...

First, because the ngrok container runs ngrok by default with the command value as an argument, this will fail with:

ngrok-1  | ERROR:  unknown shorthand flag: 'c' in -c

Secondly, because docker compose performs variable substitution on compose files, your attempt to use $AUTH_TOKEN in the embedded script will fail (compose will attempt to replace it with the value of $AUTH_TOKEN from your local environment). You need to replace $ with $$ if you want an actual $ in your shell scripts (e.g., sed "s/AUTH_TOKEN/$$AUTH_TOKEN/g").

You will need to figure out some other way to handle the secret for ngrok. For example, you can run a container before the ngrok container that reads the configuration file, performs variable subsitution, and then writes it out into a volume that is then used by the ngrok container.

Or... you can simplify things by taking advantage of the fact that you can provide ngrok with a list of configuration files, which means you don't need to perform substitution on your ngrok config file; instead, just provide a second config file that only contains the auth token.

Given this in ngrok.yaml:

version: 2
log: stderr
tunnels:
  webserver:
    addr: webserver:80
    inspect: false
    proto: http

And this in ngrok_authtoken.yaml:

authtoken: my_auth_token
version: 2

I can use this compose.yaml:

secrets:
  ngrok_authtoken.yaml:
    file: ./ngrok_authtoken.yaml

services:
  ngrok:
    image: docker.io/ngrok/ngrok:latest
    volumes:
    - ./ngrok.yaml:/etc/ngrok/ngrok.yaml
    secrets:
    - ngrok_authtoken.yaml
    command:
    - --config
    - /run/secrets/ngrok_authtoken.yaml,/etc/ngrok/ngrok.yaml
    - start
    - webserver

  webserver:
    image: docker.io/traefik/whoami:latest

And now I can access the webserver container using an ngrok-generated url.