Hyperledger Indy data is not being mounted in Kubernetes volume directory

Question

I am trying to run indy-nodes in kubernetes. These indy nodes are sandbox nodes and write data in /var/lib/indy directory inside the container. When I run the pod with a volume mounted, it does not write anything in the volume directory. Although it creates a directory inside the volume, it is empty all the time. However, when I create a pod without a volume mount option, the container writes data inside /var/lib/indy.

Following is the Dockerfile:
Hastebin: https://hastebin.com/hitinefizi.nginx

Kubernetes Deployment:

{{- $root := .}}
{{- range .Values.indy}}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  # namespace: {{$root.Values.namespace}}
  name: {{.name}}
spec:
  selector:
    matchLabels:
      name: {{.name}}
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        name: {{.name}}
    spec:
      containers:
        - name: {{.name}}
          image: {{.image}}
          volumeMounts:
            - name: {{$root.Values.pv.metadata.name}}
              mountPath: "/var/lib/indy/sandbox"
              subPath: "volume/indy/{{.name}}/sandbox"
          ports:
            - containerPort: {{ index .ports 0 }}
            - containerPort: {{ index .ports 1 }}
      nodeSelector:
        nodeType: {{$root.Values.hosts.blockchain}}
      volumes:
        - name: {{$root.Values.pv.metadata.name}}
          {{- if eq $root.Values.storage.type "nfs" }}
          persistentVolumeClaim:
            claimName: {{$root.Values.pvc.metadata.name}}
          {{- else }}
          hostPath:
            path: /var/kubeshare/
          {{- end }}
{{- end}}

The directory inside volume:

[root@centos1 kubeshare]# tree volume/indy/
volume/indy/
|-- indy-node1
|-- indy-node2
|-- indy-node3
`-- indy-node

The directory /var/lib/indy inside the container without volume:

root@indy-node1-587c4758bf-2hpp6:/var/lib/indy# tree -L 3
.
|-- plugins
`-- sandbox
    |-- data
    |   `-- Node1
    |-- domain_transactions_genesis
    |-- keys
    |   |-- Node1
    |   |-- Node1C
    |   |-- Node2
    |   |-- Node3
    |   `-- Node4
    |-- node1_additional_info.json
    |-- node1_info.json
    |-- node1_version_info.json
    `-- pool_transactions_genesis

I am not sure why it is happening. Any help/suggestions would be appreciated.

Update: This is the same thing happening with docker-compose when I try to use local volume.

Answer 1

Mounting in docker is consistent with standard behaviour of mounting on Linux. Linux mount command docs say

The previous contents (if any) and owner and mode of dir become invisible, and as long as this filesystem remains mounted

This is as well the way things work in Docker. If you mount a local directory, or an existing named docker volume, the content of filesystem in the container on the location of the mount will be shadowed (or we can call it "overriden").

Simplified example of what is going on

Having dockerfile

FROM alpine:3.9.6

WORKDIR /home/root/greetings
RUN echo "hello world" > /home/root/greetings/english.txt
CMD sleep 60000

And build it docker build -t greetings:1.0 .

Now create following docker-compose.yml:

version: '3.7'

services:
  greetings:
    container_name: greetings
    image: greetings:1.0
    volumes:
      - ./empty:/home/root/greetings

and create empty directory empty next to it.

Start it docker-compose up -d. While the container is running, let's get into container and see what the filestructure inside looks like. docker exec -ti greetings sh. Now when we are inside, if you run ls /home/root/greetings you'll see that the directory is empty - even though in the Dockerfile we have baked file /home/root/greetings/english.txt into the image's filesystem.

Named docker containers behave more desirably, if the named docker container is new and doesn't contain any data. If you mount such container on location in container where there already is some data, the named volume will get this data copied on it.

You can try this by adjusting the docker-compose.yml to this

version: '3.7'

services:
  greetings:
    container_name: greetings
    image: greetings:1.0
    volumes:
      - greetingsvol:/home/root/greetings

volumes:
  greetingsvol:
    driver: local

and if you repeat the procedure and exec yourself into the container, you'll see that file /home/root/greetings/english.txt is still there.

That's because when you cd yourself into /home/root/greetings, you are not looking at actual container's filesystem, but at mounted device - the name docker volume - which has been initialized by copy of container's original files on that given location. (Assuming docker volume greetingsvol did not previously exist.)

Solution to your problem

You are mounting directory /var/kubeshare on your host to container's /var/lib/indy/sandbox. Let's see what the container stores on that location on startup (indypool is how I named built indy sandbox image on my localhost)

docker run --rm indypool ls /var/lib/indy/sandbox
domain_transactions_genesis
keys
pool_transactions_genesis

So if you mount your local directory onto /var/lib/indy/sandbox, it will shadow these files and the pool will fail to start up (and therefore consequently won't create files such as node1_additional_info.json etc).

So I think you have 2 options:

1. Unless you have a good reason not to, use named docker volumes.
2. Copy the original image data from container's /var/lib/indy/sandbox into your /var/kubeshare. Then you keep everything else as was. That way, the directory will be shadowed by new filesystem containing exactly the same data as the container expects to find there.