I am trying to run indy-nodes in kubernetes. These indy nodes are sandbox nodes and write data in /var/lib/indy
directory inside the container. When I run the pod with a volume mounted, it does not write anything in the volume directory. Although it creates a directory inside the volume, it is empty all the time. However, when I create a pod without a volume mount option, the container writes data inside /var/lib/indy
.
Following is the Dockerfile:
Hastebin: https://hastebin.com/hitinefizi.nginx
Kubernetes Deployment:
{{- $root := .}}
{{- range .Values.indy}}
---
apiVersion: apps/v1
kind: Deployment
metadata:
# namespace: {{$root.Values.namespace}}
name: {{.name}}
spec:
selector:
matchLabels:
name: {{.name}}
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
name: {{.name}}
spec:
containers:
- name: {{.name}}
image: {{.image}}
volumeMounts:
- name: {{$root.Values.pv.metadata.name}}
mountPath: "/var/lib/indy/sandbox"
subPath: "volume/indy/{{.name}}/sandbox"
ports:
- containerPort: {{ index .ports 0 }}
- containerPort: {{ index .ports 1 }}
nodeSelector:
nodeType: {{$root.Values.hosts.blockchain}}
volumes:
- name: {{$root.Values.pv.metadata.name}}
{{- if eq $root.Values.storage.type "nfs" }}
persistentVolumeClaim:
claimName: {{$root.Values.pvc.metadata.name}}
{{- else }}
hostPath:
path: /var/kubeshare/
{{- end }}
{{- end}}
The directory inside volume:
[root@centos1 kubeshare]# tree volume/indy/
volume/indy/
|-- indy-node1
|-- indy-node2
|-- indy-node3
`-- indy-node
The directory /var/lib/indy
inside the container without volume:
root@indy-node1-587c4758bf-2hpp6:/var/lib/indy# tree -L 3
.
|-- plugins
`-- sandbox
|-- data
| `-- Node1
|-- domain_transactions_genesis
|-- keys
| |-- Node1
| |-- Node1C
| |-- Node2
| |-- Node3
| `-- Node4
|-- node1_additional_info.json
|-- node1_info.json
|-- node1_version_info.json
`-- pool_transactions_genesis
I am not sure why it is happening. Any help/suggestions would be appreciated.
Update: This is the same thing happening with docker-compose when I try to use local volume.
Mounting in docker is consistent with standard behaviour of mounting on Linux. Linux
mount
command docs sayThe previous contents (if any) and owner and mode of dir become invisible, and as long as this filesystem remains mounted
This is as well the way things work in Docker. If you mount a local directory, or an existing named docker volume, the content of filesystem in the container on the location of the mount will be shadowed (or we can call it "overriden").
Simplified example of what is going on
Having dockerfile
And build it
docker build -t greetings:1.0 .
Now create following
docker-compose.yml
:and create empty directory
empty
next to it.Start it
docker-compose up -d
. While the container is running, let's get into container and see what the filestructure inside looks like.docker exec -ti greetings sh
. Now when we are inside, if you runls /home/root/greetings
you'll see that the directory is empty - even though in the Dockerfile we have baked file/home/root/greetings/english.txt
into the image's filesystem.Named docker containers behave more desirably, if the named docker container is new and doesn't contain any data. If you mount such container on location in container where there already is some data, the named volume will get this data copied on it.
You can try this by adjusting the
docker-compose.yml
to thisand if you repeat the procedure and exec yourself into the container, you'll see that file
/home/root/greetings/english.txt
is still there.That's because when you
cd
yourself into/home/root/greetings
, you are not looking at actual container's filesystem, but at mounted device - the name docker volume - which has been initialized by copy of container's original files on that given location. (Assuming docker volumegreetingsvol
did not previously exist.)Solution to your problem
You are mounting directory
/var/kubeshare
on your host to container's/var/lib/indy/sandbox
. Let's see what the container stores on that location on startup (indypool
is how I named built indy sandbox image on my localhost)So if you mount your local directory onto
/var/lib/indy/sandbox
, it will shadow these files and the pool will fail to start up (and therefore consequently won't create files such asnode1_additional_info.json
etc).So I think you have 2 options:
/var/lib/indy/sandbox
into your/var/kubeshare
. Then you keep everything else as was. That way, the directory will be shadowed by new filesystem containing exactly the same data as the container expects to find there.