docker-registry disk full and no ideas to diet it

Question

I have a server with a docker registry, and have pushed a lot of times a build the same :latest tag now my DD is full and I can't get how to diet it.

disk is full

df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        48G   45G  397M 100% /
udev             10M     0   10M   0% /dev
tmpfs           794M   81M  713M  11% /run
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/dm-1       9.8G   56M  9.2G   1% /var/lib/docker/devicemapper/mnt/2e895760700ac3e1575e496a4ac6adde4de6129226febba8c0c3126af1655ad9
shm              64M     0   64M   0% /var/lib/docker/containers/5aa47e34d1b8be22deeae473729b4e587e6e4bfe7fb3e262eda891bad4b05042/shm

there is no dangling volume nor images

# docker volume ls -qf dangling=true
# 
# docker images -f "dangling=true" -q
#  

docker images

[root@kvm22:/etc/cron.daily] # docker images 
REPOSITORY                               TAG                 IMAGE ID            CREATED             SIZE
jwilder/nginx-proxy                      0.5.0               72b65b5a6f38        4 weeks ago         248.4 MB
registry                                 2                   c9bd19d022f6        11 weeks ago        33.27 MB
registry                                 2.5                 c9bd19d022f6        11 weeks ago        33.27 MB

disk usage

 # du -h -d 7 /var/lib/docker/volumes/
12K /var/lib/docker/volumes/24000fbe2e81da06924be8f7ce81e07101824036bca5f87d4d811f2a6f7bfa7b/_data
16K /var/lib/docker/volumes/24000fbe2e81da06924be8f7ce81e07101824036bca5f87d4d811f2a6f7bfa7b
42G /var/lib/docker/volumes/registry_docker-registry-volume/_data/docker/registry/v2/blobs/sha256
42G /var/lib/docker/volumes/registry_docker-registry-volume/_data/docker/registry/v2/blobs
5.9M    /var/lib/docker/volumes/registry_docker-registry-volume/_data/docker/registry/v2/repositories/labor-prod
5.9M    /var/lib/docker/volumes/registry_docker-registry-volume/_data/docker/registry/v2/repositories
43G /var/lib/docker/volumes/registry_docker-registry-volume/_data/docker/registry/v2
43G /var/lib/docker/volumes/registry_docker-registry-volume/_data/docker/registry
43G /var/lib/docker/volumes/registry_docker-registry-volume/_data/docker
43G /var/lib/docker/volumes/registry_docker-registry-volume/_data
43G /var/lib/docker/volumes/registry_docker-registry-volume
43G /var/lib/docker/volumes/

Output of docker version:

# docker --version 
Docker version 1.12.4, build 1564f02

Output of docker info:

# docker info 
Containers: 4
 Running: 1
 Paused: 0
 Stopped: 3
Images: 5
Server Version: 1.12.4
Storage Driver: devicemapper
 Pool Name: docker-8:1-1184923-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: ext4
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 1.058 GB
 Data Space Total: 107.4 GB
 Data Space Available: 3.036 GB
 Metadata Space Used: 2.142 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.145 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.90 (2014-09-01)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options:
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.873 GiB
Name: kvm22
ID: G6OC:EKKY:ER4W:3JVZ:25BI:FF2Y:YXVA:RZRR:WPAP:SASB:AJJA:DM6J
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
Insecure Registries:
 127.0.0.0/8

Answer 1

In case anyone still has this problem: This is the 'reset' way how I solved it:

You can stop and delete the registry

docker stop registry && docker rm -v registry 

and restart it afterwards:

docker run -d -p 5000:5000 --restart=always --name registry registry:2

Then you would have to rebuild your images localy and push them to the registry again.

Answer 2

Found another tool here. https://github.com/burnettk/delete-docker-registry-image

Included a clean_old_version.py script. I give it a try too.

Answer 3

I had the same problem. I can't believe there's no ready solution for this. Anyway I hacked a tool together and it seems to work.

You can find it here: https://github.com/Richie765/docker-tools

Basically it uses a bash script to find out which manifests are untagged. Then deletes them through the registry API. Afterwards you can run a garbage collection to actually delete the data.

I'm sure the script isn't perfect. Any improvements are welcome!