Docker overlay2 directory does not get clean

Question

We use jenkins as our CI/CD tool and kubernets as its agent. Each time our pipeline is triggered a pod in our k8s cluster comes up and starts executing our scripts. The container contains docker engine and cli to build our docker images. To speed up our pipelines we have a volume attached to container at /var/lib/docker. We only need to keep cache data for our builds. To do so we use docker-buildx-plugin.

At the end of our pipeline we run the following command:

docker buildx prune -f -a --keep-storage=5G
docker image prune -f -a

However after a while the docker volume gets filled, even though docker system df shows that docker only stores about 5G of data:

# docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          0         0         0B        0B
Containers      0         0         0B        0B
Local Volumes   0         0         0B        0B
Build Cache     213       0         4.236GB   4.236GB

However docker storage uses more than that:

# df -hT
Filesystem     Type     Size  Used Avail Use% Mounted on
overlay        overlay  194G   93G  102G  48% /
tmpfs          tmpfs     64M     0   64M   0% /dev
/dev/vda1      ext4     194G   93G  102G  48% /etc/hosts
tmpfs          tmpfs     63G  8.0K   63G   1% /root/.ssh
shm            tmpfs     64M     0   64M   0% /dev/shm
/dev/vdb       xfs       14G   12G  2.1G  86% /var/lib/docker
tmpfs          tmpfs     63G   12K   63G   1% /run/secrets/kubernetes.io/serviceaccount

# du -sh /var/lib/docker/*
78M     /var/lib/docker/buildkit
72K     /var/lib/docker/containerd
0       /var/lib/docker/containers
4.0K    /var/lib/docker/engine-id
55M     /var/lib/docker/image
44K     /var/lib/docker/network
12G     /var/lib/docker/overlay2
0       /var/lib/docker/plugins
0       /var/lib/docker/runtimes
0       /var/lib/docker/swarm
0       /var/lib/docker/tmp
24K     /var/lib/docker/volumes

And the above mentioned commands not working:

# docker image prune -f -a
Total reclaimed space: 0B
# docker buildx prune -f -a --keep-storage=5G
Total:  0B

Also there are not images or containers or other volumes:

# docker container ls -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

#docker image ls -a
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE

# docker volume ls   
DRIVER    VOLUME NAME

If I run docker system prune everything including cache data will get deleted. Can any one please help?

Other information:

# docker version
Client: Docker Engine - Community
 Version:           25.0.3
 API version:       1.43 (downgraded from 1.44)
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:14:17 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:32:12 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

# docker buildx version
github.com/docker/buildx v0.11.2 9872040