loop device setup (losetup, mount, etc) fails in a container immediately after host reboot

Question

I'm trying to populate a disk image in a container environment (podman) on Centos 8. I had originally run into issues with accessing the loop device from the container until finding on SO and other sources that I needed to run podman as root and with the --privileged option.

While this did solve my problem in general, I noticed that after rebooting my host, my first attempt to setup a loop device in the container would fail (failed to set up loop device: No such file or directory), but after exiting and relaunching the container it would succeed (/dev/loop0). If for some reason I needed to set up a second loop device (/dev/loop1) in the container (after having gotten a first one working), it too would fail until I exited and relaunched the container.

Experimenting a bit further, I found I could avoid the errors entirely if I ran losetup --find --show <file created with dd> enough times to attach the maximum number of loop devices I would need, then detached all of those with losetup -D, I could avoid the loop device errors in the container entirely.

I suspect I'm missing something obvious about what losetup does on the host which it is apparently not able to do entirely within a container, or maybe this is more specifically a Centos+podman+losetup issue. Any insight as to what is going on and why I have to preattach/detach the loop devices after a reboot to avoid problems inside my container?

Steps to reproduce on a Centos 8 system (after having attached/detached once following a reboot):

$ dd if=/dev/zero of=file bs=1024k count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.00826706 s, 1.3 GB/s
$ cp file 1.img
$ cp file 2.img
$ cp file 3.img
$ cp file 4.img
$ sudo podman run -it --privileged --rm -v .:/images centos:8 bash

[root@2da5317bde3e /]# cd images
[root@2da5317bde3e images]# ls
1.img  2.img  3.img  4.img  file
[root@2da5317bde3e images]# losetup --find --show 1.img
/dev/loop0
[root@2da5317bde3e images]# losetup --find --show 2.img
losetup: 2.img: failed to set up loop device: No such file or directory
[root@2da5317bde3e images]# losetup -D
[root@2da5317bde3e images]# exit

exit
$ sudo podman run -it --privileged --rm -v .:/images centos:8 bash

[root@f9e41a21aea4 /]# cd images
[root@f9e41a21aea4 images]# losetup --find --show 1.img
/dev/loop0
[root@f9e41a21aea4 images]# losetup --find --show 2.img
/dev/loop1
[root@f9e41a21aea4 images]# losetup --find --show 3.img
losetup: 3.img: failed to set up loop device: No such file or directory
[root@f9e41a21aea4 /]# losetup -D
[root@f9e41a21aea4 images]# exit

exit
$ sudo podman run -it --privileged --rm -v .:/images centos:8 bash

[root@c93cb71b838a /]# cd images
[root@c93cb71b838a images]# losetup --find --show 1.img
/dev/loop0
[root@c93cb71b838a images]# losetup --find --show 2.img
/dev/loop1
[root@c93cb71b838a images]# losetup --find --show 3.img
/dev/loop2
[root@c93cb71b838a images]# losetup --find --show 4.img
losetup: 4.img: failed to set up loop device: No such file or directory

Answer 1

The only other method I can think of, beyond what you've already found is to create the /dev/loop devices yourself on boot. Something like this should work:

modprobe loop  # This may not be necessary, depending on your kernel build but is harmless.
major=$(grep loop /proc/devices | cut -c3)
for index in 0 1 2 3 4 5
do
  mknod /dev/loop$i b $major $i
done

Put this in /etc/rc.local, your system's equivalent or otherwise arrange for it to run on boot.

Answer 2

I know it's a little old but I've stumbled across similar problem and here what I've discovered:

After my vm boots up it does not have any loop device configured and it's ok since mount can create additional devices if needed but:

it seems that docker puts overlay over /dev so it won't see any changes that were done in /dev/ after container was started so even if mount requested new loop devices to be created and they actually were created my running container won't see it and fail to mount because of no loop device available.

Once you restart container it will pick up new changes from /dev and see loop devices and successfully mount until it run out of them and try to request again.

So what i tried (and it seems working) I passed /dev to docker as volume mount like this

docker -v /dev:/dev -it --rm <image> <command> and it did work.

If you still have this stuff I was wondering if you could try it too to see if it helps.