I am using packer hcl to take a Rocky 8 AMI and create a new Rock 8 AMI that is partitioned with LVM2.
The build of the new AMI works and I get an AMI that I can boot but as soon as it comes up / is read-only. I can login via the console and make it RW very easily and then mount all the rest of the LVs. I have enabled systemd debug in the kernel boot line and when I look at the boot.log and journalctl I find errors except that chrony, hostname and sshd will not start as a service. The reason being is that the temp directories have not been created because / was read-only.
If I make / RW, mount the other LVs and then create the starting hostkeys for sshd it starts fine and I can ssh in.
Dmesg also shows nothing.
I thought it might have something to do with auditd and/or selinux so I disabled both in the kernel line. But no change.
Even after mounting things as RW and and doing a reboot it boots back in to the same state. I have validated the FSTab by eye and once / is RW I can run mount -a and everything mounts no errors no warnings.
Any clue where the problem could be? It seems to be very early in the boot process around where systemd switches from sysroot to /.
I was reading the code for amigen8 and for some reason started looking at the temp files and tmpfs. Thats when I noticed that the tmp files resource was not enabled in systemd and it was one of the first things systemd ran after switching from sysroot to /. Once I enabled it everything mounted correctly and all was read write.