Openstack Rocky running on three nodes, ssds as root disks, jbod's for Ceph. Ceph is running on the same nodes as OpenStack.
Before configuring Ceph, all the instances created were ephemeral (as expected ) and I was able to see the disk in /var/lib/nova/instances and play with it.
After I configure and enable the Ceph and NFS backends in Cinder, I'm unable to create ephemeral instances, all go to ceph-as-block-storage or NFS (depending on the volume created).
If i create a flavor with 10G root and 10G ephemeral, get "/" as 10G on ceph and "/mnt" as 10G on ephemeral, but I can't get the root partition to be ephemeral.
In cinder.conf I have the order like this:
enabled_backends = ceph,nfs
Any ideas on how to enable ephemeral disks again ?
Ephemeral disks on instances depends of nova, even though there are a kind of volume, the instances using ephemeral disk are called nova boot also.
Ephemeral disks in real world are bonded to the compute node, because they run locally [1], but is not common on production environments to do that because you will not achieve HA and you can crash the compute node if you use all the space in the node, if you have a ceph storage backend you can use a specific pool for that purpose, I strongly recommend you to check this guide [2], in specific the part to configure libvirt on the ephemeral client secret [3].
Cheers.
[1] https://docs.openstack.org/ha-guide/storage-ha-backend.html
[2] http://docs.ceph.com/docs/master/rbd/rbd-openstack/
[3] http://docs.ceph.com/docs/master/rbd/rbd-openstack/#setup-ceph-client-authentication