The TLDR of the problem is that if I do a 4 KiB write to a newly created file (test opens with O_CREAT and unlinks the file beforehand) and then a 4 KiB read, the read fails with EOPNOTSUPP. If the file already exists, there's no problem. Also, not all write->read patterns exhibit this problem (https://github.com/DataDog/glommio/issues/605#issuecomment-1734156888) and for whatever reason btrfs doesn't seem to repro this issue (but ext4, xfs, and f2f do). The disks I've tested this on generally have a single dm-crypt partition & then LVM partitions within that (the root disk has an extra EFI partition).
I have a very simple test case written up in https://github.com/DataDog/glommio/issues/605. I haven't been able to build a repro against liburing directly but mainly because I'm not 100% sure about what leads to the situation so it's hard to write an independent repro to confirm everything is the same to definitively determine if this is an issue within glommio or some config screwed up in my machine. That being said, I'm doubtful the problem is within glommio given how simple the test case is and that the same code works fine on other machines.
I instrumented the Linux kernel a bit and I think I've tracked the EOPNOTSUPP being generated at submit_bio_noacct in https://elixir.bootlin.com/linux/v6.5.5/source/block/blk-core.c#L732 where the BIO struct is carrying the REQ_NOWAIT but the bdev doesn't support it? At first I thought it had something to do with me disabling read/write workqueues for dm-crypt but booting with them enabled didn't seem to change the issue. From the values I've instrumented, the write->read patterns that pass seem to have the same nowait flag set on the read path so I'm assuming the read is finding success before submit_bio_noacct is invoked.
Does anyone know what might be causing this or even any tips on what I can instrument in the Linux kernel to get more insight? Or any tips on how to trace the invocation path of the functions within the kernel instead of having to constantly rebuild the linux kernel with new printfs & reboot would also be a great (have never done this before and while I assume eBPF would be the right tool, I haven't been able to find a beginner guide for it that I could extend to do what I want from it).
I've reproduced this so far on 6.1, 6.4, and 6.5 (6.4 & 6.5 are what I was running when I hit this issue while 6.1 is a random older kernel I picked - not sure if there's a better candidate to try it with).