Mariadb in raid 10 NVME SSD slow read speeds

Question

It would seem to me that we have a bottleneck we just cant seem to get over.

We have a setup which contains 4 NVME drives in Raid 10

We are using mariadb 10.4

We have indexes

The workload that we have will 99% of the time be IO bound there is no way around that fact

What I have seen while watching the performance dashboard in mysql workbench is that both the SATA SSD and NVME SSD read at about 100MB for the same data set

Now if I am searching through 200M rows(or pulling 200M) I would think that the Innodb disk read would go faster then 100MB

I mean these drives should be capable of reading 3GB(s) so I would at least expect to see like 500MB(s)

The reality here is that I am seeing the exact same speed on the NVME that I see on the SATA SSD

So the question I have is how do I get these disk to be fully utilized

Here is the only config settings outside of replication

sql_mode                          = 'NO_ENGINE_SUBSTITUTION'
max_allowed_packet                = 256M
innodb_file_per_table
innodb_buffer_pool_size           = 100G
innodb_log_file_size              = 128M

innodb_write_io_threads = 16 // Not sure these 2 lines actually do anything 
innodb_read_io_threads = 16

Answer 1

IO bound there is no way around that fact

Unless you are very confident on the suitability of indexes this seems a little presumptuous.

Assuming your right, this would imply a 100% write workload, or a data size orders of magnitude higher that RAM available and a uniform distribution of small accesses?

innodb_io_capacity is providing a default limitation and your hardware is capable of more.

Also if you are reading so frequently, your innodb_buffer_pool_size isn't sufficient.