Time consumption in Memory Optimize table

Question

I have created one demo like- first created one temp table, then one In Memory table, then some insertion query to insert into the temp table, then used the same insertion query to insert into the In Memory table. Before running each queries I've cleared the buffers. Finally found that the time consumption for In Memory table is always greater than that of Temp table by 2-4secs. But it shouldn't be. So will anyone please tell me why it is happening like this? Thanks in advance.