Performance cost of MESI protocol?

Question

The MESI (Modified, Exclusive, Shared, Invalid) protocol is used for CPU caches to communicate and ensure they are all using the latest value for a cache line. When one CPU modifies a cache line value, all other CPUs subscribed to this cache line are kept alerted on the change to the cache line.

However, in all the literature I have read regarding MESI I haven't seen if there is any performance cost whilst the protocol communicates? Would this cost just be part of the x86 LOCK prefix cost? I am fairly certain MESI can be used even when the x86 LOCK prefix is not?

NB Intel actually uses the MESIF protocol- where F is an additional "Forwarding" state.

Answer 1

MESI protocol works by exchanging messages on the inter-CPU bus (however that bus is implemented). That bus has limited throughput capacity, so that one can saturate it by using atomic instructions.

This is why poorly written applications that needlessly use atomic instructions adversely affect an entire machine.

Answer 2

Yes, the MESI(F) protocol is used on all memory operations (ie reads and writes). Imagine, you've written something into the cache (ie 'M' state) and now that line has to be evicted. The protocol says you need to write it back to memory. If the protocol wasn't used, then we'd either need to always write-thru to memory (huge bandwidth cost) or have a inconsistent memory (bad idea).

That is, if there's no sharing, the MESI protocol would still be used and in such case, the lines would be in 'E', 'I', or 'M' states and no 'S' will be used.

A side note: remember that almost all applications on a system typically share some dynamic library code. Where do you think that code will reside and how its access will be managed?

Now, to answer your question on the performance impact. Yes, implementing MESI(f) or any coherence protocol will have an impact on performance, but that impact is actually positive when compared to the case where no coherence protocol exists. In that case, every read/write would need to go to main memory (ie, your application will be 100's of times SLOWER).

So, bottom line: Although, the MESI(f) protocol does have a negative impact on bandwidth, overall it has a positive impact on performance. It actually buys us alot of performance (and power) compared to the case where we don't use a cache coherence protocol (ie no cache)