We're porting Isis2 (isis2.codeplex.com) to make better use of Infiniband verbs and have our code running. However, IB is oriented around an asynchronous receive model in which you post a bunch of receive buffers and then, as receives complete, you process the received data.
Polling is slow: if I use a blocking wait for, say, 2ms, I might delay as long as 2ms before seeing the IB data. So that's a solution, but a poor one. What I really want is a way to wait until an IB completion record is finalized and then to have my thread wake up instantly (on Windows this is easy... on Linux it isn't as natural). Does anyone know how one does this? When using Verbs, there isn't any IB file descriptor, so obviously I can't use select()
Never mind; we just realized that they offer a method (ibv_reg_notify_cq) for this. We'll try that. Not the world's best documented API...