How could I detect the completion of a RDMA read / write

Question

I am new in RDMA. Now, I am learning to use RDMA read/write. If a client posts a write/read to a server. How could the client know whether the write/read is successfully complete? In other words, how to know writes have been applied to the server, and how to know data have been read from the server.

I learn RDMA with the tutorial in https://github.com/jcxue/RDMA-Tutorial. It detects the completion by polling two memory locations, start and end.

while ((*msg_start != 'A') && (*msg_end != 'A')) {
}

Is it only this way to detect completion of a write/read? Any other way without polling the data in memory?

Thanks!

Answer 1

I'm not able to find the code you're referencing in the github repo you linked, but in any case the code is incorrect - the RDMA adapter may write data into memory from RDMA operations in any order, so it is possible that the first byte and the last byte of the buffer are filled in but the middle of the buffer has not been transferred yet. (Although in practice e.g. Mellanox adapters do have stronger ordering than is strictly required by the spec)

The right way for a client to check that an RDMA operation has completed is to poll for a completion. RDMA operations are submitted to send queues, and every send queue has a completion queue (CQ) attached to it. When the RDMA operation completes, a completion will be generated and added the that CQ, and the client can poll the CQ to see if it is there.