RDMA write and RDMA write with immediate

Question

I am trying to get more details on the RDMA write semantics. I have the following questions on the same.

1. How the RDMA write and RDMA write with immediate works?
2. If I use the RDMA write operation, can I read data from the local buffer on the peer side?
3. Can we access the data that is stored in DDR memory? How to map virtual address to physical address and Is there any Infiniband API for the same?

Thanks.

Answer 1

1. How the RDMA write and RDMA write with immediate works?

All RDMA semantics are defined in the IB specification. Briefly, an RDMA write with immediate is a network message that consumes a receive WQE, writes data to the destination MKEY, and generates a CQE with the immediate data. Note that even if no data is written to the receive WQE, the WQE is consumed because a CQE with the immediate data is generated.

2. If I use the RDMA write operation, can I read data from the local buffer on the peer side?

No, RDMA write takes local buffer and writes it to a remote buffer. To read the remote buffer, you must perform an RDMA read.

3. Can we access the data that is stored in DDR memory?

Yes, but the software semantics don't care if data is in DDR, cache are any other memory mapped with an RDMA MKEY.

How to map virtual address to physical address?

This is done using the ibv_reg_mr verbs API.