I've been struggling lately in how to get o2ib to function properly with a particular MOFED version. What I've tried so far is to install the Lustre kernel, rebuild MOFED for that Lustre kernel (which appears to be working) and then observe that ib0 is listed upon a reboot, and install the generic Lustre kmod-lustre kmod-lustre-osd-ldiskfs lustre-osd-ldiskfs-mount lustre lustre-resource-agents. However just because ib0 is there, does not mean that o2ib presents itself in Lustre. Even running "lnetctl net add --net o2ib --if ib0" gives nothing but errors that the interface cannot be found.
I have tried rebuilding Lustre serveral times to try and get the o2ib interface, but to no avail. The rpms are built, but when I install them the situation is no better. My process is as follows (for Lustre 2.12):
git clone git://git.whamcloud.com/fs/lustre-release.git cd lustre-release git checkout 2.12.0 sh autogen.sh ./configure --with-o2ib=/usr/src/ofa_kernel/default/ make rpms
Would anyone have any suggestons?
Thanks!
Actually you should do the other way around: you should compile Lustre to properly use your MOFED and your kernel. This is the order of dependencies:
Your kernel (e.g: 3.10.0-1127.8.2.el7.x86_64)
Your MOFED has to be compiled for your kernel. If your kernel is one of the ones available from Mellanox you just need to install the rpms or let the MOFED installer do it for you:
If you're using a different kernel you need to recompile MOFED (you need to install kernel-devel for this) with support for your kernel:
Now your MOFED is ready to run on top of your kernel and your Lustre RPMs are ready to run on top of your kernel and the o2ib driver will use the symbols compiled for your MOFED.