Can I use MKL libraries provided by Anaconda with Spark

Question

I want to use MKL libraries provided with Anaconda with Spark. I installed libgfortran

conda install libgfortran

linked libs

ln -s $PATH_TO_ANACONDA/lib/libmkl_rt.so  $PATH_TO_ANACONDA/lib/liblapack.so.3
ln -s  $PATH_TO_ANACONDA/lib/libmkl_rt.so $ PATH_TO_ANACONDA/lib/libblas.so.3 

and placed PATH_TO_ANACONDA/lib on LD_LIBRARY_PATH

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PATH_TO_ANACONDA/lib

but when I run Spark ALS example I see

WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK
WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK

so i think it didn't work. Is something clearly wrong with this?

Answer 1

You can use any library as long as you load it to the nodes. Use --packages with the libraries you would like to use