LightGBM not found

Question

I've encountered an error when trying to use LightGBM in my Python project. Whenever I attempt to import LightGBM in my code, I get the following error:

OSError: dlopen(/Users/hp/opt/anaconda3/envs/MachineLearning/lib/python3.9/site-packages/lightgbm/lib/lib_lightgbm.so, 0x0006): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
Referenced from: <3AF1EF0C-311C-31EC-BCE3-679F37ABEE16> /Users/hp/opt/anaconda3/envs/MachineLearning/lib/python3.9/site-packages/lightgbm/lib/lib_lightgbm.so
Reason: tried: '/usr/local/opt/libomp/lib/libomp.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/libomp/lib/libomp.dylib' (no such file), '/usr/local/opt/libomp/lib/libomp.dylib' (no such file), '/usr/local/lib/libomp.dylib' (no such file), '/usr/lib/libomp.dylib' (no such file, not in dyld cache)

I'm working in a conda environment named "MachineLearning" on macOS, and I've tried setting the LDFLAGS and CPPFLAGS environment variables as suggested in various sources, but the issue persists. Can someone please help me understand what's causing this error and provide steps to resolve it? I'm stuck at this point because I'm trying to use LightGBM for Streamlit and FastAPI projects.

Environment Details:

• Python Version: 3.9
• Conda Environment: "MachineLearning"
• Operating System: macOS

Any insights or guidance would be greatly appreciated. Thank you!

Answer 1

The error said OpenMP library is not installed. According to LightGBM doc, it is better to install LightGBM via brew install lightgbm

Answer 2

By default, distributions of the lightgbm Python package expect to be able to link to OpenMP at runtime. It appears you don't have OpenMP available at any of the standard paths that the linker will search.

Since you're already working in a conda environment, use conda / mamba to install lightgbm, not pip install. That way, you'll get conda-forge's distribution of OpenMP automatically (conda-forge/lightgbm-feedstock link) and not need to worry about installing it separately.

Run the following to fix your issue:

# remove pip-installed copy of 'lightgbm' in MachineLearning conda env
source activate MachineLearning
pip uninstall --yes lightgbm

# install conda-forge distribution instead
conda install \
   --yes \
   --name MachineLearning \
   -c conda-forge \
   'lightgbm>=4.1.0'

With that, you should not need to make any customizations to environment variables like LDFLAGS and lightgbm should load successfully in a Python process in that conda environment.

If it's absolutely necessary that you use pip install lightgbm or build lightgbm from source (e.g. to customize the build or to use unreleased versions), then install OpenMP separately in that conda environment:

conda install \
   --yes \
   --name MachineLearning \
   -c conda-forge \
   llvm-openmp