FARM-Haystack Install downgrade Pytorch Causes CUDA Incompatibility

Question

When running Haystack with GPU I am getting the following error. After digging into it, I realize that Haystack is downgrading Pytorch to a version that isn't compatible with my CUDA.

NVIDIA GeForce RTX 3060 with CUDA capability sm_86 is not compatible with the current PyTorch installation. The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70. If you want to use the NVIDIA GeForce RTX 3060 GPU with PyTorch, please check the instructions at

Steps to reproduce Haystack downgrading Pytorch

Create a fresh environment with conda Install Pytorch 1.13 (pip3 install torch torchvision torchaudio) Check Torch and Cuda versions

>>> torch.__version__
'1.13.0'
>>> torch.version.cuda
'11.7'

Install Haystack (pip3 install 'farm-haystack[docstores-gpu,faiss-gpu]'). The installer using an old version of Torch Collecting torch<1.13,>1.9 Using cached torch-1.12.1-cp38-cp38-manylinux1_x86_64.whl (776.3 MB) Checking Torch and Cuda versions

>>> torch.__version__
'1.12.1+cu102'
>>> torch.version.cuda
'10.2'

Answer 1

There are two solutions

1. Using the light-the-torch to install Pytorch 1.12 with CUDA 11.6
2. Installing from source and forcing Pytorch version by modifying the versions.

1. Using light-the-torch

a. Create an environment b. Install light-the-torch (ltt) https://pypi.org/project/light-the-torch/ c. Installing Pytorch with the version Haystack support using ltt. Ltt will install the right CUDA version.

ltt install torch==1.12.1

d. Check the version of Pytorch and CUDA

torch.__version__
'1.12.1+cu116'
torch.version.cuda
'11.6'

2. Installing From Source

Here is the solution I come up with installing from the source and configuring the Pytorch version.

Installing FARM and specifying Pytorch version

git clone https://github.com/deepset-ai/FARM.git
cd FARM
Edit requirement.txt to point to the right PyTorch torch>1.12.2,<1.14

pip install -r requirements.txt
pip install --editable .
cd ...

Installing Haystack forces it to use a specific Pytorch version

git clone https://github.com/deepset-ai/haystack.git
cd haystack
Edit pyproject.toml to point to the right PyTorch torch>1.12.2,<1.14

pip install --upgrade pip
pip install -e '.[docstores, docstores-gpu, faiss, faiss-gpu]'

Checking Torch version

>>> import torch
>>> torch.__version__
'1.13.0+cu117'
>>> torch.version.cuda
'11.7'