Using `load_in_8bit=True` requires Accelerate: `pip install accelerate`

626 Views Asked by At

Here is my model.py code

from config import config
from prompts import get_vlm_prompt, get_llm_prompt

import torch

from transformers import (
    BitsAndBytesConfig,
    InstructBlipProcessor, 
    InstructBlipForConditionalGeneration,
)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
double_quant_config = BitsAndBytesConfig(load_in_4bit=True,
                                            bnb_4bit_quant_type="nf4",
                                            bnb_4bit_use_double_quant=True,
                                            bnb_4bit_compute_dtype=torch.bfloat16)

Here is my configuration (requirements.txt)

fastapi==0.103.2
langchain==0.0.311
multion==0.2.2
openai==0.27.10
Pillow==10.0.1
pydantic==2.4.2
python-dotenv==1.0.0
torch==1.13.1
transformers==4.33.3
sentencepiece==0.1.99
accelerate==0.23.0
bitsandbytes==0.41.1
pydantic-settings==2.0.3
python-multipart==0.0.6

Running this command

uvicorn app:app --reload

While making the call I am getting this error

ImportError: Using load_in_8bit=True requires Accelerate: pip install accelerate and the latest version of bitsandbytes pip install -i https://test.pypi.org/simple/ bitsandbytes or pip install bitsandbytes

I already have accelerate and bitsandbytes, but still getting this error. What can I do here?

0

There are 0 best solutions below