Here is my model.py code
from config import config
from prompts import get_vlm_prompt, get_llm_prompt
import torch
from transformers import (
BitsAndBytesConfig,
InstructBlipProcessor,
InstructBlipForConditionalGeneration,
)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
double_quant_config = BitsAndBytesConfig(load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16)
Here is my configuration (requirements.txt)
fastapi==0.103.2
langchain==0.0.311
multion==0.2.2
openai==0.27.10
Pillow==10.0.1
pydantic==2.4.2
python-dotenv==1.0.0
torch==1.13.1
transformers==4.33.3
sentencepiece==0.1.99
accelerate==0.23.0
bitsandbytes==0.41.1
pydantic-settings==2.0.3
python-multipart==0.0.6
Running this command
uvicorn app:app --reload
While making the call I am getting this error
ImportError: Using load_in_8bit=True requires Accelerate: pip install accelerate and the latest version of bitsandbytes pip install -i https://test.pypi.org/simple/ bitsandbytes or pip install bitsandbytes
I already have accelerate
and bitsandbytes
, but still getting this error. What can I do here?