4 bit quantization in Gemma-2b

Question

Source-:

Hugging-face documentation.

enter image description here

My code-:

enter image description here

My error log

ImportError: Using bitsandbytes 8-bit quantization requires Accelerate: pip install accelerate and the latest version of bitsandbytes: pip install -i https://pypi.org/simple/ bitsandbytes

Note I have already installed accelerate and bitsandbytes

But I still have one confusion the log say that for 8-bit quantisation I need accelerate and other package, but I am doing 4 bit quantization.

Iam Trying to quantize and expecting model to download

Answer 1

I got the same error, felt like the accelerate is actually a wrapper of Cuda, so still GPU required. With GPU, I can use 4bit quantization tho.