For GPT4All, cannot seem to load downloaded model file. Getting error llama_init_from_file: failed to load model (bad f16 value 5)

Question

I am trying to use the following code for using GPT4All with langchain but am getting the above error:

Code:

import streamlit as st 

from langchain import PromptTemplate, LLMChain
from langchain.llms import GPT4All

from langchain.agents.agent_toolkits import create_python_agent
from langchain.tools.python.tool import PythonREPLTool

PATH = 'D:\Python Projects\LangchainModels\models\ggml-stable-vicuna-13B.q4_2.bin'
llm = GPT4All(model=PATH, verbose=True)

agent_executor = create_python_agent(
    llm=llm,
    tool=PythonREPLTool(),
    verbose=True
)

st.title(' GPT For Y\'all')

prompt = st.text_input('Enter your prompt here!')

if prompt: 
    response = agent_executor.run(prompt)
    st.write(response)

And the error traceback from the code being run:

llama_model_load: loading model from 'D:\Python Projects\LangchainModels\models\ggml-stable-vicuna-13B.q4_2.bin' - please wait ...
llama_model_load: n_vocab = 32001
llama_model_load: n_ctx   = 512
llama_model_load: n_embd  = 5120
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 40
llama_model_load: n_layer = 40
llama_model_load: n_rot   = 128
llama_model_load: f16     = 5
llama_model_load: n_ff    = 13824
llama_model_load: n_parts = 2
llama_model_load: type    = 2
llama_model_load: invalid model file 'D:\Python Projects\LangchainModels\models\ggml-stable-vicuna-13B.q4_2.bin' (bad f16 value 5)
llama_init_from_file: failed to load model

I have also given the same error to the gpt4all repository and no feedback yet. Is there any version dependencies? For example even though not document specified I know langchain needs to have >= python3.8 for it to be run successfully.

Answer 1

You might not have enough memory in order to run that model, I think vicuna model requires at least 16 GB of ram. maybe try something smaller like mini orca small?

Although if it is a memory error, you would expect it to tell you so, so maybe I am wrong.