LLM in LLMChain ignores prompt

Question

I'm getting an incorrect output from an LLMChain that uses a prompt that contains a system and human messages. Can't figure out why. I download the gpt4all-falcon-q4_0 model from here to my machine.

from langchain.chains import ConversationChain, LLMChain
from langchain.llms import GPT4All

from langchain.prompts import (
    PromptTemplate,
    SystemMessagePromptTemplate,
    HumanMessagePromptTemplate,
    ChatPromptTemplate
)

import sys

sys.path.insert(0,"..")
from local_config import MODEL_PATH

llm = GPT4All(
        model=MODEL_PATH,
        )

system_template = """
You are a helpful assistant who generates comma separated lists.
A user will only pass a category and you should generate subcategories of that category in a comma-separated list.
ONLY return comma-separated values and nothing more!
"""

system_message = SystemMessagePromptTemplate.from_template(
    system_template
)

human_template = '{category}'

human_message = HumanMessagePromptTemplate.from_template(human_template)

prompt = ChatPromptTemplate.from_messages([system_message, human_message])

chain = LLMChain(llm = llm,
                 prompt=prompt,
                 verbose=True)

chain.run('Machine Learning')

Instead of getting a string of comma-separated values, I get the following output:

Engineer\nSure, I can help you with that. Please provide the category for which you want to generate subcategories.

For reference, it does work if I just use a simpler template for the prompt:

template = """
Return all the subcategories of the following category
{category}
"""

prompt = PromptTemplate(input_variables=["category"],
                        template = template)

chain = LLMChain(llm = llm,
                 prompt=prompt,
                 verbose=True)

chain.run('Machine Learning')

Output

'Here\'s a Python function that takes in a list of categories and returns a list of all the subcategories of each category:\n```python\ndef get_subcategories(categories):\n    subcategories = []\n    for category in categories:\n        subcategories.append(category)\n    return subcategories\n```\nTo use this function, you can pass it a list of categories as an argument, like this:\n```python\ncategories = [\n    "Machine Learning",\n    "Deep Learning",\n    "Computer Vision",\n    "Natural Language Processing",\n    "Artificial Intelligence",\n    "Robotics",\n    "Game Development",\n    "Data Science",\n    "Big Data",\n    "Cloud Computing",\n    "Blockchain",\n    "Internet of Things",\n    "Cybersecurity",\n    "Machine Learning",\n    "Deep Learning",\n    "Computer Vision",\n    "Natural Language Processing",\n    "Artificial Intelligence",\n    "Robotics",\n    "Game Development",\n    "Data Science",\n    "Big Data",\n    "Cloud Computing",\n    "Blockchain",\n    "Internet of Things",\n    "Cybersecurity",\n]\n\nsubcategories = get_subcategories(categories)\nprint(subcategories)\n```\nThis'

Answer 1

The code looks right. But the GPT4all-Falcon model needs well structured Prompts. Here are 2 things you look out for:

Your second phrase in your Prompt is probably a little to pompous. I think youve to split it up a little to make the instructions easier to understand.

Maybe change it from:

""" You are a helpful assistant who generates comma separated lists. A user will only pass a category and you should generate subcategories of that category in a comma-separated list. ONLY return comma-separated values and nothing more! """

into:

""" You are a helpful assistant who generates comma separated lists. A Human will only pass a category. Generate subcategories of that category in a comma-separated list. Return the comma-separated list! """

Furthermore something what most people forget is to place \n (next lines) on the right places in a prompt. Having zero \n in your string can make a huge difference with this model to having an \n after a phrase for example. You can place \n either by pressing enter in your """String""" or write the string in one line like:

"""You are a helpful assistant who generates comma separated lists.\nA Human will only pass a category. Generate subcategories of that category in a comma-separated list.\nReturn the comma-separated list!\n"""


output:
* Data Science
* Artificial Intelligence
* Deep Learning
* Neural Networks
* Computer Vision
* Natural Language Processing
* Reinforcement Learning
* Game Theory
* Information Retrieval
* Knowledge Representation
......

good luck and happy prompting