Using Custom LLM to choose between doc retrieval (on SQL) or answering directly

Question

Is there a way to make LLM automatically choose between using a function to get information (Retrieval QA) or answering question directly based on his knowledge.

Eg: Suppose I am making a airline bot, where customer provides his PNR/Email for verification. I have made it to make query to get the details of customer. (Used langchain)

But for further answering the question how can I make it answer directly to skip the document retrieval when it's not required. Eg: I say I ask how he is doing. (Just a general question for llm, and to make bot for human like)

In this question doc retrieval is not required.

I have used langchain, for doc retrieval over structured data (SQL) and over unstructured data (pdfs of terms).

I am trying to use open source LLMs like Llama or any other (trying to avoid openai completely)

Answer 1

in this scenario, you can use RunnableBranch in langchain: and your question will route in 2 or more LLM

the first llm should identify the topic, if the topic is related to the document, llm should simply say document with no further words then you have to parse the output using langchain and check if the output was document pass it to the other LLM which have access to details to your customers.

here is how the first prompt should look like:

const promptTemplate =
  PromptTemplate.fromTemplate(`Given the user question below, classify it as either being about \`docuemnt\`, \`airline\`, or \`Other\`.
                                     
Do not respond with more than one word.

<question>
{question}
</question>

Classification:`);

and after the question was answered you can parse it and choose your LLM for it:

const model = new ChatAnthropic({
  modelName: "claude-2",
});

const classificationChain = RunnableSequence.from([
  promptTemplate,
  model,
  new StringOutputParser(),
]);

const classificationChainResult = await classificationChain.invoke({
  question: "how are you?",
});
console.log(classificationChainResult);
// console.log: Other

now that you know the topic, you can use if-else statements or whatever you like to do with it.

If you provide me with the code, I can offer more accurate assistance.