How to combine ConversationalRetrievalQAChain, Agents, and Tools in LangChain

Question

I'd like to combine a ConversationalRetrievalQAChain with - for example - the SerpAPI tool in LangChain.

I'm using ConversationalRetrievalQAChain to search through product PDFs that have been ingested using OpenAI's embedding API and a local Chroma vector DB. This works fine. However, the product PDFs don't have up-to-date pricing information. So when the user asks for pricing info, I'd like LangChain to use the SerpAPI tool to google for the price. I have both pieces working separately, but I'd love to combine them.

Here's the document search piece (keep in mind: this is PoC-quality code):

// Prompt used to rephrase/condose the question
const CONDENSE_PROMPT = `Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question.

Chat History:
{chat_history}

Follow Up Input: {question}

Standalone question:`;

// Prompt for the actual question
const QA_PROMPT = `You are a helpful AI assistant for sales reps to answer questions about product features and technicals specifications.
Use the following pieces of context to answer the question at the end.
If you don't know the answer, just say you don't know. DO NOT try to make up an answer.
If the question is not related to the context, politely respond that you are tuned to only answer questions that are related to the context.

{context}

Question: {question}
Helpful answer:`;

export const POST: RequestHandler = async ({ request }) => {
  const { messages } = await request.json();
  const { stream, handlers } = LangChainStream();

  const openAIApiKey = OPENAI_API_KEY;
  const embeddings = new OpenAIEmbeddings({ openAIApiKey });

  // This model is used to answer the actual question
  const model = new ChatOpenAI({
    openAIApiKey,
    temperature: 0,
    streaming: true,
  });

  // This model is used to rephrase the question based on the chat history
  const nonStreamingModel = new ChatOpenAI({
    openAIApiKey,
    temperature: 0,
  });

  const store = await Chroma.fromExistingCollection(embeddings, {
    collectionName: 'langchain',
  });

  const chain = ConversationalRetrievalQAChain.fromLLM(
    model,
    store.asRetriever(),
    {
      returnSourceDocuments: true,
      verbose: false,
      qaChainOptions: {
        type: "stuff",
        prompt: PromptTemplate.fromTemplate(QA_PROMPT)
      },
      questionGeneratorChainOptions: {
        template: CONDENSE_PROMPT,
        llm: nonStreamingModel,
      },
    }
  );

  const callbacks = CallbackManager.fromHandlers(handlers);
  const latest = (messages as Message[]).at(-1)!.content;

  chain.call({ question: latest, chat_history: (messages as Message[]).map((m) => `${m.role}: ${m.content}`).join('\n') }, callbacks).catch(console.error);

  return new StreamingTextResponse(stream);
};

This is the SerpAPI tool code:

export const POST: RequestHandler = async ({ request }) => {
  const { messages } = await request.json();
  const { stream, handlers } = LangChainStream();

  const openAIApiKey = OPENAI_API_KEY;

  // This model is used to answer the actual question
  const model = new ChatOpenAI({
    openAIApiKey,
    temperature: 0,
    streaming: true,
  });

  // Define the list of tools the agent can use
  const tools = [
    new SerpAPI(SERPAPI_API_KEY, {
      location: "Austin,Texas,United States",
      hl: "en",
      gl: "us",
    }),
  ];

  // Create the agent from the chat model and the tools
  const agent = ChatAgent.fromLLMAndTools(model, tools);

  // Create an executor, which calls to the agent until an answer is found
  const executor = AgentExecutor.fromAgentAndTools({ agent, tools });

  const callbacks = CallbackManager.fromHandlers(handlers);
  const latest = (messages as Message[]).at(-1)!.content;

  executor.call({ input: latest }, callbacks).catch(console.error);

  return new StreamingTextResponse(stream);
};

On their own, they work fine. How do I combine them?

This is the workflow I'm imagining:

1. User's question comes in.
2. LangChain decides whether it's a question that requires an Internet search or not.
3. If it does, use the SerpAPI tool to make the search and respond.
4. If it doesn't require an Internet search, retrieve similar chunks from the vector DB, construct the prompt and ask OpenAI.
5. Either path should end up in the chat history, and possibly the chat history should be utilized in either case, so that the user could ask "How much does in cost?" and the implied product can be known based on the chat history.

Is that possible? Thank you!

Answer 1

My good friend Justin pointed me in the right direction. LangChain has "Retrieval Agents". The idea is that the vector-db-based retriever is just another tool made available to the LLM. So in my example, you'd have one "tool" to retrieve relevant data and another "tool" to execute an internet search. Based on the user's input, the LLM would pick which tool to use.

More details and sample code here: https://js.langchain.com/docs/use_cases/question_answering/conversational_retrieval_agents