How to Improve Source Document Relevance in a Langchain ConversationalRetrievalQAChain Implementation?

Question

I am currently working on a project where I have implemented the ConversationalRetrievalQAChain, with the option "returnSourceDocuments" set to true. The system works perfectly when I ask specific questions related to the VectorStore database, as it returns matching sources. However, when I pose generic questions like "Is the earth round?" or "How are you today?" it returns unrelated sources that don't align with the query. It appears that the vector store always returns documents, even if they don't match the query.

I'm seeking guidance on how to enhance the relevance of the source documents retrieved by the Langchain ConversationalRetrievalQAChain. Are there specific tools or techniques within the Langchain framework that can help mitigate this behavior, or is it necessary to develop a manual process to assess document relevance? How can I effectively limit the retrieval of unrelated source documents in this scenario?

Here is relevant part of the code:

async init(): Promise<void> {
        try {
            this.driver = await this.getQdrantDriver()
            this.retriever = await this.createRetrieverFromDriver()
            this.chat = new ChatOpenAI({ modelName: aiConfig.modelName })
            this.chain = await this.createQAChain(this.chat)

            this.questionGenerationChain = await this.createQuestionGenerationChain()
            this.conversation = new ConversationalRetrievalQAChain({
                retriever: this.retriever,
                combineDocumentsChain: this.chain,
                questionGeneratorChain: this.questionGenerationChain,
                returnSourceDocuments: true,
            })
        } catch (error) {
            Logger.error(error.message)
            throw error
        }
    }

    private async createQuestionGenerationChain(): Promise<LLMChain> {
        const { default: Prompt } = await import('App/Models/Prompt')
        return new LLMChain({
            llm: this.chat,
            prompt: await Prompt.fetchCondensePrompt(),
        })
    }

    private async createRetrieverFromDriver(): Promise<VectorStoreRetriever<QdrantVectorStore>> {
        return this.driver.asRetriever(qdrantConfig.noResults ?? 5)
    }

    private async getQdrantDriver(embeddings = new OpenAIEmbeddings(), collectionName: string | null = null): Promise<QdrantVectorStore> {
        const { default: Ingest } = await import('App/Models/Ingest')
        return new QdrantVectorStore(
            embeddings,
            {
                url: qdrantConfig.qdrantUrl,
                collectionName: collectionName ?? await Ingest.lastCollection(),
            },
        )
    }

    private async createQAChain(chat: BaseLanguageModel<any, BaseLanguageModelCallOptions>): Promise<StuffDocumentsChain> {
        const { default: Prompt } = await import('App/Models/Prompt')
        return loadQAStuffChain(chat, {
            prompt: await Prompt.fetchQuestionPrompt(),
        })
    }

Answer 1

Okay I solved this problem by using Contextual compression retriever and EmbeddingsFilter. It's not perfect solution but in most cases it mitigates the unwanted resources.

https://js.langchain.com/docs/modules/data_connection/retrievers/how_to/contextual_compression/