How can I process a request to AzureChatOpenAI that is too big?

Question

I'm trying to perform a question to AzureChatOpenAI that is very big. Therefore I'm getting the following error:

openai.BadRequestError: Error code: 400 - {'error': {'message': "This model's maximum context length is 8192 tokens. However, your messages resulted in 37571 tokens (37332 in the messages, 239 in the functions). Please reduce the length of the messages or functions

I'm working on Python. How can I get rid of this error? Is there a way to send this data in chunks, so all the info can be sent but at the same time, all the message is processed as a single one?

Answer 1

When dealing with large requests in AzureChatOpenAI, you might encounter limitations due to the model’s maximum context length.

Token Limit Error: The error message you received indicates that the model’s maximum context length is 8192 tokens.

However, your messages resulted in 37571 tokens (37332 in the messages and 239 in the functions), which exceeds the limit.

Divide your input data into smaller chunks (subsets of tokens).

Process each chunk separately and then combine the results.

This way, you can stay within the token limit while still processing the entire message.