'asyncio' with OpenAI API Call Hangs After Extended Run Time

Question

I'm using asyncio alongside the OpenAI API to translate a set of texts concurrently. Initially, everything works as expected, and I see the answers from OpenAI printed in the console. However, after running for a while, the code seems to hang. Subsequent answers from OpenAI are not printed in the console, and I don't see any "retrying" messages either. Here's my code:

import asyncio
from aiohttp import ClientSession
import openai
import os
openai.api_key = os.getenv("OPENAI_API_KEY")

async def _atranslate(sem, messages, **model_kwargs):
    max_retry = 2
    async with sem:
        while max_retry > 0:
            try:
                response = await openai.ChatCompletion.acreate(
                    messages=messages,
                    **model_kwargs
                )
                answer = response.choices[0]['message']['content']
                print(answer)
                return answer
            except Exception as e:
                print(e)
                await asyncio.sleep(5)
                max_retry -= 1
                print('retrying...')
        raise ConnectionError('cannot reach openai!')

async def atranslate(text_list: list, source=None, target='English', max_workers=3, **model_kwargs):
    aio_session = ClientSession()
    openai.aiosession.set(aio_session)
    model_kwargs.setdefault('model', 'gpt-3.5-turbo')
    model_kwargs.setdefault('temperature', 1)
    model_kwargs.setdefault('timeout', 10)
    template = 'Translate the following {source} text into {target}:{text}'
    semaphore = asyncio.Semaphore(max_workers)
    tasks = []
    for text in text_list:
        messages = [{
            'role': 'user',
            'content': template.format(
                source=source,
                target=target,
                text=text
            )}
        ]
        tasks.append(asyncio.create_task(_atranslate(semaphore, messages, **model_kwargs)))
    results = await asyncio.gather(*tasks)
    await aio_session.close()
    return results

if __name__ == '__main__':
    textList = '... (some texts are omitted for brevity)'
    translations = asyncio.run(atranslate(textList*20, 'Korean', 'English',30))

When I run the above code, it starts off well but after some time, it simply hangs. What could be causing this? Are there any solutions or suggestions to address this issue?

I have tried to change the timeout parameter or simply not raise ConnectionError, but it doesn't work.I think it might be the problem of api usage limits, but I don't know why it doesn't throw any error.

Answer 1

Two things, first I would wrap the use of ClientSession in try...finally, or use it as a contextmanager as shown here.

Also, if you are pretty sure that there are some exceptions getting swallowed somewhere, try to catch BaseException instead of Exception. Yes, there is a BaseException class which except Exception: won't catch, and the asyncio library throws exceptions derived from BaseException sometimes.