Intermittent Hang in Python on openai.ChatCompletion.create() Method Call with GPT-3

Question

I am encountering a strange bug while using the openai Python library. Specifically, the method openai.ChatCompletion.create() intermittently hangs during execution without any error messages or exceptions being raised.

I am working on a script to process a dataset of 100 strings using a custom class named GPTHandler.

I'm running on windows 11, python 3.11.3, openai library v 0.28.1

The script reads strings from a spreadsheet, processes them, and sends them to the OpenAI GPT-3.5 Turbo model for text completion. Here's a simplified version of the setup:

import openai

class GPTHandler:
    def __init__(self):
        # Load base payload configuration from a config file
        # ...

    def use_gpt(self, string_to_be_processed):
        # Update the payload with the new string
        # ...
        payload = self._build_gpt_payload(string_to_be_processed)  
        # validated with logs this returns a valid payload on calls that hang
        # {
        #     "model": "gpt-3.5-turbo",
        #     "mesages": [
        #         {
        #             "role": "user",
        #             "content": "This is a test string."
        #         }
        #     ]
        # }
        try:
            print("Sending API request...")
            gpt_response = openai.ChatCompletion.create(**payload)
            print("API request complete.")
            return gpt_response
        except Exception as e:
            print(f"Exception: {e}")

# Load strings from spreadsheet
# ...

# Instantiate GPTHandler
gpt_handler = GPTHandler()

# Process each string
for string in strings:
    gpt_handler.use_gpt(string)

The issue occurs intermittently. The terminal prints "Sending API request..." but then hangs indefinitely without printing "API request complete." or raising any exception. Initially, I thought it was a delay, but even after waiting for 20 minutes, the hang persists. Notably, pressing Ctrl+C doesn’t terminate the script, and I have to kill the terminal to stop it. This hang happens on different strings and after a varying number of API calls.

I implemented a TimeoutTracer class to set a timeout using sys.settrace, hoping to interrupt the hang and gather more information.

class TimeoutTracer:
    def __init__(self, start, TOTAL_TIMEOUT):
        self.start = start
        self.TOTAL_TIMEOUT = TOTAL_TIMEOUT

    def trace_function(self, frame, event, arg):
        logger.debug(f"trace called at time: {time.time()}")
        if time.time() - self.start > self.TOTAL_TIMEOUT:
            logger.error(f"{'#'*20} ERROR {'#'*20}")
            logger.error(f"Timeout of {self.TOTAL_TIMEOUT} seconds exceeded!")
            raise Exception('Timed out!')
        return self.trace_function



class GPTHandler:
    def __init__(self):
        # ...

    def use_gpt(self, string_to_be_processed):
        payload = self._build_gpt_payload(string_to_be_processed)
        TOTAL_TIMEOUT = 60
        start = time.time()
        tracer = TimeoutTracer(start, TOTAL_TIMEOUT)
        sys.settrace(tracer.trace_function)
        try:
            print("Sending API request...")
            gpt_response = openai.ChatCompletion.create(**payload)
            print("API request complete.")
        except Exception as e:
            logger.error(f"Error: {e}")
            return "{}"
        finally:
            sys.settrace(None)  # Remove the time constraint and continue normally.
        # ... (process gpt_response)

# ... (rest of the script including loading strings and processing loop)

I had a log output for every time sys.settrace was called and found that when it hangs, sys.settrace stops being called, which suggests that no code is being executed during these hang periods. This might also explain why Ctrl+C isn’t working. Additionally, I am not hitting any rate limits as OpenAI would return a message in that case.

I am looking for insights or suggestions on what could be causing this hang and how to resolve or debug it further. Any help would be greatly appreciated!