Celery prefork multiple workers

Question

I am using the celery prefork infrastructure in order to run 4 concurrent worker in my service. The service runs in an AWS Task Definition which write logs to a log stream that is dedicated to that task. I wanted to implement a mechanism that will make sure every worker will write to a different log stream which will be created when the worker starts, that way I can monitor the logs in a more scalable way and not get multiple logs from different workers in the same main log stream.

In order to do that I created handler functions in order to listen for events and perform my operations:

def configure_signals(child_process_amount: int = None) -> None:
    cs.setup_logging.connect(get_logger)
    if child_process_amount is not None and child_process_amount > 1:
        cs.worker_process_init.connect(add_logger)

def get_logger(loglevel=logging.WARN, **kwargs) -> logging.Logger:
    return init.get_logger()

group_name = os.environ.get('GROUP_NAME')
def add_logger(*args, **kwargs):
    global group_name
    logger = init.get_logger()
    internal_handler = lhf.LogHandlerFactory().create_log_handler(group=group_name)
    handler = internal_handler.get_log_handler()
    logger.addHandler(handler)

the create_log_handler function is creating a new log stream for each worker process that spawns up and its working as expected. the problem is that for some reason which I can't figure out there comes a time when the workers stop writing to their dedicated log stream and only write to the main log stream. This makes the logs not readable and impossible to maintain.

I tried looking in the logs to see if there was some sort of an error occurring when it stops writing. I did notice that I see a "celery@4fsafds32 missed heartbeat" log some time before the logs stop coming to the new stream. I don't know if this is the reason or not. Also sometimes the "celery@4fsafds32" is even referring to a celery app that wasn't created in this task but in another one. I know that because I see for example that when the celery app start it says "celery@hwehereg3" is ready, so its a different celery app even.