Python RQ, how do I mantain an instance when running a custom worker?

24 Views Asked by At

I have some problems using Python RQ library

I have a flask application that enqueues a task, this task receives data to be used in a ML model. Here is an example:

# task.py
from utils.models import model

def dispatch_task(data):
    return model.predict(data)

model is an instance of a class that has the trained model, this model is trained in the __init__ method.

Model run training again every time I enqueue a task, this caused by the fork in the RQ library.

I found in the documentation that I can import my modules before the worker.work() method is called https://python-rq.org/docs/workers/#performance-notes

#!/usr/bin/env python
from redis import Redis
from rq import Worker

# Preload libraries
import library_that_you_want_preloaded

# Provide the worker with the list of queues (str) to listen to.
w = Worker(['default'], connection=Redis())
w.work()

But this doesn't solve my problem because dispatch_task can't find model instance.

How can I make the instance of the model before and use it inside dispatch_task?

0

There are 0 best solutions below