I have some problems using Python RQ library
I have a flask application that enqueues a task, this task receives data to be used in a ML model. Here is an example:
# task.py
from utils.models import model
def dispatch_task(data):
return model.predict(data)
model is an instance of a class that has the trained model, this model is trained in the __init__ method.
Model run training again every time I enqueue a task, this caused by the fork in the RQ library.
I found in the documentation that I can import my modules before the worker.work() method is called https://python-rq.org/docs/workers/#performance-notes
#!/usr/bin/env python
from redis import Redis
from rq import Worker
# Preload libraries
import library_that_you_want_preloaded
# Provide the worker with the list of queues (str) to listen to.
w = Worker(['default'], connection=Redis())
w.work()
But this doesn't solve my problem because dispatch_task can't find model instance.
How can I make the instance of the model before and use it inside dispatch_task?