Listening to python huey events - run listener once?

Question

I try to incorporate python huey queue into my flask application, and I got it all working and all. I use it to run task in my workflow, and while the task is running I hide it from the user (add task_id from huey.task to my taskstatus in database) - otherwise it would seem as the same task is stuck, but in reality it's running in the background.

Now the tricky part is to show my task while the huey task is finished. I incorporated event listener (iterate through huey.storage) like in huey docs but from what I understand it subscribes to redis and run indefinitely (since there is periodic task check). So from what I understand the event listener itself has to run in separate thread, so I wrote huey task to listen to huey tasks:

@huey.task(include_task=True)
def get_huey_events(task):
    from huey.consumer import EVENT_FINISHED
    app = create_huey_app('development')
    with app.app_context():
        # store huey task id and name in database
        task1 = HueyTask(task.task_id, task.name)
        db.session.add(task1)
        db.session.commit()
        for event in huey.storage:
            if event['status'] == EVENT_FINISHED:
                # consume result to remove from storage
                result = huey.result(event['id'])
                # remove huey id from my task status - inidicates the task finished - my task will be shown to user 
                status = WorkflowProcessStatuses.query.filter_by(huey_id=event['id']).first()
                if status:
                    status.huey_id = None
                    db.session.add(status)
                    db.session.commit()

But putting it that way means that such task need to run only once - since it consumes one worker forever - soon there will be no more free workers. I start above get_huey_events task while create app factory is run like this:

with app.app_context():
    task1 = HueyTask.query.filter_by(name='queuecmd_get_huey_events').first()
    pipe = redis.StrictRedis()
    if task1:
        exists = pipe.hexists('huey.tasks', task1.id)
    else:
        exists = 0
    if task1 is None or not exists:
        if task1 and not exists:
            #clean old task if not running now
            pipe.hdel('huey.tasks', task1.id)
            db.session.delete(task1)
            db.session.commit()
        result = get_huey_events()
        pipe.hset('huey.tasks',result.task.task_id,'Event listener')

So i get my stored id of get_huey_events huey task and see if it is stored in redis database in my custom created huey.tasks name with the task.id key. If there is task in database but not in redis, or there is no task in database then i run event listener, otherwise not.

My question is - is there a better way to do this? Should event listener itself be a huey.task? I run it under windows for now.

Answer 1

You can just start a regular old thread to run the event listener if you wish. It doesn't need to be a task.