Notify or callback flask after remote celery worker is completed

Question

I am running celery client(Flask) and worker in two different machines, now once the worker has completed the task, I need to callback a function on client side. Is this possible?

Celery client:-

celery_app=Celery('test_multihost', broker='amqp://test:test@<worker_ip>/test_host', backend='rpc')
result= testMethod1.apply_async((param1, param2,param3), link=testMethod2.s())

@celery_app.task
def testMethod2():
    #testMethod2 body.

Celery Worker:-

celery_app=Celery('test_multihost', broker='amqp://test:test@<worker_ip>/test_host', backend='rpc')
@celery_app.task
def testMethod1():
   #testMethod1 body

But the problem is the function testMethod2 is getting executed on the celery worker side, not on the client side.

Is there anyway that I can callback the method on client side?

Answer 1

I solved this problem using @after_task_publish from celery signals. The code snippet is as follows:-

@after_task_publish.connect(sender=<registered_celery_task>)
def testMethod2(sender=None, headers=None, body=None, **kwargs):
    #callback body

The testMethod2 will be called after the celery worker is completed on the remote machine. Here I can access the result of celery worker using headers parameter.

Answer 2

One way to do this is to have Celery write its result in a database table, and use Flask to poll for the result of the task by repeatedly querying the database. A similar construct might be to keep a register of completed tasks in Redis, but the gist would be the same.

Do you want to trigger a completion message to the user? If you can notify by email/text message, you could just let Celery handle that of course.

If you need to kickstart some Flask process - and it really needs to be inside Flask's ecosystem for some reason - use the worker with the requests module to call to an endpoint that Flask is listening to.