How to consolidate celery logs from different computers. CELERY

442 Views Asked by At

How do I get logs from different workers in a single file in Celery. I have 3 workers running python tasks and a master node which runs the broker. I want to consolidate logs from these worker machines and store it in the master machine. How can I do that?

1

There are 1 best solutions below

0
On

We have all our Celery workers on AWS EC2 instances configured to upload Celery logs to CloudWatch. This is one way to achieve what you want. It is not difficult to implement this kind of system even if you are not on AWS - all you need is an agent running on each worker machine that periodically uploads Celery logs to central place. It can even be a cron-job running a script that does the job for you.