How do I manage different celery config files on various nodes in a cluster?

Question

I have a cluster of celery nodes with different celery configurations (celeryconfig.py & celery.py) along with different start & stop scripts. It is getting cumbersome to manage the different files.

Right now I hacked up my ansible automation to incorporate different config files but this seems dirty and will not scale.

Is there any way to handle all different configurations, a centralized way to manage the config files?

Any help appreciated, thanks!

Answer 1

Ansible is usually pretty good at this kind of thing.

There are several ways you can deal with this:

• have a per-host config file and get ansible to create a symlink /path/to/celeryconfig.py --> /path/to/configs/<hostname>-celeryconfig.py
• same as previous but on a per-group basis
• use ansible templates to tweak particular parts of a generic template config file depending on host/group/whatever
• etc...