Can I reduce number of requests made to redis data store when using celery and beat

Question

I am using on a project Django + celery + beat. Is there a way how to reduce number or requests made to redis data store?

THe project runs on fly.io. Part of the fly.toml

[processes]
  app = "gunicorn --bind :8000 zzrec.wsgi"
  beat = "celery -A zzrec beat -l info -S django"
  celery = "celery -A zzrec worker -l INFO"

The project - dev version - is running on free tier on fly. But I am getting ResponseError: max daily request limit exceeded. Limit: 10000, Usage: 10000. message from upstash https://upstash.com/docs/redis/troubleshooting/max_daily_request_limit

So I want to find out if it is possible to modify settings of celery or beat so it will not thit 10.000 requests.

I need celery to run "background" tasks but expecially for dev environment it would be good enough if the are "checked" every minute or so. Not every second.

UPDATE I can see that someone already voted to close this question.. So to make sure

• I am not the author of the code of the working project mentioned above
• I am owner of the project but I am not a programer
• I DID Google search and did not find answer to my question
• the programer that set up Django + celery + beat does not know answer to that question

Answer 1

Check first if adjusting the CELERY_BEAT_SCHEDULE could help, especially if your main goal is to manage the frequency at which tasks are scheduled and reduce the number of requests made to Redis. (a bit like in "How do I run periodic tasks with celery beat?")
Since each task execution potentially involves communication with Redis (for task state management and result storage), reducing the frequency of tasks should lead to a proportional reduction in Redis requests.
Adjusting the beat schedule is generally straightforward and does not require deep changes to your application's codebase. It is a matter of configuration changes rather than code changes.

CELERY_BEAT_SCHEDULE = {
    'less_frequent_task': {
        'task': 'myapp.tasks.some_task',
        'schedule': crontab(minute='*/10'),  # Every 10 minutes
    },
    # Other tasks
}

That configuration would schedule some_task to run every 10 minutes, reducing the frequency from a potential default of running every minute or even more frequently.

The CELERY_BEAT_SCHEDULE configuration is typically done within your Django project's settings, most often the settings.py, which contains various configurations for your Django application.

myproject/
│
├── myproject/
│   ├── __init__.py
│   ├── settings.py      # Django settings file
│   ├── urls.py
│   └── wsgi.py
│
├── myapp/               # Your Django app
│   ├── migrations/
│   ├── models.py
│   ├── tasks.py         # Celery tasks
│   └── ...
│
├── manage.py
└── ...

In your project's __init__.py, make sure Celery is initialized. This is typically done by creating a new Celery instance and loading the settings from Django:

# myproject/__init__.py

from celery import Celery
from django.conf import settings
import os

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')

app = Celery('myproject')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

When you start your Django project, you also need to run Celery Beat alongside your Celery workers:

celery -A myproject beat -l info

To process the tasks, you need to run the Celery worker:

celery -A myproject worker -l info

---

Overriding the is_due method of a custom schedule class allows you to define your own logic for determining when a task is due to run next. This can be a powerful feature, especially if you need a level of control that goes beyond static schedules defined in CELERY_BEAT_SCHEDULE.

You have to define your custom schedule class. That class should inherit from celery.schedules.schedule or an appropriate base class, and you should override the is_due method to implement your custom scheduling logic.

myproject/
│
├── myproject/
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
│
├── myapp/
│   ├── migrations/
│   ├── models.py
│   ├── tasks.py
│   ├── schedules.py  # Custom schedule class defined here
│   └── ...
│
├── manage.py
└── ...

With schedules.py:

from celery.schedules import schedule

class MyCustomSchedule(schedule):
    def __init__(self, run_every):
        super(MyCustomSchedule, self).__init__(run_every)

    def is_due(self, last_run_at):
        # Implement your custom scheduling logic here
        # Return a tuple (is_due, next_time_to_run_in_seconds)
        ...

Next, use instances of your custom schedule class in the CELERY_BEAT_SCHEDULE configuration. You create an instance of your custom class and pass it as the schedule parameter.

from datetime import timedelta

CELERY_BEAT_SCHEDULE = {
    'my_custom_scheduled_task': {
        'task': 'myapp.tasks.my_task',
        'schedule': MyCustomSchedule(timedelta(minutes=10)),
        # other task options
    },
    # other scheduled tasks
}

Make sure that your Celery application is correctly loading the configuration. If you are using a Django project, this typically means ensuring your settings.py is properly set up and that your celery.py file correctly loads these settings.

After defining your custom schedule class, you can import it in your settings.py and use it in CELERY_BEAT_SCHEDULE.

# myproject/settings.py
from myapp.schedules import MyCustomSchedule
from datetime import timedelta

# Celery configuration
# ...

CELERY_BEAT_SCHEDULE = {
    'my_custom_scheduled_task': {
        'task': 'myapp.tasks.my_task',
        'schedule': MyCustomSchedule(timedelta(minutes=10)),
        # other task options...
    },
    # other scheduled tasks...
}

After making these changes, restart your Celery worker and beat processes to pick up the new schedule.