Efficient recurring tasks in celery?

1k Views Asked by At

I have ~250,000 recurring tasks each day; about a fifth of which might be updated with different scheduled datetimes each day.

Can this be done efficiently in Celery? - I am worried about this from celery's beat.py:

    def tick(self):
        """Run a tick, that is one iteration of the scheduler.

           Executes all due tasks.

       """
        remaining_times = []
        try:
            for entry in values(self.schedule):
                next_time_to_run = self.maybe_due(entry, self.publisher)
                if next_time_to_run:
                    remaining_times.append(next_time_to_run)
        except RuntimeError:
            pass

        return min(remaining_times + [self.max_interval])
0

There are 0 best solutions below