Laravel, Queue, Horizon and >10 Servers (Workers) - Creating several million jobs takes extremely long

Question

I need to create several 10 million jobs.

I have tried it with for-loops and Bus::batch([]) and unfortunately the creation of the jobs takes longer than the processing of the jobs by the 10 servers/workers. That means the workers have to wait until the job shows up in the database (redis etc). With redis-benchmark I could learn that Redis is not the problem.

Anyway... is there a way to create jobs in BULK (not batch)? I'm just thinking of something like:

INSERT INTO ... () VALUES (), (), (), (), ...

Anyway, creating several million jobs in a for-loop or in batch seems to be very slow for some reason. Probably because it's always just 1 query at a time and not an "upsert".

For any help I would be very grateful!

Answer 1

Writing a million records will be kind of slow at any condition. I'd recommend maximize your queue performance using several methods:

1. Create job that will create all other jobs if possible
2. Use only QUEUE_CONNECTION=redis for your queues as redis stores data in RAM which is fastest possible.
3. Create your jobs after response was processed already