Scalability Challenges with Laravel and Redis Queue Worker for Email Verification

Question

I have developed an email verification tool using Laravel, hosted on EC2 instances with t3.xlarge, auto scaling, and load balancing. The verification process involves a Redis queue worker and Laravel Horizon for job management.

Bulk Email Verification Logic:

1. When a user uploads a list of 10k emails,
2. I dispatch a StartBulkEmailVerificationJob.
3. This job utilizes Laravel's job batching to dispatch
4. VerifyChunkBulkEmailVerificationJob, with each job processing a chunk of 35 emails.
5. For a 10k-email list, this results in 286 VerifyChunkBulkEmailVerificationJob jobs being dispatched.

Challenge:

The current setup works smoothly for a single user, but when multiple users simultaneously upload 10k-email lists, subsequent users have to wait for the completion of the previous users' jobs.

Specific Questions:

1. Are there any recommended strategies or tools to optimize the parallel processing of bulk email verification jobs for multiple users in a Laravel + EC2 setup?

2. Are there additional features or configurations in Laravel Horizon or Redis queue that I may not be aware of, which could improve parallel processing and reduce waiting times for users?

Conclusion:

I'm looking for insights, recommendations, or best practices to enhance the scalability of my email verification tool in a multi-user scenario. Any advice on optimizing the existing setup or introducing new tools/strategies would be greatly appreciated. Thank you!