Best approach for PHP mass-mailing routine?

Question

I'm creating a site that sends out a daily news email to about 800 users, at a time they can specify. My problem is that my script takes a long time to run and times out, so I'm looking for some advice on how I could be approaching this better.

Current approach:

Users are placed in a 'mailing queue' database table with their ID, receive time, and a sent flag.

I'm then running a CRON script every minute which does the following:

1. Grab all from mailing queue with a 'receive time' less than or equal to now, that haven't sent
2. Loop through the users, joining a preferences table to get their chosen categories (up to about 30 per user).
3. For each category, find the latest 3 articles
4. Prepare an HTML email with this content using PHPMailer
5. PHPMailer is using Mailgun SMTP to avoid overloading my SMTP server
6. Send mail to user, mark as sent in database

My observations so far are:

1. When testing the script by running in-browser, it runs incredibly slowly for a few minutes then times out (without sending any emails).
2. When running every minute via CRON, it sends way over the number of emails (about 1400) over the course of 40 minutes, I guess because the script is overlapping itself and the sent flag is not reliably updated.

The majority of users are set to receive their email at the same time, so I'm doing 'worst case scenario' testing on this basis

Questions

1. Is my script far too heavy, by querying the database and generating the HTML email content for each user on the fly? I'm wondering if it would be better to generate the content ahead of time and store against the user in the mailing queue.
2. Would a queue manager like Beanstalkd help? I've had a look into it, but am struggling to see how to implement into my routine.

Ultimately I need the emails to be sent reliably to each user at the time they expect.

Any advice much appreciated!

Answer 1

You can do this in PHP, but you probably shouldn't. You're trying to build a mail server when there are much better ways of doing this, which mainly involve using a mail server.

Sending high volumes of email during page loads is not workable – it can be troublesome even for single messages, yet many still try. Approach it like this:

• Store your list in a database.
• When you want to send, generate a record representing each message to send (essentially a copy of the list).
• Have a daemon (a long-running task) or cron job that sends the messages in chunks.
• Create messages one at a time and submit them to a local mail server.
• Use DKIM signatures.
• As each message is sent, mark them as sent in the database, but you need to be very aware of how database transactions and locks work for this to work safely and avoid duplicates – do this right and overlapping processes work just fine.
• You can generate messages as fast as you like, and your mail server will deal with queuing, onward delivery, retries, bounces.
• Use VERP addressing, and feed bounces into a bounce handler (be warned, writing these is not fun!), and have that prevent sending to bouncing addresses in future.

This approach works well – it is exactly how my own system ([Smartmessages.net](https://info.smartmessages.net/ , which is built in PHP) works, and I can sustain over 200 messages per second using multiple message generators running in parallel (database transactions FTW!).

If you find all this a bit too much (it is very difficult), you're probably better off using a commercial sending service (like my own) or hosting a DIY solution, such as Mailcoach by my good friends at Spatie. Either of these would work well, and your list is pretty small – I'm often handling lists of over 100,000.