How to efficiently setup broadcasts:send and messenger:consume?

Product Support
1

My Mautic version is: 5.1.0
My PHP version is: PHP Version 8.1.2-1ubuntu2.19
My Database type and version is: 10.6.18-MariaDB-0ubuntu0.22.04.1

I send about 50k emails every day. They are created via API and then sent using this cron job:

*/4 * * * * sudo -u www-data php /opt/mautic/bin/console mautic:broadcasts:send --channel=email --limit=700 --batch=700

Additionally, I have a queue worker running with systemd. It repeatedly executes this command:

/opt/mautic/bin/console messenger:consume email --time-limit=3600 --memory-limit=512M --limit=700

The emails are sent, but with the above settings the table messenger_messages keeps filling up. I have not found any documentation on this, but I suppose that the emails that are “sent” by the cron command are actually pushed, images and all, into this table and the messenger:consume command is supposed to read and process (e.g. send to SMTP provider) them from there.

What happens is that the table will, in the course of 1-2 hours, grow about 60GB and then everything crashes because space is full. I will have to manually clear and optimize the table to make it work again.

I suppose that my workflow is fine but that the settings are faulty. I have tried multiple workers with no effect. Also, I can’t find a way to actually log how many emails are processed by a worker.

When I put it very small values (like 100 emails every five minutes), this doesn’t happen and everything gets sent just fine. But then sending the daily newsletter will take about 40 hours.

If you read this far, please consider commenting. Any idea will help :slight_smile: Thanks

2

Its a combination of your server speed, the sending speed etc.

You are right - if you use queuing, messages will be placed in that table.

This cronjob will send them.
https://docs.mautic.org/en/5.x/configuration/cron_jobs.html#process-email-queue-cron-job

You specified, that your job runs up to 1h or up to 700 pieces of mail. Depending on how you set it up, at the same time you start 11 other instances of your cronjob (each 5 min right?). Maybe you run into trouble with your STMP interface (rate limit?).

How many mails do you send to your newsletter?

Some inspiration: Mautic cron script with mail rate limit - Mautic v5 · GitHub

I needed to run a script, that makes sure it will not send more than 14 mails per second (AWS SES Limit).

1 Like
3

Hey Dirk,

thank you for your answer!

I set up the cronjob to send 700 emails per 4 minutes. I set up the systemd service to consume emails continuously, but I don’t know how to find out its processing speed.

My SMTP provider will allow me to send 200.000 emails per hour. I intend to send 50.000 emails per day.

My VM has 16GB RAM and 4 vCPUs that are never, even during sending, using more than one third of capacity.

Do you know a way to discover how many emails are processed by the messenger:consume job? I understand that I need to balance the settings with my hardware, but I don’t know how.

4

I also think that your hardware looks decently sized…

You probably could start issuing the commands on your server shell and track times. Maybe enhance a script with some output to console to see the progress. You may then identify the bottleneck.

With my script I quickly found out, that it already took more than a second to send those 14 mails and I won’t hit the rate limit this way…

5

If Doctrine struggles with high volumes, consider switching to a more scalable queue system like Redis or RabbitMQ.