Mautic: 5.2.2
PHP: 8.3
We’re only sending segment emails at this time, and to do this I’m using messenger:consume email cron to process my doctrine queue.
I have emails loaded in messenger_messages, but no matter what I try for --limit and --time-limit I can’t seem to get anything more than an average of .6 emails per second when calculating out how many are cleared over time.
I’m using DuoCircle as our SMTP relay, and trying to determine where the bottleneck is as I see no evidence of memory or cpu issues.
@dirk_s I’ve using your script, keeping the defaults that were set to 14 email per second, I’ve tried tweaking a few things as well, but can’t seem to get any difference in actual throughput regardless of settings.
I’m in the process of setting up a RabbitMQ option in case that will help, I have it running on a dev server but need to clear this queue first.
I guess I’m mostly just not clear on where the issue is, at this point I suspect it’s just the long handshake with smtp and it doesn’t seem to send any more than a single email in a batch at all.
Our previous system was able to send send different emails in parallel, but that doesn’t seem to be a solid option that I can see without danger of duplicate sends. The current queue of 33,000 emails is made up of 9 separate emails being sent to 9 individual segments. Is it possible to have each email use a separate worker somehow in parallel?
I do no have a dev site with rabbitmq to test out. I’ve noticed that queueing messages into rabbit only happens at 2.0/s so that’s pretty slow as well.
I ran a test to queue messages directly and got
Published 1000 messages in 0.022 seconds
Average: 45454.55 msg/sec
So, I’m still chasing bottlenecks here, wondering if my expectations are maybe not inline with what Mautic can do?
I was able to process a sample email of 1420 messages, I got the send speed using rabbit up to .9 per second, so only a slight improvement. My dev server has 12 cores and 64GB memory, so I don’t think this is a resource issue, as nothing is showing full utilization.
Any input welcome on what are fair expectations for mautic performance and where I might look to make adjustments.
We just completed some testing over the last few days and I was meaning to come back and up date this as well -
I read some other posts that indicated that multiple crons would end up sending duplicate emails, so that made me a little nervous. I tried using some of the suggested scripts out there that manage things with a lock file and flock, but those weren’t increasing our send rate. I’m not sure if the duplication issue was only when using the standard queue settings?
We only send segment emails at the moment, so my understanding is that the only cron that works here is the messenger:consume email. boadcast:send doesn’t appear to read from the messenger_messages table? Or at least when I had that cron running, no emails went out.
Since I moved to rabbitMQ instead of the messenger_messages table, I started using supervisor to spin up additional workers. With this we were able to get up to 3 or 4 emails a second running 3 or 4 workers.
I would assume that I could accomplish the same things with the multiple crons you suggested with rabbitMQ and still avoid duplicates as well, but I like the additional control with supervisor, at least as we are still testing out Mautic.
I’ve actually hit exactly the same wall on larger campaigns in Mautic 5+.
Tried the usual stuff: Redis, RabbitMQ, Doctrine queue… they all end up doing basically one-by-one processing. On the same box, especially during big sends, those queues start chewing serious RAM and CPU, and everything else suffers.
Weirdly enough, going back to how Mautic 4 handled it with the filesystem spool (those .message files) was actually way faster in raw throughput — the only real downside was crappy locking that let duplicates sneak in if you ran multiple processes.
So I figured: why not revive that idea but fix the flaws and make it better for modern Mautic?
Ended up coding a small plugin that adds a custom file-based messenger transport:
Proper file locking → no duplicates, even with parallel workers
Batches messages on read/write (configurable size, like 100 or 500) → kills the 1-at-a-time bottleneck
Barely uses any extra resources — no heavy services, no big memory footprint
Still works with standard messenger:consume email
Threw in a few extra commands that mimic the old Mautic:emails:send style (run once, exit when empty, locked so you can’t double-start)
On my setups it ends up a lot faster than Redis/RabbitMQ while sipping way less CPU/RAM. For really big ones I even have a simple bash supervisor script that watches load + queue depth and scales workers up/down automatically.
It’s production-tested now but still a bit rough around the edges (needs cleaner config, better docs, strip some debug lines…). If you’re interested in trying it or poking at the code, let me know — can share via GitHub once I tidy it a bit more.
Hi @esio interesting! Where exactly will your plugin hook into the process? Is it the message workers filling the queue? Consumer stays as before, but reads from the files?
Is the plugin replacing core files or adding a new way to handle messages - as in setting a new way to queue messages?
ok, let me explain. The plugin hooks in exactly where Symfony Messenger expects a transport - so producers (the parts of Mautic that generate messages like sending emails or tracking hits) keep working the same way, they just write to files instead of the DB table when you set the DSN to filesystem://…
The consumer stays 100% the standard messenger:consume email (or hit, etc.) - no core files touched or overridden. It just reads from the filesystem directories (var/queue/email/*.message etc.) using Symfony’s FilesystemTransport under the hood, with some extra atomic locking to make sure multiple workers don’t step on each other. So you get safe parallel processing without the DB polling/locking nightmare.
It’s basically a new transport option you activate via Queue Settings or config, then clear cache and you’re good. Nothing gets replaced in core, just a new way to handle queuing.
One extra thing I added that might be useful: there’s support for running the consumer in a “legacy-ish” mode where it processes everything available and then exits cleanly instead of hanging around forever polling an empty queue. That way you can use custom supervisor which can spin up/down workers based on queue length without them eating CPU/idle forever (similar as I do).
It may work on Mautic 5/6 without any changes, as it is very simple code but I didn’t tested it as I already migrated from Mautic 4 to 7.1 RC directly.
Just a quick update.
I’ve added a dedicated supervisor command to the plugin as well (so now it is all-in-one solution to dispatch email queue efficiently without 3rd party queue engines).
The command is:
mautic:emails:supervisor
It works as a smart, auto-scaling process manager for your email workers. Instead of manually configuring a fixed number of messenger:consume email processes, this command starts with an initial number of threads (default: 1) and dynamically scales them up to a configurable maximum (default: 8) based on current send rate, system load average, and memory usage.
Key features:
Monitors server load and memory in real time and prevents spawning new threads if limits are exceeded (e.g. max load 4.0 or 80% memory usage).
Restarts individual worker threads after a set time limit or memory threshold to keep things fresh and stable.
Supports dynamic rate-based scaling so it adds more threads when there’s a backlog and scales down when the queue is light.
Configurable via local.php settings (initial threads, max threads, check interval, settle time, etc.).
Logging of scaling decisions (can be enabled/disabled) and optional benchmark mode.
Runs in the foreground - easy to wrap with systemd, nohup, or existing process manager.
Its entire logic is directly based on the bash supervisor script I’ve been running in my production environment for years to manage outgoing email processes. It behaves very similarly: it watches the queue and system resources, intelligently spins workers up or down, and avoids idle processes constantly polling an empty queue.
You can run it with extra verbosity like this:
php bin/console mautic:emails:supervisor -vv
I believe these kinds of rate limits should be handled directly by the sending transport itself, instead of at the Messenger level.
For AWS SES, I’m using the etailors_amazon_ses plugin from pm-pmaas:
It automatically adjusts the sending speed and batch sizes according to your actual AWS account limits and works nicely with Mautic 7.
The original version had some trouble when running multiple workers in parallel. BuggerSee created PR #130 that adds shared token bucket rate limiting (making it safe for multiple workers) and fixes a few other things:
I’m running with that PR applied, so I didn’t need to build any custom limiting on my own.
In my setup I haven’t hit any SES limits yet. My account allows 2 million emails per day plus 170 per second, which is pretty hard to reach in normal use. I just limit the number of parallel processes (via shared supervisor implementation) based on my own benchmarks of the real average speed. Most of the time I’m not exceeding 50% of the per-second quota, so it runs smoothly and stable.
