My Amazon SES was just approved out of sandbox. My sending rate is 14 emails per second. Apparently if I send faster than that the emails are rejected for exceeding service levels.
I’m wondering if I should set this in Mautic’s email configuration. “Message limit for queue processing” at 14 and “Time limit for queue processing” at 1 second.
Is that correct?
My Amazon SES was just approved out of sandbox. My sending rate is 14 emails per second. Apparently if I send faster than that the emails are rejected for exceeding service levels.
I’m wondering if I should set this in Mautic’s email configuration. “Message limit for queue processing” at 14 and “Time limit for queue processing” at 1 second.
Is that correct?
@darrelt Did you ever get an answer on this? Any other gotchas you experienced when timing into SES?
I’m wondering if Mautic automatically matches SES send rate? Or I should follow OPs settings? I also have 14 emails / sec limit and would like to know if I should do something in mautic to match this
Hi @ezzat !
There are different ways how to handle SES with Mautic, I will share you my practice, which I use for all our customers at Friendly Automate and which works like a Swiss clockwork. There are other good solutions as well, I’m sure someone will post their in this thread. (Always use the proper path, since I’m not paying attention to it in this guide)
Too fast sending
If you send too fast, your campaign will fall into a cliff. That means you start sending, depending on the speed first 500-1000 emails are delivered, and the rest just goes into Valhalla, as won’t be deleviered nor will you get a message from Mautic, that something is wrong. You’ll see these messages with delivered status in Mautic, never received status in Amazon SES, and there is no way to know what got sent out and what didn’t. (At leas no easy way.)
I address this the following way:
1. Queue is your friend
As I’m sending thousands of emails, I’m using the queue. You need to turn it on in the Mautic settings.
Configuration > Email Settings > Queue processing
You can use the above settings, we will overwrite them anyway in the crontab. (See later)
When you create an email, and send it out in your browser, you’ll see a line while the send is completed.
If you don’t set the queue processing, than your emails will be sent out right there on the fly by your browser basically. It is a good way to send out lists like this if you like to play with fire. You’ll run into php timout, amazon SES limit ban, etc.
If you have queue processing allowed, the progress bar will represent the emails pushed into your queue. If you were an internet user already in the 1990s, and you enjoy staring at progress bars, do it by all means, but you can also time your broadcast in the future and let the cronjob take care of it.
2. Set up the proper cronjobs
-
Let the cron handle the progress bar: app/console mautic:broadcasts:send
This cronjob will place the emails into the queue. (You can find these files in app/spool/defualt) If you send 500 emails at once, you won’t need to limit this process. But if you send 50.000, you will need to add some settings. (More later.) -
The cron that sends to Amazon SES is: app/console mautic:emails:send
This cron will check the queue folder and push the emails to Amazon SES. You don’t want to do this faster than 14/ sec. (Or whatever your limit is.) -
There is a third aspect as well. You don’t want to create a jam in your system. What I mean is, you want to make sure, that filling up the queue and sending out the emails is happening in a controlled way, without one script blocking the other one. You have other cronjobs running too: segments, campaign triggers, etc, you have to let them run as well.
3. Time your scripts
-
First you need to see how often you run your cronjobs. Let’s say you run 5 different cronjobs every minute. If you want things to happen smooth, you need to make sure, that each cronjob is finishing before the 60 seconds are up.
-
Let’s simulate a sending: turn off your cronjobs, and make a send, like you would do normally. Let’s see what happens in the background. Open the stopwatch in your phone. (Seriously.) Start it and enter the command: php path/console mautic:broadcasts:send
-
Figure out how many you are queueing up within 50 seconds. Write it down.
-
If your limit is 14/sec, you don’t want to send faster than 850/min. So our limit will be around 800 / min. Just to make sure you reaching this amount make a test, and see how fast you send out 800 emails. Go back to your queue, count the files (find . -type f | wc -l) and send out 800 emails: app/console mautic:emails:send --message-limit=800
Did it finish under 1 minute? If yes, than you’ll need to limit this. If not, than we have some work to do.
4. Optimize
You want to push into the queue just a bit faster than you are sending out to reach optimal delivery speed. If you can build the queue and send with 800/min, than do the settings accordingly:
app/console mautic:broadcasts:send --limit=800
app/console mautic:emails:send --message-limit=790
If building the queue is slower, than it makes sense to limit the send as well. Sending too fast is not good for the deliverability anyway.
So in this particular example you’ll have 5 cronjobs running. The first 3 (Segment rebuild, campaign rebuild, campaign trigger) each will run 0-30 seconds, and the email related segments will run 50-55 sec.
As you can see, you are not being efficient here, you are doing almost nothing for 3 minutes, and sweating your below part off for 2 minutes. Can this be more efficient? Glad you asked.
5. Run your cronjobs sequentially
You can chain up your cronjobs into one bash file. This way once a command is finished, the next one runs.
If you run your batched cron commands every 5 min, you can achieve something like this:
You can also batch run this every minute, but make sure you limit all your crons. For example (Again, this depends on your server speed.)
mautic:broadcasts:send --limit=150
mautic:emails:send --message-limit=150
mautic:campaigns:rebuild --batch-limit=300
mautic:segment:update --batch-limit=900
mautic:campaigns:trigger
mautic:import --limit=500
mautic:webhooks:process
mautic:reports:scheduler
I hope this all makes sense, if you have any questions, let me know and I’ll try to clear it up for you!
Joey
14 Likes
@joeyk Thank you for the very throughout explanation, this is great, I will be trying to set up something similar and come back here should I stumble on any issues, thanks again!
Yes this really helps! We setup things sequentially.
But in our current setup, it’s taking about 1 minute to send out 60 mails for example and this will take too long for us in case of 2+ mil e-mails. So we’re looking into running multiple jobs of mautic:emails:send at the same time, with different lock-names… as the help of that command suggests;
bin/console mautic:emails:send --help
–lock-name[=LOCK-NAME] Set name of lock to run multiple mautic:emails:send command at time
Could anyone verify this is possible to run multiple jobs of the e-mail sending at once? And also at the same time maybe filling the queue up with the mautic:broadcasts:send command? We saw the file locking of the emails is atomic, so this wouldn’t be an issue aside from some warnings, but also saw something being written to /tmp//body and we’re not sure if multiple processes will try to access the same file there. (And thus eventually sending the wrong e-mail to the wrong person…).
We’re using Amazon SES API, our current rate limit is about 300/sec which we don’t even get close to.
Our server and DB load is very low under all of this, so I don’t think that is causing the slowdown.
1 Like
Hi, I do something similar for a client, this is possible. There the idea is to send out emails asap, and all other crons are stopped during “sending time”. We manage all the crons with a bash script. We use a simple command to check if anything is in the spool:
if [ -z "$(ls -A /var/www/html/mautic/app/spool/default)" ]; then
If there are files in the pool, we only send. (and fill the queue.)
We also run our crons every minute in a cascading way using
sleep 10, sleep 20 etc - not sure if it helps, I just didn’t feel comfortable starting 5 cron jobs at the same time.
We are only sending 200k, but the multiple threads do the job. You should be ready to recieve the feedback as well, cause a 2+ mill send generates a bunch of feedback and Mautic should be able to manage that.
1 Like
Ah that’s also a good idea, thanks! We’ve tried it as well, also with multiple threads. The FS queue was still too slow for us, so now we tried switching to immediate sending and using the mautic:broadcasts:send command together with --min-contact-id= and -max-contact-id (0-200000, the next one 200000-400000 etc.) and a limit of 1000 in a bash script which has it’s own pid file locking :).
So about 12 threads at once, and running every minute and enabled us to send out about 100 mails/sec on our current infrastructure which is pretty good for us.
We’re running all the other crontabs every 5 minutes in sequence with a bash script.
So far it’s been running smoothly, also with the open tracking etc. We plan on evaluating sometime soon once again.
Thanks again for this, it really helped a lot!
3 Likes
Hi,
I am new to Mautic. We have to send out lists of 100k regularly.
Usually we don’t use planned send outs therefore don’t use php console mautic:broadcasts:send.
But currently it takes ages to send out the emails.
The spool folder is getting filled up with 100k files when we do a send out.
We would like to speed it up and thought about starting more processes php console mautic:emails:send with different lock files.
We are wondering whether doing so could cause sending out emails twice or even more times. Or does the system prevent it happening?
Thanks!
You should be safe with using lockfiles, we are doing that as well. Try to set 1 minute cronjobs to make sure the emails are regularly batched up.
J
Hi,
created a bash file to start the parallel jobs with a time difference of 2 seconds:
#!/bin/bash
SLEEP_INTERVAL=“2”
php /var/www/html/bin/console mautic:emails:send --lock-name=email01 &
sleep ${SLEEP_INTERVAL}
php /var/www/html/bin/console mautic:emails:send --lock-name=email02 &
sleep ${SLEEP_INTERVAL}
php /var/www/html/bin/console mautic:emails:send --lock-name=email03 &
sleep ${SLEEP_INTERVAL}
php /var/www/html/bin/console mautic:emails:send --lock-name=email04 &
sleep ${SLEEP_INTERVAL}
php /var/www/html/bin/console mautic:emails:send --lock-name=email05 &
sleep ${SLEEP_INTERVAL}
php /var/www/html/bin/console mautic:emails:send --lock-name=email06 &
sleep ${SLEEP_INTERVAL}
php /var/www/html/bin/console mautic:emails:send --lock-name=email07 &
sleep ${SLEEP_INTERVAL}
php /var/www/html/bin/console mautic:emails:send --lock-name=email08 &
sleep ${SLEEP_INTERVAL}
php /var/www/html/bin/console mautic:emails:send --lock-name=email09 &
sleep ${SLEEP_INTERVAL}
php /var/www/html/bin/console mautic:emails:send --lock-name=email10 &
Did a try with a segment of 314 addresses. The logs in the SMTP service we use, shows 348 processed queries and I could verify that some email addresses received the emails up to 5 times.
How can I prevent this happen?
–message-limit and --time-limit are both set to to empty values in Mautic->Configuration.
Any ideas?
Would be really appreciated to find a solution to make sending out much faster without sending multiple emails to one recipient.
Thanks!
Gerd
Hello Gerd, I saw your message on my channel as well. You are almost there. Try to set up file lock mode with each command:
–lock_mode=LOCK_MODE
Allowed value are “pid” , “file_lock” or “flock”. By default, lock will try with pid, if not available will use file system [default: “pid”]
Hi,
I just tried it, it makes no difference.
The problem is, that some emails are getting renamed as “tryagain” and “finaltry” (or similar) although they were send out and then send out again by another process.
Maybe a new process tries to access the file, but can’t do it because of another process, runs into an error and renames the file, so that it will be sent out again.
Strange behaviour.
Thanks!
Gerd
Any further ideas?
What sending provider do you use? And which capacity (emails/second) do you achieve with running only one process? We are at 1 email per second. It would be cool to get to 10 or 20 per second…
Further ideas to fiddle with these commands:
–do-not-clear
By default, failed messages older than the --recover-timeout setting will be attempted one more time then deleted if it fails again. If this is set, sending of failed messages will continue to be attempted.
–recover-timeout[=RECOVER-TIMEOUT]
Sets the amount of time in seconds before attempting to resend failed messages. Defaults to value set in config.
–clear-timeout[=CLEAR-TIMEOUT]
I had a look into the source code. I can’t really see anything protecting a file from being send out several times.
I also tried to fiddle around with the parameters. But it doesn’t work really. No change in behaviour.
I now decided to tweak the sending process using different spool folders to work through depending on a parameter I pass. Before sending out I will run a script to move all emails from the spool folder balanced into 20 other process spool folders.
Those folders will then be worked through individually by each process.
This will prevent an email being send out twice or more times.
Interesting. We are waiting 10 sec between scrips and works perferctly. The scripts are executed every minute with a limit of 100 -200 emails / batch depending on the server.
So what are your parameters in config?
Maybe having exactly those could help.
What is the criteria for the message limit to be put on 100/200? Is it what one process is able to send out in one minute on average?
Are we able to book you for assistance and support?
Thanks!
Hi,
Sorry I can’t spend working time on troubleshooting this, but here is the way we did it:
Create a script, that holds all the cronjobs
execute the script every minute with limiting how long a thread runs (1 min)
If you don’t know how many emails you will send, you can just use the time limit to limit the runtime.
php /pathtomautic mautic:emails:send --time-limit=47s –lock_mode=flock & sleep 15 &
php /pathtomautic mautic:emails:send --time-limit=47s –lock_mode=flock & sleep 15 &
php /pathtomautic mautic:emails:send --time-limit=47s –lock_mode=flock & sleep 15
Why 47 sec? because we need around 8 seconds to get the list of emails to be sent and I’m giving 5 second buffer.
Your solution about multiple spools also sounds great btw.
I have set it up with multiple spools, was a little tricky to get it working. But it is possible.
And its working so far. Thanks for your input!
1 Like