Mautic sending email is too slow

[sun123]

I am using Mautic 4.2 and Amazon SES which has currently 14/sec max. send rate. However, I am not being able to send more than 7/sec of emails using queue. I have to be ready for future which might need about 300/sec.

1. I have checked that infrastructure is not a problem as I cant see spikes on CPU or Memory or Network etc in Mautic server
2. I am using cron jobs to send emails but to test emails/sec I directly run command in console as below
   a. Firstly I populate queue in spool using campaign rebuild and trigger
   b. Count total no. of emails in queue say 1000
   c. Run command to send email say 800 per minute
   /bin/php /var/www/html/mautic/bin/console mautic:emails:send --time-limit=60 --message-limit=800
   d. Count total no. of remaining emails in queue and it is around 500 to 550

So that is the problem there, I was expecting Mautic to complete 800 per min as in command but something is wrong there. Can you please help me with this problem?

Also I tried to run same command with diff. locks which exceeds max send rate, However, it does not reach 800 per min anyway.

Any help is appreciated, thanks

[joeyk]

Hi, check this forum post:

Matching SES send rate

I’m wondering if Mautic automatically matches SES send rate? Or I should follow OPs settings? I also have 14 emails / sec limit and would like to know if I should do something in mautic to match this

[sun123]

Thanks @joeyk for reply.

I looked into the post you referred but as mentioned in Optimize part, I try to send 790 with below command but I find that Mautic is not able to send 790 per minute, it just sends 500 per minute

app/console mautic:emails:send --message-limit=790 --time-limit=60

Can you please help here?

[joeyk]

The reason:
Your server is too slow. You’d need to investigate the bottle neck. (DB, memory, CPU?)
At the same time I would be cautions with sending so fast. If you exceed the limit, you lose emails.
Joey

[sun123]

Thanks for getting back to me

We already checked our server CPU, memory and network which is ok.
As long as we are close to SES max send rate, we don’t worry sending to fast.

Can you please suggest anything else that is delaying sending emails? Is multithread with locks only way?

Have a good day

[joeyk]

Server CPU /memory showing OK values when you run HTOP doesn’t mean that you fully utilize it. You need to tweak your DB and PHP settings in a way to speed up send.
Mautic 5 will speed up email sending by 3,5x. I’s wait for that couple of months.
Joey

[sun123]

Thanks @joeyk for replying.

I agree with you that we need to tweak our PHP setting and DB. Can you please let me know what is setting I need to look into?

Excited for new release already.

Have a great day

[joeyk]

Hi, I can’t give you optimization advice as I don’t know your environment. I suggest you post a job in the forums.
I hope someone can solve it. Joey

[sun123]

Hi @joeyk

I applied multi-threading to send emails and got success to achieve a faster send rate like with one thread is 10/sec and I have applied 5 threads which gives 50/sec throughput.

However, I have found that there is an issue with duplicates. Same email gets sent multiple times when multiple threads are used.

Can you please help here?

Command below:

/bin/php /var/www/html/mautic/bin/console mautic:emails:send --lock-name=“1” & sleep=2 & /bin/php /var/www/html/mautic/bin/console mautic:emails:send --lock-name=“2” & sleep=2 & /bin/php /var/www/html/mautic/bin/console mautic:emails:send --lock-name=“3” & sleep=2 & /bin/php /var/www/html/mautic/bin/console mautic:emails:send --lock-name=“4” & sleep=2

[joeyk]

Hi, try to set limits according to your cron cycle. And run this command every minute with a time expiry of 45 sec.

[sun123]

Thanks @joeyk for your advice. I set time limit of 45 sec and message limit of 300 but there are still duplicate emails in the inbox. Can you please review below command? Thanks, appreciate your help

/bin/php /var/www/html/mautic/bin/console mautic:emails:send --lock_mode=flock --lock-name=“1” --message-limit=300 --time-limit=45 & /bin/php /var/www/html/mautic/bin/console mautic:emails:send --lock_mode=flock --lock-name=“2” --message-limit=300 --time-limit=45 & /bin/php /var/www/html/mautic/bin/console mautic:emails:send --lock_mode=flock --lock-name=“3” --message-limit=300 --time-limit=45

[joeyk]

Can u plz check what happens if you keep 15 sec break between the 3 threads?

sleep 15

[sun123]

@joeyk , Thanks for your suggestion, I tried this and ended with duplicates again. Then I investigated a little bit and I found that when using multi threading, some threads collide with each other and they give some warning that there is no file *.sending and thus results in failed message.

These failed messages are then picked up again as configured in system. So, I configured not to retry failed message and wipe them out. Then there are no duplicates now. I know that this is not 100% accurate solution as there might be true failure sometimes that we want system to resend again.

Anyways, I am still surprised at why multithreading is not working as expected even though I have added lock_mode=flock.

Thanks for your help, appreciated

[neiman]

Hi, @sun123 Im facing the same issue with the duplicates, how you did to configure not to retry the failed messages and wipe them out??
Thank you in advance!!

[carlosatfongo]

Script to replace the mautic:emails:send cronjob

Hey Joey There was a merged pull request regarding this multi threading sending: Relevant part is this info from kuzmany: [Bildschirmfoto 2023-10-13 um 10.33.21] The thing is I don’t know if this ever happened in mautic swiftmailer: Also need fix in swiftmailer Add flock lock on message during queue flushing by Dcoutelle · Pull Request #1270 · swiftmailer/swiftmailer · GitHub So maybe a patched is needed. This would be the patch: https://patch-diff.githubusercontent.com/raw/swiftmail…

You’ll want to see the above post^ This patch to swiftmailer should solve your duplicate send issue. works well for us, and we were able to run multiple sending crons at the same time

What we’re using for now:

sudo php /srv/mautic/bin/console mautic:emails:send --lock_mode=flock --lock-name="1" & sleep 2 & sudo php /srv/mautic/bin/console mautic:emails:send --lock_mode=flock --lock-name="2" & sleep 2 & sudo php /srv/mautic/bin/console mautic:emails:send --lock_mode=flock --lock-name="3"

Using the above code, we benchmarked some email tests. This code runs 3 send commands at the same time, which is about 18 emails per second to our SMTP sever… which is too fast. We’re going to only run 2, or about 12.5 emails per second.

[joeyk]

Hello @carlosatfongo
Are you sure you need to run this as sudo?
Doesn’t it disobey the file lock?

[carlosatfongo]

For some reason, the script fails if we run it something other than sudo/root. So my user, the IT admin’s user, or the Mautic owner user.

Going to keep digging to see how we can run it with the mautic owner user, which is what we intend.

[carlosatfongo]

Yeah we cannot figure out how to run our own multithread script as a user other than root. Any suggestions?

The terminal command we use to trigger the script, logged in as any user (www-data, my account, IT Admin’s account) but root:

./multithread_send_test.sh

The script contents:

#!/bin/bash

echo "------------------------------------------------ START SCRIPT "
date
d=`date +%Y-%m-%d`
echo $d

echo "------------------------------------------------ START PHP Commands "
sudo -u www-data php /srv/mautic/bin/console mautic:emails:send --lock_mode=flock --lock-name="1" & sleep 2 & sudo -u www-data php /srv/mautic/bin/console mautic:emails:send --lock_mode=flock --lock-name="2"
echo "------------------------------------------------ END PHP Commands "

date
echo "------------------------------------------------ END SCRIPT "

We have the same issue with trying to user this user’s SmartSend script.

Script to replace the mautic:emails:send cronjob

I’ve always had issues with my mautic:emails:send cronjob. If I put the send limit too low my emails don’t go out fast enough. If I put the limit too high I exceed the Amazon SES rate limit of 14 emails per second. So I made this script that does the best of both worlds… you can run it every 30sec-1min and it will check if there are mails to send. If no mails need to be sent it just exits without running mautic. If there are mails that need to go out, it will send EXACTLY 14 emails, wait 1 sec,…

[carlosatfongo]

Looks like we managed to solve the issue. I double checked permissions on our mautic path, and our script works using the mautic owner.

sudo -u www-data php /srv/mautic/bin/console mautic:emails:send --lock_mode=flock --lock-name="1" & sleep 2 & sudo -u www-data php /srv/mautic/bin/console mautic:emails:send --lock_mode=flock --lock-name="2"