Matching SES send rate

Hi,
created a bash file to start the parallel jobs with a time difference of 2 seconds:

#!/bin/bash
SLEEP_INTERVAL=“2”
php /var/www/html/bin/console mautic:emails:send --lock-name=email01 &
sleep ${SLEEP_INTERVAL}
php /var/www/html/bin/console mautic:emails:send --lock-name=email02 &
sleep ${SLEEP_INTERVAL}
php /var/www/html/bin/console mautic:emails:send --lock-name=email03 &
sleep ${SLEEP_INTERVAL}
php /var/www/html/bin/console mautic:emails:send --lock-name=email04 &
sleep ${SLEEP_INTERVAL}
php /var/www/html/bin/console mautic:emails:send --lock-name=email05 &
sleep ${SLEEP_INTERVAL}
php /var/www/html/bin/console mautic:emails:send --lock-name=email06 &
sleep ${SLEEP_INTERVAL}
php /var/www/html/bin/console mautic:emails:send --lock-name=email07 &
sleep ${SLEEP_INTERVAL}
php /var/www/html/bin/console mautic:emails:send --lock-name=email08 &
sleep ${SLEEP_INTERVAL}
php /var/www/html/bin/console mautic:emails:send --lock-name=email09 &
sleep ${SLEEP_INTERVAL}
php /var/www/html/bin/console mautic:emails:send --lock-name=email10 &

Did a try with a segment of 314 addresses. The logs in the SMTP service we use, shows 348 processed queries and I could verify that some email addresses received the emails up to 5 times.
How can I prevent this happen?
–message-limit and --time-limit are both set to to empty values in Mautic->Configuration.
Any ideas?
Would be really appreciated to find a solution to make sending out much faster without sending multiple emails to one recipient.
Thanks!
Gerd

Hello Gerd, I saw your message on my channel as well. You are almost there. Try to set up file lock mode with each command:

–lock_mode=LOCK_MODE
Allowed value are “pid” , “file_lock” or “flock”. By default, lock will try with pid, if not available will use file system [default: “pid”]

Hi,
I just tried it, it makes no difference.
The problem is, that some emails are getting renamed as “tryagain” and “finaltry” (or similar) although they were send out and then send out again by another process.
Maybe a new process tries to access the file, but can’t do it because of another process, runs into an error and renames the file, so that it will be sent out again.
Strange behaviour.

Thanks!
Gerd

Any further ideas?

What sending provider do you use? And which capacity (emails/second) do you achieve with running only one process? We are at 1 email per second. It would be cool to get to 10 or 20 per second…

Further ideas to fiddle with these commands:

–do-not-clear
By default, failed messages older than the --recover-timeout setting will be attempted one more time then deleted if it fails again. If this is set, sending of failed messages will continue to be attempted.
–recover-timeout[=RECOVER-TIMEOUT]
Sets the amount of time in seconds before attempting to resend failed messages. Defaults to value set in config.
–clear-timeout[=CLEAR-TIMEOUT]

I had a look into the source code. I can’t really see anything protecting a file from being send out several times.
I also tried to fiddle around with the parameters. But it doesn’t work really. No change in behaviour.

I now decided to tweak the sending process using different spool folders to work through depending on a parameter I pass. Before sending out I will run a script to move all emails from the spool folder balanced into 20 other process spool folders.
Those folders will then be worked through individually by each process.
This will prevent an email being send out twice or more times.

Interesting. We are waiting 10 sec between scrips and works perferctly. The scripts are executed every minute with a limit of 100 -200 emails / batch depending on the server.

So what are your parameters in config?
Maybe having exactly those could help.
What is the criteria for the message limit to be put on 100/200? Is it what one process is able to send out in one minute on average?
Are we able to book you for assistance and support?
Thanks!

Hi,
Sorry I can’t spend working time on troubleshooting this, but here is the way we did it:

Create a script, that holds all the cronjobs
execute the script every minute with limiting how long a thread runs (1 min)
If you don’t know how many emails you will send, you can just use the time limit to limit the runtime.

php /pathtomautic mautic:emails:send --time-limit=47s –lock_mode=flock & sleep 15 &
php /pathtomautic mautic:emails:send --time-limit=47s –lock_mode=flock & sleep 15 &
php /pathtomautic mautic:emails:send --time-limit=47s –lock_mode=flock & sleep 15

Why 47 sec? because we need around 8 seconds to get the list of emails to be sent and I’m giving 5 second buffer.

Your solution about multiple spools also sounds great btw.

I have set it up with multiple spools, was a little tricky to get it working. But it is possible.
And its working so far. Thanks for your input!

1 Like

Sounds great. May I have some questions?

  • How are you distributing the spools upon email creation?
  • How do you tell Mautic which spool to use?
    Thx,
    Joey

I am interested in the “many-spool-solution” as well.
Could you give some insights @gerdr?

I don’t have that solution at this point.
But I can imagine to have multiple spools where concurrent email:send commands are sending emails with a parameter, that describes the spool.

Seems like @gerdr was here only to steal but not to share, too bad.

@geimsdin I wasn’t sure whether I should reply to such a bs and offending post.
I am no longer using Mautic at all. So you are lucky that I forgot to deactivate my profile here.
At the time I had the problem, nobody could give me an answer. So what did I steal @geimsdin?
Almost one year ago I tried a quick and dirty solution to get it run within a day or two.

The below solution did work for us with Mautic 4.1.1 AND I can’t say if it will still works with newer Mautic versions, as we stopped using Mautic since March 2022. The main reason was the dodgy behaviour of the template generator and the email generation (even when using MJML). It was not reliable and easy enough for us.

What I basically did:

  1. create a shell script, which creates n spool folders
/var/www/html/var/spoolemail01
/var/www/html/var/spoolemail01
/var/www/html/var/spoolemail01
/var/www/html/var/spoolemailNN
  1. create a shell script with n calls of the email sending command:
    File /var/www/html/bin/multiemails.sh
#!/bin/bash
threads=20
threadsa=$[$threads+1]

/var/www/html/bin/movebalanced.sh

i=1

while [ $i -lt $threadsa ]; do
  fileext="$(printf "%02d" $i)"
  php /var/www/html/bin/console mautic:emails:send --lock-name=email$fileext &
  i=$[$i+1]
done
  1. Create a shell script to move the spooled emails into the n spool folders.
    File /var/www/html/bin/movebalanced.sh
#!/bin/bash

threads=20
countfiles=`ls /var/www/html/var/spool/*.message | wc -l`

if [ $countfiles -gt 0 ]
then
        files_per_folder=$((($countfiles/$threads)))
        rest=$(($countfiles-($files_per_folder*$threads)))
        resta=$(($rest+1))
        k=1
        threadsa=$((($threads+1)))
        while [ $k -lt $threadsa ]; do
                i=0
                fileext="$(printf "%02d" $k)"
                if [ "$k" -lt $resta ];
                then
                        new_files_per_folder=$((($files_per_folder+1)))
                else
                        new_files_per_folder=$((($files_per_folder)))
                fi
                echo $new_files_per_folder

                for x in /var/www/html/var/spool/*.message; do
                  if [ "$i" = $new_files_per_folder ]; then break; fi
                  mv -n -- "$x"  /var/www/html/var/spoolemail$fileext/
                  i=$((i+1))
                done
                k=$[$k+1]
        done
fi
  1. change the console command so that it allows to use the parameter
    a. File /var/www/html/bin/console

Insert

define('V_SPOOL_FOLDER_EXT',$input->getParameterOption('--lock-name') ?? '');

after

$input = new ArgvInput();

b. File /var/www/html/app/bundles/EmailBundle/Command/ProcessEmailQueueCommand.php

Replace in line 102

$spoolPath = $this->parametersHelper->get('mautic.mailer_spool_path');

with

$spoolPath = $this->parametersHelper->get('mautic.mailer_spool_path').$lockName;

c. File /var/www/html/app/bundles/EmailBundle/Config/config.php

Enter

if (defined("V_SPOOL_FOLDER_EXT")) 
{
    $vSpoolFolderExt = "". V_SPOOL_FOLDER_EXT;
}
else
{
    $vSpoolFolderExt = "";
}

before

return [ 'routes' => [

d. File /var/www/html/app/bundles/EmailBundle/Swiftmailer/Spool/DelegatingSpool.php, function private function getSpoolDir(): string

Enter


if (defined("V_SPOOL_FOLDER_EXT")) 
{
   $vSpoolFolderExt = "". V_SPOOL_FOLDER_EXT;
}
else 

{
    $vSpoolFolderExt = "";
}

before

$filePath = $this->coreParametersHelper->get('mailer_spool_path');

and change

$filePath = $this->coreParametersHelper->get('mailer_spool_path');

to

$filePath = $this->coreParametersHelper->get('mailer_spool_path').$vSpoolFolderExt;

e. crontab
Enter

* * * * * /var/www/html/bin/multiemail.sh

Deactivate

#* * * * * php /var/www/html/bin/console mautic:emails:send

I can’t guarantee that this solution will work for anyone.

I am still surprised that this issue hasn’t been solved by the developers as it is very crucial.
The reason I went this path was, that multiple processes of the email sending command run on the one default spool folder caused multiple send outs of one email spool file. We basically didn’t want to spam our customers. We had to sent out 100,000 emails within a short period of time, so needed a parallel sent out functionality. But the one offered by Mautic caused lots of headaches and this solution helped us for the time we were using Mautic.

1 Like

@gerdr didn’t want to sound offensive, just stating that as soon as you found a solution you stopped replying. Thanks for sharing your dirty (as you stated) but very smart solution, could be a very good starting point for me or anyone else stumbling upon this post.
Much appreciated.

I would like to add to this thread. There are couple of improvements to sending in the meanwhile.
This PR is soon to be added to M5:

Add threads support for broadcast command

It ensures LOT faster thread processing.

Hi could someone help me out with getting my cron jobs set correctly. I am running Mautic using Cloudron. PHP Version 7.4.3

I have tried setting up my cron jobs a few ways but I am not sure how to set them sequentially in the way they are built out in cloudron.

I have tried staggering them, but not working , and when I set a campaign it shows emails sent, but in Amazon Ses shows none have sent. If I send a test email it works, or change to send instantly. I am also not getting any errors and not logging anything abnormal.

This is what I have in my cron jobs. How would I make this sequential?


*/5 * * * *   (echo "==> cron: mautic:segments:update --batch-limit=1600"; sudo -E -u www-data php /app/code/bin/console mautic:segments:update) > /proc/$(cat /var/run/crond.pid)/fd/1 2>&1

*/5 * * * *  (echo "==> cron: mautic:campaigns:rebuild --batch-limit=1600"; sudo -E -u www-data php /app/code/bin/console mautic:campaigns:rebuild) > /proc/$(cat /var/run/crond.pid)/fd/1 2>&1

*/5 * * * *   (echo "==> cron: mautic:emails:send --message-limit=790"; sudo -E -u www-data php /app/code/bin/console mautic:emails:send) > /proc/$(cat /var/run/crond.pid)/fd/1 2>&1

*/5 * * * *   (echo "==> cron: mautic:broadcasts:send --limit=800"; sudo -E -u www-data php /app/code/bin/console mautic:broadcasts:send) > /proc/$(cat /var/run/crond.pid)/fd/1 2>&1


Normally you would make a bash file and put all the scripts in there and run it every minute.
I can help you with the Mautic part, but the cloudron part I have no idea from. Maybe it’s better to ask that part in the cloudron forum:
“How can I make a bash file and run scripts sequentially?”

@gerdr thanks for taking the time to post your solution. I have been trying to send 500k+ emails in a timely fashion via AWS SES as well. Your solution worked on mautic 4.4.9 with almost no modifications.

1 Like

@joeyk My Amazon SES account’s sending rate is 14/sec. I don’t want to exceed it anyway.

How can I make sure that Mautic will not send emails faster than 14/sec with the following command?

bin/console mautic:emails:send --message-limit=800