Sounds great. May I have some questions?
- How are you distributing the spools upon email creation?
- How do you tell Mautic which spool to use?
Thx,
Joey
I am interested in the “many-spool-solution” as well.
Could you give some insights @gerdr?
I don’t have that solution at this point.
But I can imagine to have multiple spools where concurrent email:send commands are sending emails with a parameter, that describes the spool.
Seems like @gerdr was here only to steal but not to share, too bad.
@geimsdin I wasn’t sure whether I should reply to such a bs and offending post.
I am no longer using Mautic at all. So you are lucky that I forgot to deactivate my profile here.
At the time I had the problem, nobody could give me an answer. So what did I steal @geimsdin?
Almost one year ago I tried a quick and dirty solution to get it run within a day or two.
The below solution did work for us with Mautic 4.1.1 AND I can’t say if it will still works with newer Mautic versions, as we stopped using Mautic since March 2022. The main reason was the dodgy behaviour of the template generator and the email generation (even when using MJML). It was not reliable and easy enough for us.
What I basically did:
- create a shell script, which creates n spool folders
/var/www/html/var/spoolemail01
/var/www/html/var/spoolemail01
/var/www/html/var/spoolemail01
/var/www/html/var/spoolemailNN
- create a shell script with n calls of the email sending command:
File /var/www/html/bin/multiemails.sh
#!/bin/bash
threads=20
threadsa=$[$threads+1]
/var/www/html/bin/movebalanced.sh
i=1
while [ $i -lt $threadsa ]; do
fileext="$(printf "%02d" $i)"
php /var/www/html/bin/console mautic:emails:send --lock-name=email$fileext &
i=$[$i+1]
done
- Create a shell script to move the spooled emails into the n spool folders.
File /var/www/html/bin/movebalanced.sh
#!/bin/bash
threads=20
countfiles=`ls /var/www/html/var/spool/*.message | wc -l`
if [ $countfiles -gt 0 ]
then
files_per_folder=$((($countfiles/$threads)))
rest=$(($countfiles-($files_per_folder*$threads)))
resta=$(($rest+1))
k=1
threadsa=$((($threads+1)))
while [ $k -lt $threadsa ]; do
i=0
fileext="$(printf "%02d" $k)"
if [ "$k" -lt $resta ];
then
new_files_per_folder=$((($files_per_folder+1)))
else
new_files_per_folder=$((($files_per_folder)))
fi
echo $new_files_per_folder
for x in /var/www/html/var/spool/*.message; do
if [ "$i" = $new_files_per_folder ]; then break; fi
mv -n -- "$x" /var/www/html/var/spoolemail$fileext/
i=$((i+1))
done
k=$[$k+1]
done
fi
- change the console command so that it allows to use the parameter
a. File /var/www/html/bin/console
Insert
define('V_SPOOL_FOLDER_EXT',$input->getParameterOption('--lock-name') ?? '');
after
$input = new ArgvInput();
b. File /var/www/html/app/bundles/EmailBundle/Command/ProcessEmailQueueCommand.php
Replace in line 102
$spoolPath = $this->parametersHelper->get('mautic.mailer_spool_path');
with
$spoolPath = $this->parametersHelper->get('mautic.mailer_spool_path').$lockName;
c. File /var/www/html/app/bundles/EmailBundle/Config/config.php
Enter
if (defined("V_SPOOL_FOLDER_EXT"))
{
$vSpoolFolderExt = "". V_SPOOL_FOLDER_EXT;
}
else
{
$vSpoolFolderExt = "";
}
before
return [ 'routes' => [
d. File /var/www/html/app/bundles/EmailBundle/Swiftmailer/Spool/DelegatingSpool.php, function private function getSpoolDir(): string
Enter
if (defined("V_SPOOL_FOLDER_EXT"))
{
$vSpoolFolderExt = "". V_SPOOL_FOLDER_EXT;
}
else
{
$vSpoolFolderExt = "";
}
before
$filePath = $this->coreParametersHelper->get('mailer_spool_path');
and change
$filePath = $this->coreParametersHelper->get('mailer_spool_path');
to
$filePath = $this->coreParametersHelper->get('mailer_spool_path').$vSpoolFolderExt;
e. crontab
Enter
* * * * * /var/www/html/bin/multiemail.sh
Deactivate
#* * * * * php /var/www/html/bin/console mautic:emails:send
I can’t guarantee that this solution will work for anyone.
I am still surprised that this issue hasn’t been solved by the developers as it is very crucial.
The reason I went this path was, that multiple processes of the email sending command run on the one default spool folder caused multiple send outs of one email spool file. We basically didn’t want to spam our customers. We had to sent out 100,000 emails within a short period of time, so needed a parallel sent out functionality. But the one offered by Mautic caused lots of headaches and this solution helped us for the time we were using Mautic.
1 Like
@gerdr didn’t want to sound offensive, just stating that as soon as you found a solution you stopped replying. Thanks for sharing your dirty (as you stated) but very smart solution, could be a very good starting point for me or anyone else stumbling upon this post.
Much appreciated.
I would like to add to this thread. There are couple of improvements to sending in the meanwhile.
This PR is soon to be added to M5:
Add threads support for broadcast command
It ensures LOT faster thread processing.
Hi could someone help me out with getting my cron jobs set correctly. I am running Mautic using Cloudron. PHP Version 7.4.3
I have tried setting up my cron jobs a few ways but I am not sure how to set them sequentially in the way they are built out in cloudron.
I have tried staggering them, but not working , and when I set a campaign it shows emails sent, but in Amazon Ses shows none have sent. If I send a test email it works, or change to send instantly. I am also not getting any errors and not logging anything abnormal.
This is what I have in my cron jobs. How would I make this sequential?
*/5 * * * * (echo "==> cron: mautic:segments:update --batch-limit=1600"; sudo -E -u www-data php /app/code/bin/console mautic:segments:update) > /proc/$(cat /var/run/crond.pid)/fd/1 2>&1
*/5 * * * * (echo "==> cron: mautic:campaigns:rebuild --batch-limit=1600"; sudo -E -u www-data php /app/code/bin/console mautic:campaigns:rebuild) > /proc/$(cat /var/run/crond.pid)/fd/1 2>&1
*/5 * * * * (echo "==> cron: mautic:emails:send --message-limit=790"; sudo -E -u www-data php /app/code/bin/console mautic:emails:send) > /proc/$(cat /var/run/crond.pid)/fd/1 2>&1
*/5 * * * * (echo "==> cron: mautic:broadcasts:send --limit=800"; sudo -E -u www-data php /app/code/bin/console mautic:broadcasts:send) > /proc/$(cat /var/run/crond.pid)/fd/1 2>&1
Normally you would make a bash file and put all the scripts in there and run it every minute.
I can help you with the Mautic part, but the cloudron part I have no idea from. Maybe it’s better to ask that part in the cloudron forum:
“How can I make a bash file and run scripts sequentially?”
@gerdr thanks for taking the time to post your solution. I have been trying to send 500k+ emails in a timely fashion via AWS SES as well. Your solution worked on mautic 4.4.9 with almost no modifications.
1 Like
@joeyk My Amazon SES account’s sending rate is 14/sec. I don’t want to exceed it anyway.
How can I make sure that Mautic will not send emails faster than 14/sec with the following command?
bin/console mautic:emails:send --message-limit=800