Working through how we config our Kubernetes at the moment
Important question - how can we safely split up the sending queue so we can multi thread it.
What protective systems are in place to help prevent an email being sent twice?
Looking over our cron’s when I try to push through multiple request
What does the bypass locking do?
Script with pid 4103645 in progress.
Script in progress. Can force execution by using --bypass-locking.
Any prior learnings or anything that comes to mind to help prevent us from doing anything silly would be appreciated
What is currently planned for scaling the cronjobs on mautic:
Set the schedule to execute the cronjobs
Let say we set to execute command mautic:emails:send every 5 minutes
At the low email queue to be sent, say just 100 emails, it will be safe and will be finished under 5 minute, don’t need to spawn another worker
At the high email queue to be sent, say it 10k-100k emails, it won’t be finished under 5 minute, so at the 5th minute it will spawn another worker and execute the same command to send the emails on parallel, and it will repeat at the 10th, 15th, and so on
They do not work together in the sense they are not aware of each other.
Thy do work well in parallel and I am using the exact same method you just described.
Beware that as you add new contacts, you might need to adjust your worker’s settings.
I am running one specific instance on Mautic 4.4.6 with 7 email workers, sent millions of emails using this method, it works well with large amounts of emails, no problem there, but it consumes a lot of CPU and RAM.
It’s both, depending on the email size and complexity it will be the email creation or the sending.
The increase in RAM demand is not negligible but not too much.
The CPU requirements are relatively big.
If your emails are very heavy and take a while to get generated, you might want to use the queue and build them, let’s say overnight, send them in the morning. I’ve tried this and it worked wonderfully and was fast, however, there is a problem, if you have any emails that need sending immediately (like a password reminder or a welcome email), they might get stuck until you open the gates.
So right now I have the queue disabled and I regulate the sending speed (so I don’t get over the SES limit) with how many cronjobs I run in parallel.
It’s all “intumath” with very generous “padding”.
I can’t give you the numbers cause those depend on both your SES limits and the hardware you’re using.
Looking at a separate script to count contacts in db and contact id range on campaigns.