Scaling with Workers - What happens if you force a cron script?

Product Support
1

Working through how we config our Kubernetes at the moment

Important question - how can we safely split up the sending queue so we can multi thread it.
What protective systems are in place to help prevent an email being sent twice?

Looking over our cron’s when I try to push through multiple request
What does the bypass locking do?

Script with pid 4103645 in progress.
Script in progress. Can force execution by using --bypass-locking.

Any prior learnings or anything that comes to mind to help prevent us from doing anything silly would be appreciated

What is currently planned for scaling the cronjobs on mautic:

  • Set the schedule to execute the cronjobs
  • Let say we set to execute command mautic:emails:send every 5 minutes
  • At the low email queue to be sent, say just 100 emails, it will be safe and will be finished under 5 minute, don’t need to spawn another worker
  • At the high email queue to be sent, say it 10k-100k emails, it won’t be finished under 5 minute, so at the 5th minute it will spawn another worker and execute the same command to send the emails on parallel, and it will repeat at the 10th, 15th, and so on
2

Think I found our solution with lock names
@joeyk - any other learnings here I gotta be mindful of?

3

Did you test with a real email send? For me lock names seemed to be buggy. Many emails were sent 2x

4

What quantity were you sending when it wigged out and double sent?

Really really keen to not have a repeat of my multiple sms headaches I had years ago haha

5

I did a send of 10k.
Batch size 100. 4 workers.
When I ran the command errors popped up - workers tried to send emails that were not available. But anyway some emails were sent 2x

6

Did you find the cause of them not being available?
Did the server run out of resources or anything during the send?

My last headache was when I sent the sms, the server maxed out on resources, then it started double sending.

My solution would to cap the number of batches per worker and potentially a pause between each batch

7

When sending emails the files are picked up from the spool folder. Missing files happened when 2 workers try to pick up the same file. This should not happen…

8

How do the workers work together and know which message to pick up?

is there anyway to put a prefix on the messages and allocate it to a worker?

Havent fully read what these name locks do either - is this the prefix?

Or we spin up a worker per batch once a worker has been a assigned a batch then kill the worker on completion of that batch … (donno how much the devops would like that haha)

This should limit the cron per worker
mautic:broadcasts:send --min-contact-id=0 --max-contact-id=2500
mautic:broadcasts:send --min-contact-id=2501 --max-contact-id=5000
… etc

9

know of any cheaper alternatives to mailtrap?

10

They do not work together in the sense they are not aware of each other.
Thy do work well in parallel and I am using the exact same method you just described.

Beware that as you add new contacts, you might need to adjust your worker’s settings.

I am running one specific instance on Mautic 4.4.6 with 7 email workers, sent millions of emails using this method, it works well with large amounts of emails, no problem there, but it consumes a lot of CPU and RAM.

Is Mailtrap cheaper than SES?

11

Broadcast send is fixed already - i tested a PR 1 year ago. You can do multi thread on it

12

Hey mate

Testing was more to validate duplicates and ingess with stats n validation

Looking at a separate script to count contacts in db and contact id range on campaigns.

What specifically consumes the cpu/memory?

The email create or broadcast/send?
Stats ill run through a horizontal rabbit

13

It’s both, depending on the email size and complexity it will be the email creation or the sending.
The increase in RAM demand is not negligible but not too much.
The CPU requirements are relatively big.

If your emails are very heavy and take a while to get generated, you might want to use the queue and build them, let’s say overnight, send them in the morning. I’ve tried this and it worked wonderfully and was fast, however, there is a problem, if you have any emails that need sending immediately (like a password reminder or a welcome email), they might get stuck until you open the gates.

So right now I have the queue disabled and I regulate the sending speed (so I don’t get over the SES limit) with how many cronjobs I run in parallel.
It’s all “intumath” with very generous “padding”.
I can’t give you the numbers cause those depend on both your SES limits and the hardware you’re using.

Looking at a separate script to count contacts in db and contact id range on campaigns.

That sounds very interesting!

1 Like