Using smtp4dev (mock SMTP server) - I was able to get a throughput of 550 emails per minute. Sent in batches of 100 emails per session.
When I switch the SMPT server to a live server, I start out getting a half-way decent number of sends per minute (I know it won’t be 550) - but after a few minutes, it slows down to a crawl. Right now, I’m getting about 1 message every 90 seconds.
Conferring with my SMTP provider, it looks like the messages are NOT being sent over in batches (they have not provided the logs for me) - so I can’t confirm nor post.
So - this looks like it’s on my side - but I can’t for the life of me figure it out.
There is no logging that comes out of messenger:consume (-vvv doesn’t give me anything).
I’ve got 13k messages in the queue - at this rate, it’ll take almost 2 weeks to send out.
Don’t do that. You’ll kill your server. It won’t make it faster. It will bring it to a crawl.
48GB of RAM for a PHP script… That’s not serious my friend.
Some of our customers are sending over 20K emails within an hour. And none require extreme configuration like that.
128GB + 8 cores VPS for a Mautic install?
Stop that crazy spending, seriously. Hit me up if you’re looking for a way to optimize your infrastructure spending.
Last bit;
The specified HowTo is wrong at the third step. Badly executed and totally miss the point of a lock file. Use “flock” instead. Easier and faster to implement. On top of it, it’s much more effective.
I started with a much smaller environment. 16GB Ram / 2 Cores. When these problems started happening, I read that resources can be a bottleneck. I have a physical VM environment with 4TB of RAM and hundreds of cores. So - I decided to give it power. I’m not spending a nickel.
I’ll look at flock.
When you say “don’t do that” - are you referring to the memory limit being so huge?
PHP is very bad when it comes to RAM management. Having it using too much RAM won’t make it faster (literally the opposite). Let’s keep in mind that PHP is a single-thread process (it will never go above 100% of a single core).
On the other side, if a PHP script ever go wild like that, then there will be an issue somewhere else. More RAM won’t solve anything but only delay the crash.
Maybe you could skip the queue for such a small sending volume?
**flock is most likely already installed on your system. flock --help should be helpful.
I could try skipping the queue - but I also want to be efficient for our smtp provider. I’m really hoping to set up batches - which is why the queue.
I see what you mean. But without any reference, it can be quite misleading. Still, 15k is very small.
Anyways, to get in “batch” as you said, that’s the “–limit=” argument that you are looking for.
i.e. To send in batch of 500 emails at every 5 minutes: */5 * * * * /usr/local/bin/php $HOME/public_html/bin/console mautic:broadcasts:send --limit=500 >/dev/null 2>&1
To run it continually without overloading the system and overlapping (the step #3 in a few letters), that would be my way: * * * * * flock --timeout=55 /tmp/sending "/usr/local/bin/php $HOME/public_html/bin/console mautic:broadcasts:send --limit=200"
So in v5.2.1, it would be… * * * * * flock --timeout=55 /tmp/sending "/usr/local/bin/php $HOME/public_html/bin/console messenger:consume --time-limit=50 --limit=200"
If your script crash, chances are that it will ignore any argument.
On the other end, flock is already taking care of it (I suspect the --time-limit to be a second attempt at the Step #3: Reinventing the wheel, but with a built-in flat)
I’m pretty new at this - so basically trying to piece together hundreds of pages of “help”.
So - If I’m understanding it correctly.
Flock places a lock on a file - and runs messenger:consume --limit:200.
My understanding of the timeout is - if flock can’t get a lock on the file in n seconds, it terminates the attempt. So - if consume runs longer than a minute - it just stops, and waits for the next cron trigger. Makes sense to me.
Do I have it right?
I appreciate your patience and your help! Writing code since 2000, and I’m still learning every day!
flock will take care of the lock (flock => file lock, shortened)
As you said;
If the script already run, it will wait X seconds then leave it there.
If the previous run do finish during that wait (or if not running at all when triggered), the new process will kick in, sending an extra 200.
And I hear you about putting everything together. Read back the step #3, the script part with what you just learned;
It does emulate flock, but can also delete the lock file (!?) - even if a process is still running in the background (assuming that it’s “too old” !?). Potentially stacking an extra unwanted process on a maybe stale one (which can lead to unexpected results and defeat a lock purpose).
In short: Often over complicated leading to errors/bugs and incomprehension.
