I’m a Mautic newbie who just did his first Mautic installation. I like the software so much that I’m considering using it both for:
moderate/low volume marketing automation type emails
sending so-called email blasts (to double opt-in recipients) … typically weekly newsletters, or broadcast email to a segment in Mautic terms.
My goal is to centralize email sending activities in a single system (automation + newsletters), which raises a few questions about mass sending with Mautic:
Is Mautic up to the job of sending a weekly newsletter to ~100,000 recipients?
Which email transport provider would you advise in this case?
I expect peaks of email open/click events coming to the Mautic server at a rate of maybe 100 requests/second.
What server hardware is required to handle such a peak rate with Mautic?
I’ve seen that Mautic can be configured to use RabbitMQ or Beanstalkd queues to process event peaks without loss.
What kind of event rate justifies introducing queue systems like these in a Mautic deployment?
Would you recommend RabbitMQ over Beanstalkd?
Any real-word experience of mass sending with Mautic will be very much appreciated.
… and many thanks to the Mautic team for this great software!
Hi @lhpz!
Mautic is built for that.
We have customers with lists way over 100.000 contacts.
They are running several campaigns (autoresponders with email and push notifications) side by side PLUS sending at least weekly newsletters.
All our servers are on google cloud platform and use Amazon SES for sending emails.
I think for your volume a simple VM with 2cpu and 4GB Ram does the job.
Maybe it works with even one cpu. You can scale up at any point if you need to.
What we found so far is that a queue system is not yet necessary for our clients. That depends on server capacities though.
Conclusion: Mass sending is an absolute treat with Mautic. I totally love it and would never switch back to any of the systems we provided before.
Hi, a quick question:
If you send 250.000 / week, do you have an issue with bloated db since mautic saves each sent email content in the database (email_stats i think)? After 3 months you have 3 million entries just there.
Are you cleaning that out? Or that is just a bug in 2.16.3 ?
Thx.
Thank you very much @alexhammer for this real-world feedback. May I ask you how long does it take to send a newsletter to 100.000 contacts on your setup?
Additionally, would you recommend limiting the overall send rate at the Mautic level - if this is possible - to avoid raising any red flags with spam filters at mailbox operators?
I’ve never used Amazon SES, I don’t known if they will limit the send rate to protect clients from sending too fast, shooting themselves in the foot with deliverability problems.
@joeyk, I’m a Mautic newbie but I’ve seen my share of MySQL servers. You can handle 500 million rows in a MySQL table on a a modest server, even on a beefy laptop with SSD storage. Backups will be slow, as well as queries building stats over a long time-frame, but the storage bloat should not be much of a problem on this email_stats table till you reach 5GB or 10GB. I can see that only the tokens are stored in the table, not the full message.
At 1 million emails per month, you can probably store a few decades of sending on a single server. That being said, removing anything older than 3 years is probably good hygiene.
Yeah I wasn’t worried either first time when I started to use Mautic for mass sending.
I just checked my email_stats (using regularly.)
On this account I sent 167.000 emails, where the email_stats size is 167.000 rows, 637 MB. A send of 250k will be around 800MB. Montly 3 Gig.
(160k page hit also generates 200 MB, so by 20% open rate you can add 300MB monthy to track the opens to your list.)
So you need to calculate 10 GIG for 3 months, 40 GIG for a year. And this is just sending emails and registering clicks. As you mentioned it starts to be ‘slow’ over 5-10 GIGs. It means you have a great 2-3 months without cleaning.
So I’m still wondering: why does Mautic save all the actual full email content for every single email sent? How do you guys clean?
I’m still a newbie … so I don’t have the Mautic operational data that you gave here. Obviously, your table grows much faster than want I originally thought just looking at the table structure.
why does Mautic save all the actual full email content for every single email sent?
I guess you’re referring to the email_copies table. I’m not sure what it is used for. It will clearly facilitate rendering the personalized message in a full page when accessed directly via URL (if you cannot read this message, click this). But you could arguably just store the tokens + email template_id, regenerate the full personalized message on demand, and save on storage! Probably a good feature request for people using Mautic for mass sending.
In the mean time, if you have some DB administration skills, you can apply InnoDB page compression on tables like email_stats or email_copies, see InnoDB Page Compression - MariaDB Knowledge Base. Verbose PHP serialized values stored in email_stats.tokens and email_stats.open_details, or HTML code stored in email_copies.body are prime candidates for good compression. That being said, a real server can handle many 10G tables without problems, especially when these tables are not written very often, which is the case here. With large tables, the problems are mostly with backups, and the time to restore in case of a catastrophic crash. At some point, a two database servers setup with real-time replication is the best strategy.
How do you guys clean?
I do not have much real-world Mautic operation under my belt, but from experience, I would avoid deleting old rows at the database level directly. I fear this would distort stats stored elsewhere. I’d rather use the application layer through a CLI command like console mautic:maintenance:cleanup --days-old=365 --dry-run. This dry-run command will list these record types:
Audit log entries
UTM tag history
User notifications
User tokens
Visitor page hits
Visitors
I’m not sure which table is behind “User Tokens”, but you can clearly save space on old page hits here. May be some more experienced Mautic admin can say more about what the cleanup command actually do.
Thx for the answer. I’m okay with my instances some running for 3 years, sending I’m just trying to give you a heads up based on real life experience.
No, I’m not referring to the email_copies, that table is just 1,5 MB. I’m referring to the email_stats.
Of course you can always have a beefier server. Of course you can clean after 1 year.
I’m just trying to give you heads up what to scale for in the first 3-9 months.
If you use other (just bulk) email tools, you can will collect 5th of the data, have 10% of database size, and just keep emailing. Why use Mautic for sending 3 mill emails / month?
It is lot better to use another software for bulk sending, and push openers into Mautic. You can even track opens with Mautic while sending with another solution. You’ll end up with lot better overall performance, and less maintanance, less headache, less server cost, and more flexibility.
The genie of Mautic is with tracking the relevant leads, and I will always use it for this purpose.
But sending bulk seems to me like trying to shoot mosquitos with a machine gun.
