Your software My Mautic version is: v4.4.1 My PHP version is: 7.4.23 My Database type and version is: 10.3.37-MariaDB-0ubuntu0.20.04.1
Your problem
My problem is on the ability of the server to handle the volume. Want to know if Mautic is capable of taking care of higher traffic etc: The current server is a AWS EC2 r6a.2xlarge with following setting.
Total RAM Memory : 64 GB
RAM Allocated to DB: 39 GB
Total Hard disk Space : 380 GB
DB Size : 161 GB
Top five Table :
Table Name Size(MB)
dsijlead_event_log
59567.98
dsijaudit_log
22909.92
dsijcampaign_leads
18405.92
dsijemail_stats
16871.34
dsijleads
11958.73
We keep data of table dsijemail_stats and dsijcampaign_lead_event_log for 1 month and dsijpage_hits for 5 days .
The issue we face is that Mautic becomes unresponsive a few times during the day. It stays in a hung state for some minutes and then comes back online. This happens more frequently when the traffic on our site increases. So, is the solution all about sizing up the server (increase instance level) or is it so that Mautic is not built for higher load etc? Do note the these hanging happens even though the CPU usage stays at 15-20% throughout. This should indicate the CPU is not stuck as well.
Please guide.
database size
What strikes me most is 12G for the leads table… What is that, 100 Mio entries? If that number is really what you have to deal with, then you might indeed be in a region that vanilla Mautic code cannot handle. But some million known leads plus a good multiple of that in anonymous should be doable if you maintain your Mautic properly.
My guess is, though, that you should be able to drastically reduce the database size by taking a good look at comprehensive housekeeping, and cleaning out the junk aggressively from all relevant tables. Some starting points here.
Performance / Load
As long as the database content is in good shape, you should be able to handle everything else with regular LAMP and hardware (or cloud) tuning. With Mautic, I have not seen that being a bottleneck that cannot be overcome.
Thanks Ekke for the inputs. In the leads table we have 9977157 records. Do note we have added a large number of fields in this table to maintain various information about each user. So, information like user’s subscription start and end date for each service etc. Did not know where to place these as these fields are required to create apt segments. Owing to these added fields in the table the size could be showing 12 GB etc. So, is that an issue and if the above makes sense?
Thanks @rpadode for sharing your example…we just started on Mautic and are moving to memory optimized EC2 instances.
Interesting that you’re on r6a… based on this article I was going to move to r6g as it seems like it has similar compute & memory but 10% cheaper by using ARM architecture AWS EC2 Instance Comparison: R6g vs R6a vs R6i Update: I realize now that switching from x86_64 to ARM is a real effort… r6a seems like a great choice!
Have you had a chance to set up CloudWatch agent to track memory consumption? This is next on my list as we just had our first OOM error. I guess another option would be to run top CLI command (Shift + M to sort processes by memory) if you can catch it during a slow down.
Thanks… We are trying to optimise our segments and Lead table size. Meanwhile, could you clarify whether in Mautic we can have different segments rebuild at different times
For example, Can I have Segment A rebuild once a day while segment B rebuild twice a day etc? If not, anyway to do so as all our segments need note rebuilt every time. Also, we notice that segment building appears to be overwhelming the resources.
Thanks
Thanks. This video is interesting. Will work towards enrolling on your site for the script.
Meanwhile, had some fundamental question with regards to segment creation.
Is the effort to rebuild the segment same each time the cron runs i.e. is it one big run if done daily versus small runs multiple times if done say every 5 mins to form the same segment. Are the re-runs incremental effort or the same as if building the segment from scratch?
Do the segments get created in the sequence of the increasing order of the segment id?
Is there a way to be alerted if segment creation is failing so that manual intervention can be acted upon?
Thanks in advance
1 - depends how you use the segments and what they control. If you need to launch campaigns (automations) frequently, and they are based on segments, do it.
2 - good question
3 - There is a built in alert of 48 hours I think. You can look in the code and modify it.
Setting batch limits in your crons and running crons in sequence using the & symbol between the crons will help smooth performance issues with large segments as well.
@rpadode I setup Cloudwatch memory tracking on our EC2 instance and I’ve noticed I get a large spike every time our mautic:contacts:deduplicate cron job runs.
We had it set to run on the 19th and 49th minute of the hour like this with logging to papertrail:
I’ve noticed that the deduplicate job (Mautic 4.4.10) doesn’t seem to have a default batch size. It’s running de-duplication queries for every contact in our database (19K contacts).
Are you running a deduplicate cron job? If so, then does your system freeze up when you call it manually (for example on an ec2 instance… sudo -u www-data php /var/www/html/bin/console mautic:contacts:deduplicate -vvv)
Anyway just troubleshooting things for our own install and wondered if this might be a common issue.
Hi ekke… It so appears that the Mautic site tends to stall (become non responsive) when there is high traffic on the main site and in particular on a specific page that has a focus element poping up. Is this a known issue with Mautic that too many requests to popup (focus element) a dialog box can stall it?
Hi Joey… It so appears that the Mautic site tends to stall (become non responsive) when there is high traffic on the main site and in particular on a specific page that has a focus element poping up. Is this a known issue with Mautic that too many requests to popup (focus element) a dialog box can stall it?
