After long testing I’ve finally managed to use Mautic as the most important E-mail Marketing Pillar in my company.
Now we’re mostly using it (along with Amazon SES) for E-mail marketing campaigns aimed to new potential customers but in a couple of weeks I would like to use it for Drip Campaigns targeting our existing customers (instructions, tips, re-marketing, offers).
In order to work with existing customers I need to costantly (daily basis) import and update contacts and custom fields from our custom Company Software.
After long testing I’ve finally managed to use Mautic as the most important E-mail Marketing Pillar in my company.
Now we’re mostly using it (along with Amazon SES) for E-mail marketing campaigns aimed to new potential customers but in a couple of weeks I would like to use it for Drip Campaigns targeting our existing customers (instructions, tips, re-marketing, offers).
In order to work with existing customers I need to costantly (daily basis) import and update contacts and custom fields from our custom Company Software.
Hey @g.storelli Sorry to awaken this old thread, but I’m running into the same question today with Mautic 5.2.4, was wondering if you found a solution that worked well?
IMO the best way to integrate two systems would be using the REST API that is provided by Mautic. You can also use Mautic campaigns to send off webhooks to your other systems.
If this is to technical there is always the ability to do a custom import, i know there is a plugin that does this I am just not sure it is supporting Mautic 5 yet.
Thanks @mikew I did see someone mention a plugin, I was checking mailertizer but wasn’t seeing any plugins for importing listed. Although several interesting options there for other tasks I’ll have to explore.
I do have a couple API scripts that run, it seemed to be a little slow. Our contact list is about 5.5M records, we would be lo0king at monthly updates where probably a solid 20-30% of the records would have some custom field change. A big part of what I use the API for is to build segments, unfortunately, there doesn’t seem to be a mechanism to empty a segment or even query the membership of the segment without querying every contact which adds a layer of complexity to updating them.
I have a mautic dev site running, and I’m playing with just uploading a CSV file of changes, so that might be workable, but those files will get large, which could be problematic. And update of 74,000 I tested last night only took 50 minutes to run, we could chunk these into smaller groups and run them through too. Do you know if there’s a way to trigger imports without using the UI top upload and create the import?
I haven’t looked at webhooks, not too familiar with that structure, but If I’m understanding it would require the source system to have something to connect to?
So I will check out my dev team availability to look into the custom import plugin, it does also allow for parallel imports which could speed things up in your case. Have you checked that there is an index on the unique identifier you are using in the leads table, usually this is email, however we have often changed it to support different environments..
Are you running RDS or some other independent database seperated from the app ?
We have many iGaming customers that we are brining data in via tracking script, api and then the majority of the chunk on custom imports due to limitations on API and queing was becoming problematic. We do this running hourly delta imports…
Interesting to understand the processing power on your MySQL server/RDS
We have 2 fields as unique, the stadnard email and then an internal ID we added as a custom field. This is set to indexable and unique identifier in Mautic config. I see an index in the database on this field as well as one for the combination of the two fields.
I did also recently set the operation to AND for the purpose of merging.
Our database is running in a docker container on the same machine currently, this server hosts a few small wordpress sites and a couple other custom php sites, so I wanted to insulate the database in case we needed mautic to have a special version or decided to maybe move to Maria.
A couple recent imports went pretty quick I think. We did a 9439 record import in 10 minutes and a 5546 import in 5 minutes. The largest previous ones we did were 552562 in 12 hours and 904875 in 26 hours.
I’m definitely interested in the plugin, we are still in sort of a trial phase, although I think Mautic is performing well enough that we will end up fully adopting it. As we expand the use, our use case might be a little different than some, but we’ll have a couple dozen maybe users on the system. We would potentially allow them to upload lists on their own, but our setup is that we have a single central list, we don’t want individuals to “own” contacts per se, so it would require a lot of trust that their uploads will be well formatted and not stomp on contacts that other users need.
I’m also interested in your hourly delta import setup. We wouldn’t need something that often, but interested in the process you use. I have some python scripts that we use to generate segments based on criteria from an external system, and part of that would include updates and new contact creation in the future. I’m running some tests to fully understand what happens when we import existing contacts that may have different values or missing values.
What kind of resources are you running ? Import seems rather slow.
On the delta what we do is we pull a CSV that different platforms drop on an S3 bucket, have a bash script running behind a cron that does this for us on an hourly basis, use the custom import (after we have defined the import template) and run it from the bash script (like in command line, so it is mautic:import:directory which will create the file for import in the directory, then mautic:import which takes care of the import.
The delta is obviously given from the platform side, this is a great way for platforms that do not want to spend the time/money on API integration but still want relatively “live” data.
Hi @mikew
I’m running a dedicated server from Dreamhost, 64GB RAM. I have memory_limit set to 4G at the moment, as this server does host a few of our custom apps and a couple small wordpress sites, but I can bump that up (kind of forgot I had it set that low while running some tests until you asked )
Thanks for the workflow description on the import, I think I may have a misunderstanding - does mautic:import use whatever file is in var/import? I can certainly write a process that produces the file in that folder, but I was thinking there was more to it than just putting a file there and calling the console import command?
There is definitely more than just calling that command, you need to do the mapping as well, that is why the plugin was so good. I think i say that it is now also ported to Mautic 5, check our “mautic custom import github” and you should find the plugin, it basically takes your last manual import mapping settings, uses this for mapping, specifies your import directory and carries out the import.
I figured it would require more : )
Tonight I completed a python script that reads a CSV, identifies existing contacts, creates any that aren’t in Mautic, then creates a segment from the full list. So I can certainly use the same procedure to generate a general update list from our main data source. I think an import would be more efficient than the API call process, but this works well for creating specific segments as we need.
I feel dense here - but can you point me to this github project, the only one coming up for me is from mtcextendee but it’s not been updated in 5 years.
Nope that is the correct one that you have found. I can confirm that it still works with Mautic 4.x (as we use it there) I have not tested in on Mautic 5 yet.
Agreed on the import vs API, I have found that when hitting the API with high traffic it can eat away at server resources and then you need to get into setting up queues etc…
)