Large Contact Sync Best Practices

Development
1

Hey all,

I’m exploring syncing contacts from our proprietary database into Mautic. This will be a one-way sync, from our database into Mautic, not the other way around. The database has over 300,000 unique users and that number is growing quickly. What’s the recommended architecture for doing something like this?

My thoughts:

  1. We run our internal system on PHP and MySQL so it would be easy to make additions/updates to the mt_leads table directly (our Mautic database is hosted on AWS RDS) but, while performant, that doesn’t seem right for a bunch of other reasons.
  2. I could use the Mautic API to check over the entire listing of contacts but it seems like it would be relatively slow and inefficient to check for changes on this many records via a REST API.
  3. I could try to make updates to Mautic in real-time as we make updates in our system but I don’t want to slow down our production system at all for this.
  4. I could run a cron job on our main system that uses the Mautic API to sync only records that have changes since the last cron job ran.

Option 4 is currently my best option but I don’t know if there are other options out there or someone has tackled this type of thing before and can offer some insight? What do you guys recommend?

Thanks!

2

I agree that option 4 sounds best.
You can also check this plugin That will help you to automate CSV file imports to Mautic

3

For future readers, I ended up going with a mix of option 1 and option 4. I first pull the full list of relavent Mautic contacts directly from the mautic database. I then use that to compare to my list and when there are differences in the relavent fields, I use the Mautic API to make the updates. This ended up being far more performant than trying to pull thousands of contacts via the Mautic API, but it still uses the API to make changes so I don’t have any of the drawbacks of direct database manipulation.

Also, I tried to use the batch update API endpoints on the write, but those are not documented so it’s difficult. Also, it appears that the batch update endpoints only work on certain elements (contacts) and not others. So, for example, I could bulk update or bulk add contacts but then I still had to loop through each of them to add them to their respective companies. Ended up not being worth the hassle for me…