Bulk import CSV recommendations needed 500K contacts

Product Support
1

We have 500k contacts to bring in.



I chunked them into 20k lines in a CSV and started importing. We have about 20 custom fields we are bringing in for these contacts, plus the regular fields for contact address and IP.



The import became very slow after 200k contacts were loaded. We are getting about 1 user per second right now importing.



I tried setting a config to import 2 files at the same time to see if I could get more utilization on the server. So far nothing has worked. Seeing about 15% utilization on the ec2 instance and about 20% on the dB.



Setup is in AWS: these are dedicated for this mautic instance.

Ec2 2cpu / 4gb ram (php 7.1)

RDS 2 cpu / 4gb ram (MySQL)



Any help or info on best practices would be appreciated.



We did try using the API at first and that did not go well. We were seeing about a contact importing every 4 - 10 seconds.

2

We have 500k contacts to bring in.

I chunked them into 20k lines in a CSV and started importing. We have about 20 custom fields we are bringing in for these contacts, plus the regular fields for contact address and IP.

The import became very slow after 200k contacts were loaded. We are getting about 1 user per second right now importing.

I tried setting a config to import 2 files at the same time to see if I could get more utilization on the server. So far nothing has worked. Seeing about 15% utilization on the ec2 instance and about 20% on the dB.

Setup is in AWS: these are dedicated for this mautic instance.
Ec2 2cpu / 4gb ram (php 7.1)
RDS 2 cpu / 4gb ram (MySQL)

Any help or info on best practices would be appreciated.

We did try using the API at first and that did not go well. We were seeing about a contact importing every 4 - 10 seconds.

3

@paulc we can directly import from PHPmyadmin what you looking for ?

4

Ohh… please explain. That is not mentioned anywhere. I’d prefer that method.

5

add me on skype

6

Since this is a community, why not sharing the script with the rest? :slight_smile:

I actually don’t recommend direct database queries (although I have done it myself as well) but in such case, I believe you could code your own import script (e.g. PHP+mysqli)

  • read CSV (in chunks, of course)
  • insert into … on duplicate key …
  • clear Mautic cache at the end

@bizcrony , could you possibly elaborate your solution?

Best,
D.

7

@denispavliha He is not going to share the solution since he is charing for this job.

8

Too bad.