# Bulk import CSV recommendations needed 500K contacts

**URL:** <https://forum.mautic.org/t/bulk-import-csv-recommendations-needed-500k-contacts/9934>\
**Category:** Product Support\
**Created:** [November 20, 2017, 8:09am UTC](https://forum.mautic.org/t/bulk-import-csv-recommendations-needed-500k-contacts/9934 "2017-11-20T08:09:56Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![paulc](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mautic.org/paulc/32/239_2.png) [@paulc](https://forum.mautic.org/u/paulc)\
**Post date:** [November 20, 2017, 8:09am UTC](https://forum.mautic.org/t/bulk-import-csv-recommendations-needed-500k-contacts/9934/1 "2017-11-20T08:09:56Z")

</div>

We have 500k contacts to bring in.  
  
  
  
I chunked them into 20k lines in a CSV and started importing. We have about 20 custom fields we are bringing in for these contacts, plus the regular fields for contact address and IP.  
  
  
  
The import became very slow after 200k contacts were loaded. We are getting about 1 user per second right now importing.  
  
  
  
I tried setting a config to import 2 files at the same time to see if I could get more utilization on the server. So far nothing has worked. Seeing about 15% utilization on the ec2 instance and about 20% on the dB.  
  
  
  
Setup is in AWS: these are dedicated for this mautic instance.  
  
Ec2 2cpu / 4gb ram (php 7.1)  
  
RDS 2 cpu / 4gb ram (MySQL)  
  
  
  
Any help or info on best practices would be appreciated.  
  
  
  
We did try using the API at first and that did not go well. We were seeing about a contact importing every 4 - 10 seconds.

---

<div class="post-metadata">

**Author:** ![paulc](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mautic.org/paulc/32/239_2.png) [@paulc](https://forum.mautic.org/u/paulc)\
**Post date:** [November 20, 2017, 8:09am UTC](https://forum.mautic.org/t/bulk-import-csv-recommendations-needed-500k-contacts/9934/2 "2017-11-20T08:09:56Z")

</div>

We have 500k contacts to bring in.

I chunked them into 20k lines in a CSV and started importing. We have about 20 custom fields we are bringing in for these contacts, plus the regular fields for contact address and IP.

The import became very slow after 200k contacts were loaded. We are getting about 1 user per second right now importing.

I tried setting a config to import 2 files at the same time to see if I could get more utilization on the server. So far nothing has worked. Seeing about 15% utilization on the ec2 instance and about 20% on the dB.

Setup is in AWS: these are dedicated for this mautic instance.  
Ec2 2cpu / 4gb ram (php 7.1)  
RDS 2 cpu / 4gb ram (MySQL)

Any help or info on best practices would be appreciated.

We did try using the API at first and that did not go well. We were seeing about a contact importing every 4 - 10 seconds.

---

<div class="post-metadata">

**Author:** ![bizcrony](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mautic.org/bizcrony/32/409_2.png) [@bizcrony](https://forum.mautic.org/u/bizcrony)\
**Post date:** [November 20, 2017, 9:57am UTC](https://forum.mautic.org/t/bulk-import-csv-recommendations-needed-500k-contacts/9934/3 "2017-11-20T09:57:20Z")

</div>

@paulc we can directly import from PHPmyadmin what you looking for ?

---

<div class="post-metadata">

**Author:** ![paulc](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mautic.org/paulc/32/239_2.png) [@paulc](https://forum.mautic.org/u/paulc)\
**Post date:** [November 20, 2017, 5:03pm UTC](https://forum.mautic.org/t/bulk-import-csv-recommendations-needed-500k-contacts/9934/4 "2017-11-20T17:03:41Z")

</div>

Ohh… please explain. That is not mentioned anywhere. I’d prefer that method.

---

<div class="post-metadata">

**Author:** ![bizcrony](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mautic.org/bizcrony/32/409_2.png) [@bizcrony](https://forum.mautic.org/u/bizcrony)\
**Post date:** [November 20, 2017, 6:17pm UTC](https://forum.mautic.org/t/bulk-import-csv-recommendations-needed-500k-contacts/9934/5 "2017-11-20T18:17:38Z")

</div>

add me on skype

---

<div class="post-metadata">

**Author:** ![denispavliha](https://avatars.discourse-cdn.com/v4/letter/d/e0b2c6/32.png) [@denispavliha](https://forum.mautic.org/u/denispavliha)\
**Post date:** [November 24, 2017, 2:32pm UTC](https://forum.mautic.org/t/bulk-import-csv-recommendations-needed-500k-contacts/9934/6 "2017-11-24T14:32:34Z")

</div>

Since this is a community, why not sharing the script with the rest? 🙂

I actually don’t recommend direct database queries (although I have done it myself as well) but in such case, I believe you could code your own import script (e.g. PHP+mysqli)

- read CSV (in chunks, of course)
- insert into … on duplicate key …
- clear Mautic cache at the end

@bizcrony , could you possibly elaborate your solution?

Best,  
D.

---

<div class="post-metadata">

**Author:** ![ninjoan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mautic.org/ninjoan/32/1005_2.png) [@ninjoan](https://forum.mautic.org/u/ninjoan)\
**Post date:** [November 24, 2017, 4:26pm UTC](https://forum.mautic.org/t/bulk-import-csv-recommendations-needed-500k-contacts/9934/7 "2017-11-24T16:26:52Z")

</div>

@denispavliha He is not going to share the solution since he is charing for this job.

---

<div class="post-metadata">

**Author:** ![denispavliha](https://avatars.discourse-cdn.com/v4/letter/d/e0b2c6/32.png) [@denispavliha](https://forum.mautic.org/u/denispavliha)\
**Post date:** [November 24, 2017, 4:40pm UTC](https://forum.mautic.org/t/bulk-import-csv-recommendations-needed-500k-contacts/9934/8 "2017-11-24T16:40:47Z")

</div>

Too bad.
