Large leads import - Stuck

Product Support
1

Hi guys,



I’m trying to import a lot of 10,000 leads using the CSV import (Very easy by the way), but it stucks at 4800 leads (100 leads per batch)

I have actually >500,000 leads to import (Cross countries), any idea how to solve this? Or any other way to import the leads?



Thank you!

1 Like
2

Hi guys,

I’m trying to import a lot of 10,000 leads using the CSV import (Very easy by the way), but it stucks at 4800 leads (100 leads per batch)
I have actually >500,000 leads to import (Cross countries), any idea how to solve this? Or any other way to import the leads?

Thank you!

3

It would help to know the reason the import got stuck. Could you check Mautic logs (app/logs folder) and server log if there are some error messages?

4

Thanks for the hint.
From the log I saw some error on PHP execution time out. I’ve raised the max executime time and it works well now.

But I though to upload per batch was solving this issue haha.

Thank you

5

Hi I was wondering if you could lend some assistance. I am having a similar issue with a much smaller batch (1200 records) it seems to stop at ~700, I have considered just uploading the remainder but I thought if there was a solution it would be worth it for the future. What action was necessary to increase the execution time value? I would like to test if this is the appropriate solution for my issue.

6

It is limitation of the server. Here is article about how to increase that limit https://www.devside.net/wamp-server/apache-and-php-limits-and-timeouts

7

Wow, from reading the forums I now realize that the “limit” in the import is BATCH SIZE…I thought it meant TOTAL limit of the import. I have been failing at this all day. Might be awesome to have “batch size” instead of “limit”. That would have saved me a day. :wink: Importing now…

8

THANK YOU! Super helpful

9

Enough of my contacts are not getting imported.
The imports work but miss out the vast majority of emails from the .csv file and only adds a small percentage to the mautic contacts.

Repeatedly importing on the command line with multiple parallel processes helps a bit but still missing lots of email addresses.

[quote]bitnami@debian:/opt/bitnami/apps/mautic/htdocs/app/logs$ sudo php /opt/bitnami/apps/mautic/htdocs/app/console mautic:import -l 17000
Import 25 with 426907 rows is starting.
17000/17000 [============================] 100%
Import has been delayed: unknown
308600 lines were processed, 9 items created, 306406 items updated, 2185 items ignored in 195.13 s[/quote]

I have a cron job set up as well.

0,15,30,45 * * * * sudo php /opt/bitnami/apps/mautic/htdocs/app/console mautic:import -l 1000     --no-interaction --no-ansi 2>&1 >> /opt/bitnami/apps/mautic/htdocs/app/logs/mautic.import.txt

The only hack seems to be repeatedly manually importing the .csv files again and again even though their status is already Imported & 100%. Each time a few more emails are added!

The import process is too slow and is not working properly, especially for large lists.

Status Source file Runtime Progress Total rows Inserted Updated Ignored Date Added ID Imported Internet_Marketing Real Estate.csv 1 minute 22 seconds 100% 61656 24098 37557 1 Today, 6:59 am 22 Imported 1million p3.csv 4 minutes 20 seconds 100% 340350 6672 331635 2043 Today, 5:56 am 21 Imported 1million p3.csv 1 minute 39 seconds 100% 340350 5343 332973 2034 Yesterday, 10:09 pm 20 Imported 1million p2.csv 2 minutes 36 seconds 100% 426907 8543 415368 2996 Yesterday, 10:08 pm 19

Changes to these settings, does improve the above but doesn’t fix the “Import has been delayed: unknown” issue:

  1. Increase PHP limits:

max_execution_time = 600 max_input_time = 600 memory_limit = 1024M

https://www.devside.net/wamp-server/apache-and-php-limits-and-timeouts

  1. .CSV must be UTF-8 encoded charset:
    Excel > FILE > Save As > Tools > Web Options > Encoding > Save this document as > Unicode UTF-8
    https://help.surveygizmo.com/help/encode-an-excel-file-to-utf-8-or-utf-16

  2. Excel > FILE > Save As… “CSV (MS-DOS)”