Export to CSV and Excel creates contact duplicates

Development
1

I noticed that there is a problem with exporting 11000 of contacts to CSV or Excel. Some contacts are duplicated and others are not. The total amount of contacts in the export is the expected 11000.

So far I noticed that the later contacts (with higher ids) are more affected.

It’s a problem with the latest v3 branch and with the latest 4.x branch.

2

Can you clarify if you mean export from reports or export from the contact list? The export from reports can be done via a command which has much higher time and memory limits than a HTTP request. The export feature from the contact list does not have an option to do it on a background so it will run on large number of records until the memory or time limit is hit for the HTTP request. You can increase those limits on server if that’s what you want to do.

I’d recommend to go with the scheduled reports where the report is generated in the background and sent to your email:

3

Thanks @escopecz for that insight. I will try the way via report as well.

However, it would be great if the export via contacts works too:

image
image2048×770 138 KB

Or at least shows a message that it didn’t work.

Since I get a CSV, just with duplicate entries, I don’t think it is a problem with the memory limit or the timeout?

image
image1746×494 85.4 KB

4

The duplicates looks like a bug. But I’m not able to reproduce it. When I export my contacts from the contact list I get only unique IDs. Please try to find steps to replicate the duplicates. Do you have another Mautic instance where it also exports duplicates? There must be something unique on that instance.

5

Hi,
Do you have multiple keys / unique identifiers in this instance?
They seem to have the same ID, which is unique…

1 Like
6

It looks like the problem is the sorting of the database query. When the query sorts by last_activity we have the problem. When the query sorts by id everything is ok. Sorting by last_active is used by default.

7

We replicated it on 4 different instances. But always with the same database. So far I did only see it with contact lists >10k. But we have a test sample of 10k contacts now that we plan to import and export again in a fresh Mautic instance. Will update you soon.

8

Hi,

Seems the OFFSET clause doesn’t work well when the table is sorted by a column containing all NULL values. We get this issue when there are more than 10,000 client records in the DB.

Could you please try the next steps to reproduce it

  1. Import the mock data (10,000 or more)
  2. Open the contact list
  3. Order by the field with no value (e.g. last_activity if data was imported)
  4. Export to CVS
  5. Open file, sort by id
  6. There are twice
  7. Open the contact list again
  8. Order by id
  9. Export to CVS
  10. Open file, sort by id
  11. Everything ok

I can send mock clients to import.

9

Related PR Export to CSV and Excel creates contact duplicates by volha-pivavarchyk · Pull Request #10849 · mautic/mautic · GitHub