Thanks, @rcheesley for tagging me on this topic.
I will describe how things work and what kind of changes we bring to M5, I work a lot on the email bundle and I can support on this issue.
The current implementation of M4, has two ways to handle email sending:
-
Immediate: where the calls are happening to the transport that you want to send through directly, and the errors are handled by that transport. For example, if you use SendGrid API in the immediate mode, if the API authentication fails, the send will not be counted, and it will not be retried, nor the contact will be marked as DNC.
mautic/SendGridWrapper.php at 9cb69551fbff6046a1e1476aa23d2219e6e41c36 · mautic/mautic · GitHub
This sending mode has advantages of handling large batch sizes in a single API call, and direct API calls to the MTA.
It is disadvantage,that it consume lots of memory from PHP process, it timeouts, and may result in sending duplicates.
The queue mode: it store the emails as files in the var/spool folder, each email is saved as a file and processes each file in a single run. This mode is the source of all the issues
How it works?
When you select the queue mode, the queue transport is selected which basically takes the email message and converts it into a file that is stored on the disk, regardless of the status of the MAT. therefore Mautic marks all the emails as sent when you use the queue mode because it set the files on disk for the cron job to process them later. It fails in one case and emails are not marked as sent when the disk is full, has writing permissions issues, etc.
The issue of DNC, happens here
This command connects to the real transport (MTA), which tries to send, so if the API request fails for some reason like authentication error, rate limit issues, etc. the command will mark the file as .tryagain and retry up to 3 times before it marks the contact as DNC, even if the email was not sent.
Of course this implementation is not logical and DNCs should happen using webhooks and callbacks by MTA.
The good news is that we will be moving away form swiftmailer to a new mailing system that uses symfony/mailer/ and symfony/messeneger. you can see the PR here
Next Tuesday we will have a call about the new changes in the mailing systems, so anyone who likes to join please message me on slack with your email and I will add you.
I hope this covers the issue and help you better understand how EmailBundle works