Changing how Mautic 5 handles email from send immediately to queue

[hotstryker]

Your software
My Mautic version is: V5.0.0-rc2
My PHP version is: 8.1
My Database type and version is: MariadDB 10.6

When I did the original install, on Step 2 I was asked “How should emails be spooled and delivered?” Options being “Send Immediately” or “Queue”. I had selected “Send Immediately” and proceeded with the install.

Now however, after reading some posts on deliverability, I have determined that it’s in my best interest to use the Queue feature to start with.

Everything I can find in the documentation and forums say to go to Configuration → Email Settings → How should email be handled? and change it to Queue. But I do not have that setting anywhere in M5.

How do I change this?

[hotstryker]

[joeyk]

I think its Queue settings (In the left submenu)

[hotstryker]

Looked at that and I don’t think that’s it either. Not sure as there’s nothing in the docs about that.

I did find in the docs this morning that it’s saying to switch to queued sending all you need to do is set a cron job…

bin/console messenger:consume email_transport

If that’s the case, then what is sending them now? the 3 cron jobs I have running are:

bin/console mautic:segments:update
bin/console mautic:campaigns:update 
bin/console mautic:campaigns:trigger 

Do you know which one of those I would need to remove when adding in the one required for queuing in order to prevent the immediate sending?

Thanks.

[hotstryker]

I am also not understanding how that one cron job alone determines the “rate” in which the queued emails are sent out (i.e. say I only want to send 100/hour and the cron job is running every 5 minutes, that would be 8 emails per cron execution).

[hotstryker]

Is there anyone at all in the community that can assist me in getting M5 properly setup to send emails via queue?

[marko008]

https://docs.mautic.org/en/5.x/configuration/settings.html#mautic-doesn-t-use-queues-by-default
You need to set queue setting scheme to doctrine and host to default
In CRON jobs after messenger:consume email_transport you can add --batch-limit=X to limit how many messages you want to send at once.

[marko008]

don’t forget to add messenger:consume email

[hotstryker]

Thanks for the help @marko008. It’s not working just yet. I’m getting errors from the cron jobs. When I look at the logs, the first error I get is that The "--batch-limit" option does not exist. I tried it on both command of messenger:consume email_transport and then messenger:consume email

Also, when I don’t have the --batch-limit= on it, I get the error message that The receiver "email_transport" does not exist.

[tcastelnau]

Hi guys,

I have the same issue. I can’t find a way to limit the number of emails sent every time a cron job runs.

Does anyone have an idea ?

[tcastelnau]

I found more information. The syntax is not correct. It would be /bin/console messenger:consume email --limit=20 which would send 20 emails each time a cron job runs.

For more options about messenger:consume, you can run :
php /var/www/html/bin/console messenger:consume -- help

I got rid of the errors you mentioned but I still can’t send emails with my campaigns, although Mautic marks the emails as sent.

I have the email configured in the “Email Settings”, tested, OK.
In “Queue Setting”, Emails are configured to “Doctrine” and “Default” for the host.

And I have a cron job that runs every 10 minutes :

*/1 * * * * php /var/www/html/bin/console messenger:consume email --limit=1

What’s the problem?

By checking the log i can see the errors are gone.

[jet1]

To debug, I recommend to comment out your cron commands and launch everything directly from terminal.

1. Make sure the particular email is “published” and date is set to a moment in past:

2. Make sure your “queue” settings are set to doctrine
   

   

3. run emails generator

php -d memory_limit=-1 ./bin/console mautic:broadcasts:send --batch=30 --limit=2000 --id=[YOUR-EMAIL-ID]

in terminal, from mautic working dir (note that you need to get your email id from mautic emails list - this arg helps to isolate sending and debugging for 1 particular campaign)

This will start feeding your queue with generated emails.
If the process exists fast - this means you have a problem with your email generation setup. If it starts working - you will see it, it takes many seconds.

4. Run actual email sending (you can start another terminal instance for this):

php ./bin/console messenger:consume email --memory-limit=8280M -vv

So, you need two processes running when you want to send queued email campaign: first one is generating emails and putting it into queue; another one is working on the queue.

If it works from terminal, you can start moving these commands either into crontab or daemon like supervisorctl (my preferred method).

[tcastelnau]

Thank you so much! I followed your suggestions and I was finally able to troubleshoot and set my desired limit rate! Emails are now sent properly.

Awesome!

[hotstryker]

@jet1

Thank you for that! When running from CLI, everything seems to work.

The only thing I’m noticing is that the php ./bin/console messenger:consume email -vv doesn’t exit when you run it. This causes a problem when I have it running via cron, as I end up with many many instances of it running until the server bogs down to the point of locking up the server.

What do I need to do to get it to exit after each execution?

Also, the -vv isn’t reporting anything to the console. Is that normal?

[jet1]

messenger:consume is designed to be a long-lived process so it can be run via supervisortctl or systemd (restarted from time to time, e.g. via --memory-limit=1280M. you can condition it to work inside cron via --time-limit=360 arg or/and --limit=10 (amount of emails) as well.

to see all available options, run ./bin/console help messenger:consume

[hotstryker]

Awesome. Thank you!

[hotstryker]

I just thought of 1 more question. All this has been done with testing Broadcast emails. What about Campaign emails? Those have been sending out immediately even without the bin/console mautic:broadcasts:send and bin/console messenger:consume email commands.

It’s probably not as critical in my situation as the likelihood of a campaign triggering that many emails at once is fairly rare, but would be nice to understand what would be needed.

[mzagmajster]

Mautic brodcast send is processed in batches how often it gets called it depends on your cron config.

[ahiniesta]

Hello. I have Sendgrid as delivery method, but I get:

php ./bin/console messenger:consume email --memory-limit=8280M -vv

[OK] Consuming messages from transport “email”.

// The worker will automatically exit once it has exceeded 8280M of memory or received a stop signal via the
// messenger:stop-workers command.

// Quit the worker with CONTROL-C.

In SyncTransport.php line 37:

[Symfony\Component\Messenger\Exception\InvalidArgumentException]
You cannot receive messages from the Messenger SyncTransport.

Exception trace:
at /var/www/html/vendor/symfony/messenger/Transport/Sync/SyncTransport.php:37
Symfony\Component\Messenger\Transport\Sync\SyncTransport->get() at /var/www/html/vendor/symfony/messenger/Worker.php:101
Symfony\Component\Messenger\Worker->run() at /var/www/html/vendor/symfony/messenger/Command/ConsumeMessagesCommand.php:234
Symfony\Component\Messenger\Command\ConsumeMessagesCommand->execute() at /var/www/html/vendor/symfony/console/Command/Command.php:298
Symfony\Component\Console\Command\Command->run() at /var/www/html/vendor/symfony/console/Application.php:1058
Symfony\Component\Console\Application->doRunCommand() at /var/www/html/vendor/symfony/framework-bundle/Console/Application.php:96
Symfony\Bundle\FrameworkBundle\Console\Application->doRunCommand() at /var/www/html/vendor/symfony/console/Application.php:301
Symfony\Component\Console\Application->doRun() at /var/www/html/vendor/symfony/framework-bundle/Console/Application.php:82
Symfony\Bundle\FrameworkBundle\Console\Application->doRun() at /var/www/html/vendor/symfony/console/Application.php:171
Symfony\Component\Console\Application->run() at /var/www/html/bin/console:16

and campaign messages are not sent. Any help?

[mzagmajster]

You do not have async consumer configured.