# Messenger:consume redis concurrency for better queued emails performance. Error: "Could not acknowledge redis message"

**URL:** <https://forum.mautic.org/t/messenger-consume-redis-concurrency-for-better-queued-emails-performance-error-could-not-acknowledge-redis-message/30855>\
**Category:** Product Support\
**Created:** [February 9, 2024, 8:45pm UTC](https://forum.mautic.org/t/messenger-consume-redis-concurrency-for-better-queued-emails-performance-error-could-not-acknowledge-redis-message/30855 "2024-02-09T20:45:16Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![jet1](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mautic.org/jet1/32/13361_2.png) [@jet1](https://forum.mautic.org/u/jet1)\
**Post date:** [February 9, 2024, 8:45pm UTC](https://forum.mautic.org/t/messenger-consume-redis-concurrency-for-better-queued-emails-performance-error-could-not-acknowledge-redis-message/30855/1 "2024-02-09T20:45:16Z")

</div>

**Your software**  
My Mautic version is: 5.0.3  
My PHP version is: 8.1.2  
My Database type and version is: mysql 8.0.36-0ubuntu0.22.04.1 for Linux on x86\_64

**Your problem**  
My problem is:  
I am trying to send 500k emails ( channels → emails )  
I am trying to increase throughput of my worker:

```auto
php /var/www/mtc/bin/console messenger:consume email --memory-limit=8280M

```

I managed to use supervisorctl to run this command in 1 thread successfully - thanks to documentations - but it’s very slow (around 4-5 emails per second) and I would like to run it in 5 threads instead to match my allowed Amazon SES speed (100+ emails per second).

These errors are showing in the log:  
Symfony\Component\Messenger\Exception\TransportException  
Could not acknowledge redis message “1584889235265–0”.

Steps I have tried to fix the problem:

> <https://github.com/symfony/symfony/issues/35358>
>
> \*\*Symfony version(s) affected\*\*: 4.3.0+
> 
> \*\*Description\*\*  
> As far as I've see…n it isn't possible to have two consumers on the same transport. We have the possibility to define the stream, group and consumer per transport. But none of this config options allow to have two or more consumer commands \`messenger:consume\` running on the same transport.
> 
> 
> \*\*How to reproduce\*\* 
> \- Use a config similar to this:
> \`\`\`
> framework:
> messenger:
> transports:
> async\_priority\_high:
> dsn: 'redis://redis-server/priority\_high'
> \`\`\`
> \- Run two or more consume commands in parallel for the same transport \`bin/console messenger:consume async\_priority\_high\`
> \- Send a message through the \`async\_priority\_high\` transport.
> 
> Now there is the chance that one of the commands exits with an exception like:
> \> Symfony\\Component\\Messenger\\Exception\\TransportException: Could not acknowledge redis message "1579079882286-4".
> 
> This also means that the message was consumed twice.
> 
> ~~Unfortunately this happens because the message is only being acknowledged (Redis \`XACK\`) after the message is handled and leaves it for the duration of the handling in the stream to be consumed by the other consumer command too.~~
> EDIT: This is not true. The issue does only happen once if the two consumers are started around the same time and at least one message is already in the queue.
> 
> \*\*Possible Solution\*\*  
> ~~Using redis \`XACK\` right after \`XREADGROUP\` in \`Symfony\\Component\\Messenger\\Transport\\RedisExt\\Connection:get()\` and using the \`Symfony\\Component\\Messenger\\Transport\\RedisExt\\Connection:reject()\` method to re-add the message to the redis stream.~~
> EDIT: With the edit above, this isn't a solution.
> 
> \*\*Additional context\*\*  
> I am sorry if I did get the usage of messenger wrong, but IMO having multiple consume commands running at the same time should be supported.

I am having trouble setting this up in mautic environment. I am wondering if anyone managed to do it?

---

<div class="post-metadata">

**Author:** ![jet1](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mautic.org/jet1/32/13361_2.png) [@jet1](https://forum.mautic.org/u/jet1)\
**Post date:** [February 9, 2024, 9:07pm UTC](https://forum.mautic.org/t/messenger-consume-redis-concurrency-for-better-queued-emails-performance-error-could-not-acknowledge-redis-message/30855/2 "2024-02-09T21:07:11Z")

</div>

here are the docs describing how ENV vars can be used for symfony `messenger.conf` file (which is absent in case of mautic setup) to achieve redis concurrency:

> **[Messenger: Sync & Queued Message Handling (Symfony Docs)](https://symfony.com/doc/current/messenger.html#supervisor-configuration)**

I would like to understand how I can incorporate this approach into mautic use case.

---

<div class="post-metadata">

**Author:** ![jet1](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mautic.org/jet1/32/13361_2.png) [@jet1](https://forum.mautic.org/u/jet1)\
**Post date:** [February 9, 2024, 9:21pm UTC](https://forum.mautic.org/t/messenger-consume-redis-concurrency-for-better-queued-emails-performance-error-could-not-acknowledge-redis-message/30855/3 "2024-02-09T21:21:05Z")

</div>

I would like to clarify that the error “Could not acknowledge redis message” arises only when `numprocs=2` is used in supervisorctl (to have several instances of the worker)

---

<div class="post-metadata">

**Author:** ![jet1](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mautic.org/jet1/32/13361_2.png) [@jet1](https://forum.mautic.org/u/jet1)\
**Post date:** [February 10, 2024, 9:03am UTC](https://forum.mautic.org/t/messenger-consume-redis-concurrency-for-better-queued-emails-performance-error-could-not-acknowledge-redis-message/30855/4 "2024-02-10T09:03:49Z")

</div>

After struggling with RAM usage on my server (64GB + 120GB swap couldn’t bear with Mautic spool on Redis!)  
I have managed to implement a bash script with backpressure algorithm, which I think is potentially a better approach than cron, because spool queue generation script and worker (which actually sends emails via SES) are adapting to each other: 1000 emails are generated and then the script checks how spool emails count reduces before sending another chunk of emails.

**Attention:** this email requires a small patch to `/app/bundles/ChannelBundle/Command/SendChannelBroadcastCommand.php`:  
(because I couldnt find an easy way to use SQL to retrieve “pending” emails count for a particular email item.

```php
        $table->render();
 // new lines which allow awk to extract amount of emails put into spool. 
// @TODO: find a console command to extract "pending" emails count from email and replace this!
        $output->writeln('');
        $output->writeln('emails_sent:' . $counts['success']);

        $this->completeRun();

```

The sending script:

```bash
#!/bin/bash

# This is the email ID from [mautic]/s/emails: make sure the email is published and published data is in past!
val1=1246

while true; do
    # Run the PHP command directly and stream its output, including both stdout and stderr
    output=$(sudo -uwww-data -s -- php -d memory_limit=-1 /var/www/mtc/bin/console mautic:broadcasts:send --batch=50 --limit=1000 --id=$val1 2>&1 | tee /dev/tty)
    php_exit_status=${PIPESTATUS[0]} # Capture the exit status of the PHP command

    # Check if PHP command execution was successful
    if [$php_exit_status -ne 0]; then
        echo "PHP command exited with error code $php_exit_status"
        exit $php_exit_status
    fi

    # Extract the number of emails sent from the output
    emails_sent=$(echo "$output" | awk -F ':' '/^emails_sent/{gsub(/ /, "", $2); print $2}')

    # Print debug
    echo "[$(date --iso-8601=seconds)] emails_put_to_queue: $emails_sent"

    # Exit with '0' success if emails_sent is =0 or undefined or less than 0
    if [-z "$emails_sent"] || ["$emails_sent" -le 0]; then
        echo "Exiting with success because emails_sent is undefined, equal to 0, or less than 0"
        exit 0
    fi

    while true; do
        # Get the length of the messages stream, extracting the integer part
        len=$(redis-cli --raw XLEN messages)

        echo "[$(date --iso-8601=seconds)] emails_in_queue: $len"

        # Break the loop if the stream length is less than 50
        if ["$len" -lt 50]; then
            break
        fi

        # Wait for 30 seconds before checking again
        sleep 30
    done
done

```

## Usage:

I put this script into mautic folder into `send.sh` file and then run this script via `tmux` as `bash ./send.sh` (or allow members of my team to execute via our Jenkins pipeline) - I don’t recommend to run it directly on remote server for 100K+ subs because ssh connection tends to break..

I managed to achieve ~26K emails delivered per hour via AWS SES with this script (with single worker thread), which is a nice improvement compared to ~12K emails per hour which we used to get via older naive scripts which occasionally spammed our /spool directory (Mautic 3) which regularly put our Ubuntu server on its knees…

I would like to note that the script is targeted at Mautic 5 + Redis email queue.

If I manage to setup Redis run concurrently in 3 threads via supervisorctl, I think I would get another performance increase.

---

<div class="post-metadata">

**Author:** ![jet1](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mautic.org/jet1/32/13361_2.png) [@jet1](https://forum.mautic.org/u/jet1)\
**Post date:** [February 17, 2024, 6:51pm UTC](https://forum.mautic.org/t/messenger-consume-redis-concurrency-for-better-queued-emails-performance-error-could-not-acknowledge-redis-message/30855/5 "2024-02-17T18:51:52Z")

</div>

Wow not a lot of responses here, I am wondering if anybody from mautic team even reading these forums from time to time?..

I have finally managed to achieve 50K emails per hour rate today - after more tuning and switching to Doctrine driver for messenger (instead of Redis). This is a huge boost compared to what I have started from.

 ![2024-02-17 at 19.48](https://us1.discourse-cdn.com/flex020/uploads/mautic/original/2X/e/e8aaf40a4bd876b8af8ddef8c8e2353c69749cd5.png)

---

<div class="post-metadata">

**Author:** ![magma](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mautic.org/magma/32/433_2.png) [@magma](https://forum.mautic.org/u/magma)\
**Post date:** [February 17, 2024, 8:48pm UTC](https://forum.mautic.org/t/messenger-consume-redis-concurrency-for-better-queued-emails-performance-error-could-not-acknowledge-redis-message/30855/6 "2024-02-17T20:48:39Z")

</div>

Wow, can you describe what you did? Make some kind of tutorial? Thanks!

---

<div class="post-metadata">

**Author:** ![jet1](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mautic.org/jet1/32/13361_2.png) [@jet1](https://forum.mautic.org/u/jet1)\
**Post date:** [February 21, 2024, 9:03am UTC](https://forum.mautic.org/t/messenger-consume-redis-concurrency-for-better-queued-emails-performance-error-could-not-acknowledge-redis-message/30855/7 "2024-02-21T09:03:40Z")

</div>

Sure, compared to my first post in this thread here is what changed:

1. I installed symfony mailer doctrine driver  
[Messenger: Sync & Queued Message Handling (Symfony 5.x Docs)](https://symfony.com/doc/5.x/messenger.html#doctrine-transport)

2. I changed Mautic queue mail settings to doctrine

 ![2024-02-21 at 09.56](https://us1.discourse-cdn.com/flex020/uploads/mautic/original/2X/a/a28b3a81be1b1a037744fcc8c4e934495a23ef1a.png)

1. I set supervisorctl to run `consume` command in 2 threads

2. I changed my `send.sh` script to check doctrine queue instead of redis queue:

```auto
#!/bin/bash

# put this script in mautic project folder root and run to send single email.
# pass email ID via --id=xxx. This is the email ID from [mautic]/s/emails: make sure the email is published and published data is in past!

val1=""

# Parse command line arguments for --id
for arg in "$@"
do
    case $arg in
        --id=*)
        val1="${arg#*=}"
        shift # Remove --id= from processing
        ;;
    esac
done

# Check if val1 (email ID) is not set and exit if required
if [-z "$val1"]; then
    echo "id required, pass as --id=333"
    exit 1
fi

echo "running for email id: $val1"

while true; do
    # Run the PHP command directly and stream its output, including both stdout and stderr
    output=$( php -d memory_limit=-1 /var/www/mtc/bin/console mautic:broadcasts:send --batch=30 --limit=2000 --id=$val1 2>&1 | tee /dev/tty)
    php_exit_status=${PIPESTATUS[0]} # Capture the exit status of the PHP command

    # Check if PHP command execution was successful
    if [$php_exit_status -ne 0]; then
        echo "PHP command exited with error code $php_exit_status"
        exit $php_exit_status
    fi

    # Extract the number of emails sent from the output
    emails_sent=$(echo "$output" | awk -F ':' '/^emails_sent/{gsub(/ /, "", $2); print $2}')

    # Print debug
    echo "[$(date --iso-8601=seconds)] emails_put_to_queue: $emails_sent"

    # Exit with '0' success if emails_sent is =0 or undefined or less than 0
    if [-z "$emails_sent"] || ["$emails_sent" -le 0]; then
        echo "Exiting with success because emails_sent is undefined, equal to 0, or less than 0"
        exit 0
    fi

    while true; do
        # Get the length of the messages stream, extracting the integer part
        len=$(./bin/console doctrine:query:sql --no-ansi -- "SELECT count(*) from messenger_messages" | awk '/[0-9]+/{print $1; exit}')

        echo "[$(date --iso-8601=seconds)] emails_in_queue: $len"

        # Break the loop if the stream length is less than 50
        if ["$len" -lt 50]; then
            break
        fi

        # Wait for 30 seconds before checking again
        sleep 30
    done
done

```

so now I run the script like this:  
`bash ./send.sh --id=xxxx` (from my `tmux` session or our team Jenkins instance)

Suprisingly, Doctrine queue is working noticeably faster compared to Redis, even in single thread. I think the reason this is happening is that our emails have a lot of links and Mautic uses very weird approach to generated email links; which results that each rendered email put in queue is huge, I mean, HUGE: 1 entry can take up to megabytes, this is a very, very large serialized object if we talk about newsletter for 1M recipients. Redis chokes in write and read. Mysql 8 + Doctrine manages to process this better.

---

<div class="post-metadata">

**Author:** ![raffye.memon](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@raffye.memon](https://forum.mautic.org/u/raffye.memon)\
**Post date:** [April 9, 2024, 3:06am UTC](https://forum.mautic.org/t/messenger-consume-redis-concurrency-for-better-queued-emails-performance-error-could-not-acknowledge-redis-message/30855/8 "2024-04-09T03:06:57Z")

</div>

Hello,

thanks for the script… i am getting output below … mautic email also mark emails as sent but they are never sent . ! they are not even in spool folder… ! i am not where they are sent !

```auto
running for email id: 4

+-----------------------------------------+--------+----------+
| Channel | # sent | # failed |
+-----------------------------------------+--------+----------+
| Email: Resgate seu presente 🎁 - Copy 2 | 0 | 0 |
+-----------------------------------------+--------+----------+
[2024-04-09T02:53:23+00:00] emails_put_to_queue: 
Exiting with success because emails_sent is undefined, equal to 0, or less than 0
root@webmin:/home/copasorte/domains/mautic.copasorte.com/public_html# ./send.sh 
running for email id: 4

Resgate seu presente 🎁 - Copy 2

 1/1 [============================] 100%

+-----------------------------------------+--------+----------+
| Channel | # sent | # failed |
+-----------------------------------------+--------+----------+
| Email: Resgate seu presente 🎁 - Copy 2 | 1 | 0 |
+-----------------------------------------+--------+----------+
[2024-04-09T02:55:21+00:00] emails_put_to_queue: 
Exiting with success because emails_sent is undefined, equal to 0, or less than 0
root@webmin:/home/copasorte/domains/mautic.copasorte.com/public_html# ./send.sh 
running for email id: 4

+-----------------------------------------+--------+----------+
| Channel | # sent | # failed |
+-----------------------------------------+--------+----------+
| Email: Resgate seu presente 🎁 - Copy 2 | 0 | 0 |
+-----------------------------------------+--------+----------+
[2024-04-09T03:02:59+00:00] emails_put_to_queue: 
Exiting with success because emails_sent is undefined, equal to 0, or less than 0

```

---

<div class="post-metadata">

**Author:** ![jet1](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mautic.org/jet1/32/13361_2.png) [@jet1](https://forum.mautic.org/u/jet1)\
**Post date:** [December 19, 2024, 9:59am UTC](https://forum.mautic.org/t/messenger-consume-redis-concurrency-for-better-queued-emails-performance-error-could-not-acknowledge-redis-message/30855/9 "2024-12-19T09:59:08Z")

</div>

You need to apply a patch to Mautic file - see [Messenger:consume redis concurrency for better queued emails performance. Error: "Could not acknowledge redis message" - #4 by jet1](https://forum.mautic.org/t/messenger-consume-redis-concurrency-for-better-queued-emails-performance-error-could-not-acknowledge-redis-message/30855/4)