Messenger:consume redis concurrency for better queued emails performance. Error: "Could not acknowledge redis message"

Product Support
7

Sure, compared to my first post in this thread here is what changed:

  1. I installed symfony mailer doctrine driver
    Messenger: Sync & Queued Message Handling (Symfony 5.x Docs)

  2. I changed Mautic queue mail settings to doctrine

2024-02-21 at 09.56
2024-02-21 at 09.563450×1696 485 KB

  1. I set supervisorctl to run consume command in 2 threads

  2. I changed my send.sh script to check doctrine queue instead of redis queue:

#!/bin/bash

# put this script in mautic project folder root and run to send single email.
# pass email ID via --id=xxx. This is the email ID from [mautic]/s/emails: make sure the email is published and published data is in past!


val1=""

# Parse command line arguments for --id
for arg in "$@"
do
    case $arg in
        --id=*)
        val1="${arg#*=}"
        shift # Remove --id= from processing
        ;;
    esac
done

# Check if val1 (email ID) is not set and exit if required
if [ -z "$val1" ]; then
    echo "id required, pass as --id=333"
    exit 1
fi

echo "running for email id: $val1"

while true; do
    # Run the PHP command directly and stream its output, including both stdout and stderr
    output=$( php -d memory_limit=-1 /var/www/mtc/bin/console mautic:broadcasts:send --batch=30 --limit=2000 --id=$val1 2>&1 | tee /dev/tty)
    php_exit_status=${PIPESTATUS[0]}  # Capture the exit status of the PHP command

    # Check if PHP command execution was successful
    if [ $php_exit_status -ne 0 ]; then
        echo "PHP command exited with error code $php_exit_status"
        exit $php_exit_status
    fi

    # Extract the number of emails sent from the output
    emails_sent=$(echo "$output" | awk -F ':' '/^emails_sent/{gsub(/ /, "", $2); print $2}')

    # Print debug
    echo "[$(date --iso-8601=seconds)] emails_put_to_queue: $emails_sent"

    # Exit with '0' success if emails_sent is =0 or undefined or less than 0
    if [ -z "$emails_sent" ] || [ "$emails_sent" -le 0 ]; then
        echo "Exiting with success because emails_sent is undefined, equal to 0, or less than 0"
        exit 0
    fi

    while true; do
        # Get the length of the messages stream, extracting the integer part
        len=$(./bin/console doctrine:query:sql --no-ansi -- "SELECT count(*) from messenger_messages" | awk '/[0-9]+/{print $1; exit}')

        echo "[$(date --iso-8601=seconds)] emails_in_queue: $len"

        # Break the loop if the stream length is less than 50
        if [ "$len" -lt 50 ]; then
            break
        fi

        # Wait for 30 seconds before checking again
        sleep 30
    done
done

so now I run the script like this:
bash ./send.sh --id=xxxx (from my tmux session or our team Jenkins instance)

Suprisingly, Doctrine queue is working noticeably faster compared to Redis, even in single thread. I think the reason this is happening is that our emails have a lot of links and Mautic uses very weird approach to generated email links; which results that each rendered email put in queue is huge, I mean, HUGE: 1 entry can take up to megabytes, this is a very, very large serialized object if we talk about newsletter for 1M recipients. Redis chokes in write and read. Mysql 8 + Doctrine manages to process this better.

1 Like