Configuring Memcache or Redis in local.php?

Product Support
1

I have seen a number of posts on the forum around the topic of how to install and configure Redis or Memcache with Mautic. Alas there is not one person that has put the solution. Also the documentation in Mautic docs seems super outdated and not explanatory at all: Mautic Developer Documentation

Some background of what I have done, yet I am unclear if it is working as I think my syntax is incorrect in local.php file (but it is not breaking it) - all done with the help of Google & chatGPT.

I am running on ubuntu 20.04 and did the following:
Installed memcach and php module:
sudo apt-get install memcached
sudo apt-get install php-memcached
restarted php7.4-fpm service
added the following to my local.php

'cache' => array(
    'default' => array(
        'driver' => 'memcached',
        'options' => array(
            'servers' => array(
                array(
                    'host' => '127.0.0.1',
                    'port' => 11211,
                    'weight' => 100
                )
            )
        )
    )

Saved and exited.

However when I telnet into memcache server and check stats items I am not seeing anything being cached there…

telnet 127.0.0.1 11211
stats item

I I am not getting anything in response.

My hunch is my configuration in local.php is incorrect.

So anyone ever done this ?

Sorry have to tag you @rcheesley :slight_smile:

2

Sadly this section has not yet been updated and brought over to the new developer docs so I can’t help here!

If someone does have a working config, we are happy to help with documenting - the repo is here:

Here is the issue for the cache bundle:

Also worth noting this PR in the new end-user docs:

We have the entire configuration section documented now (see here) so that PR will also add the things which you can document in the local config file. We could add the specifics for setting up cache providers - I don’t know if this sits best in the dev docs or the end-user docs? What do you think @mikew?

2 Likes
3

I would think dev docs. I think the new docs looks super slick! Nice job.

Another things to add would be getting Mautic to work with Master/Master DB or Master/Slave on read and write.

My last post Mautic, MySQL, Master & Slave, Read/Write still no replies though :frowning: I am sure I am not the only one looking to do this

1 Like
4

I think that @m.abumusa has been doing some work on the read/write and primary/secondary DBs, I recall seeing some PRs. Maybe he can help with the docs too :slight_smile:

5

Thanks @rcheesley appreciate that. Look forward to a reply from @m.abumusa

I am really hoping that someone (@ekke , @joeyk , @Yosu_Cadilla - sorry all had to tag you) can shed some light on the two main issues I am trying to face at the moment which are:
implementing redis/memcached for Mautic including API
setting up master/master master/slave environment.

6

@mikew I created two PRs for Master/Replica implementation, and I am using both of them in two production environments one that uses AWS RDS and the other one using Google Cloud MySQL.

As you can see from the file changes in PR #12038, I am only targeting a few areas of Mautic like reports, segments creation, and campaigns.

A better implementation may be in the future I can direct all the SELECT (except ones inside a COMMIT) statements to the read replica, all INSERT, UPDATE, DELETE, and COMMIT to the master node.

For the time being you can use these two PRs on both 5.x and 4.4 and it will work you and distribute the load in your database

2 Likes
7

I can start a PR on that regard, I will create an issue on the documentation repo so I do not forget that

1 Like
8

@mikew regarding cache, the configuration variables are in this file

1 Like
9

There was a group of 4-6 relatively young DEVOPS guys a few years ago, pushing this and other high level initiatives, like RabbitMQ integration and other similar stuff.

Since these community people are long gone, as almost everyone else from that time, no one is supporting those integrations anymore and they have degraded over time, meaning they will work out of the box on, let’s say M2.14 and probably on 2.26.5 , but after that, it’s a coin flip.

The initial Master/Slave integration in particular, I believe was written by Heath Dutton and was added to the main stable version of Mautic at the time, hence there is no independent repo, however this is the repo he used for most Mautic related things, so maybe you can find something buried there…

If you also take into account the 2 great engineers working at Mautic first and then at Acquia are also gone, there’s no one to clarify those long forgotten issues anymore.
Glad to see that m.abumusa is picking the baton for M4 and M5, but if you are on M3, you might need to do a bit of scavenging…

Personally, Master-Slave is of no use to me, reads and caches are plenty fast already, what I usually need is faster writes, for that I prefer to use master-master also called multi-master replication, commercial name: MariaDB Galera Cluster.

Additional advantages of this type of setup:

  • Every database in the cluster behaves just as a regular DB and no extra configuration is required for Mautic, the data replication is entirely managed by the DB layer over the local network. However, it is not as simple as master-slave to set it up on the MySQL level, but it’s also not rocket science.
  • If you are using Multi-master, you can switch to one Mautic to another instantly, hence enabling full read-write High Availability to any number of nodes.
  • You can define all nodes individually as read only, write only or write/read, and you can change that while the DB is running, whithout changing anything in Mautic.
1 Like
10

Thank you @m.abumusa.

Do you happen to know if there’s any documentation on how/if Mautic can handle in-memory (instead of in-disk) Symfony cache, which is available since Symfony 3 and as you shared, configurable on the local.php file?

Other related topics I am looking for info about (all related to the piece of code you shared) Is there any information somewhere about how to properly configure memcached or how to check if it is working?

Also for Redis and/or RabbitMQ: Do you know about any info about the configuration of these parameters that are on the local.php file?

It would be great if we could put together what parameters are usable or legacy or just history and do not do anything at all in current Mautic versions.
If they are usable, how to configure them and how to check proper configuration is in place.
And things like queues, can be used on ingress and egress, for example for page visit tracking and email queuing, some info about the supported use cases would be great too, don’t you agree?

11

Oh! Since we are discussing documentation, The API docs were quite good a few years ago, but have had little to no love since. It also never had much useful documentation for non-Symfony programmers.

Currently, besides the very basic information available in the docs, the only place to be able to understand how the Mautic API really works is by looking at the code, which is not just PHP, but also Symfony based, making it anything from hard to impossible for programmers of the other 999 programming languages and frameworks out there to use the API.
Additionally, the comments on the code are not very explicit, so sometimes it is hard, even looking at the code, to understand which parameters are mandatory.

API requirements are not documented and also not enforced in the code and error codes aren’t always returned, so if you want to do anything out of the basic few things, you need to test the limits and requirements “by hand”, not complaining, but also does not help Mautic’s image as a enterprise-ready solution…

Since an API’s reason to exist is to be able to communicate with other systems, regardless of language, stack, etc… it would make a lot of sense to have an API documentation that does a bit more than pointing to some PHP implementation of the API.

12

Hi @m.abumusa - thanks for all the information.

I will start with memcached. Before reaching out I had done a bit of research and found a similar post on Redis: Mautic Cache Redis config - #2 by mzagmajster
which references a fix for the Redis adapter :Add RedisAdapter from Symfony 4 for CacheBundle by kuzmany · Pull Request #10889 · mautic/mautic · GitHub

However after implementing a number of variations of the code placed above in my local.php file, I constantly get a broken Mautic.

I add the following code:

        'parameters' => (
                'cache_adapter'           => 'mautic.cache.adapter.filesystem',
    'cache_prefix'            => '',
    'cache_lifetime'          => 86400,
    'cache_adapter_memcached' => (
        'servers' => ('memcached://localhost'),
        'options' => (
            'compression'          => true,
            'libketama_compatible' => true,
            'serializer'           => 'igbinary',
        ),
    ),

which results in the following error:

2023/03/07 07:12:10 [error] 3856#3856: *8874 FastCGI sent in stderr: "PHP message: PHP Notice - Undefined variable: inline - in file /var/www/mautic/offline.php - at line 77PHP message: PHP Notice:  Undefined variable: inline in /var/www/mautic/offline.php on line 77PHP message: ParseError: syntax error, unexpected '=>' (T_DOUBLE_ARROW) - in file /var/www/ams/app/config/local.php - at line 769" while reading response header from upstream, client: 46.116.227.13, server: ams1.surge-ams.com, request: "GET /s/dashboard HTTP/1.1", upstream: "fastcgi://unix:/run/php/php7.4-fpm.sock:", host: "mtest.surge-ams.com", referrer: "https://mtest.surge-ams.com/s/login"

I have played around a lot with positioning the code and changing parenthesis with the square brackets however am unable to get it to not break Mautic.

I have checked and bot memcached and the php extension are installed and I am also able to telnet into memcached port and run stats items there to see if there is any data - unfortunately not, but the service is running.

Have you managed to get this work

13

Hi @m.abumusa

I followed the first PR that you posted and updated the following files:

  1. app/bundles/CoreBundle/Config/config.php
  2. Created file: app/bundles/CoreBundle/Doctrine/Connection/PrimaryReadReplicaConnectionWrapper.php and crowned it to www-data:www-data
  3. app/config/config.php

I then went and added the new parameter ‘db_host_ro’ => ‘10.10.0.3’, to my local.php file. I have confirmed that my slave is working and accessible

I restarted php7.4-fpm service.
Cleared cache manually rm -rf var/cache/*

When I tried to get to my Mautic instance, I get a broken site and in the error log I am seeing the following error:

2023/03/07 08:27:41 [error] 3856#3856: *9000 FastCGI sent in stderr: "PHP message: PHP Notice - Undefined variable: inline - in file /var/www/mautic/offline.php - at line 77PHP message: PHP Notice:  Undefined variable: inline in /var/www/mauitc/offline.php on line 77PHP message: Symfony\Component\DependencyInjection\Exception\LogicException: Messenger support cannot be enabled as the Messenger component is not installed. Try running "composer require symfony/messenger".

I am not using composer and running Mautic 4.4.4 so I went and installed this manually with this command:

sudo apt-get install php-symfony-messenger

Unfortunately I am still getting the same error message.

If I change back the config.php file back to the original my instance comes up again…

14

Hey @Yosu_Cadilla ,

I have done some research, in order to test if memcached you can telnet 127.0.0.1 11211, once inside the terminal write: stats items and then you are meant to see keys and some other info there, for me I am not seeing anything.

15

Yup, exactly.

16

That is a pity to hear :frowning:

I think these two aspects are rather crucial for getting Mautic enterprise ready.

Some other things that might be of interest that I have done is setup an environment where I have created a nfs server and multiple Mautic workers sitting behind a load balancer for traffic, another instance outside of the load balancer but still pulling from the nfs server to run cron jobs, I was even thinking of splitting this up even further to have one server just taking care of campaign jobs and the other for segment update and sending. I am seeing on big installations even when there is ample resources campaigns triggers are being killed in the middle. I also setup an Redis server to share the php resources as well.

Currently giving better improvement but I believe if I was able to set up master-master or master-slave as well as memcache it could help as well.

Some other performance issues I have seen is API rate limit which I would be interested in hearing some thoughts around. I read an article where certain applications (not mautic), would have a separate server for api requestes on a different domain like api.mautic.com…

Interested in hearing what everyone thinks of the above. Maybe together we can come up with a nice cook book on best practices and howtos for big traffic sites.

1 Like
17

@escopecz I was wondering if you have some insights on this particular thread.

Would very much appreciate any pointers here.

18

I don’t have any experience with the QueueBundle. But there are discussions in Slack that the QueueBundle should be removed and replaced with Messenger: Sync & Queued Message Handling (Symfony 5.4 Docs) which is already part of Mautic 5 dependencies. I haven’t seen any work started on that though.

1 Like
19

Thanks @escopecz for jumping in and giving some more info on this.

20

Hello Guys / @joeyk (you might like this one too)

I would like to share some tricks regarding performance tuning with redis and the symfony cache files.

This are Mautic 4 tricks. Don’t know about Mautic 5. But I think most of them might work with 5 too.

Note: I assume redis is already installed and running with default config.

1) Activate the Redis Cache in local.php

Status: Running in production

Add this to local.php and clear the cache:

    'cache_adapter' => 'mautic.cache.adapter.redis',
	'cache_adapter_redis' => array(
		'dsn' => 'redis://127.0.0.1:6379/0',
		'options' => array(
			'lazy' => '',
			'persistent' => '0',
			'persistent_id' => '',
			'timeout' => '30',
			'read_timeout' => '0',
			'retry_interval' => '0'
		)
	),
    'cache_prefix' => 'pa_',
    'cache_lifetime' => 3600,

Check if it’s working on the shell with command:
redis-cli keys “pa_*”

(Make sure to do some page hits / traffic with mautic page tracking …)

Output should be like this:

Bildschirmfoto 2024-07-21 um 12.57.21
Bildschirmfoto 2024-07-21 um 12.57.211380×730 134 KB

2) Activate the doctrine metadata_cache_driver and query_cache_driver

Status: Development / Running fine on my development server

We use 2 of 3 doctrine caches for metadata and query building. This should reduce the time for doctrine to build queries and classes.

Add a config_override.php to app/config/ with this code:

<?php 


$container->loadFromExtension("doctrine", array(
    "orm" => array(
        "metadata_cache_driver" => array('type' => 'pool', 'pool'=>'doctrine.system_cache_pool'),
        "query_cache_driver"    => array('type' => 'pool', 'pool'=>'doctrine.result_cache_pool'),
    )
));

$container->loadFromExtension("framework", array(
    "cache" => array(
        "pools" => array(
            "app.cache.redis" => array(
                "adapter" => "cache.adapter.redis",
                "public" => false,
                "default_lifetime" => 86400
            ),
            "doctrine.result_cache_pool" => array(
                "adapter" => "app.cache.redis"
            ),
            "doctrine.system_cache_pool" => array(
                "adapter" => "app.cache.redis"
            ),        
        )
    )
));

Check if it’s working on the shell with command:
redis-cli keys “*CLASSMETADATA_”

(Make sure to do some page hits / traffic with mautic page tracking …)

Output should be like this:

Bildschirmfoto 2024-07-21 um 13.02.08
Bildschirmfoto 2024-07-21 um 13.02.081360×636 174 KB

3) Mount the mautic cache dir in a ram disk

Status: Running in production

Harddrive is slower than ram. We place the whole mautic cache dir in a ram disk.
But for mautic you don’t need to change any source code or configuration.

Add a line to /etc/fstab with:

tmpfs /home/m/web/m.dev.testserver.online/public_html/app/cache/prod tmpfs defaults,noatime,nosuid,nodev,noexec,mode=1777,size=60M 0 0

and do
mount -a

Check with shell command:
df -h

Output should be like this:

Bildschirmfoto 2024-07-21 um 13.05.25
Bildschirmfoto 2024-07-21 um 13.05.251522×352 239 KB

4) Place a static copy of the mtc.js file in the mautic document root

Status: Running in production

Mautic mtc.js is processed by php. This is slower than having a static copy in the document root.

You can do via shell from the mautic document root with
wget https://m.dev.testserver.online/mtc.js

5) PHP Sessions in Redis

Status: Running in production

Configure in the php.ini like this:

[Session]
; Handler used to store/retrieve data.
; PHP: Runtime Configuration - Manual
session.save_handler = redis
session.save_path = “tcp://127.0.0.1:6379?database=10”
redis.session.locking_enabled = 1
redis.session.lock_retries = -1
redis.session.lock_wait_time = 10000

Check in Mautic under <your_mautic-domain>/s/sysinfo under PHP info

Should look like this:

Bildschirmfoto 2024-07-21 um 13.50.35
Bildschirmfoto 2024-07-21 um 13.50.352056×196 19.4 KB

6) Master/Slave Database Handling

Status: Development / Needs testing for this version

Mautic supports Master/Slave Configurations. It’s already part of the source code:

See Source Code from Mautic 4:

Bildschirmfoto 2024-07-21 um 14.05.06
Bildschirmfoto 2024-07-21 um 14.05.061920×258 112 KB

In the past (Mautic 2.15) I used this already.

I added a db_host_ro param to local.php to define a read only mysql slave.
I try to adapt the configuration for mautic 4 but could not test it.

Important place this before trick 2) because both are changing the same config.

Anyways here is my code snippet for changing config_override.php:

<?php 

// Read Only db cluster support.
// In future this may be best as a modification of the core config.php.
$dbHostRO = $container->hasParameter('mautic.db_host_ro') ? $container->getParameter('mautic.db_host_ro') : null;
if (!empty($dbHostRO)) {

    // Default from config.php
    $dbalSettings = [
        'driver' => '%mautic.db_driver%',
        'host' => '%mautic.db_host%',
        'port' => '%mautic.db_port%',
        'dbname' => '%mautic.db_name%',
        'user' => '%mautic.db_user%',
        'password' => '%mautic.db_password%',
        'charset'               => 'utf8mb4',
        'default_table_options' => [
            'charset'    => 'utf8mb4',
            'collate'    => 'utf8mb4_unicode_ci',
            'row_format' => 'DYNAMIC',
        ],
        'types' => [
            'array' => \Mautic\CoreBundle\Doctrine\Type\ArrayType::class,
            'datetime' => \Mautic\CoreBundle\Doctrine\Type\UTCDateTimeType::class,
            'generated' => \Mautic\CoreBundle\Doctrine\Type\GeneratedType::class,
        ],
        // Prevent Doctrine from crapping out with "unsupported type" errors due to it examining all tables in the database and not just Mautic's
        'mapping_types' => [
            'enum' => 'string',
            'point' => 'string',
            'bit' => 'string',
        ],
        'server_version' => '%mautic.db_server_version%',
    ];

    // Add a single slave (which is a load balanced Aurora read-only cluster).
    $dbalSettings['keep_replica'] = true;
    $dbalSettings['slaves'] = [
        'slave1' => [
            'host' => $dbHostRO,
            'port' => '%mautic.db_port%',
            'dbname' => '%mautic.db_name%',
            'user' => '%mautic.db_user%',
            'password' => '%mautic.db_password%',
            'charset' => 'UTF8',
        ],
    ];
    
    // Default from config.php
    $container->loadFromExtension('doctrine', [
        'dbal' => $dbalSettings,
        'orm' => [
            'auto_generate_proxy_classes' => '%kernel.debug%',
            'auto_mapping' => true,
            'mappings' => $bundleMetadataBuilder->getOrmConfig(),
            'dql' => [
                'string_functions' => [
                    'match' => \DoctrineExtensions\Query\Mysql\MatchAgainst::class,
                ],
            ],
        ],
    ]);
}

I learned this trick from the great DSMGroup:

(btw: the repository also teaches me a lot of other mautic tricks. It’s gold)

2 Likes