Upgrade Mautic from v2.16.3 to v5.0.3 via docker

I’d like to share story and my experience with mautic migration to new server with upgrade to latest version.

Story:

We have Mautic v2.16.3 running by our web provider, which is not willing to support instance anymore, so I decided to migrate instance to own server.

Input data

mautic: v2.16.3
mysql: 5.5 (database dump)
dump size: 639M

First try was very naive, but it was try :slight_smile:

Run v5.0.3 directly with database from v2.16.3. Heres are errors:

[notice] Migrating up to Mautic\Migrations\Versionzz20230929183000
[error] Migration Mautic\Migrations\Version020230615115326 failed during Execution. Error: "An exception occurred while executing a query: SQLSTATE[22032]: <<Unknown error>>: 3140 Invalid JSON text: "The document is empty." at position 0 in value for column '#sql-1_7d.headers'."

In ExceptionConverter.php line 117:

  An exception occurred while executing a query: SQLSTATE[22032]: <<Unknown error>>: 3140 Invalid JSON text: "The document is empty." at position 0 in value for column '#sql-1_7d.headers'.


In Exception.php line 28:

  SQLSTATE[22032]: <<Unknown error>>: 3140 Invalid JSON text: "The document is empty." at position 0 in value for column '#sql-1_7d.headers'.


In Connection.php line 71:

  SQLSTATE[22032]: <<Unknown error>>: 3140 Invalid JSON text: "The document is empty." at position 0 in value for column '#sql-1_7d.headers'.

=> mautic/app/migrations/Version020230615115326.php at 1bf73cce081f948c1945e4679a455d6cb508435c · mautic/mautic · GitHub

One of tables probably did not contains JSON valid value (headers column) => SO based on source of migration, I was checked data inside tables and fix was easy, so try again:

UPDATE emails SET headers = "[]";

=> but then error

An exception occurred while executing a query: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'conditions' in 'form_fields'

This says to me that I must migrate over each major version…

Start with upgrade to v3

I migrated in sandobx, I recommend this for everybody, do not play with production, use sandbox!

Here is my docker-compose.yml

name: "v3-mautic"
version: "3"

services:
  database:
    image: percona:5.7
    environment:
      MYSQL_ROOT_PASSWORD: root
      MYSQL_DATABASE: mautic
    ports:
      - "3306"
    volumes:
      - db_data:/var/lib/mysql
      - ./docker-entrypoint-initdb.d:/docker-entrypoint-initdb.d
      - ./percona.cnf:/etc/my.cnf.d/mautic.cnf # <- explained later
    command: --character-set-server=utf8mb4 --collation-server=utf8mb4_czech_ci --sql-mode=""

  mautic:
    image: mautic/mautic:v3-apache
    volumes:
      - mautic_data:/var/www/html
    environment:
      - MAUTIC_DB_HOST=database
      - MAUTIC_DB_USER=root
      - MAUTIC_DB_PASSWORD=root
      - MAUTIC_DB_NAME=mautic
      - PHP_INI_DATE_TIMEZONE=Europe/Prague
      - MAUTIC_TRUSTED_PROXIES=["0.0.0.0/0"]
      - MAUTIC_RUN_CRON_JOBS=false
      - MAUTIC_RUN_MIGRATIONS=false
    depends_on:
      - database
    ports:
      - "8080:80"
volumes:
  db_data:
  mautic_data:

! docker image v3-apache includes version 3.3.4

Probably its mentioned somewhere but I missed it, docker images officialy on dockerhub ARE NOT UPTODATE!

step-by-step:

  1. place dump into docker-entrypoint-initdb.d = in compose file its binded to database container, percona automatically load dump on first start
  2. start database contianer and wait until data are loaded a db is ready = docker compose up -d database --wait
  3. Now needs to start mautic container, update mautic inside and run migrations:
docker compose run --rm mautic bash

run it again, v3 copy mautic data into persistant layer and exit…
then attach container without entrypoint

docker compose run --rm --entrypoint "" mautic bash
php bin/console mautic:update:apply
php bin/console mautic:update:apply --finish

Now I received error

Error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 66002784 bytes)

This is known limit mysql5.7 with default config = database is to heavy, so bypass it by override mysql setting.
In my example I created configuraton called “percona.cnf” and bind it into database volume (its already in example docker-compose.yml above)

[mysqld]
innodb_large_prefix=1
innodb_file_format=Barracuda
innodb_file_per_table=1

After restart databa se container, run again update script of mautic => :tada:

Success! You are now running version 3.3.5 of Mautic.

Continue with upgrade to v4

Thanks to topic: Update Mautic Docker from v3 to v4

Just updated docker-compose.yml, => use image mautic/mautic:v4-apache. Remove mautic volume!

!! docker image v4-apache includes version 4.4.9 !! But for success upgrade must be latest v4

Existing containers and volumes must be destroyed before start

docker compose down -v mautic

Then you can create new container and prepare mautic:

docker compose run --rm mautic bash

then attach container without entrypoint

docker compose run --rm --entrypoint "" mautic bash

and run migration command…

note: mautic:update:apply failed without error details

php bin/console mautic:update:apply
php bin/console mautic:update:apply --finish

These update command will updated mautic codebase to latest 4.4.11 => with this version manual migrations pass

php bin/console doctrine:migration:migrate

At this point, database is v4.4.11 migrated, I updated all emails record for default empty JSON array:

UPDATE emails SET headers = "[]";

Continue with upgrade to v5

This is stright forward.

  1. export database (because migration between percona5.7 and 8) by docker compose exec database mysqldump -u root -p mautic > docker-entrypoint-initdb.d/v4_mautic.sql
  2. use new docker-compose.yml for v5 according docker-mautic/examples/basic at mautic5 · mautic/docker-mautic · GitHub, I use one of example and just add docker-entrypoint-initdb.d into db volumes.
  3. make sure DOCKER_MAUTIC_RUN_MIGRATIONS=true is enabled in .mautic_env
  4. start db container for create database and load dump
  5. then start mautic_web container, in my case everything went fine :slight_smile:

config/local.php

local.php contains aside credentials to db also paths and other settings. I must manually adjusted credentials (easy) and paths to uploads, logs, etc…

For change cache_path, log_path I was change our original path from /data/websites/my.mautic/www/app to path correspond with mautic5 docker image => /var/www/html/var

sed -i 's|/data/websites/my.mautic/www/app|/var/www/html/var|g' local.php

Rest paths to media, uploads was specified reltive to previous, and mautic5 image use docroot folder

sed -i 's|/var/www/html/var/../media|/var/www/html/docroot/media|g' local.php

Do not forget to backup your local.php first !

Conclusion

Throughout the migration process, I encountered numerous errors, particularly with constraints conflict. An example of such errors is when attempting to add a foreign key constraint:

mysql> ALTER TABLE campaign_summary ADD CONSTRAINT FK_6692FA4FF639F774 FOREIGN KEY (campaign_id) REFERENCES campaigns (id);
ERROR 1215 (HY000): Cannot add foreign key constraint

Similarly, another issue arose with:

mysql> ALTER TABLE campaign_summary ADD CONSTRAINT FK_6692FA4F71F7E88B FOREIGN KEY (event_id) REFERENCES campaign_events (id) ON DELETE CASCADE;
ERROR 1215 (HY000): Cannot add foreign key constraint

After spending several hours manually executing migrations and dealing with these errors, I discovered that the Mautic image was not the latest version. Running an update with php bin/console mautic:update:apply resolved all these issues.

Using Docker (Compose) for migration greatly simplified setting up Mautic, eliminating the need to install all dependencies manually and allowing for quick switching between versions.

I hope this topic isn’t duplicity and that it will help someone in their migration journey.

4 Likes

Thank you.
Sharing this in the next newsletter.

Hey luk4s, your tutorial is so helpful, it should be a standard for everybody dockerizing Mautic, congrats!

I’ve looked all over and could not find a good way to migrate from my baremetal Mautic 4.x to Dockerized Mautic 5.x, but with your help i was able to do it.

You should add some words about the other containers, “cron” and “workers”. I have just started them, need to have a look if there is something to setup or check.

For an unknown reason i could not migrate the database using the docker entrypoint as i got an error. But that may was caused because i also move from MariaDB 10.5 to MySQL 8.x with Docker. Therefore i just startup the DB without data and use mysqldump outsite on the bare host to import and that did the trick.

Then i also copied media/images/* and media/files/* from my baremetal install to the corresponding docker volume.

I tried to follow the v4 to v5 steps, but end up with NO data from the previous v4 db in the new v5 mautic setup. ALL tables were renamed to “bak_” and NONE of the data was preserved.

Anyone have thoughts on why this might be?

You mean data is not presented in bak_ tables too?

There is one route via installer that tries to backup tables by changing the prefix. Check the data in bak_ tables if the data is there just change the config options in local.php and everything is back to normal.

However if the data is not presented in bak tables, then you might have accidently deleted the docker volume. In that case I think you will have to pull your mautic database from the backup.

I think I was not very clear. All of the existing tables are prefixed with bak_ and do exist.

I am not clear on what needs to be done in the local.php but I will take a look and see if that clears things up. Those directions do not mention the bak_ tables so I was thinking it would just to a migration of schema and such but leave all of the data.

I will look and report back.

Make sure you v4 mautic DB contains tables without prefix bak_ - so its production mautic DB. Then make sure your docker-compose have properly setup env variables. Start DB container of v4 mautic and load your v4 mautic database into its DB.
After that start mautic container and check its address in browser = it should be first setup page. That’s fine because local.php is missing, so fill first setup form = be aware there is checkbox for “backup existing tables by prefix” => do not check this :wink:

When setup is ready, follow guide above, update mautic inside in container to latest v4 and continue to mautic5 .