Hi Mauticians, Today morning, our Mautic server crashed and I see the following logs in my Apache log right before the crash:
185.220.205.246 - - [07/Nov/2023:10:31:16 +0000] "GET /s/login HTTP/1.1" 200 2815
185.220.205.246 - - [07/Nov/2023:10:31:16 +0000] "GET /text/css HTTP/1.1" 301 370
185.220.205.246 - - [07/Nov/2023:10:31:16 +0000] "GET /media/dashboards/ HTTP/1.1" 301 426
185.220.205.246 - - [07/Nov/2023:10:31:16 +0000] "GET /image/jpeg HTTP/1.1" 301 378
185.220.205.246 - - [07/Nov/2023:10:31:16 +0000] "GET /s/dashboard HTTP/1.1" 302 366
185.220.205.246 - - [07/Nov/2023:10:31:16 +0000] "GET /upgrade/upgrade.php?task=moveBundles&updateState= HTTP/1.1" 200 228
185.220.205.246 - - [07/Nov/2023:10:31:16 +0000] "GET /media/js/upgrade/upgrade.php?task=clearCache&updateState= HTTP/1.1" 500 1124
It looks as if someone tried to login and/or perform some upgrade. All further requests down the line throw a 500, so that’s the last time points when requests succeeded.
In the Mautic logs, I see this:
[2023-11-07 00:35:06] mautic.WARNING: Received SES webhook of type 'Notification' but couldn't understand payload [] []
[2023-11-07 00:44:04] mautic.WARNING: Received SES webhook of type 'Notification' but couldn't understand payload [] []
[2023-11-07 00:44:05] mautic.WARNING: Received SES webhook of type 'Notification' but couldn't understand payload [] []
[2023-11-07 00:44:05] mautic.WARNING: Received SES webhook of type 'Notification' but couldn't understand payload [] []
[2023-11-07 01:02:04] mautic.WARNING: Received SES webhook of type 'Notification' but couldn't understand payload [] []
[2023-11-07 02:53:04] mautic.WARNING: Received SES webhook of type 'Notification' but couldn't understand payload [] []
[2023-11-07 06:44:06] mautic.WARNING: Received SES webhook of type 'Notification' but couldn't understand payload [] []
[2023-11-07 08:14:39] mautic.WARNING: Received SES webhook of type 'Notification' but couldn't understand payload [] []
[2023-11-07 08:49:01] mautic.WARNING: Received SES webhook of type 'Notification' but couldn't understand payload [] []
[2023-11-07 09:36:00] mautic.WARNING: Received SES webhook of type 'Notification' but couldn't understand payload [] []
[2023-11-07 10:01:25] mautic.WARNING: Received SES webhook of type 'Notification' but couldn't understand payload [] []
[2023-11-07 10:31:16] mautic.NOTICE: PHP Notice - Undefined index: path - in file /opt/bitnami/apps/mautic/htdocs/app/bundles/CoreBundle/Controller/ExceptionController.php - at line 120 {"request":"[object] (Symfony\\Component\\HttpFoundation\\Request: GET //addons HTTP/1.1\r\nAccept: */*\r\nAccept-Encoding: gzip\r\nHost: mautic.marblear.com\r\nReferer: https://52.207.144.195/addons/\r\nUser-Agent: Opera/9.80 (Windows NT 6.1; WOW64; U; en) Presto/2.2.15 Version/10.00\r\nX-Php-Ob-Level: 1\r\n\r\n)","exception":"[object] (Symfony\\Component\\Debug\\Exception\\FlattenException: {})","logger":null,"class":"Symfony\\Component\\HttpKernel\\Exception\\NotFoundHttpException","currentContent":"","layout":"Error","code":404,"anonymous":true,"baseTemplate":"MauticCoreBundle:Default:slim.html.php","templatePage":false,"template":"MauticCoreBundle:Error:404.html.php","templating":"[object] (Symfony\\Bundle\\FrameworkBundle\\Templating\\DelegatingEngine: {})","statusText":"Not Found","url":"//addons","urlParts":{"host":"addons"}} []
[2023-11-07 10:31:17] mautic.CRITICAL: Uncaught PHP Exception InvalidArgumentException: "The "/opt/bitnami/apps/mautic/htdocs/themes" directory does not exist." at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php line 548 {"exception":"[object] (InvalidArgumentException(code: 0): The \"/opt/bitnami/apps/mautic/htdocs/themes\" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php:548)"} []
[2023-11-07 10:31:17] mautic.CRITICAL: Exception thrown when handling an exception (InvalidArgumentException: The "/opt/bitnami/apps/mautic/htdocs/themes" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php line 548) {"exception":"[object] (InvalidArgumentException(code: 0): The \"/opt/bitnami/apps/mautic/htdocs/themes\" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php:548)"} []
[2023-11-07 10:31:17] mautic.CRITICAL: Uncaught PHP Exception InvalidArgumentException: "The "/opt/bitnami/apps/mautic/htdocs/themes" directory does not exist." at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php line 548 {"exception":"[object] (InvalidArgumentException(code: 0): The \"/opt/bitnami/apps/mautic/htdocs/themes\" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php:548, InvalidArgumentException(code: 0): The \"/opt/bitnami/apps/mautic/htdocs/themes\" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php:548)"} []
[2023-11-07 10:31:17] mautic.CRITICAL: Exception thrown when handling an exception (InvalidArgumentException: The "/opt/bitnami/apps/mautic/htdocs/themes" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php line 548) {"exception":"[object] (InvalidArgumentException(code: 0): The \"/opt/bitnami/apps/mautic/htdocs/themes\" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php:548)"} []
[2023-11-07 10:31:17] mautic.ERROR: InvalidArgumentException: The "/opt/bitnami/apps/mautic/htdocs/themes" directory does not exist. - in file /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php - at line 548 [] []
[2023-11-07 10:31:17] mautic.CRITICAL: Uncaught PHP Exception InvalidArgumentException: "The "/opt/bitnami/apps/mautic/htdocs/themes" directory does not exist." at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php line 548 {"exception":"[object] (InvalidArgumentException(code: 0): The \"/opt/bitnami/apps/mautic/htdocs/themes\" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php:548)"} []
[2023-11-07 10:31:17] mautic.CRITICAL: Exception thrown when handling an exception (InvalidArgumentException: The "/opt/bitnami/apps/mautic/htdocs/themes" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php line 548) {"exception":"[object] (InvalidArgumentException(code: 0): The \"/opt/bitnami/apps/mautic/htdocs/themes\" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php:548)"} []
[2023-11-07 10:31:17] mautic.CRITICAL: Uncaught PHP Exception InvalidArgumentException: "The "/opt/bitnami/apps/mautic/htdocs/themes" directory does not exist." at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php line 548 {"exception":"[object] (InvalidArgumentException(code: 0): The \"/opt/bitnami/apps/mautic/htdocs/themes\" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php:548, InvalidArgumentException(code: 0): The \"/opt/bitnami/apps/mautic/htdocs/themes\" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php:548)"} []
[2023-11-07 10:31:17] mautic.CRITICAL: Exception thrown when handling an exception (InvalidArgumentException: The "/opt/bitnami/apps/mautic/htdocs/themes" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php line 548) {"exception":"[object] (InvalidArgumentException(code: 0): The \"/opt/bitnami/apps/mautic/htdocs/themes\" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php:548)"} []
[2023-11-07 10:31:17] mautic.ERROR: InvalidArgumentException: The "/opt/bitnami/apps/mautic/htdocs/themes" directory does not exist. - in file /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php - at line 548 [] []
[2023-11-07 10:31:17] mautic.CRITICAL: Uncaught PHP Exception InvalidArgumentException: "The "/opt/bitnami/apps/mautic/htdocs/themes" directory does not exist." at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php line 548 {"exception":"[object] (InvalidArgumentException(code: 0): The \"/opt/bitnami/apps/mautic/htdocs/themes\" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php:548)"} []
[2023-11-07 10:31:17] mautic.CRITICAL: Exception thrown when handling an exception (InvalidArgumentException: The "/opt/bitnami/apps/mautic/htdocs/themes" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php line 548) {"exception":"[object] (InvalidArgumentException(code: 0): The \"/opt/bitnami/apps/mautic/htdocs/themes\" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php:548)"} []
[2023-11-07 10:31:17] mautic.CRITICAL: Uncaught PHP Exception InvalidArgumentException: "The "/opt/bitnami/apps/mautic/htdocs/themes" directory does not exist." at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php line 548 {"exception":"[object] (InvalidArgumentException(code: 0): The \"/opt/bitnami/apps/mautic/htdocs/themes\" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php:548, InvalidArgumentException(code: 0): The \"/opt/bitnami/apps/mautic/htdocs/themes\" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php:548)"} []
[2023-11-07 10:31:17] mautic.CRITICAL: Exception thrown when handling an exception (InvalidArgumentException: The "/opt/bitnami/apps/mautic/htdocs/themes" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php line 548) {"exception":"[object] (InvalidArgumentException(code: 0): The \"/opt/bitnami/apps/mautic/htdocs/themes\" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php:548)"} []
[2023-11-07 10:31:17] mautic.ERROR: InvalidArgumentException: The "/opt/bitnami/apps/mautic/htdocs/themes" directory does not exist. - in file /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php - at line 548 [] []
[2023-11-07 10:31:17] mautic.CRITICAL: Uncaught PHP Exception InvalidArgumentException: "The "/opt/bitnami/apps/mautic/htdocs/themes" directory does not exist." at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php line 548 {"exception":"[object] (InvalidArgumentException(code: 0): The \"/opt/bitnami/apps/mautic/htdocs/themes\" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php:548)"} []
[2023-11-07 10:31:17] mautic.CRITICAL: Exception thrown when handling an exception (InvalidArgumentException: The "/opt/bitnami/apps/mautic/htdocs/themes" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php line 548) {"exception":"[object] (InvalidArgumentException(code: 0): The \"/opt/bitnami/apps/mautic/htdocs/themes\" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php:548)"} []
[2023-11-07 10:31:17] mautic.CRITICAL: Uncaught PHP Exception InvalidArgumentException: "The "/opt/bitnami/apps/mautic/htdocs/themes" directory does not exist." at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php line 548 {"exception":"[object] (InvalidArgumentException(code: 0): The \"/opt/bitnami/apps/mautic/htdocs/themes\" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php:548, InvalidArgumentException(code: 0): The \"/opt/bitnami/apps/mautic/htdocs/themes\" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php:548)"} []
[2023-11-07 10:31:17] mautic.CRITICAL: Exception thrown when handling an exception (InvalidArgumentException: The "/opt/bitnami/apps/mautic/htdocs/themes" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php line 548) {"exception":"[object] (InvalidArgumentException(code: 0): The \"/opt/bitnami/apps/mautic/htdocs/themes\" directory does not exist. at /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php:548)"} []
[2023-11-07 10:31:17] mautic.ERROR: InvalidArgumentException: The "/opt/bitnami/apps/mautic/htdocs/themes" directory does not exist. - in file /opt/bitnami/apps/mautic/htdocs/vendor/symfony/finder/Finder.php - at line 548 [] []
I restarted the server (runs on Amazon EC2 using the outdated Bitnami package), but the server is still down. If I try to open Mautic in a web browser, no new log messages are being created in Mautic’s log. It stopped at the 10:31:17 time mark.
I cleared Mautic cache already, as recommended on the troubleshooting page, but still no change. Apache itself is still running, but reports only 500.
Any idea what might be going on here?
Edit: If I call the URL that broke the installation:
/media/js/upgrade/upgrade.php?task=clearCache&updateState=
I get this response:
{
"complete": true,
"error": false,
"updateState": "eyJwbHVnaW5Db21wbGV0ZSI6ZmFsc2UsImJ1bmRsZUNvbXBsZXRlIjpmYWxzZSwiY2FjaGVDb21wbGV0ZSI6ZmFsc2UsImNvcmVDb21wbGV0ZSI6dHJ1ZSwidmVuZG9yQ29tcGxldGUiOmZhbHNlfQ==",
"stepStatus": "In Progress"
}