Marketing Metrics Daily Rollup - MySQL Procedure - Reporting for Google Data Studio

Next share for the evening

My marketing metrics rollup SQL query we run daily.

This allows us to this tap our data studio into the marketing reports.

Reporting from Mautic has always been my headache, this helps resolve some of that ! :slightly_smiling_face:

This is a database MySQL Procedure or an operation you can run from the likes of Integromat
It does consume a bit of server resources to run, so I wouldn’t recommend running a report from this :), best to data warehouse.

https://github.com/Torn-Marketing/Mautic-SQL-Database-Marketing-Stats-Rollup

Shared with love from the Torn Marketing team
https://tornmarketing.com.au

Some items may not work on your db depending on your version, so comment out any lines that may conflict ie google plus field.

Obviously, change your database table prefixes ‘mauticdb_’ to match your local db

 --	Daily rollup logic for compiling Mautic marketing stats into a central table for data studio

DROP TABLE IF EXISTS `email_stats_daily_rollup`;
CREATE TABLE `email_stats_daily_rollup`  AS 
SELECT
	l.email AS email1,
	e.`name` AS e_name1,
	l.firstname AS firstname1,
	l.lastname AS lastname1,
	DATE( es.date_sent ) AS date_sent1,
	DATE( es.date_read ) AS date_read1,
IF
	( cut.hits IS NULL, 0, 1 ) AS is_hit1,
	IFNULL( cut.hits, 0 ) AS hits1,
	IFNULL( cut.unique_hits, 0 ) AS unique_hits1,
IF
	( dnc.id IS NOT NULL AND dnc.reason = 1, 1, 0 ) AS unsubscribed1,
	c.title AS category_title1,
	e.`subject` AS subject2,
	e.sent_count AS sent_count1,
	es.is_read AS is_read1,
	es.viewed_in_browser AS viewed_in_browser1,
--	vp.id AS id1,
--	vp.`subject` AS subject1,
	DATE( e.variant_start_date ) AS variant_start_date1,
IF
	( dnc.id IS NOT NULL AND dnc.reason = 2, 1, 0 ) AS bounced1,
	cmp.`name` AS name1,
	clel.campaign_id AS campaign_id1,
	c.id AS category_id1,
	comp.companyaddress1 AS companyaddress11,
	comp.companyaddress2 AS companyaddress21,
	comp.companyannual_revenue AS companyannual_revenue1,
	comp.companycity AS companycity1,
	comp.companyemail AS companyemail1,
	comp.companyname AS companyname1,
	comp.companycountry AS companycountry1,
	comp.companydescription AS companydescription1,
	comp.companyfax AS companyfax1,
	comp.id AS comp_id1,
	comp.companyindustry AS companyindustry1,
	comp.companynumber_of_employees AS companynumber_of_employees1,
	comp.companyphone AS companyphone1,
	comp.companystate AS companystate1,
	comp.companywebsite AS companywebsite1,
	comp.companyzipcode AS companyzipcode1,
	l.address1 AS address11,
	l.address2 AS address21,
	l.attribution AS attribution1,
	l.attribution_date AS attribution_date1,
	l.city AS city1,
	l.company AS company1,
	l.country AS country1,
	l.facebook AS facebook1,
	l.fax AS fax1,
	l.foursquare AS foursquare1,
	l.googleplus AS googleplus1,
	l.id AS contactId1,
	l.instagram AS instagram1,
	l.linkedin AS linkedin1,
	l.mobile AS mobile1,
	l.phone AS phone1,
	l.points AS points1,
	l.position AS position1,
	l.preferred_locale AS preferred_locale1,
	l.skype AS skype1,
	l.state AS state1,
	l.title AS title1,
	l.twitter AS twitter1,
	l.website AS website1,
	l.zipcode AS zipcode1,
	e.created_by_user AS e_created_by_user1,
	e.date_added AS e_date_added1,
	e.date_modified AS e_date_modified1,
	e.description AS e_description1,
	es.email_address AS email_address1,
	e.id AS e_id1,
--	i.ip_address AS ip_address1,
	es.is_failed AS is_failed1,
	e.is_published AS e_is_published1,
	companies_lead.is_primary AS is_primary1,
	e.lang AS lang1,
	e.modified_by_user AS e_modified_by_user1,
	es.source AS source1,
	es.source_id AS source_id1,
	e.publish_down AS e_publish_down1,
	e.publish_up AS e_publish_up1,
IF
	( es.date_read IS NOT NULL, TIMEDIFF( es.date_read, es.date_sent ), '-' ) AS read_delay1,
	es.retry_count AS retry_count1,
	e.revision AS revision1 
FROM
	mauticdb_email_stats es
	LEFT JOIN mauticdb_emails e ON e.id = es.email_id
	LEFT JOIN mauticdb_leads l ON l.id = es.lead_id
--	LEFT JOIN mauticdb_ip_addresses i ON i.id = es.ip_id
--	LEFT JOIN mauticdb_emails vp ON vp.id = e.variant_parent_id
	LEFT JOIN mauticdb_categories c ON c.id = e.category_id
	LEFT JOIN (
	SELECT
		COUNT( ph.id ) AS hits,
		COUNT(
		DISTINCT ( ph.redirect_id )) AS unique_hits,
		cut2.channel_id,
		ph.lead_id 
	FROM
		mauticdb_channel_url_trackables cut2
		INNER JOIN mauticdb_page_hits ph ON cut2.redirect_id = ph.redirect_id 
		AND cut2.channel_id = ph.source_id 
	WHERE
		cut2.channel = 'email' 
		AND ph.source = 'email' 
	GROUP BY
		cut2.channel_id,
		ph.lead_id 
	) cut ON e.id = cut.channel_id 
	AND es.lead_id = cut.lead_id
	LEFT JOIN mauticdb_lead_donotcontact dnc ON e.id = dnc.channel_id 
	AND dnc.channel = 'email' 
	AND es.lead_id = dnc.lead_id
	LEFT JOIN mauticdb_campaign_lead_event_log clel ON clel.channel = "email" 
	AND e.id = clel.channel_id 
	AND clel.lead_id = l.id
	LEFT JOIN mauticdb_campaigns cmp ON cmp.id = clel.campaign_id
	LEFT JOIN mauticdb_companies_leads companies_lead ON l.id = companies_lead.lead_id
	LEFT JOIN mauticdb_companies comp ON companies_lead.company_id = comp.id 
WHERE
	es.date_sent IS NOT NULL 
	limit 10
	
	;
	
	
	
	-- Phase 2 of query

TRUNCATE TABLE `email_stats_daily_rollup`;
INSERT IGNORE `email_stats_daily_rollup`


SELECT
	l.email AS email1,
	e.`name` AS e_name1,
	l.firstname AS firstname1,
	l.lastname AS lastname1,
	DATE( es.date_sent ) AS date_sent1,
	DATE( es.date_read ) AS date_read1,
IF
	( cut.hits IS NULL, 0, 1 ) AS is_hit1,
	IFNULL( cut.hits, 0 ) AS hits1,
	IFNULL( cut.unique_hits, 0 ) AS unique_hits1,
IF
	( dnc.id IS NOT NULL AND dnc.reason = 1, 1, 0 ) AS unsubscribed1,
	c.title AS category_title1,
	e.`subject` AS subject2,
	e.sent_count AS sent_count1,
	es.is_read AS is_read1,
	es.viewed_in_browser AS viewed_in_browser1,
--	vp.id AS id1,
--	vp.`subject` AS subject1,
	DATE( e.variant_start_date ) AS variant_start_date1,
IF
	( dnc.id IS NOT NULL AND dnc.reason = 2, 1, 0 ) AS bounced1,
	cmp.`name` AS name1,
	clel.campaign_id AS campaign_id1,
	c.id AS category_id1,
	comp.companyaddress1 AS companyaddress11,
	comp.companyaddress2 AS companyaddress21,
	comp.companyannual_revenue AS companyannual_revenue1,
	comp.companycity AS companycity1,
	comp.companyemail AS companyemail1,
	comp.companyname AS companyname1,
	comp.companycountry AS companycountry1,
	comp.companydescription AS companydescription1,
	comp.companyfax AS companyfax1,
	comp.id AS comp_id1,
	comp.companyindustry AS companyindustry1,
	comp.companynumber_of_employees AS companynumber_of_employees1,
	comp.companyphone AS companyphone1,
	comp.companystate AS companystate1,
	comp.companywebsite AS companywebsite1,
	comp.companyzipcode AS companyzipcode1,
	l.address1 AS address11,
	l.address2 AS address21,
	l.attribution AS attribution1,
	l.attribution_date AS attribution_date1,
	l.city AS city1,
	l.company AS company1,
	l.country AS country1,
	l.facebook AS facebook1,
	l.fax AS fax1,
	l.foursquare AS foursquare1,
	l.googleplus AS googleplus1,
	l.id AS contactId1,
	l.instagram AS instagram1,
	l.linkedin AS linkedin1,
	l.mobile AS mobile1,
	l.phone AS phone1,
	l.points AS points1,
	l.position AS position1,
	l.preferred_locale AS preferred_locale1,
	l.skype AS skype1,
	l.state AS state1,
	l.title AS title1,
	l.twitter AS twitter1,
	l.website AS website1,
	l.zipcode AS zipcode1,
	e.created_by_user AS e_created_by_user1,
	e.date_added AS e_date_added1,
	e.date_modified AS e_date_modified1,
	e.description AS e_description1,
	es.email_address AS email_address1,
	e.id AS e_id1,
--	i.ip_address AS ip_address1,
	es.is_failed AS is_failed1,
	e.is_published AS e_is_published1,
	companies_lead.is_primary AS is_primary1,
	e.lang AS lang1,
	e.modified_by_user AS e_modified_by_user1,
	es.source AS source1,
	es.source_id AS source_id1,
	e.publish_down AS e_publish_down1,
	e.publish_up AS e_publish_up1,
IF
	( es.date_read IS NOT NULL, TIMEDIFF( es.date_read, es.date_sent ), '-' ) AS read_delay1,
	es.retry_count AS retry_count1,
	e.revision AS revision1 
FROM
	mauticdb_email_stats es
	LEFT JOIN mauticdb_emails e ON e.id = es.email_id
	LEFT JOIN mauticdb_leads l ON l.id = es.lead_id
--	LEFT JOIN mauticdb_ip_addresses i ON i.id = es.ip_id
--	LEFT JOIN mauticdb_emails vp ON vp.id = e.variant_parent_id
	LEFT JOIN mauticdb_categories c ON c.id = e.category_id
	LEFT JOIN (
	SELECT
		COUNT( ph.id ) AS hits,
		COUNT(
		DISTINCT ( ph.redirect_id )) AS unique_hits,
		cut2.channel_id,
		ph.lead_id 
	FROM
		mauticdb_channel_url_trackables cut2
		INNER JOIN mauticdb_page_hits ph ON cut2.redirect_id = ph.redirect_id 
		AND cut2.channel_id = ph.source_id 
	WHERE
		cut2.channel = 'email' 
		AND ph.source = 'email' 
	GROUP BY
		cut2.channel_id,
		ph.lead_id 
	) cut ON e.id = cut.channel_id 
	AND es.lead_id = cut.lead_id
	LEFT JOIN mauticdb_lead_donotcontact dnc ON e.id = dnc.channel_id 
	AND dnc.channel = 'email' 
	AND es.lead_id = dnc.lead_id
	LEFT JOIN mauticdb_campaign_lead_event_log clel ON clel.channel = "email" 
	AND e.id = clel.channel_id 
	AND clel.lead_id = l.id
	LEFT JOIN mauticdb_campaigns cmp ON cmp.id = clel.campaign_id
	LEFT JOIN mauticdb_companies_leads companies_lead ON l.id = companies_lead.lead_id
	LEFT JOIN mauticdb_companies comp ON companies_lead.company_id = comp.id 
WHERE
	es.date_sent IS NOT NULL 
--	limit 1000
	
	;
5 Likes

Great stuff @tornmarketing, thank you for sharing!

1 Like

Could someone give a “for dummies” run down of how to use this?

I have added the procedure to the SQL section of PHPMyAdmin but the data returned was not decipherable. What tech stack does one need to be able to produce a human-readable report and how would one implement all the steps required?

Few youtube clips around for MySql procedures, ie this one below

But to be honest I run mine through Integromat now as I have error logging, easy scheduling tools etc and the ability to manage multiple instances quickly from the one interface.

If you do go down the Integromat method, you will need to break down the queries like below

Link → Integromat
Significantly easier than having to deal with MySQL config assuming you have remote db access …
You can use the free tier for 1000 operations per month, so using a similar setup to below that would allow you a refresh of ~6 refreshes a day assuming you werent using the advanced scheduling tools :slight_smile:

Push comes to shove and if you are an enterprise client,
Ping me and I can help sort your out via our agency Torn Marketing

2 Likes

Thanks Torn,

What does the resulting report or interface look like?

It’s a cleaned and structured table as per the primary fields in the SELECT

but ultimately covers emails sent per contact, then back fills against each send event.
Ie primary contact details, email name, email subject, email category

Then sent date, read, read count, last read, clicked, click count, unsubscribe, bounces etc

From this I build out my reports in Google data studio and Business Intelligence tools from which I blend the data where required with other data sets

Hello, thanks for the stored procedure, hope I can make it run.

I am getting this error:
MySQL said: #1064 - You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ‘CREATE TABLE email_stats_daily_rollup AS SELECT l.email AS email1, e.`n’ at line 2

And line 2 is:
CREATE TABLE email_stats_daily_rollup AS

What could be the syntax error? I am not seeing it.

The possible reasons for this error can be
Reserved Words
Missing Data
Mistyped Commands
Obsolete Commands

There are 2 key variations as to why someone maybe getting errors

  1. database prefixes - ie mauticdb_email_stats
    Where my database prefix is “mauticdb” this needs to be customised as per your database, this is a variable depending on how you have installed your mautic instance

  2. missing fields in the database what are in the query
    you will noticed I have commented out missing fields with the – comment out function. You will want to make sure that all the queried fields actually exist otherwise will throw an error saying fields dont exist.

I would add to that - if there are fields you dont need in your report, then you should comment them out to speed things up. ie social accounts like foursquare googleplus instagram linkedin etc etc
googleplus may have actually been removed from new default installs too which maybe a missing field

  1. database version or database user permissions - this is rather variable and complex, this report assumes you have a level of skill when it comes to db management.
    if you dont have access/rights or the skills to manage a database I suggest you upskill on database management before going through this more complex reporting method.

Hi everyone, I came here looking for how to connect mautic to data studio. His reports really need to mature a lot.

Could someone create a step-by-step guide on how to use this feature? From what I understand when executing this script in phpmyadmin it will return another file, is it a CSV?

Is there any way to do this procedure in the terminal?

Hey mate

Been using this for years and its not an officially supported solution - something I have personally crafted for my own needs.

The procedure creates a summary table in the database which I can hookup to my data studio/looker via the mysql connector.

The purpose of this method above was to simplify my reporting so I took the load off my database every time I ran a report and the procedure ran it at 3am every morning when I knew that there was a low chance of emails and resources being pulled on the server.

If you have followed the instructions correctly it should create the table email_stats_daily_rollup.
The sql query provided is relevant to my setup and of course has to be updated depending on your mysql database setup/config, table prefixes and the like.
That part will require some technical mysql knowledge which I am unable to run you through in a step by step as the above assumes a baseline mysql knowledge and experience.

TABLE email_stats_daily_rollup

The other alternative is you can create your own mysql query directly in data studio/looker based on your own campaign structures.

Alternatively again
you have the API docs or you can use data ingests and transformation tools such as singer.io, airbyte or the like

1 Like

Hi Tom, I find it amazing that so few people are talking about your approach to generating good statistics for Mautic. I did some internet research to see if anyone had already found a more user-friendly solution for sending this data to Data Studio, but it looks like you managed to solve something too complicated for most people in a simple way.

I tried to follow in your footsteps, but to no avail. I manage to get to the integration with my mautic and Make (formerly integomat). I manage to pull some contacts but I can’t understand how I could integrate with the MySQL table.

If the data were exported to a CSV from Google Sheets I would know exactly what to do, but this is not the case.

Can you record a video for YouTube presenting your brilliant idea so that those of us who would like to get to where you are can also do this?