Register new objects using custom fields architecture

Ideas and Feature Requests
1

My idea is:

At the moment there are two objects registered with custom fields: Contact, Company. I would like to add the ability to Mautic core so that its possible to register more objects (for example Product) leavering the existing Mautic custom fields architecture (I know custom fields do not leverage the best design, but it is the one we have).

In order to register new custom object and enable adding new custom fields to that object you have to do the following:

  • We have to create new entity that implements CustomFieldEntityInterface, CustomFieldEntityTrait IdentifierFieldEntityInterface and use CustomFieldEntityTrait inside that entity.
  • In this enttiy we also need to define relationships to other tables (how this custom table is connected to leads, companies, etc. tables - define PK & FKs)
  • Next we need to add entry to this form: app/bundles/LeadBundle/Form/Type/FieldType.php to show object in dropdown when we are creating new custom field.
  • Finally we need to modify this file: app\bundles\LeadBundle\Field\DTO\CustomFieldObject.php to include the object name that points to actual custom table in the database.

Everything mentionted above except the last point can one way or another be done via custom plugin.

So for modifications required in this file: app\bundles\LeadBundle\Field\DTO\CustomFieldObject.php I did not find any other way to modify it except changing the core file.

Would it be possible that this $objects variable inside the file above is somehow populated dynamically based on records in database. It can also be that CustomFieldObject.php file is somehow generated based on records in the database?

I think these groups of people would benefit from this idea:

Mautic Administrators.

Why I think they would benefit from this idea:

Well, with this solution we enable adding new objects to Mautic shipping this with Mautic core can be really powerful as you only need developer to define relationships between objects, fields can be added via user admins, whenever they please.

Any code or resources to support this idea:

I created a small demo for myself to check if registering new objects is possible.

Are you willing to work on this idea?:

Yes.

What skills and resources do you need to explore this further?

Q1. If I decide to move forward and create a PR for this, would Mautic core team be open to merging that into the core (this kind of needs to be confirmed before hand, so I do not waste time on something that at the end does not get merged)?

Q2. Is there anything to watch out implementing this feature (any potential problems you see with my approach)?

Q3. Do you have any other suggestions in regards to this feature?

Thank you.

Regards, M.

2 Likes
2

The concern I have here is that the current custom field architecture does not scale. It creates new columns to database tables when a new custom field is created. It works nicely in a development environment but it is hell in production where you have thousands of queries running on that table per minute and the table is huge. So adding the column will lock the table that will add all the queries into a queue and they will start timing out which leads to data loss and bad user experience. In our experience, it will also cause the hit of the connection limit as so much queries are backed up and it can lead to the database server restart during the schema change and the database won’t be recoverable.

It has also other limits like number of indexed columns in a single table.

So I wouldn’t base a new feature on this bad architecture, but if you send a PR, I don’t see why it wouldn’t be merged if it won’t break anything.

2 Likes
3

It might be good to mention that there is also this custom objects plugin that might help someone reading this thread:

4

This is an interesting idea for sure. It’s easy to see how useful this could be, especially in the context of e-commerce and subscription membership platforms.

Also, I see what @escopecz is saying about the db tables. I ran an instance of Mautic once that maxed out the custom fields and it was a small disaster.

I wonder if new custom objects creation could come with a different structure that avoids the time out issue…

EP

5

Thank you both for feedback.

About custom objects… While I really appreciate the plugin, I would advise anyone to really test the plugin before they decide to base a project on it (especially if a project is more complex).

About the max. number of cusom fields, I think this issue largely becomes less relevant with this feature since you can always just create a new object that is just extension of the previous one and connect it using FK.

Regards, M.

6

Hi, i am on a finishing expedition here, I am asking this questions before so I can better estimate if I should invest time implementing this feature. I am trying to be as useful as possible to the community.

  1. Is there any plan to improve Mautic custom fields architecture (to move away from custom fields as we know them and onto new architecture)?

If yes, can you please elaborate on it a little bit:

  1. What is the time frame for this new architecture to be brought into the core (maybe this is on the horizon for Mautic 6)?

  2. If similar initiative already exists, where do you need help with (I can offer development time for it)?

I am also available to discuss this into further detail with the core team to ensure implementation goes in the right direction. In fact I think crucial that before I start anything of such magnitude to sync. with the community - hence why I am writing this post :slight_smile:

Looking forward to your feedback.

Regards, M.

7

There is no such initiative as far as I know. It would be major change to Mautic as everything evolves around contacts and the custom fields. We should gather information about what the optimal architecture (DB schema) would look like. In my head it could be:

  1. Store custom fields in a JSON column like a noSQL doc. MySql 8 allows to query such JSON but we’d have to compare the performance with other options. This would be the simplest to implement.
  2. Create new table for all the value types (int, varchar, date, …) so it would have many columns and most of the values would be blank
  3. Create new table for each value type. That would not waste too much space but then we’ll have many tables to join or union.
  4. Create new table for each object and custom field? That would result in many tables but they shouldn’t be that long as if all objects would share the same tables

Perhaps there are more approaches? We’d have to make sure that all the queries that we do today, mostly segment queries are performant on the new architecture with tens of millions of records. Would you be interested in research and creating several POCs so the community could choose the best one?

8

Hi,

thank you for your feedback, yes I would be interested in research and testing different approaches, I just have to think about how this effects some of the projects that would leverage this new architecture (time-wise).

And most likely, I would need some assistance outlining the process of testing these approaches, so we get the reliable info on which we can choose the optimal solution.

If I estimate that this would take too long, I would have to go with just extending the custom fields architecture we have now.

Thank you for now, I will get back to you for sure in regards to this topic :slight_smile: .