As a developer I want to be able to apply patches and upgrade mautic/core within the existing composer.json

Ideas and Feature Requests
1

Right now it is not possible to run composer update and update mautic.

For example, in Drupal the command is the following:

composer update drupal/core --with-dependencies

Within the current mautic composer.json it is not possible to upgrade mautic. It is, because this is not possible, also not possible to apply patches like this in the composer.json. Not having this makes it very difficult to maintain patches in production. It would also require to disable upgrading within the source code. Our hosting setup doesnā€™t allow to modify the PHP files from within the application other than the media files. We only want to update using GIT and make sure changes are kept in git. This is the defacto regular workflow that is needed to move this to a QA flow so we can have a flow for updates that is compatible with git and multiple environments. Using Pull Request is fun for development but not an option if you want to work with a stable branch from the original source + a patch that you verified.

    "extra": {
            "patches": {
                "mautic/core": {
                    "#8352 Allow custom fields in search for companies": "https://github.com/mautic/mautic/pull/8352.patch"
                }
            }
        }

Therefor I suggest we discuss how we can add mautic/core as a new composer package using the type mautic-core, similar as to how Drupal is structured.

Drupal example project structure:

Important snippet:

 "require": {
        "composer/installers": "^1.0.24",
        "drupal/core": "self.version",
        "drupal/core-project-message": "self.version",
        "drupal/core-vendor-hardening": "self.version",
        "wikimedia/composer-merge-plugin": "^1.4"
    },

Drupal core composer example:

Important snippet:
"type": "drupal-core",

I tried to make this work using the existing sources but because the project is setup as composer type:project it is not possible to move the app folder somewhere else, nor is it possible to move the vendor folder outside of the docroot without making a fork of the project, causing a split from the authoritative source, which I do not want.

Iā€™m sure that people within Acquia can help to add more feedback here, as what Iā€™m suggesting is quite normal in the Drupal world :slight_smile:

4 Likes
2

Hey @Nick_vh! Sounds like a great idea to me, would make it easier for companies that forked Mautic to apply patches as necessary. Would also help them to stay in sync with the community version more easily and vice versa.

I donā€™t know all the specifics about Mauticā€™s update system (yet), but e.g. what the CLI update command (php app/console mautic:update:apply) does also includes things like clearing cache & performing database migrations. So I guess, if we were to introduce an update system through Composer, it would involve having a post-update-cmd command event which would trigger the CLI command above? Would that be a viable option? Maybe @escopecz could elaborate a bit more on that :slight_smile:

1 Like
3

Yep, this is great idea. As I tested that we can already install Mautic with Composer right now. However as @Nick_vh pointed out, thatā€™s not optimal. I played with doing it the proper way a couple months back but weā€™d have to change the folder structure a little. But that change would complicate life to users who installed Mautic to shared hostings. I donā€™t know if anyone does that. It would be good to discus that in the community.

2 Likes
4

If it helps, I will attend some hours in the community sprint. Can we maybe talk about that in virtual at a specific time? Also willing to help and spent some time in getting a hacky version together. I alsready created a couple helper repoā€™s similar to Drupal here: https://github.com/nickveenhof/mautic-project & https://github.com/nickveenhof/mautic-finder & https://github.com/nickveenhof/mautic-core-composer-scaffold but need to make a specific branch of Mautic to get this up and running where the app folder also has a composer.json file similar to Drupalā€™s core folder.

1 Like
5

Thanks @Nick_vh for just sharing your POC during the Mautic Community Sprint! The recording can be found here:

Letā€™s continue the discussion about this feature here in this topic :slight_smile: thanks a lot so far!!

1 Like
6

My comments on the POC so far (based on what @Nick_vh explained):

  • Weā€™d need to add the Mautic version to composer.json (e.g. 2.16.1)
  • Move index.php into a separate folder (e.g. public/index.php) so that index.php and the vendor folder are not in the same directory. Take insipration from Drupal here, where apparently itā€™s possible to choose during installation if you want index.php in the same folder as vendor (which might be needed on shared hosting platforms)
  • The type in composer.json needs to be changed from project to e.g. mautic-core, otherwise Composer wonā€™t work

Letā€™s continue the discussion here :slight_smile: according to Nick, it might be already a quick fix to change the type in composer.json, but not sure if that would have any unwanted side-effects. @escopecz what do you think?

1 Like
7

So, here is an update.

Right now itā€™s possible to install mautic through composer AND patch it AND update it through the following command. It is also possible to patch mautic. See the composer.json from the mautic-project repository.

ā€œcomposer update mautic/core --with-dependenciesā€

How to test it yourselves?

COMPOSER_MEMORY_LIMIT=-1 composer create-project nickveenhof/mautic-project:3.x-dev my-local-mautic-folder --no-interaction

How does it work?

The main magic is in this small PR:

The other magic is in the composer project, and mainly in the composer.json file.

More magic can be found in the accompanying projects:

And

How to proceed? Iā€™m not sure why, but the config_override.php does not want to take effect from the start. When it is defined in local.php it does work but it seems silly that this is needed. I would love some help understanding config_override.php and why I canā€™t set the cache, tmp & other folders more easily. To reproduce this, use the command above, and see that even though the config_override.php is in the correct spot, the cache is being built up from app/ā€¦/var/cache instead of the configured app/ā€¦/ā€¦/var/cache.

1 Like
8

https://www.youtube.com/watch?v=SicommdDH7M also shows this in action.

1 Like
9

Also added the possibility to install 2.x now

COMPOSER_MEMORY_LIMIT=-1 composer create-project nickveenhof/mautic-project:2.x-dev my-local-mautic-folder --no-interaction

Source:

Same single line change is needed in the develop branch to support this flow

1 Like
10

This seems so promising. Has the acquisition effected this? If not, where is this project at this point?

11

Thank you so much @Nick_vh ! Using composer is definitely a must have, hopefully I found this thread.

Just did a pull request because the install was crashing due to an old patch: https://github.com/nickveenhof/mautic-project/pull/3

12

Hi folks,

On Tuesday, 3rd November we discussed the ā€˜composerificationā€™ of Mautic further, but discussions spun out in several threads so Iā€™m going to add them below for reference as they arenā€™t featured in the meeting notes.

Read the discussion in the team meeting notes here.

Other discussions here:

Thread 1
Yosu Cadilla As for the composer options, not something the average Mautic needs or is going to use, ever!
ekke for others would be great, though! :wink:
Yosu Cadilla Yes, I understand the practical benefits of better composer compatibility for both developers and sysadmins, but guess what, we are currently doing pretty well on the developer side, while we are failing terribly on the user side.If it was a simple, non-breaking improvement, by all means!!!
Yosu Cadilla Turns out is is not and would require quite a lot of work.
ekke so ā€œend user features are more importantā€? No doubt about that!
Yosu Cadilla :ok_hand::skin-tone-2:
Nico Grienauer (he/him) Drupal/Analytics/Styleguides for bigger more profession projects it is currently an obstacle to not have composer. but yes. it should be not prio 1.
Nick Veenhof I believe we can do both at the same time :wink:
Yosu Cadilla @Nico Grienauer (he/him) Drupal/Analytics/Styleguides What are the specific use cases where it becomes an obstacle?
Nico Grienauer (he/him) Drupal/Analytics/Styleguides what I have in mind: at our company we have a build process and we do not ā€œclick buttonsā€ on the frontend to make updates of a system. if something has changed in a docker compose etc, then our gitlab runner starts to build a new system and checks everything. after everything is fine, we have a stage to test the system. if the stage is fine, we can merge this into master and this can be deployed then to the live server. Option 1) on the live server only the fragments which have changes will be addded/changed to the live system: done. option 2) a skript will do a git pull on the live system which is changing the missing files: done.the verisoning etc is set in the composer file, so we are in control, when and what will be updated and developers can count on, that they have the same system locally to develop.there will be no database pulled from live etc for a developer setup. if a db is needed, we could pull a dsgvo ready db with anonymoused data.so no editor/marketer should have the ability to click the update button or install any plugins out of the blue.I we donā€™t have control over the system, we cant guarantee stuff and ā€œcantā€ help customer with support.
Nick Veenhof Fully agree with Nico
john The users doesnā€™t have to know they are using Composer but they do need it. If we need a good marketplace and the extendability of Mautic is the main drawback of the whole project as I see it then Composer gives us 90% of features that the Marketplace needs. Iā€™ve already built a basic UI on top of it. The only blocker to finish the marketplace is the directory structure of Mautic.To sum it up, if users want Marketplace and one button extension installations then they want Composer.
Red Composer can help a lot for building and distributing plugins. It would be great to have a example plugin to demonstrate how to use it. I try to build a plugin but it take too much time to set it up. ( Iā€™m new to Mautic plugin drvelopment).

Thread 2
Jordan Ryan (Facet Interactive) (Thread 2) Composerification vs. Marketplace priority
Jordan Ryan (Facet Interactive) Going off of @Yosu Cadillaā€™s comments here. I see the core of this issue as two fold:
Jordan Ryan (Facet Interactive) Marketplace is required in order to centralize the visibility of Mautic plugins, customizations, and functionality.
Jordan Ryan (Facet Interactive) 2. The design and maintenance of Mautic updates predicates the marketplace in that there is no easy way for developers, let alone marketers, to maintain their core updates along with plugin updates in a version controllable way.
Jordan Ryan (Facet Interactive) 2.a. - I donā€™t think that enterprises (the kinds of customers that are essentially bank rolling the development of this community, can scale collaboration in Mautic without better tools / features ā€” and they all want to do this with the security and stability of version controlled systems.
Jordan Ryan (Facet Interactive) Solving the composerification problem of the marketplace is a separate issue than composerification of core, though, and Iā€™m reminded of hacky solutions in Drupal 7 such as https://www.drupal.org/project/composer_manager which handled a separate set of composer.json and vendor files in the installation.
Jordan Ryan (Facet Interactive) Would this not be a potential solution in the short term? It is ideal that the Mautic project is fully handled from core composerificationā€¦ However in the Mautic 3.x short term a Mautic Plugin that enables management of additional plugins as a post-install step could be the path towards marketplace viability.
Yosu Cadilla Marketplace is required in order to centralize the visibility of Mautic plugins, customizations, and functionality.How badly do we need centralized visibility of plugins? There are already several pages, both official and non-official listing all the pluginsā€¦
Yosu Cadilla 2. The design and maintenance of Mautic updates predicates the marketplace in that there is no easy way for developers, let alone marketers, to maintain their core updates along with plugin updates in a version controllable way.Why is that?
Yosu Cadilla 2.a. - I donā€™t think that enterprises (the kinds of customers that are essentially bank rolling the development of this community, can scale collaboration in Mautic without better tools / features ā€” and they all want to do this with the security and stability of version controlled systems.All of them? or a very specific few?
Yosu Cadilla Solving the composerification problem of the marketplace is a separate issue than composerification of core,Indeed (or at least it should).
Yosu Cadilla No one doubts it is a good idea to do it, we just think there are other prioritiesā€¦https://forum.mautic.org/c/ideas/14And not in that list, things like re-visiting all the major integrations with other MKTG platforms, some seem to be broken since the pre-Acquia daysā€¦
Nick Veenhof Actually, if we start by reshuffling directories, and make keep composer optional (as it is today) we can already achieve a whole lot. Iā€™ll see if I can make a PR that showcases this
john @Yosu Cadilla as you represent an average user of Mautic you must admit that current way of installing plugins is not user friendly. And plugins updates are even harder.
Yosu Cadilla Yes, I completely agree @john. However:a) It is daunting if you compare it with a tool like WP for example, but not compared with other tools more geared towards business, that tend to be more ā€œcorkyā€, because there is supposed to be some tech guy managing them.b) We got used to it, no one is complaining about it, or is it and I missed it?c) I believe the point is not whether it would be good to have this, the point is that given the very limited resources we have, we canā€™t do everything we want/need.If having a better way to install plugins means rewriting Mautic, then it is probably not worth the effort at this point in time.In my opinion, our scarce resources would be better used making Mautic more competitive/compelling on the business/marketers/features side of things. (edited)
Yosu Cadilla My concern is that Mautic has long parted from the cutting edge of MA technology and it is now at serious risk of becoming obsolete, if it isnā€™t already. (edited)
Yosu Cadilla The advantage of being a free tool (as in free beer), was plenty enought when the other comparable tools costed thousands of dollars/euros/month
Yosu Cadilla Right now, there are amazing tools out there with all the features Mautic has and then some more, at under $100/month!!! (edited)
Yosu Cadilla At this price point, for any decent business, the difference between $0 and $99 is not worth using an obsolete tool.
Yosu Cadilla That is the danger we are facing!!!
Yosu Cadilla And that is why I believe we need to take ā€œfeaturesā€ as a life and death matter and give it maximum priority over anything and everything else.
Yosu Cadilla Cause 6-9-12 or 18 months from now, it might be too lateā€¦
john Right features. Thatā€™s why I pursue improved extendability. If you notice there are companies building plugins for WP, Joomla, Drupal, and other PHP projects. But Iā€™ve seen very few attempts for Mautic ecosystem. The Marketplace is not only to help Mautic users but also to help plugin developers to get their plugins to the customers. The current process is so scary that no very few users are actually using them. And if very few users use the plugins then there is no reason for plugin developers to build them (features).It is also not about rewrite. Only about moving some folders around. Itā€™s not that much about changing the code.I maintain one plugin and I must say that users have problem with the installation process. I donā€™t understand why you think that writing ā€˜composer install some/pluginā€™ is less convenient than moving folders around with SFTP or via SSH.
Slackbot Hey there, it sounds like you might be asking for support? We ask that you please post requests for support on https://forum.mautic.org - for more information on how the Mautic community uses Slack and our Forums please see this post: Using the Community Forums with Slack - thank you! :mautibot:
Yosu Cadilla It is also not about rewrite. Only about moving some folders around. Itā€™s not that much about changing the code.Then by all means, move those folders. The marketplace is undoubtedly a great improvement!At some point of the discussion, it seemed (at least to me) that some people thought that rewriting Mautic was a necessity in order to allow better Composer compatibility. If it is just a matter of reorganizing folders, I donā€™t see why not.I guess most of the comments I just did here apply to the conversation on the channel and not here on the Composer Threadā€¦
Jordan Ryan (Facet Interactive) At this price point, for any decent business, the difference between $0 and $99 is not worth using an obsolete tool.I agree with this point, the reason why Mautic remains competitive at the enterprise level is because there isnā€™t a low-cost comparison that can be made.Conversely, I would not choose Mautic for a budget client. There is too much testing and maintenance work in between updates to make it cost effective. Invariably there are regressions, so for this price point I would wholly recommend a SaaS solution over Mautic.I would ask if this is really the audience we are trying to serve? Free is not better. Open source, extensible, choose-where-you-host your data are strategic advantages. I agree Features are lagging, but unless the core team can add in more features, then the core team should focus on enabling more developers to contribute custom plugins.
Yosu Cadilla I agree with most of what you just stated. However, if the issue at hand is to make things simpler for developers, I believe those developers are the small company developers and the self hosting entrepreneurs with programming skills, I really doubt corporate Mautic users have any issues with the current level of complexity.AND, If we are aiming Mautic at at larger companies, then we need no marketplace nor a simpler way to install plugins, cause there are engineers managing those matters at those kinds of companies.
Jordan Ryan (Facet Interactive) The question still remains, is it the responsibility of the Core team to add more features, or to enable more features to be added and contributed by the community? I donā€™t think these are opposed but thatā€™s the root of the discussion here.
Jordan Ryan (Facet Interactive) Personally I think the lack of composerification makes Mautic very brittle. mautic-eb does a good job of handling it and we copied this for our approach with mautic-k8s. Itā€™s still brittle between updates and maintaining a bunch of bash scripts to replicate what isā€¦ simply offered out of box in composer is kind of wonky.
Yosu Cadilla Right, I guess if we are willing to get to the root of the problems, we will indeed find some structural decisions might have to be taken in order to prevent those (oddly recurring) issues come back again in the future.
Jordan Ryan (Facet Interactive) I think @Dennis Ameling (he/him)'s points are worth considering in this context, too:https://mautic.slack.com/archives/CQMKV0RU1/p1604569409186200?thread_ts=1604501184.174700&cid=CQMKV0RU1 (edited)
Yosu Cadilla The question still remains, is it the responsibility of the Core team to add more features, or to enable more features to be added and contributed by the community?One thing I always found to be very odd is thet the Dev team is called Product team.
Yosu Cadilla And we donā€™t have a Strategic, multidisciplinary team dealing with Product and Market related issues.

Participants:

ekke, Yosu Cadilla, Nico Grienauer (he/him) Drupal/Analytics/Styleguides, Nick Veenhof, john, Red, Jordan Ryan (Facet Interactive), Slackbot

13

Hi folks,

In case your missed the news, this is one of the Strategic Initiaitives for 2020-21.

Weā€™re having the kick-off call on 17th December at 1400 UK time.

Hereā€™s a Zoom link:https://acquia.zoom.us/j/98670327912?pwd=TnJ3MitkTVQvWTRhZUpkbWtEbm4wQT09

Meeting ID: 986 7032 7912
Passcode: 742589

Please do join us in #i-composer-support on Slack as well!

14

Closing this as we deployed support for Composer with Mautic 4.0! :tada: Thank you to everybody who worked on this!