Mautic Marketplace

Ideas and Feature Requests
,
1

I have a prove of concept for the Marketplace plugin that would list available plugins, let users install them and update with one button push. However, there are some drawbacks that I need help with.

Details: https://johnlinhart.com/blog/mautic-marketplace-proposal

5 Likes
2

@escopecz
Very interesting POC. I guess ideally you’d want to find a good balance between end user experience and developer experience.

End user experience: install plugins with a click of a button in the UI, and fast (<30 seconds)
Developer experience: stay as close as possible to Composer/Packagist, as it’s battle-tested and many PHP developers are already familiar with it (lower barrier to entry)

What I see in the WordPress world, is that developers are starting to use Composer in their plugins, but in the WordPress plugin directory, the plugins are packaged into a ZIP file (WordPress takes care of this if I’m correct), so users who want to install the plugins don’t have to install all Composer packages separately (the /vendor folder is included already). This way, installation of WordPress plugins is mostly done within 15-30 seconds. Not sure if this approach would work for Mautic though, but might be worth considering! Happy to help working on a more detailed concept + programming work :slight_smile:

3 Likes
3

My main concerns with this are forcing people out of the UI in order to update Mautic/extensions at the command line using Composer.

I think from a UX perspective, if we are informing them of an update via the notifications in the UI, they will expect to perform the update there as well. To then not be able to do so is frustrating enough, but to only be able to update via SSH is quite an ask for the non-technical user (thinking of business owners, marketers etc who may only be familiar with FTP/UI-based tooling at best).

If it’s a problem offering a UI-based update we could maybe have some kind of pre-flight check in the installation process which checks for the minimum requirements to perform a UI-based update, and if they are not met, only offer updates via command line and make that really clear from the start? At least then they know what they are getting into.

If we force composer-based command line updates my concern is that we risk a lot of out of date instances.

Many apps are switching to automated under the hood updates (e.g. WordPress, Drupal, etc) - maybe it’s worth looking at how they manage it if they are composer-based?

1 Like
4

I may not be clear enough in the article. My main goal is to let Mautic users to click one button in the GUI and install a plugin they want. There will be also option to do the same via CLI and I started with that to prove the concept as it’s a lot simpler to implement.

I consider installing Mautic with Composer a lot simpler than via FTP. Compare this:

Composer way:

  1. Execute command composer create-project mautic/core
  2. Open Mautic in the folder and follow the installation wizard.

Warning: This does not work yet. It’s just an example how the installation would look like.

FTP way:

  1. Download a zip package from https://not.important
  2. Unzip the package in some local folder.
  3. Upload all files via FTP to your server
  4. Open Mautic in the folder and follow the installation wizard.

Is there anyone who consider Composer way more complicated?

I know that some shared hostings does not let their customers to execute a command. I would like to know how many Mautic instances are running on such shared hostings. I noticed in the Slack discussions that Mautic power users are discouraging newcomers to use this option as they will hit the wall sooner or later. So I would even suggest to stop optimizing for shared hostings. But that’s another topic.

Automate updates is just question of running another cron job with another command. But Mautic does not have good reputation with updates. I don’t think we are at the right point where users would willingly chose the automatic updates.

Anyway, I think I got stuck with installing Mautic via Composer anyway. I’ll write an article about it if anyone else would like to pick it up. But I think I found a way around it and I’ll focus on plugins now.

1 Like
5

I like this idea and great progress with last commits.

Execution time

With last updates you speed up execution time for non-cached composer for 50s. Congrats. My Android take tens of seconds to install app from store, then it’s not bad.

But maybe bundles without composer dependencies, we can install manually without composer

  • find last version on GitHub
  • download, unzip to plugins related directory
  • make any additional modifications If needed

Do you think It should work?
But of course this can make marketplace code more uselessly.
The composer way for all would be more straight.

1 Like
6

I’m in the middle of testing something in that area. It will allow us to divide the work into more steps and we won’t have to modify Mautic that much.

  1. Fetch all plugin versions from Composer API
  2. Decide which version to install. Mostly latest
  3. Download the distribution zip file
  4. Unzip it to the right directory
  5. Then run composer install in that plugin dir. (I plan to modify kernel to load all composer autoload files for plugins automatically if exist)
  6. Clear Mautic’s cache
  7. Run mautic:plugins:install

This should solve the 2 obstacles I had with all Composer approach. But it has other downsides.

  1. We’ll have to make some package version decisions ourselves. Composer would do it for us better.
  2. Installing 2 plugins with same library will bring a Mautic installation down. Composer won’t resolve it for us as there will be composer.json for each plugin. Not one for everything.
7

Nice to hear it. I’m excited
Also we can talk how to allow any bundle add own private plugins to the list.
I am not sure If Mautic/Acquia want provide marketplace in same way like https://addons.prestashop.com/en/ (with some %, support and bussines )
But I would like build extension with premium paid plugins based on this marketplace ecosystem.

8

The marketplace is my free time project. No company is behind it. I want to get the marketplace for open source plugins to Mautic 3. It can be extended for more features later. Packagist allows private repos. That would be one way to distribute paid packages. But we could also build some payment gate for Mautic org so each dev would not have to build their own. Feel free to look into that.

9

I’ve already spent a lot of time on marketplace concept too (free/paid).
I have idea how to integrate http://packagist.com private repositories, also build stripe payment gateway etc etc. I suggest continue discussion in our private chat for more details, then get back to this forum.

1 Like
10

This is great @john

  • Using composer - yes.
  • Disable xdebug and using prestissimo - both yes.
  • Using a cron task - double yes. Especially if we are planning to eventually move to a in-code scheduler with a “every minute” cron task, then the most they’d have to wait is a minute, which is fine. I wouldn’t want to do any kind of update process via web anyway, too much can go wrong.

Suggestions:

  1. You might consider including composer itself as part of the vendor contents, so that you can activate/use it directly without any kind of shell_exec or worries about the wrong version being used. (I haven’t tried this myself, but I worry about leaning on outdated composer versions in the wild).
  2. I also recommend you explore wikimedia/composer-merge-plugin which is what the October CMS uses to handle plugin sub-dependencies. I’m sure there are other, more “propper” ways… but this ensures that if someone just download’s a plugin and plops it in the folder, your composer update/require will see those dependencies and include them. https://github.com/TheDMSGroup/mautic-eb/blob/04b84e5a2c0b49eb22f5b48cfcf19fca735042d1/composer.json an example of how we’re using this with ./mautic/plugins/**/composer.json as the pattern to search for.
  3. I think it’s fair to require plugins to have a packagist entry… If they don’t, and someone just provides a url, your script could add a “repo” entry and attempt to install it with composer that way. Similar end result, still leaning on composer awesomeness.

It’s worth using composer 100% because it is smart enough to roll back when there are failures. You could (if you are paranoid) version the composer.lock contents in a folder by date so that someone could dig themselves out of a failed update without breaking plugins.

1 Like
11

Heath, that is great! Your mautic-eb project is similar to what I failed to prepare. However, that does not solve the issue until all Mautic instances will be installed that way. Until then I’ll stick with installing dependencies for every plugin separately. It wouldn’t be hard to migrate to your way of installing plugins when we decide it is the right way in the future.

Here is how my install command looks like now with timing:

$ bin/console mautic:marketplace:install thedmsgroup/mautic-contact-client-bundle
145 versions of the plugin metadata fetched in 160 ms
2.15.1.0 version is considered to be latest stable
Package distribution downloaded in 1759 ms
Loading composer repositories with package information
Updating dependencies
Package operations: 17 installs, 0 updates, 0 removals
  - Installing mautic/composer-plugin (1.0.1): Downloading (100%)         
  - Installing mustache/mustache (v2.13.0): Downloading (100%)         
  - Installing psr/log (1.1.2): Downloading (100%)         
  - Installing ralouphie/getallheaders (3.0.3): Downloading (100%)         
  - Installing psr/http-message (1.0.1): Downloading (100%)         
  - Installing guzzlehttp/psr7 (1.6.1): Downloading (100%)         
  - Installing guzzlehttp/promises (v1.3.1): Downloading (100%)         
  - Installing guzzlehttp/guzzle (6.5.2): Downloading (100%)         
  - Installing namshi/cuzzle (2.0.3): Downloading (100%)         
  - Installing mtdowling/jmespath.php (2.4.0): Downloading (100%)         
  - Installing aws/aws-sdk-php (3.130.2): Downloading (100%)         
  - Installing league/flysystem (1.0.61): Downloading (100%)         
  - Installing league/flysystem-aws-s3-v3 (1.0.23): Downloading (100%)         
  - Installing dropbox/dropbox-sdk (v1.1.7): Downloading (100%)         
  - Installing league/flysystem-dropbox (1.0.4): Downloading (100%)         
  - Installing phpseclib/phpseclib (2.0.23): Downloading (100%)         
  - Installing league/flysystem-sftp (1.0.22): Downloading (100%)         
<warning>Package league/flysystem-dropbox is abandoned, you should avoid using it. Use spatie/flysystem-dropbox instead.</warning>
Writing lock file
Generating optimized autoload files
Dependencies installed in 33810 ms
Mautic cache cleared in 527 ms
Plugin schema installed in 3513 ms
Total execution time: 39774 ms

Some key points from that command output dump:

  1. Installing a plugin from the zip dist package takes ~2 seconds.
  2. Installing composer dependencies will be different for each plugin but in case of thedmsgroup/mautic-contact-client-bundle which has quite a lot dependencies it takes ~34 seconds. Only ~16 seconds if Composer cache is used. But I can’t find a way how to warm up the cache before installing. --dry-run does not do it. It would help to split this step into 2.
  3. Deleting contents of the cache dir takes ~0.5 seconds.
  4. Installing schema (if any) takes ~3.5 seconds.

In the GUI it can be 4 distinct requests to decrease the risk of the timeout.

This means we can save ~1 minute on resolving Mautic dependencies when installing plugin dependencies into separate vendor dir. On the down side, we’ll lose potential conflict handling this way. I also noticed that Prestissimo doesn’t seem to work. It could be faster when downloaded in parallel.

I’m a bit worried that installing guzzlehttp/guzzle in the plugin dir will conflict with Mautic’s dependencies though. I cannot confirm as I develop it for Mautic 3 and your plugin is failing there.

I would like to ditch the GUI and use commands only. It would make everything simpler. But that is not acceptable for our endusers. Read the comment from Ruth.

1 Like
12

Gotcha… This looks like a big step in the right direction, and I’m sure 90% of plugins would have no external dependencies (or would include a yucky sub-vendor folder), so this (my) plugin is probably a worst case scenario. With that in mind this approach is likely just fine.

You could have the dependency installing portion detect a failure, and continue retrying till it completes without error/timeout. Composer should typically be able to resume downloads where it left off. You can safely use “prefer-dist” and “no-dev” flags I think. And you can set the composer timeout to be the same timeout as web requests (minus ~3 seconds) so that composer is the one doing the timing out, and thereby failing a bit more gracefully (Using “process-timeout” composer var).

2 Likes
13

I circled back to the beginning. I want the solution to work for all plugins and dependencies. So the approach I’m trying now is to let Composer install the plugins but the Marketplace will use separate composer-combined.json. It will be the same as Mautic’s composer.json but it will also contain all the plugins.

1 Like
14

Hi John,

This is a wonderful initiative!
Kudos and power to you…

Please consider one (I think) important point: Non-devs do not have composer installed, which adds a barrier…
I think simple Unix/Linux commands would serve most people well.

wget https://not.important/file.zip
unzip file.zip
rm file.zip

or in one line:
wget https://not.important/file.zip -O file.zip; unzip file.zip; rm file.zip

  • No need for extra packages in your server (which might be a production server for most people, as most people only have one server).
  • No need to learn what Composer is or how it works or the dangers for your production server.
  • It’s a One Liner! It doesn’t get any simpler than this!
15

Hi Yosu,

good points there!

Composer is a PHP package which can be installed by Composer. A bit of inception there. So my plan is to add Composer into Mautic dependencies (the vendor folder). Then we can build GUI and commands around that. Mautic users doesn’t need to know what Composer is.

I say there will be GUI, but it will also check if the time and memory limit is high enough to install the package safely. If not, there will be a warning with 2 options:

  1. Install it with the command and the command will be just copy-paste from the warning message.
  2. Suggestion to increase the timeout or memory limit in php.ini.

Composer can do more than installing a plugin. It can install the right version of the plugin that will fit your PHP and Mautic version. It can automatically update all plugins considering they follow semantic versioning. And notify the user that there is a breaking change in the new version and ask to review the change log first.

3 Likes
16

If there’s a GUI then that’s the best solution indeed!

:love_you_gesture:

17

What I see in the WordPress world, is that developers are starting to use Composer in their plugins

I like that we use an approach that is around and many people are used to it. (low entry barrier)

1 Like
18

One thing that might be a major issue here is the max_execution_time in PHP, which only applies to the UI (not to the command line).

For example, if I set my max_execution_time in WordPress to 5 seconds, and try to install a plugin, it fails. So they also haven’t found a way to work around that issue.

What I’d like to suggest is the following:

  • Build a UI in Mautic to install plugins from the Marketplace, but only allow installation of plugins if users have their max_execution_time set to at least 120 seconds (can be checked with ini_get('max_execution_time'). Would 120 secs be enough @escopecz?
19

Hi folks,

In case you missed the news, this is one of our Strategic Initiatives for 2020-21 and we’re having the kick-off call on Thursday 10th at 1700hrs UK time.

Here’s a Zoom link:Join Zoom Meeting
https://acquia.zoom.us/j/95708798179?pwd=MVdSKzVqS1pnQTcyOHI1NXU4SHQ1Zz09Meeting ID: 957 0879 8179
Passcode: 672967

Please join the Slack channel #i-mautic-marketplace if you’d like to be involved with this initiative and do join us for the kick-off call!