Composer Setup for Mautic 5 Now Needs NPM

[davidgorges]

Hey there!

Just a heads up for those who might run into the same issue I did—looks like there’s a little detail that’s not in the documentation yet. If you try to run the composer create-project command, you might bump into an error that goes a bit like this:

npm: not found
Script npm ci --prefer-offline --no-audit handling the npm-ci event returned with error code 127
Script @npm-ci was called via post-update-cmd

Turns out, the server where Mautic is hosted needs NodeJS, which wasn’t the case before. This means if your current hosting doesn’t play nice with NodeJS, you might hit a snag.

I noticed there’s already some chatter about this over at this GitHub pull request, and it seems like it’s open for community feedback.

[fjdh]

I would see it as a big downside to no longer be able to install mautic on shared hosting with management software like cPanel.

[Chris76]

I ran into the same issue.
Really bad idea to require npm, just as stated in the ticket.

What are the workarounds now?
Can I install mautic on my local machine and then simply copy all files via FTP to my VPS / webhosting folder?
I just don’t want to run npm online, if I can avoid it.

[escopecz]

I’d like to point some attention to the great work done in the Docker department lately:

GitHub - mautic/docker-mautic: Docker Image for Mautic

It installs Mautic (with NPM) and then use different image to actually run Mautic (without NPM) if you worry about security or whatever the dislike of NPM comes from.

From the technical point of view, it’s very normal to install PHP dependencies with Composer and Javascript dependencies with NPM. Both are standards for both languages that Mautic is written in.

All the JS files were hard-coded prior to Mautic 6 and because of it they were out of date. NPM comes with simple updates and tools like Dependabot can even find security issues in the JS versions Mautic uses and sends pull requests with updates.

Symfony 7 comes with a new way to install JS dependencies without NPM. It took us 1 year and 8 months to upgrade from Symfony 4 to 5 (Mautic 5) so it may be a few more years to get to Symfony 7 and update all the JS ecosystem in Mautic. Any help in that department is very welcome!

Anyway, I digress. I like that this forum post exists. The product team is watching this to see how many people are having this issue. And we can work around this by for example generating the prod JS assets in a GitHub Action and then if NPM is not present, Mautic can download the assets from GitHub. Is anyone facing this blocker able to build this?

[peter_k]

Thanks for your detailed answer.

Tangential issue: NPM build process is resource hungry

The thing I am wondering:

Is it really necessary that Mautic requires the “end user” to run NPM?

I’m sure that utilizing NPM is a great step forward and makes Mautic more reliable and secure.

However, I believe one could argue that NPM might be burden for end users/installers.

And I don’t think they need to bear it.

My suggestion is simple: Could we/Mautic consider providing pre-generated assets for release versions of Mautic?

That way:

• Assets are built using NPM.
• Pre-built assets are shipped during installation.
• No need to run NPM during installation.
• If you need to update your assets you can install and run the NPM processes on your web server anyways.

Advantages:

• For users who aren’t familiar with NPM or (more importantly) don’t have access tho NPM for some reason, shipping pre-generated assets can make the installation process much smoother.

• By removing the need to run NPM during setup, we lower the barrier to entry. This means more people can use Mautic.

Would this be feasible?

[escopecz]

It is technically possible. Nobody will stand in the way if anyone makes a pull request that will allow this.

In Mautic 5 there are features only available if you install via Composer. Mautic 6 will very likely be installable only via Composer (+docker). Is Composer the same issue as NPM?

[peter_k]

Genuine question: Why would a PR be necessary for this?

Isn’t that “just” someone running the NPM scripts and building the assets right before adding the release to GitHub?

• All PR done for 5.x.y
• runs npm ci --prefer-offline --no-audit
• Merge built assets into 5.x.y
• Release 5.x.y

Mautic 5.x.y gets shipped with pre-built static assets → no end user needs to run NPM.

That is my thinking. Am I mistaken?

I’d compare it with jQuery. It is built using NPM, too. But the end user doesn’t have to built jQuery themselves.

Or maybe Neos CMS is a better comparison. The JS, etc. are built with yarn. But the production package is (installed via composer, but) shipped with pre-built files.

[escopecz]

Please answer also my question I’d like to understand what is fine and what is not as I don’t have this issue personally.

The idea is that we don’t want to keep autogenerated files in git. All those were removed from M5. It’s causing conflicts between pull requests. It’s confusing developers whether they should push those files or not. It’s causing issues for testers as they don’t know whether their version of those files are up to date. Those files should be generated during the build process.

Are we talking about installing Mautic with Composer or from the zip package?

And to answer the question, yes, every change requires a PR. The release process is automated so a new step should be added to the script if we want to add something new.

[peter_k]

Thanks for pointing that out.

I feel like your argument is very development-centric, and my argument is very user-centric. Both are fine. Both are valid.

And I feel both do not contradict each other.

TL;DR:

1. Separate development and production via different repositories.
2. Ship Mautic with pre-built assets, because NPM serves the devs, but not the user.
3. I am not against NPM! I just don’t think it adds that much of a benefit during installation, so that the added complexity is worth it.

---

@escopecz

The idea is that we don’t want to keep autogenerated files in git. All those were removed from M5. It’s causing conflicts between pull requests. It’s confusing developers whether they should push those files or not. It’s causing issues for testers as they don’t know whether their version of those files are up to date. Those files should be generated during the build process.

Ah, I totally understand this. I’d make the same point. I can even envision me being confused by that.

At the same time, I feel like this mixes up two things that should be separated: Development and shipping for production.

Separating dev and production

From my personal and anecdotal experience, it’s a common practice to maintain different repositories for development and production.

In development, it’s absolutely reasonable for contributors to build static files themselves. To prevent the issues you pointed out.

For production releases, however, bundling static files not only makes installation for users more smooth. It also is—from my experience—common practice (see Neos or WordPress).

Examples

AFAIK the way Neos deals with this is having a dedicated repository for development and one for “production”. I.e. a repository the tagged versions get committed to, (along with the built static assets) and where composer pulls from.

A lazy 30 sec search seems to show the same for WordPress—I don’t know if this is actually the case, THB. But fact is, WordPress comes with JS built already.

A way for Mautic?

So how about adding an additional production repository? It basically mirrors the already existing development repository.

Something like this:

This makes development easier for devs as there will be no static file conflicts.

It keeps installing as easy as it was with M4.

Only disadvantage is that someone hast to

1. switch to the mautic-production-distribution
2. (force) checkout the 5.x branch
3. build the assets
4. commits the assets
5. tag the latest commit

I (maybe naively) don’t think that this is something that takes more than 2 minutes. You’d probably automate this very easily.

Role of composer vs. NPM

Mautic is fundamentally a PHP project. This makes Composer a natural fit.

Composer simplifies many things for devs as well as users, like plugin installation and overall management.

Hence, shipping with composer makes totally sense. (And Mautic has to be shipped somehow )

NPM on the other hand does nothing, really (yes, I’m totally exaggerating). What I mean by that:

If 100 people install Mautic, all of them need to build the JS files. This can be avoided if just one runs NPM, builds the JS and Mautic gets shipped with pre-built JS.

So, there is no gain for the user. It’s just one more step, one thing that adds complexity and, most significantly, one thing that introduces a new breaking point.

No user advantage

The only “advantage” I see for people running NPM is to get security patches for the JS files.

BUT:

1. I’m pretty certain most people won’t run npm after an installation, just to get JS updates. They run it along with an Mautic update. So this literally leaves them at the same place as when Mautic comes with pre-built files. (Plus: Someone who really wants JS updates still can run NPM, no matter if Mautic comes with pre-built files or not).
2. Most of Mautics JS (if I am not mistaken) is used in the backend. Exploiting a JS vulnerability requires someone who has access to the dashboard. If this is the case, the problem goes far beyond a JS vulnerability.
3. Even in the case of a severe JS security issue: People still can run NPM.

Plus: NPM will introduce new problems

There are already forum posts about issues with NPM. The more people try to upgrade, the more problems will arise. It is inevitable.

This consumes valuable time that could be spent helping people with more substantial issues (or contributing code).

Shipping pre-built JS will prevent this.

User perspective and Mautic growth

As NPM adds friction to users. This will lead to people giving up on Mautic, turning to Mailchimp or what have you.

We don’t know how big the impact might be. But I’d say every user lost to issues that can (from my POV easily) avoided would be a bummer.

@escopecz

Please answer also my question

Which one did I miss? This?

@escopecz

Is Composer the same issue as NPM?

I hope I answered that. If not please tell me what I missed.

---

I’m positive that if the two domains (dev and prod) are separated there (hopefully) won’t be any confusion for people contributing to the code. Additionally, there won’t be any burden on the Mautic user.

[Chris76]

As for me (former PHP dev, now more Python or general tech consulting) I am happy about composer.
But I like to keep NPM off my live server if possible.

Still, I can see how people with less tech knowledge will just not consider using Mautic anymore, now that NPM and composer are required.

After all: Businesses are looking for value, not for more complex setups and IT constraints.

[escopecz]

I want to know what installation method we are discussing here. There are 4 AFAIK:

1. Composer (recommended)
The users need NPM to install JS dependencies. But they do not have to run any NPM commands. It’s in the post-install scripts. Automatic. Installs Mautic with 1 command. Is this method the one we are discussing here?

2. Docker compose (recommended)
You don’t need to install Composer nor NPM. In fact, you don’t need either MySql nor PHP for this. Just Docker Compose. NPM is present only during the image build, it’s not in the production PHP container. From the simplicity of installation and security, this is the winner.

3. Zip package (Legacy)
The production assets are pre-built in the package. No need for NPM. User must install PHP, MySql, move files via SFTP or SSH. Terrible experience. But users love it because they are used to it.

4. Git (for development only)
Building assets is OK here as we all agree.

Now, I hear that Composer is OK but NPM is not. Mautic isn’t just a PHP app. It has a bunch of JS and CSS libraries. From GitHub and Mautic 5 that moved most of the libraries out:

JS + CSS + Less = 20% of the codebase. I think this deserves its own package manager. But I totally agree that it’s another issue for a Mautic user.

What I suggest is to teach users to go with the Docker Compose setup where they aren’t exposed to all the internals. They have to install just 1 tool, run 1 command and they are all set.

But again. I welcome any contribution that will improve the installation experience for the Composer way of installing Mautic.

[peter_k]

@escopecz

I want to know what installation method we are discussing here

Sorry! I’m talking about the composer installation setup.

@escopecz

Is this method the one we are discussing here?

Yes

@escopecz

Mautic isn’t just a PHP app. It has a bunch of JS and CSS libraries.

I am totally aware. Like I said I exaggerated to make my point more clear.

@escopecz

JS + CSS + Less = 20% of the codebase. I think this deserves its own package manager.

I’m absolutely with you. For developers.

@escopecz

But I totally agree that it’s another issue for a Mautic user.

This is exactly my point. A user installing Mautic via composer doesn’t need to get in touch with NPM.

@escopecz

What I suggest is to teach users to go with the Docker Compose setup where they aren’t exposed to all the internals.

Personally, I have never dealt with docker. It’s just not available on the server I run my web apps on.

I guess, you use Mautic purely along with docker?

It would be interesting to know what the percentage of users using docker vs composer will be (after legacy zip fades out).

@escopecz

But again. I welcome any contribution that will improve the installation experience for the Composer way of installing Mautic.

I shared my 2 cents on that. Unfortunately—as I have 0 experience with build processes / deployment—I can’t contribute more ATM.

But I feel I could make the point I was trying to make regarding the composer installation clear to you

Maybe, hopefully, we find a way to remove the NPM “dependency” for composer users.

[arcmedia]

Also just ran into to the NPM requirement, resulting in a total show-stopper for various installations I maintain.

Generally my hosting provider doesn’t supply npm/node and even if, I’d be running it on a different container, to prevent it messing with the production web environment. So until I find a solution, upgrading is a no-go. If I cannot find one, I’ll be forced to find a different product.

I don’t mind npm being used for development, but there should be a workaround solution for people who cannot use it. Setting requirements this high is definitely going to hurt the distribution and future success of Mautic.

[escopecz]

When you mention containers and that you don’t want NPM/Node in a production environment, take a look at this Dockerfile:

mautic/docker-mautic/blob/6ebb1d6a2751fbb029d1e5ebb6c9ae076f55d47f/fpm/Dockerfile

It uses 2 steps. The first one builds Mautic for production and the other is used for running it. So NPM is never in production. Does it help?

[MikeKMiller]

I have to agree with everyone’s concerns here. I will NEVER allow NPM/NODE or any other compilation on a production server, let alone one with the extreme importance as my lead gen efforts. I also will NEVER use Docker containers for production, self-host home server, maybe, but it has no place in production.

I have so far told 250 clients(and counting) to find other solutions since this became a requirement.

[escopecz]

@MikeKMiller thanks for joining the discussion! Can you please give me some links to read through about why is NPM/Node and Docker in production a problem? Many of the top tech companies are using these technologies in production for years (source). Maybe you know something I’m missing. I’d like to learn.

If you download Mautic from Download - Mautic Community then you have the CSS, JS assets prebuilt and no NPM nor Docker is required. Is that a solution?

If you insist on installing Mautic with Composer then NPM as well as Composer is required only when you need to install or upgrade Mautic. You can always uninstall those once you finish and install them for the next upgrade if you just cannot stand having it in production.

Ideally, if you are managing 250 Mautic instances, I’d suggest to build some automation for installing, upgrading so that you build the production package yourself and deploy it to all without NPM. I’d use Docker for that but you can just move files around. Whatever way you prefer.

And as a last idea, this is an open source project. If you don’t like one aspect of it then you can change it for the good of the whole community. Perhaps those 250 clients would appreciate it rather than migrating to something else. But you know that better than I do.

[mzagmajster]

Hi @MikeKMiller ,

I understand the NPM thing is not ideal. I would appreciate if you can elaborate a bit why Docker container is a big no-no in production from your point of view (also are you talking just not running mautic inside docker container or in general). I am curious about the reasoning.

For any answer, thanks in advance.

[bilbobaggage]

So is there any workaround on the following error or there isn’t? I’ve tried installing node v14 to 20 from the official website but still a no go. What really is going on here lol

Script npm ci --prefer-offline --no-audit handling the npm-ci event returned with error code 2
Script @npm-ci was called via post-update-cmd

[mzagmajster]

Try to run npm ci command directly to get the full error, that might give us more hints