Cron & Co: What is the perfect scheduling logic for the future?

[ekke]

My idea is:
We should bring the whole “cron jobs” thing to the next level.

Limitations so far:

• all jobs of a type run with the same frequency OR you have complex “-i” jobs which are hard to maintain
• This means that some run less frequent than desired, and some way more frequent than makes sense. No flexibility, and lots of wasted CPU cycles.
• Cron syntax is a barrier and/or cause of mistakes for many
• No transparency in the UI as to what runs when, with what frequency
• Proper error detection and alerting is tricky

Thoughts:

• Let’s have only one cron job (running once per minute), and take the rest to a Mautic-internal scheduler
• In that scheduler, allow adding console commands with frequency & hours
• Allow custom frequencies per item (like “run campaign Update for ID 16 only once per day, at 23:00 UTC”), ideally directly in the UI of that item.
• Detect & report errors (e.g. helpful log file entry)
• Make sure that an error will not block other jobs.
• Allow dependencies between jobs - e.g.
  “run the campaign Trigger for each ID directly after the campaign Update for that ID has finished”
  or
  “after segment Update for ID 7 has finished, run campaign Update for campaigns ID 4”
  or
  “after segment Update for ID 7 has finished, run campaign Update for all campaigns using that segment”
  etc…
• Define error behavior for dependent jobs
• Introduce “Run manually” button

Obviously, this needs some pondering, so please chime in

I think these groups of people would benefit from this idea:

• Admins

Why I think they would benefit from this idea:

• overcome issues listed above

Any code or resources to support this idea:

• possible starting point? typo3/typo3/sysext/scheduler at main · TYPO3/typo3 · GitHub with TYPO3 Scheduler — Scheduler main documentation

Are you willing to work on this idea?:

• sure

What skills and resources do you need to explore this further?

• requirements
• dev
• ui

[mzagmajster]

Hi @ekke ,

in my humble opinion: I understand your concerns in regards to cron jobs. The thing is that what is currently achieved with the help of cron jobs can and should be implemented using mechanisms such as queues (celery and similar libraries). In the past, there was/were talk(s) that discussed that idea.

And I think that would have much greater impact on stuff like emails, campaigns & segments then creating an internal scheduler. If we do use proper queues I am not sure how much cron jobs do we need after that or how complex they need to be.

Even though I am more of a CLI guy myself, I understand your view in regards to UI and support it. But we can also create a UI that would enable non-techy person to setup periodic updates of mautic elements when we have proper queues.

Summary

1. UI Where you can configure details for update of segments, emails, campaigns: YES
2. Internal scheduling system inside Mautic based on current update system (cron jobs): NO
3. Proper queuing mechanism (like Celery or something else): YES

If we are gonna spend time on improving the updating mechanism, I think introducing queues is step in the right direction.

That being said, I really appreciate the suggestion and the effort you put into writing the idea down.

[mikew]

Hey @ekke - love the idea.

We have felt the pains of this extensively with huge instances of Mautic both on a segment level and on a campaign level.

@joeyk has written a very cool segment script that runs segment updates according to a specific variable put into the name of the segment, stuff like [1] - runs every minute, [h] - runs once an hour, [d] runs once a day. This has been a major improvement to system performance and has allowed for heavy segments to be run only at specific hours when resources are more freed up.

We have written out a campaign cron scheduler (not released yet) which is in it’s infant phase and currently only deals with running specific actions inside a campaign that are defined by specific date and time. It does not take care of relative actions/triggers. Basically it looks through campaigns for scheduled tasks and writes out a cronjob using the -i flag for specific campaigns.

It is really interesting to understand maybe how Acquia is taking care of this on a large scale.

[joeyk]

I think this is a great idea.
The most problems of Mautic users are coming from cronjobs in my opinion. In order to be able to set the one minute cronjob requires the complete understanding of commands, syntax, Mautic’s inner working and server resources.

You can’t imagine how many times happens, that someone’s system is crashing because they tried to speed up the cronjobs without knowing what they are doing.

As Mautic can be used for a bunch of different things, the 15 minute cronjob might be okay for most of the users. However we could give options to those who would like to fine-tune the inner rythm of their Mautic instance.

We (at Friendly) use a bash script, that sequentially runs the most important cron tasks every minute. We also do some logging of problems and performance peaks. The script checks spool folder as well and is able to give greenlight to the broadcast command if a large amount of email needs to be spooled before sending.

In order to do everything fast in Mautic, you need to run the 3 major command efficiently. We have been experimenting a lot with that.

The first tier was the segment filter @mikew mentioned, and it’s available on my website for Supporters. (Why not for everyone? Because not everyone needs it. It’s a super specific script, that 99% of Mautic users won’t need, but cause problems for them if they don’t know what they doing. And it’s still a WIP.)

The second tier is campaign update and trigger. We run the 2 command after each other, first update then trigger. Both can take a long time, and won’t finish within 1 minute. In order to make sure the cronjob - that we execute every minute - doesn’t pile up, we need to shoot instead of one cannon with multiple cannons simlutaniously. The way we solved it is to divide the contacts into equal chunks and execute the command for the separate contact ranges. Like thread one is id=1 - 788, thread 2 is 789 - 3000, etc. The ranges have same amount of existing contacts inside making sure the threads have to deal with the exact number of contacts. A medium server can execute even 10 threads at the same time without problems.

The third tier is email sending. Normally I’m suggesting no to send too fast, but sometimes you need to push the gas pedal to the floor. Multi thread sending is possible with Mautic by using --lock_mode=file_lock switch and a lock name. Again: 99% of the users won’t need it. Maybe 50% think they do, but the risks of landing in spam and getting suspended from their SMTP is a lot higher.

Any of the scripts above can seriously mess up your server performance, domain reputation, bring down your Mautic if you are using the wrong number of threads, you have the wrong number of segments, or wrong speed on email sending.

I agree with Ekke’s idea to make Mautic crons more sophisticated. I think a single cron is very much needed. Acquia doesn’t use cronjobs, they are using some kind of queue system. I know, that other Mautic providers also use crons, and tweak them like us, otherwise it would be impossible to manage the number of contacts they manage. They should all chime in. I can contribute with my findings to show the current bash script, but we also need people who know Symfony, and can code the UI.

[ekke]

I agree with all of the above. And yes, we all use some bash magic to work around the limitations (including @joeyk’s segment control tweak) but that’s exactly the point: Things like that should become easier over time, by internalizing them to Mautic.

If that includes a switch to something like Celery as @mzagmajster suggests - sounds cool, but I am not fully on top of the implications. Anyway, should be part of the discussion for sure!

(And yes, there will still be the need for tuning and expertise e.g. in adjusting email queuing and delivery, or whatever of that will remain in M5+ )

[miraclemaxx]

This is a great idea. For an excellent example of how this has been solved very effectively before for other platforms, look no further than the Drupal Ultimate Cron module.Ultimate Cron | Drupal.org which includes logging, scheduling, manual runs, etc.

[rcheesley]

To some extent, switching to Symfony Mailer / Messenger will bring about some changes to things in relation to messaging queues at least, but I will allow @m.abumusa to chime in on that more specifically.

I agree that it could be much better for the user when it comes to managing cron jobs and it’s the number one thing we see in the forums and search engines that folks are struggling with - Joomla does it with scheduled tasks since last year as well - so we definitely have a lot of places where we can draw inspiration.

Thanks for starting the conversation @ekke!