Cron using too many resources.

[bobemoe]

Preconditions (*)

1. Magento CE 2.3.4
2. PHP 7.2

Steps to reproduce (*)

1. Upgrade existing store from 2.2.9 to 2.3.4

Expected result (*)

1. Site and server continue to function as prior to upgrading

Actual result (*)

1. Huge increase in CPU and RAM usage.

You can clearly see the increased load on our system graphs here. It has forced the server to start using swap. Graphs showing 12 hours before and after the M2.2.9 -> 2.3.4 upgrade. The upgrade was done at Tue 4 @ 18:00.

Same graphs but for 3 days before and 4 days after the M2.2.9 -> 2.3.4 upgrade.

I did some benchmarking on the memory and duration of the cron before and after the upgrade:

• 2.2.9 Completed in 2.44 seconds using 330MB RAM
• 2.3.4 Completed in 6.15 seconds using 714MB RAM

That is over double the time and memory used to do the same thing.

Measured with while [ true ] ; do echo $(date +%s.%3N ; free | tail -n2 ) ; sleep .1 ;done | tee cron.csv and plotted in libre office calc.

2.2.9:

2.3.4:

I see similar issues here but the solutions do not make much difference:

#25634
#26507

I did truncate the cron table Wed morning... you can see on the graphs the swap usage is lowered slightly but creeps back up over the next few days, so although that may well also be an issue, I think the increase in mem usage is something different.

[m2-assistant]

Hi @bobemoe. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

@bobemoe do you confirm that you were able to reproduce the issue on vanilla Magento instance following steps to reproduce?

• yes
• no

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[devchris79]

Are you seeing cron jobs stuck in the running status?

[Swahjak]

Just a heads up (not sure if that's the issue, but it might have been overlooked); Since Magento 2.3.3 the cron spawns several consumer processes (about 5). Each of these processes uses resources (~200Mb in RAM, not sure on the cpu but it's more than nothing). In our case it will also occasionally leave queue:consumers:start async.operations.all processes running while starting new ones (resulting in 4-5 duplicate processes).

So all in all it could easily consume an additional ~1GB in memory compared to earlier versions.

[bobemoe]

@devchris79 nothing in running status, quite a few missed though:

MariaDB> select status,count(*) from cron_schedule group by status;
+---------+----------+
| status  | count(*) |
+---------+----------+
| missed  |    30722 |
| pending |      486 |
| success |      633 |
+---------+----------+

I did empty the cron_schedule table soon after upgrading and it only made a slight improvement.

@Swahjak that could well be the issue here and would explain the increase in RAM usage. I think the increase in CPU is partly a knock on effect form the RAM (IO Wait from the swapping)

Is there any way to limit the number of these consumer process? What do they do? We only use a very small number of Magento features, is it possible to disable unneeded cron services?

[devchris79]

We have been seeing stuck processes, but maybe its an effect of running out of ram resources on the VPS. Although I have made two changes that have so far seemed to improve things:

1, Truncated the cron log as it was getting very large
2, We were seeing errors in the system log relating to aqmp not being configured. We don't use RabbitMQ so I just added the code below to env.php:

,
'cron_consumers_runner' => [
        'cron_run' => false,
        'max_messages' => 1000,
        'consumers' => [
            'async.operations.all'
        ]
    ] 

The stuck processes could be an effect of a few things happening at once, so it might require a longer period to see if things really have improved....

[Swahjak]

We've also disabled the cron_consumers_runner/cron_run = false but also added queue/consumers_wait_for_messages = 0 (both in env.php). This makes the processes stop when the're not receiving any messages. We're now running each individual process as a separate cron process. The enables you to control how often the process spawns (Magento will just spawn it on each cron run). For example the product_action_attribute.update is started every minute, but queue:consumers:start exportProcessor only runs once an hour (we don't really use the export functionality, but did't want to completely disable it).

Word of caution: This can delay certain processes. As an example product_action_attribute.update is responsible for processing mass attribute updates. If you only run this process once every hour, every mass update can take up to an hour before it's actually processed.

Reading material: magento/community-features#180 #17951

[Swahjak]

After checking our staging server (we haven't deployed the solution to production yet, we're currently watching a number of performance related issues for 2.3.4 before deploying). It runs on limited resources, but funny enough the drop in RAM expected from the solution above is clearly visible but the CPU clearly increases. This might be related to the fact that the processes require more resources during startup. But I thought it was worth mentioning it.

(The spikes are the result of deployments / container spawning). This is application is not actively used so the cron shouldn't really be doing anything.

[ilnytskyi]

Try to install each group separetely with locks
This way you only run cron groups you want
Processes are not overlaped and stay locked

#25987 (comment)
e.g. add in crontab smth like this

#RUN MAGENTO CRON GROUPS START

* * * * * chronic flock -w 0 /tmp/crongroup_default /usr/bin/php7.1 /var/www/magento/bin/magento cron:run --group=default --bootstrap=standaloneProcessStarted=1 >> /var/www/magento/var/log/magento.cron.log
* * * * * chronic flock -w 0 /tmp/crongroup_index /usr/bin/php7.1 /var/www/magento/bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1 >> /var/www/magento/var/log/magento.cron.log
* * * * * chronic flock -w 0 /tmp/crongroup_import /usr/bin/php7.1 /var/www/magento/bin/magento cron:run --group=import --bootstrap=standaloneProcessStarted=1 >> /var/www/magento/var/log/magento.cron.log

#do not run magento consumers in cron for now
#* * * * * chronic flock -w 0 /tmp/crongroup_consumers /usr/bin/php7.1 /var/www/magento/bin/magento cron:run --group=consumers --bootstrap=standaloneProcessStarted=1 >> /var/www/magento/var/log/magento.cron.log
#do not run ddg automation (Engagement Cloud)
#* * * * * chronic flock -w 0 /tmp/crongroup_ddg_automation /usr/bin/php7.1 /var/www/magento/bin/magento cron:run --group=ddg_automation --bootstrap=standaloneProcessStarted=1 >> /var/www/magento/var/log/magento.cron.log

#RUN MAGENTO CRON GROUPS END

[roni-sooryen]

Seems like this issue is able to fix this problem #26507

[MartinAarts]

Try to install each group separetely with locks
This way you only run cron groups you want
Processes are not overlaped and stay locked

#25987 (comment)
e.g. add in crontab smth like this

#RUN MAGENTO CRON GROUPS START

* * * * * chronic flock -w 0 /tmp/crongroup_default /usr/bin/php7.1 /var/www/magento/bin/magento cron:run --group=default --bootstrap=standaloneProcessStarted=1 >> /var/www/magento/var/log/magento.cron.log
* * * * * chronic flock -w 0 /tmp/crongroup_index /usr/bin/php7.1 /var/www/magento/bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1 >> /var/www/magento/var/log/magento.cron.log
* * * * * chronic flock -w 0 /tmp/crongroup_import /usr/bin/php7.1 /var/www/magento/bin/magento cron:run --group=import --bootstrap=standaloneProcessStarted=1 >> /var/www/magento/var/log/magento.cron.log

#do not run magento consumers in cron for now
#* * * * * chronic flock -w 0 /tmp/crongroup_consumers /usr/bin/php7.1 /var/www/magento/bin/magento cron:run --group=consumers --bootstrap=standaloneProcessStarted=1 >> /var/www/magento/var/log/magento.cron.log
#do not run ddg automation (Engagement Cloud)
#* * * * * chronic flock -w 0 /tmp/crongroup_ddg_automation /usr/bin/php7.1 /var/www/magento/bin/magento cron:run --group=ddg_automation --bootstrap=standaloneProcessStarted=1 >> /var/www/magento/var/log/magento.cron.log

#RUN MAGENTO CRON GROUPS END

This seems to have fixed our issues on Magento 2.3.4. When using this, be sure to check in your magento admin for additional crons that should be run for extensions. We also needed to enable some Amasty extension crons.

[driskell]

#28007 can help with high CPU as it removes some deadlocks that prevent cron schedule table from being cleaned. It can also fix things stuck in running status.

[magento-engcom-team]

Hi @bobemoe. Thank you for your report.
The issue has been fixed in #28007 by @driskell in 2.4-develop branch
Related commit(s):

• aa12503923ac994873724ecaa759e1eb14b318dc
• b0b72b0f8486ca119015e84b7786dfab13d2e475
• 315bd3af8c5ed3dd711abd40db9a163b0c953855
• 96029b55f7cd00b58967637aedac67ddd1767bad
• 7df183b6c6bc3f57f0c3d29d279c3c6d631cf455
• 68d679b7bd7d371c5266d4803f03f204893f7af2
• a90161907aedae371cb307759b685cc254efe2d8
• dda5c72c9303f9476ce7041743c87079cada87f6
• 4d4fa685c0b507b510ad899f5afe94bd6c836115
• e95d2fefefb62150e566f000a4a17f58f6716af5
• 916bc93138f7a81f65c2e8b5069eeb0f35fceb43
• edd77531f8847f7b13dbaca9aefa315ba4aa5e57
• e70027f838f766b816bd1f7063514c1a9caf7a20
• 92dc76438063e68f7523d5ab036a57d7faf7bc03
• d1b22ff825052d836b56f8ce33760d42b2241980
• a76482c4614a7660ab2d324ca4c086f66906c701

The fix will be available with the upcoming 2.4.3 release.

[kilis]

Not fixed. Issue still exits after real world tests with this fix

[kandy]

@kilis Can you describe your concern/steps in a new issue?

[kilis]

@kandy please check @ilnytskyi issue here
Its exact the same problem mentioned by @ilnytskyi

[hostep]

@kilis: before people put more time in this, have you checked/tried this comment?