Background Jobs

[etmartinkazoo]

One of the most important architectural features my company deploys are background jobs; effectively getting things off of the request / response cycle. Has anyone successfully implemented background jobs with their Keystone JS project? If so, if there are code examples, articles, etc... I would be most grateful.

Thank you!

[jshimko]

Definitely recommend checking out BullMQ for that. I've been using BullMQ on almost every project I've built in the last 5+ years and it has been highly reliable. Even for high volume queues (thousands of jobs per hour running 24/7). It's quite simple to set up and the docs have plenty of examples for all of the most common use cases. Also see Bull Board if you want a handy dashboard for monitoring and debugging your queues/jobs.

To be clear, this stuff is all way outside of the scope of Keystone, but the good news is it really doesn't matter what framework you're using with BullMQ as long as it's javascript/typescript. Something you may also want to consider is having your queue workers (the code that actually processes the jobs) run as a separate Node process/service and then just use the queue API for adding jobs to the queue via your Keystone backend. So both services connect to the same Redis instance. Then the Keystone backend adds your jobs to the queue (via Redis) and the workers listen for those new jobs and either process them immediately, at a specified time in the future, or on a cron schedule.

Hope that helps. Good luck!

[jschuur]

Then the Keystone backend adds your jobs to the queue (via Redis) and the workers listen for those new jobs and either process them immediately, at a specified time in the future, or on a cron schedule.

This is eactly what I did with Keystone and BullMQ on a project a while ago. Highly recommended.

[gautamsi]

I have a large project uses 10+ worker process running independently using pg-boss
It is using PostgreSQL for the queue management, just use a different database name.

I use this for persistence feature

[dcousens]

Something that demonstrates a canonical setup for this would be great to add to our examples/

[pahaz]

I'm not sure if this will help or not. In our project, we use Bull, but we are using Keystone 5.