[SE-4860] Upgrade Celery to v5.2.1

[gabor-boros]

Description

Upgrade Celery to the latest version and bump its dependencies.

Supporting information

The release of Redis 6 introduced the ability to share this service among multiple accounts, securing access to specific keys using a username/password and ACLs.

Celery uses kombu to provide Redis brokering for the message queues, but kombu was missing a couple of key enhancements to support multi-tenant Redis:

• Add support for setting Redis username celery/kombu#1351
• Add global key prefix for keys set by Redis transporter celery/kombu#1349

As edX have identified that the best way to bring our kombu fixes from the original PR into edx-platform is to remove the constraints keeping celery < 5.0 and upgrade celery.

OpenCraft wants to use multi-tenant redis for their shared hosting service, so we can stop maintaining RabbitMQ.

The celery upgrade also requires a newer version of xblock-poll.

Dependencies

• https://github.com/edx/configuration/pull/6586

Testing instructions

The extended heartbeat check for celery is sufficient to test that these changes don't disrupt the functionality on the sandbox as configured below.

1. Visit https://pr29046.sandbox.opencraft.hosting/heartbeat?extended
2. Ensure that the celery check passes.

Deadline

None

Other information

N/A

Reviewers

Settings

DEMO_CREATE_STAFF_USER: true
EDXAPP_FEATURES:
  ENABLE_COMBINED_LOGIN_REGISTRATION: true

[openedx-webhooks]

Thanks for the pull request, @gabor-boros! I've created OSPR-6146 to keep track of it in JIRA, where we prioritize reviews. Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

• supporting documentation
• Open edX discussion forum threads
• timeline information ("this must be merged by XX date", and why that is)
• partner information ("this is a course on edx.org")
• any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

[natabene]

@gabor-boros Thank you for your contribution. Please let me know once this is ready for our review.

[pomegranited]

👍 The multi-tenant redis sandbox is working really well @gabor-boros ! So glad to see these updates working in the platform.

• I tested this by checking the celery heartbeat task, by running an instructor task grade report, and by ensuring that problems get scored.
• I read through the code
• I checked for accessibility issues N/A
• Includes documentation N/A
• I made sure any change in configuration variables is reflected in the corresponding
  client's configuration-secure repository. N/A

[Agrendalath]

⚠️ We should first merge open-craft/xblock-poll#96.

👍

• I tested this: tested that the celery tasks are working correctly on the sandbox instance
• I read through the code
• I checked for accessibility issues: n/a
• Includes documentation: n/a
• I made sure any change in configuration variables is reflected in the corresponding client's configuration-secure repository: n/a

[gabor-boros]

@Agrendalath I just rebased onto master and pushed. FYI as the JS unit tests were failing -- which is unrelated to this change.

[Agrendalath]

@sarina, I'll merge this in 2 days, if you have no objections. We will need to coordinate merging this and https://github.com/edx/configuration/pull/6586 (by @pomegranited).

[sarina]

Hi @Agrendalath - since I joined tCRIL, I don't really know what's happening with edx.org production :) According to https://openedx.atlassian.net/wiki/spaces/COMM/pages/2800255156/Current+Core+Contributor+Committers+and+Champions you can ping @jmbowman and he can help field queries.

[pomegranited]

Thanks @Agrendalath -- I've merged edx/configuration#6586 so you're clear to merge here when ready.

[Agrendalath]

Ah, thanks, @sarina! I didn't notice the update on that page.

@jmbowman, I can merge this tomorrow if you have no objections.

[iamsobanjaved]

@gabor-boros We were caught with some other priority work, we are planning to deploy this tomorrow and test this on stage, can you make this ready to be merged?

[gabor-boros]

can you make this ready to be merged?

@iamsobanjaved Rebased

[openedx-webhooks]

@gabor-boros 🎉 Your pull request was merged! Please take a moment to answer a two question survey so we can improve your experience in the future.

[iamsobanjaved]

Merged the PR, will test this flow on stage (#29046 (comment))

[edx-pipeline-bot]

EdX Release Notice: This PR has been deployed to the staging environment in preparation for a release to production.

[edx-pipeline-bot]

EdX Release Notice: This PR has been deployed to the production environment.

[robrap]

Thanks for all your work in this @gabor-boros and @iamsobanjaved!

[pomegranited]

So fantastic this got merged, thank you @gabor-boros and @iamsobanjaved !!

[kdmccormick]

Congrats on the Celery upgrade 👍🏻

As an FYI: Celery 5's CLI changes caused some breakage in Tutor Nightly. I made a PR to fix it here: overhangio/tutor#614

[robrap]

Thanks @kdmccormick. I approved it.