feat: incremental reindex_studio management command

[DanielVZ96]

Description

Adds an incremental mode to the reindex studio management command, and also other few utilities for managing the index: reset and init.

Supporting information

• Refactor reindex_studio management command to support large instances modular-learning#235

Testing instructions

First open the home page of any library. eg.: http://apps.local.openedx.io:2001/course-authoring/library/lib:test:2test

1. Using tutor, exec into the cms instance: tutor dev exec cms bash
2. Verify that the command requires the experimental flag: ./manage.py cms reindex_studio should do nothing, and only reindex if you pass the --experimental flag
3. Test out the different flags for the command
   • ./manage.py cms reindex_studio --experimental --init: Should do nothing since the index already exist
   • ./manage.py cms reindex_studio --experimental --reset: Should recreate an empty index. This means that searching in studio page should not return any results.
   • Run and interrupt ./manage.py cms reindex_studio --experimental --incremental right after it finishes indexing collections. Run it again and assert it continues from where it was interrupted. Try searching content now.
   • Run ./manage.py cms reindex_studio --experimental and verify the index is recreated by searching blocks after it finishes. And also assert that there are no records of a current incremental update lingering: ./manage.py cms shell -c 'from openedx.core.djangoapps.content.search.models import IncrementalIndexCompleted; print(IncrementalIndexCompleted.objects.all())'

Deadline

Asap

Other information

Private-Ref: https://tasks.opencraft.com/browse/FAL-3902

[openedx-webhooks]

Thanks for the pull request, @DanielVZ96!

What's next?

Please work through the following steps to get your changes ready for engineering review:

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

• If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
  • This process (including the steps you'll need to take) is documented here.
• If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

• Dependencies

  This PR must be merged before / after / at the same time as ...

• Blockers

  This PR is waiting for OEP-1234 to be accepted.

• Timeline information

  This PR must be merged by XX date because ...

• Partner information

  This is for a course on edx.org.

• Supporting documentation
• Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

🔘 Let us know that your PR is ready for review:

Who will review my changes?

This repository is currently maintained by @openedx/wg-maintenance-edx-platform. Tag them in a comment and let them know that your changes are ready for review.

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

• Overview of Review Process for Community Contributions
• Pull Request Status Guide
• Making changes to your pull request

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

• The size and impact of the changes that it introduces
• The need for product review
• Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

[ChrisChV]

@DanielVZ96 could you add a comment here? It is not easy at first glance to realize why this call_count is 7

[DanielVZ96]

I modified the mocking mechanism to make it easier to understand and also added some comments.

[ChrisChV]

@DanielVZ96 @bradenmacdonald @pomegranited I am concerned about the functionality of this reset.

If we need to create and populate a new index on a large instance, we need to run the --reset, which removes the old index, and then I need to run the --incremental. However, the search results will be broken when the new index is populated.

Wouldn't it be better to add an option to --incremental, so that it creates a temporary index and does swap, so as not to break the search results, as the non-incremental form does?

[bradenmacdonald]

@ChrisChV I'm assuming that most times an administrator is running this incremental build, they either have no existing index or their existing index is from an old version and is missing some configuration/columns/etc. (so it actually is broken). In that case, removing the old index and starting an incremental build will result in incomplete search results for a while, but the search will be working without errors. And, the incremental index rebuilds the newest courses first, so the results should fill in relatively quickly.

I think for large instances, (where the reindex can take several days) it's better to have a working search with incomplete results, than to have a totally broken search (that displays errors because the old index does not exist or has the wrong configuration).

For Teak I would like to find a way to simplify this, maybe by only having the incremental option and allowing it to be either using a temporary index or not. But I think we'll need to see how this works first and hear from people testing it out.