Fix new boefjes issue for scheduler

[jpbruinsslot]

Changes

Wrong cache gets flushed during flushing of another cache

While working on #2786 I encountered an issue with how newly enabled boefjes were being handled by the scheduler. By taking note of the logs it seemed that the scheduler identified all current enabled boefjes to be new and would check if for those boefjes ooi's were available to run on.

This was mainly because of the following line in the code:

nl-kat-coordination/mula/scheduler/connectors/services/katalogus.py

Line 65 in 7e333a3

self.organisations_new_boefjes_cache[org.id] = {}

It would clear the new_boefjes cache every time the plugin_cache was being flushed. These caches are unrelated and flushing one should not clear the other one.

However the plugin cache in main isn't used and it's cache is also not being flushed by either get_plugin_by_id_and_org_id() or get_plugins_by_org_id() methods in the katalogus service. So we don't encounter this issue on main.

Thread locking mechanism for new_boefjes cache added

I noticed the new_boefjes cache did not implement a locking mechanism. Since multiple threads are accessing and writing to the shared cache I've added a thread lock for this cache as well.

Implemented separate locks for individual katalogus caches

Instead of one thread lock for all the katalogus caches, I've implemented individual locks for the caches. This ensures that calls to one cache will not block calls from other threads to the other caches.

Added thread safety tests

To ensure everything would be working accordingly and safeguard any future regression this PR specifically adds thread safety tests.

General refactoring of variable names, and code clean-ups

The long variable cache names were unnecessary and did not help with making the variable names more descriptive. Additionally, this PR also added several closures to some of the methods that access the caches and need to lock the resource, this to reduce some code duplication and enhance readability.

Issue link

Closes #3287

QA notes

This is issue is not seen on main but I can show its workings on the flexible scheduling feature branch. The main focus for QA would be checking if enabling of plugins will continue working correctly. E.g. when enabling a boefje that would start and execute scan because ooi's are the correct scan level.

---

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.

---

Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

[ammar92]

Looks good! I just have a few remarks/ suggestions to improve it even more. The provided PR context is really good and helpful, very much appreciated it. Keep up the good work

[jpbruinsslot]

Looks good! I just have a few remarks/ suggestions to improve it even more. The provided PR context is really good and helpful, very much appreciated it. Keep up the good work

Thanks! Made changes according your suggestions

[stephanie0x00]

Checklist for QA:

• I have checked out this branch, and successfully ran a fresh make reset.
• I confirmed that there are no unintended functional regressions in this branch:
  • I have managed to pass the onboarding flow
  • Objects and Findings are created properly
  • Tasks are created and completed properly
• I confirmed that the PR's advertised feature or hotfix works as intended.
• I checked the logs for errors and/or warnings and made issues where necessary

What works:

Seems to look good. Have enabled multiple boefjes, rescheduled some tasks (boefjes and normalizers). Findings are created and tasks complete.

What doesn't work:

Nothing identified.

Bug or feature?:

n/a