Add interval to Boefje

[madelondohmen]

Changes

Follow up of #3521

This PR adds an interval input field to the Boefje forms. It also adds an "Interval" column to the variants table on the Boefje detail page.

Demo

QA notes

Please add some information for QA on how to test the newly created code.

---

Code Checklist

• All the commits in this PR are properly PGP-signed and verified.
• This PR only contains functionality relevant to the issue.
• I have written unit tests for the changes or fixes I made.
• I have checked the documentation and made changes where necessary.
• I have performed a self-review of my code and refactored it to the best of my abilities.

• Tickets have been created for newly discovered issues.
• For any non-trivial functionality, I have added integration and/or end-to-end tests.
• I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
• I have included comments in the code to elaborate on what is not self-evident from the code itself, including references to issues and discussions online, or implicit behavior of an interface.

---

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.

---

Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

[ammar92]

Looks good, minimal remarks. Can be QA'd

[jpbruinsslot]

Wanted to test out this branch in combination with #3529, both in combination and individual checkouts, I'm getting the following error on rocky when starting openkat:

  File "/app/rocky/manage.py", line 23, in <module>
    main()
  File "/app/rocky/manage.py", line 19, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 107, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 100, in handle
    self.check(databases=[database])
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 486, in check
    all_issues = checks.run_checks(
                 ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/checks/registry.py", line 88, in run_checks
    new_errors = check(app_configs=app_configs, databases=databases)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/checks/urls.py", line 14, in check_url_config
    return check_resolver(resolver)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/checks/urls.py", line 24, in check_resolver
    return check_method()
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/urls/resolvers.py", line 519, in check
    for pattern in self.url_patterns:
                   ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/functional.py", line 47, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/urls/resolvers.py", line 738, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
                       ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/functional.py", line 47, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/urls/resolvers.py", line 731, in urlconf_module
    return import_module(self.urlconf_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/app/rocky/rocky/urls.py", line 205, in <module>
    path("<organization_code>/kat-alogus/", include("katalogus.urls"), name="katalogus"),
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/urls/conf.py", line 39, in include
    urlconf_module = import_module(urlconf_module)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/app/rocky/katalogus/urls.py", line 3, in <module>
    from katalogus.views.boefje_setup import AddBoefjeVariantView, AddBoefjeView, EditBoefjeView
  File "/app/rocky/katalogus/views/boefje_setup.py", line 198
    produces = [] not if form_data["produces"] else form_data["produces"].split(",")
                      ^^
SyntaxError: invalid syntax

[madelondohmen]

Wanted to test out this branch in combination with #3529, both in combination and individual checkouts, I'm getting the following error on rocky when starting openkat:

Should already be fixed with the last commit. Should work now, can you check again?

[jpbruinsslot]

can confirm that the error has been fixed

[jpbruinsslot]

while editting a boefje variant (http://localhost:8000/en/dev/kat-alogus/plugins/boefje/dns-sec/) and setting the interval to 60 I encounter the following error:

HTTPStatusError: Client error '400 Bad Request' for url
'http://katalogus:8000/v1/organisations/dev/boefjes/dns-sec'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400

Internal Server Error: /en/dev/kat-alogus/plugins/boefje/dns-sec/edit/
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 104, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/contrib/auth/mixins.py", line 109, in dispatch
    return super().dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 143, in dispatch
    return handler(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/edit.py", line 151, in post
    return self.form_valid(form)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/rocky/katalogus/views/boefje_setup.py", line 154, in form_valid
    get_katalogus(self.organization.code).edit_plugin(plugin)
  File "/app/rocky/katalogus/client.py", line 230, in edit_plugin
    response.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 763, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '400 Bad Request' for url 'http://katalogus:8000/v1/organisations/dev/boefjes/dns-sec'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400

[ammar92]

while editting a boefje variant (http://localhost:8000/en/dev/kat-alogus/plugins/boefje/dns-sec/) and setting the interval to 60 I encounter the following error:

Do you by any chance also have the logs of the katalogus service?

[jpbruinsslot]

while editting a boefje variant (http://localhost:8000/en/dev/kat-alogus/plugins/boefje/dns-sec/) and setting the interval to 60 I encounter the following error:

Do you by any chance also have the logs of the katalogus service?

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /app/boefjes/boefjes/katalogus/plugins.py:168 in update_boefje                                   │
│                                                                                                  │
│   165 │   storage: PluginStorage = Depends(get_plugin_storage),                                  │
│   166 ):                                                                                         │
│   167 │   with storage as p:                                                                     │
│ ❱ 168 │   │   p.update_boefje(boefje_id, boefje.model_dump(exclude_unset=True))                  │
│   169                                                                                            │
│   170                                                                                            │
│   171 @router.delete("/boefjes/{boefje_id}", status_code=status.HTTP_204_NO_CONTENT)             │
│                                                                                                  │
│ ╭────────────────────────────────────── locals ──────────────────────────────────────╮           │
│ │    boefje = BoefjeIn(                                                              │           │
│ │             │   name='DNSSEC',                                                     │           │
│ │             │   version=None,                                                      │           │
│ │             │   created=None,                                                      │           │
│ │             │   description='Validates DNSSec of a hostname',                      │           │
│ │             │   scan_level=1,                                                      │           │
│ │             │   consumes={'Hostname'},                                             │           │
│ │             │   produces={'boefje/dns-sec'},                                       │           │
│ │             │   boefje_schema=None,                                                │           │
│ │             │   cron=None,                                                         │           │
│ │             │   interval=60,                                                       │           │
│ │             │   oci_image='ghcr.io/minvws/openkat/dns-sec:latest',                 │           │
│ │             │   oci_arguments=[]                                                   │           │
│ │             )                                                                      │           │
│ │ boefje_id = 'dns-sec'                                                              │           │
│ │         p = <boefjes.sql.plugin_storage.SQLPluginStorage object at 0x7facf6ec86d0> │           │
│ │   storage = <boefjes.sql.plugin_storage.SQLPluginStorage object at 0x7facf6ec86d0> │           │
│ ╰────────────────────────────────────────────────────────────────────────────────────╯           │
│                                                                                                  │
│ /app/boefjes/boefjes/sql/plugin_storage.py:50 in update_boefje                                   │
│                                                                                                  │
│    47 │   │   instance = self._db_boefje_instance_by_id(boefje_id)                               │
│    48 │   │                                                                                      │
│    49 │   │   if instance.static:                                                                │
│ ❱  50 │   │   │   raise NotAllowed(f"Plugin with id '{boefje_id}' is static, so updating it is   │
│    51 │   │                                                                                      │
│    52 │   │   boefje = self.to_boefje(instance).copy(update=data)                                │
│    53 │   │   self.session.merge(self.to_boefje_in_db(boefje, instance.id))                      │
│                                                                                                  │
│ ╭────────────────────────────────────── locals ──────────────────────────────────────╮           │
│ │ boefje_id = 'dns-sec'                                                              │           │
│ │      data = {                                                                      │           │
│ │             │   'name': 'DNSSEC',                                                  │           │
│ │             │   'description': 'Validates DNSSec of a hostname',                   │           │
│ │             │   'scan_level': 1,                                                   │           │
│ │             │   'consumes': {'Hostname'},                                          │           │
│ │             │   'produces': {'boefje/dns-sec'},                                    │           │
│ │             │   'interval': 60,                                                    │           │
│ │             │   'oci_image': 'ghcr.io/minvws/openkat/dns-sec:latest',              │           │
│ │             │   'oci_arguments': []                                                │           │
│ │             }                                                                      │           │
│ │  instance = <boefjes.sql.db_models.BoefjeInDB object at 0x7fad0c60b790>            │           │
│ │      self = <boefjes.sql.plugin_storage.SQLPluginStorage object at 0x7facf6ec86d0> │           │
│ ╰────────────────────────────────────────────────────────────────────────────────────╯           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
NotAllowed: Plugin with id 'dns-sec' is static, so updating it is not allowed

2024-09-26T09:19:13.928283 [error] An error occurred: Plugin with id 'dns-sec' is static, so updating it is not allowed. Rolling back session
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /app/boefjes/boefjes/sql/db.py:44 in session_managed_iterator                                    │
│                                                                                                  │
│   41 │   service = service_factory(session)                                                      │
│   42 │                                                                                           │
│   43 │   try:                                                                                    │
│ ❱ 44 │   │   yield service                                                                       │
│   45 │   except Exception as error:                                                              │
│   46 │   │   logger.exception("An error occurred: %s. Rolling back session", error)              │
│   47 │   │   session.rollback()                                                                  │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │           error = NotAllowed("Plugin with id 'dns-sec' is static, so updating it is not      │ │
│ │                   allowed")                                                                  │ │
│ │         service = <boefjes.sql.plugin_storage.SQLPluginStorage object at 0x7facf6ec86d0>     │ │
│ │ service_factory = <function create_plugin_storage at 0x7fad59f0dc60>                         │ │
│ │         session = <sqlalchemy.orm.session.Session object at 0x7fad30604250>                  │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
NotAllowed: Plugin with id 'dns-sec' is static, so updating it is not allowed
Traceback (most recent call last):
  File "/app/boefjes/boefjes/sql/db.py", line 44, in session_managed_iterator
    yield service
boefjes.storage.interfaces.NotAllowed: Plugin with id 'dns-sec' is static, so updating it is not allowed
2024-09-26T09:19:13.932950 [info] Closing session for <class 'boefjes.sql.plugin_storage.SQLPluginStorage'>
2024-09-26T09:19:13.933054 [error] An error occurred: Plugin with id 'dns-sec' is static, so updating it is not allowed. Rolling back session
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /app/boefjes/boefjes/sql/db.py:44 in session_managed_iterator                                    │
│                                                                                                  │
│   41 │   service = service_factory(session)                                                      │
│   42 │                                                                                           │
│   43 │   try:                                                                                    │
│ ❱ 44 │   │   yield service                                                                       │
│   45 │   except Exception as error:                                                              │
│   46 │   │   logger.exception("An error occurred: %s. Rolling back session", error)              │
│   47 │   │   session.rollback()                                                                  │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │           error = NotAllowed("Plugin with id 'dns-sec' is static, so updating it is not      │ │
│ │                   allowed")                                                                  │ │
│ │         service = <boefjes.sql.organisation_storage.SQLOrganisationStorage object at         │ │
│ │                   0x7fad0c4c80d0>                                                            │ │
│ │ service_factory = <function create_organisation_storage at 0x7fad5a062a20>                   │ │
│ │         session = <sqlalchemy.orm.session.Session object at 0x7fad59f7eb10>                  │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
NotAllowed: Plugin with id 'dns-sec' is static, so updating it is not allowed
Traceback (most recent call last):
  File "/app/boefjes/boefjes/sql/db.py", line 44, in session_managed_iterator
    yield service
boefjes.storage.interfaces.NotAllowed: Plugin with id 'dns-sec' is static, so updating it is not allowed
2024-09-26T09:19:13.937496 [info] Closing session for <class 'boefjes.sql.organisation_storage.SQLOrganisationStorage'>
172.30.0.3:47962 - "PATCH /v1/organisations/dev/boefjes/dns-sec HTTP/1.1" 400
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to jaeger:4317, retrying in 16s.

[jpbruinsslot]

I guess this is related: #3573

[TwistMeister]

Checklist for QA:

• I have checked out this branch, and successfully ran a fresh make reset.
• I confirmed that there are no unintended functional regressions in this branch:
  • I have managed to pass the onboarding flow
  • Objects and Findings are created properly
  • Tasks are created and completed properly
• I confirmed that the PR's advertised feature or hotfix works as intended.
• I checked the logs for errors and/or warnings and made issues where necessary

What works:

• Adding a variant with an interval
• Editing a newly added variant with an interval

What doesn't work:

• Editing an out of the box existing Boefje and adding an interval results in a 404 error page. Not sure if that's created in this PR or in the "add/edit variant PR, since editing without adding an interval yields the same result. Will check this asap.
  Edit: It's also an issue in the add/edit variant branch, but I failed to catch it there: Update a Boefje #3521

Bug or feature?:

• Bug: 404 after editing an "out of the box" boefje (Editting an "out of the box" plugin/boefje results in a 404 error #3586)

[madelondohmen]

Bug or feature?:

* Bug: 404 after editing an "out of the box" boefje ([Editting an "out of the box" plugin/boefje results in a 404 error #3586](https://github.com/minvws/nl-kat-coordination/issues/3586))

Bug is fixed in #3573