Add: Notus rust implementation and integrate it into openvasd

This commit contains the complete notus implementation and integration in the openvas rust framework.

To enable the notus endpoint, it has to be configured in the `openvasd.toml` file, e.g.:
```
[notus]
advisory_path = "/var/lib/notus/products"
```
Note: adjust path according to your local setup, be sure to pick the products directory

If everything is done correctly, the notus endpoint is available at `/notus` and `/notus/{os}`.
For the following examples the URL and X-API-KEY must be changed according to the used configuration.

With a get request to `/notus` all available products (OS) are returned, e.g.:
```
curl --verbose  --insecure  --request GET https://127.0.0.1:3000/notus -H "X-API-KEY: changeme" |jq .
```

With a post request to `/notus/{os}`, a notus scan is requested. For the `{os}` key the required operating system must be given, e.g. for Debian 12 the corresponding os string is debian_12. The general rule is, the os string in lowercase and all white spaces are replacec by `_`. Here an example for a small notus scan execution:
```
curl --verbose  --insecure  --request POST https://127.0.0.1:3000/notus/debian_12 -H "X-API-KEY: changeme" -d '["firefox-esr-102.12.0esr-1~deb12u1"]' |jq .
```

The following changes are contained in the notus implementation/integration:

* SC-941: Add Notus endpoint to OpenAPI documentation

* SC-942: Implement notus entrypoint

* SC-943: Implement tests for notus functionality

* SC-944: Implement Hashsum file loader for Notus

Load Notus files from the feed using the Sha256sums file. Hashsums are verified, as the files are needed.

* SC-945: Implement version comparison algorithm

* SC-946: Create Package type models

Supported package systems: ebuild, rpm, deb, slackware

* SC-947: Use openvasd Notus endpoint for LSC in openvas

- add URI parser to get the schema, host and port to communicate with openvasd
- use curl for simple http/https client
- add structures and functions to parse the openvasd response, the notus json object containing the advisories and vulnerabilities
- function to process the advisories and to store them in the redis kb in the right format.
- dont init mqtt if openvasd lsc is enabled. Fix OS format. And other small improvements

* SC-948: Implement Notus Scanning logic

* SC-951: Change Hashsum loader to only verify necessary Hashsums

For the verification of the Hashsum used in the feed, an Iterator is used to check each line of the Hashsum file. The Hashsum was immediately verified, regardless if each iteration Item was even necessary. With this change, it is possible to only verify the Hashsum, when the corresponding File is needed.

* SC-954: Add /notus endpoint to get a list of supported product OS

* SC-967 Add: perform notus signature check

If enabled in the congiguration file (or passed via command line option) and the GNUPGHOME env variable is set pointing to the gnupg keyring, it will perform the signature check each time that a new product file is uploaded.

* Add notus standalone executable

* Change: add libcurl to docker images.

* Fix: CodeQL: uses build image

Although the build image construct got deprecated codeql relies on it.

To change that CodeQL now uses gvm-libs and install the dependencies like
the other build steps.


---------

Co-authored-by: Kraemii <[email protected]>
Co-authored-by: Philipp Eder <[email protected]>

Author: 3 people

.docker/build.Dockerfile

@@ -28,4 +28,5 @@ RUN apt-get update && apt-get install --no-install-recommends --no-install-sugge
   libroken19-heimdal \
   libhdb9-heimdal \
   libpopt0 \
+  libcurl4-gnutls-dev \
   && rm -rf /var/lib/apt/lists/*

.docker/prod-oldstable.Dockerfile

@@ -31,6 +31,8 @@ RUN apt-get update && apt-get install --no-install-recommends --no-install-sugge
     libroken18-heimdal \
     libhdb9-heimdal \
     libpopt0 \
+    libcurl4-gnutls-dev \
+    libcurl4 \
     libcgreen1-dev \
     && rm -rf /var/lib/apt/lists/*
 
@@ -64,6 +66,7 @@ RUN apt-get update && apt-get install --no-install-recommends --no-install-sugge
   libroken18-heimdal \
   libhdb9-heimdal \
   libpopt0 \
+    libcurl4 \
   zlib1g\
   && rm -rf /var/lib/apt/lists/*
 COPY .docker/openvas.conf /etc/openvas/

.docker/prod-testing.Dockerfile

@@ -39,6 +39,7 @@ RUN apt-get update && apt-get install --no-install-recommends --no-install-sugge
   libroken19-heimdal \
   libhdb9-heimdal \
   libpopt0 \
+  libcurl4 \
   zlib1g\
   && rm -rf /var/lib/apt/lists/*
 COPY .docker/openvas.conf /etc/openvas/

.docker/prod.Dockerfile

@@ -35,7 +35,8 @@ RUN apt-get update && apt-get install --no-install-recommends --no-install-sugge
   libroken19-heimdal \
   libhdb9-heimdal \
   libpopt0 \
-  zlib1g\
+  libcurl4 \
+  zlib1g \
   && rm -rf /var/lib/apt/lists/*
 COPY .docker/openvas.conf /etc/openvas/
 # must be pre built within the rust dir and moved to the bin dir

.dockerignore

@@ -1,3 +1,4 @@
 .vscode/
 .mergify.yml
 build/
+rust/target

.github/install-openvas-dependencies.sh

@@ -26,6 +26,8 @@ apt-get update && apt-get install --no-install-recommends --no-install-suggests
     libroken19-heimdal \
     libhdb9-heimdal \
     libpopt0 \
+    libcurl4 \
+    libcurl4-gnutls-dev \
     && rm -rf /var/lib/apt/lists/*
 
 curl -L -o cgreen.tar.gz https://github.com/cgreen-devs/cgreen/archive/refs/tags/1.6.2.tar.gz -k

.github/workflows/codeql-analysis-c.yml

@@ -16,7 +16,7 @@ jobs:
       actions: read
       contents: read
       security-events: write
-    container: ${{ github.repository }}-build:unstable
+    container: greenbone/gvm-libs:unstable
 
     strategy:
       fail-fast: false
@@ -26,7 +26,9 @@ jobs:
     steps:
     - name: Checkout repository
       uses: actions/checkout@v4
-
+    - name: install dependencies
+      run: |
+          sh .github/install-openvas-dependencies.sh
     - name: Initialize CodeQL
       uses: github/codeql-action/init@v2
       with:

CMakeLists.txt

@@ -200,7 +200,7 @@ configure_file (src/openvas_log_conf.cmake_in src/openvas_log.conf)
 enable_testing ()
 
 add_custom_target (tests
-                   DEPENDS attack-test pcap-test ipc-openvas-test)
+                   DEPENDS attack-test pcap-test ipc-openvas-test lsc-test)
 
 ## Program
 

INSTALL.md

@@ -25,6 +25,7 @@ Prerequisites:
 * libssh >= 0.6.0
 * libksba >= 1.0.7
 * libgnutls >= 3.6.4
+* libcurl4-gnutls-dev
 * libbsd
 
 Prerequisites for building documentation:
@@ -54,7 +55,7 @@ Install prerequisites on Debian GNU/Linux 'Bullseye' 11:
 
     apt-get install gcc pkg-config libssh-gcrypt-dev libgnutls28-dev \
     libglib2.0-dev libjson-glib-dev libpcap-dev libgpgme-dev bison libksba-dev \
-    libsnmp-dev libgcrypt20-dev redis-server libbsd-dev
+    libsnmp-dev libgcrypt20-dev redis-server libbsd-dev libcurl4-gnutls-dev
 
 
 Compiling openvas

doc/full_installation_guide.md

@@ -295,6 +295,7 @@ sudo apt install -y \
   rsync \
   nmap \
   libjson-glib-dev \
+  libcurl4-gnutls-dev \
   libbsd-dev
 ```
 

misc/CMakeLists.txt

@@ -14,6 +14,7 @@ endif (NOT PKG_CONFIG_FOUND)
 pkg_check_modules (GLIB REQUIRED glib-2.0>=2.42)
 pkg_check_modules (GLIB_JSON REQUIRED json-glib-1.0>=1.4.4)
 pkg_check_modules (GNUTLS REQUIRED gnutls>=3.6.4)
+pkg_check_modules (CURL REQUIRED libcurl>=7.74.0)
 
 pkg_check_modules (LIBGVM_BASE REQUIRED libgvm_base>=22.4)
 pkg_check_modules (LIBGVM_UTIL REQUIRED libgvm_util>=22.4)
@@ -68,7 +69,7 @@ add_definitions (-DOPENVAS_MISC_VERSION="${PROJECT_VERSION_STRING}")
 
 include_directories (${GLIB_INCLUDE_DIRS} ${GLIB_JSON_INCLUDE_DIRS}
                      ${LIBGVM_BASE_INCLUDE_DIRS}
-                     ${GNUTLS_INCLUDE_DIRS})
+                     ${GNUTLS_INCLUDE_DIRS} ${CURL_INCLUDE_DIRS})
 
 # Library
 
@@ -92,7 +93,7 @@ set_target_properties (openvas_misc_shared PROPERTIES VERSION "${PROJECT_VERSION
 target_link_libraries (openvas_misc_shared LINK_PRIVATE
                        ${GNUTLS_LDFLAGS} ${UUID_LDFLAGS}
                        ${GLIB_LDFLAGS} ${GLIB_JSON_LDFLAGS}
-                       ${PCAP_LDFLAGS} ${LIBGVM_BOREAS_LDFLAGS}
+                       ${PCAP_LDFLAGS} ${LIBGVM_BOREAS_LDFLAGS} ${CURL_LDFLAGS}
                        ${LINKER_HARDENING_FLAGS})
 
 if (OPENVAS_STATE_DIR)
@@ -135,6 +136,7 @@ set (LINK_LIBS_FOR_TESTS cgreen
                        ${LIBGVM_BASE_LDFLAGS}
                        ${GLIB_LDFLAGS}
                        ${PCAP_LDFLAGS}
+                       ${CURL_LDFLAGS}
                        ${LINKER_HARDENING_FLAGS} ${CMAKE_THREAD_LIBS_INIT}
                        ${ALIVEDETECTION_TEST_LINKER_WRAP_OPTIONS})
 
@@ -161,9 +163,25 @@ target_include_directories (ipc-openvas-test PRIVATE ${CGREEN_INCLUDE_DIRS})
 target_link_libraries (ipc-openvas-test cgreen
                        ${GLIB_LDFLAGS}
                        ${GLIB_JSON_LDFLAGS}
+                       ${CURL_LDFLAGS}
                        ${LINKER_HARDENING_FLAGS})
 
 add_custom_target (tests-ipc-openvas
                   DEPENDS ipc-openvas-test)
 
+# lsc-tests
+add_executable (lsc-test EXCLUDE_FROM_ALL table_driven_lsc_tests.c kb_cache.c plugutils.c scan_id.c)
+add_test (lsc-test lsc-test)
+target_include_directories (lsc-test PRIVATE ${CGREEN_INCLUDE_DIRS})
+target_link_libraries (lsc-test cgreen
+                       ${LIBGVM_BASE_LDFLAGS}
+                       ${LIBGVM_UTIL_LDFLAGS}
+                       ${GLIB_LDFLAGS}
+                       ${GLIB_JSON_LDFLAGS}
+                       ${CURL_LDFLAGS}
+                       ${LINKER_HARDENING_FLAGS})
+
+add_custom_target (tests-lsc
+                  DEPENDS lsc-test)
+
 ## End