Dockerized End-To-End Tests (Proof of Concept)

.dockerignore

@@ -0,0 +1,3 @@
+Makefile
+build/
+sample_defs/

docker-compose.yml

@@ -0,0 +1,29 @@
+version: '2'
+
+services:
+  cfs:
+    build:
+        context: ./
+        dockerfile: ./tools/e2eTests/platforms/Ubuntu/18.04/Dockerfile
+        args:
+          - ENABLE_UNIT_TESTS=false
+          - SIMULATION=native
+          - BUILDTYPE=debug
+          - OMIT_DEPRECATED=true
+    cap_add: 
+      - CAP_SYS_RESOURCE
+    networks:
+      - default
+    depends_on:
+      - gsw
+
+  gsw:
+    build:
+        context: ./
+        dockerfile: ./tools/e2eTests/gsw/Dockerfile
+    networks:
+      - default
+
+networks:
+  default:
+    internal: true    

tools/e2eTests/README.md

@@ -0,0 +1,133 @@
+# Core Flight System - Dockerized End-To-End Testing Tool
+
+A tool to perform dockerized end-to-end tests of cFS.
+
+This tool simulates network-based interactions between the cFS executable and a pseudo-ground system software acting as a test runner.
+
+## Getting Started
+
+### Requirements
+
+* Docker 19 or higher
+* Docker Compose compatible with compose file format 2.0 or higher
+
+### Run the test
+
+From the cFS top-level directory, run the following command:
+
+```
+docker-compose up --abort-on-container-exit --exit-code-from gsw
+```
+
+By default, the test runs with the following options:
+
+* Operating System: Ubuntu 18.04
+* Unit tests disabled
+* Simulation: native
+* Build type: debug
+* Deprecated omitted
+
+When the test passes, it exits with a zero exit code. Otherwise, it exits with a non-zero exit code.
+
+Note: `--abort-on-container-exit` makes `cfs` stop when `gsw` (i.e. the test runner) exits, and `--exit-code-from gsw` returns `gsw` exit code (that is: the test status).
+
+### Optional: Customize the cFS Image
+
+The default options to build the images are specified in `docker-compose.yml` (located at the cFS top-level directory), lines 7-12:
+
+```yaml
+dockerfile: ./tools/e2eTests/platforms/Ubuntu/18.04/Dockerfile
+args:
+  - ENABLE_UNIT_TESTS=false
+  - SIMULATION=native
+  - BUILDTYPE=debug
+  - OMIT_DEPRECATED=true
+```
+
+To customize these options:
+
+1. First, make `docker-compose.yml` refer to the relevant `Dockerfile` to specify the expected operating system (cFS `dockerfile` configuration option, line 7):
+
+    ```yaml
+    cfs:
+      build:
+        ...
+        dockerfile: ./tools/e2eTests/platforms/Ubuntu/18.04/Dockerfile
+    ```
+
+    To this end, use one of the following options.
+
+    | Operating System | Path to the corresponding Dockerfile                       |
+    |------------------|--------------------------------------------------------|
+    | Alpine 3         | `./tools/e2eTests/platforms/Alpine/3/Dockerfile`       |
+    | CentOS 7         | `./tools/e2eTests/platforms/CentOS/7/Dockerfile`       |
+    | Ubuntu 18.04     | `./tools/e2eTests/platforms/Ubuntu/18.04/Dockerfile`   |
+    | Ubuntu 20.04     | `./tools/e2eTests/platforms/Ubuntu/20.04/Dockerfile`   |
+
+2. Second, build the relevant cFS Docker image (tagged `cfs`) by passing the expected option(s) using `--build-arg`:
+
+   ```
+    docker-compose build --no-cache --build-arg <key1=value1> [--build-arg <key2=value2> ...] cfs
+    ```
+
+	For instance, to enable unit tests and set the build type to release:
+
+    ```
+    docker-compose build --no-cache --build-arg ENABLE_UNIT_TESTS=true --build-arg BUILDTYPE=release cfs
+    ```
+
+3. Finally, run the test:
+
+	```
+	docker-compose up --abort-on-container-exit --exit-code-from gsw
+	```
+
+## Alternative: Direct Use of cFS Containers
+
+It is possible to build cFS images and run the corresponding containers independently of the Docker Compose solution.
+
+### Build the cFS image
+
+To build a cFS image, run the following command from the cFS top-level directory (indeed, the working directory for the build context is expected to be the root of the cFS directory):
+
+```
+docker build \
+-t <tag of the image> \
+-f <path to the relevant Dockerfile> \
+--build-arg ENABLE_UNIT_TESTS=false \
+--build-arg SIMULATION=native \
+--build-arg BUILDTYPE=<build type> \
+--build-arg OMIT_DEPRECATED=<boolean value> .
+```
+
+Example:
+
+```
+docker build \
+-t cfs_ubuntu18 \
+-f ./tools/e2eTests/platforms/Ubuntu/18.04/Dockerfile \
+--build-arg ENABLE_UNIT_TESTS=false \
+--build-arg SIMULATION=native \
+--build-arg BUILDTYPE=debug \
+--build-arg OMIT_DEPRECATED=true .
+```
+
+Note that unit tests should preferably be disabled to decouple unit testing from end-to-end testing.
+
+### Run the cFS Container
+
+One the cFS image has been built, it can be run with the following command:
+
+```
+docker run --cap-add CAP_SYS_RESOURCE --net=host <tag of the image>
+```
+
+Example:
+
+```
+docker run --cap-add CAP_SYS_RESOURCE --net=host cfs_ubuntu18
+```
+
+The cFS ground system can then interact with the containerized cFS executable on localhost (on Linux and Windows hosts).
+
+Note: `--cap-add CAP_SYS_RESOURCE` is required for queues to be handled properly. Failing to add the `CAP_SYS_RESOURCE` capability would prematurely terminate the execution of the cFS container.

tools/e2eTests/gsw/Dockerfile

@@ -0,0 +1,11 @@
+FROM python:3.7.7-alpine3.11 
+
+ENV PYTHONUNBUFFERED=1
+
+WORKDIR /test_runner
+
+COPY tools/e2eTests/gsw/gsw.py gsw.py
+
+RUN chmod +x gsw.py
+
+ENTRYPOINT [ "./gsw.py" ]

tools/e2eTests/gsw/gsw.py

@@ -0,0 +1,138 @@
+#!/usr/bin/env python3
+
+"""
+Proof of concept:
+Pseudo-GSW to perform a simple end-to-end test of a dockerized cFS
+"""
+
+from enum import IntEnum
+import socket
+import time
+import sys
+
+class Test_status(IntEnum):
+    PASS = 0
+    FAIL = 1
+
+SAMPLE_APP_NOOP_PACKET = bytearray([
+        0x18, 0x82, 0xC0, 0x00, 0x00,
+        0x01, 0x00, 0x00
+    ])
+
+ENABLE_TELEMETRY_PACKET_TEMPLATE = bytearray([
+        0x18, 0x80, 0xC0, 0x00, 0x00,
+        0x12, 0x06, 0x00, 0x00, 0x00,
+        0x00, 0x00, 0x00, 0x00, 0x00,
+        0x00, 0x00, 0x00, 0x00, 0x00,
+        0x00, 0x00, 0x00, 0x00, 0x00
+    ])
+
+CFS_PORT = 1234
+GSW_PORT = 1235
+
+try:
+    # Host name shall match the cFS service name in docker-compose.yml
+    CFS_IP = socket.gethostbyname('cfs')
+except:
+    print('Error: cFS IP is unknown')
+    exit(1)
+
+try:
+    # Host name shall match the GSW service name in docker-compose.yml
+    GSW_IP = socket.gethostbyname('gsw') 
+except:
+    print('Error: GSW IP is unknown')
+    exit(1)
+
+
+def craft_telemetry_packet() -> bytearray:
+    """
+    Craft telemetry packet from enable telemetry packet template
+    """
+
+    packet = ENABLE_TELEMETRY_PACKET_TEMPLATE
+
+    i = 8 # IP starts at index 8
+    for c in GSW_IP:
+       packet[i] = int(hex(ord(c)), 16)
+       i += 1
+
+    return packet
+
+def enable_telemetry() -> bool:
+    """
+    Try to enable the telemetry
+    """
+
+    print('Enable telemetry (cFS IP: ' + CFS_IP + ', GSW IP: ' + GSW_IP + ')')
+
+    testSocket.sendto(craft_telemetry_packet(), (CFS_IP, CFS_PORT))
+
+    if telemetry_contains("telemetry output enabled for IP {}".format(GSW_IP), 10):
+        return True 
+
+    return False
+
+def telemetry_contains(expected_event, max_attempts) -> bool:
+    """
+    Check whether the telemetry contains an expected event or not
+    """
+
+    print ('Listening to cFS telemetry (expected event: "' + expected_event + '")')    
+
+    attempts = 0
+    while attempts < max_attempts:
+        try:
+            datagram, _ = testSocket.recvfrom(4096)
+            actual_event = ''.join(map(chr, datagram))
+            #print(attempts, ': ', actual_event)
+            if expected_event in actual_event:
+                return True
+        except:
+            pass
+
+        attempts += 1
+        time.sleep(1)
+
+    return False 
+
+def assert_sample_app_noop():
+    """
+    Send a NOOP command (SAMPLE_APP) and ensure that it is acknowledged by cFS
+    """
+
+    print ('SAMPLE_APP: Sending NOOP command')
+    testSocket.sendto(SAMPLE_APP_NOOP_PACKET, (CFS_IP, CFS_PORT))
+
+    if telemetry_contains('SAMPLE: NOOP command', 10):
+        return Test_status.PASS
+
+    return Test_status.FAIL
+
+
+#
+# Main
+#
+if __name__ == "__main__":
+
+    # Setup
+    time.sleep(5) # Warming up
+    testSocket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+    testSocket.bind(('', GSW_PORT))
+    testSocket.settimeout(.1)
+
+    # Test - SAMPLE_APP NOOP command is acknowledged by cFS
+    if enable_telemetry():
+        test_status = assert_sample_app_noop()
+
+    # Teardown
+    testSocket.close()
+
+    # Exit
+    print('Test status: ', end = '')
+    if test_status == Test_status.PASS:
+        print('PASS')
+        sys.exit(0)
+    else:
+        print('FAIL')
+        sys.exit(1)

tools/e2eTests/platforms/Alpine/3/Dockerfile

@@ -0,0 +1,53 @@
+# Warning: This image is currently not operational 
+
+FROM alpine:3.11 AS builder
+
+ARG ENABLE_UNIT_TESTS
+ENV ENABLE_UNIT_TESTS=${ENABLE_UNIT_TESTS} 
+
+ARG SIMULATION
+ENV SIMULATION=${SIMULATION}
+
+ARG BUILDTYPE
+ENV BUILDTYPE=${BUILDTYPE}
+
+ARG OMIT_DEPRECATED
+ENV OMIT_DEPRECATED=${OMIT_DEPRECATED}
+
+RUN echo http://nl.alpinelinux.org/alpine/edge/testing >> /etc/apk/repositories
+
+RUN set -ex &&                          \
+    apk add --update --no-cache         \
+        make=4.2.1-r2                   \
+        cmake=3.15.5-r0                 \
+        git=2.24.3-r0                   \
+        gcc=9.2.0-r4                    \
+        g++=9.2.0-r4
+
+RUN if [ "${ENABLE_UNIT_TESTS}" = true ]; \
+    then { echo http://nl.alpinelinux.org/alpine/edge/testing >> /etc/apk/repositories && \
+    apk add --update --no-cache lcov=1.14-r0; } fi
+
+WORKDIR /cFS
+
+COPY . . 
+
+RUN git submodule init \
+    && git submodule update \
+    && cp cfe/cmake/Makefile.sample Makefile \
+    && cp -r cfe/cmake/sample_defs .
+
+RUN make prep
+RUN make
+RUN make install
+RUN if [ "${ENABLE_UNIT_TESTS}" = true ]; then { ( make test | grep 'Failed' ) && ( make lcov | grep '%' ); } fi
+RUN if [ "${ENABLE_UNIT_TESTS}" = true ]; then { cat ./build/native/Testing/Temporary/LastTest.log | grep 'FAIL' | grep -v 'FAIL::0'; } fi
+
+
+FROM alpine:3.11
+
+COPY --from=builder /cFS/build /cFS/build
+
+WORKDIR /cFS/build/exe/cpu1
+
+CMD [ "./core-cpu1" ]