Add framework for automated regression tests (#232)

* More details in the documentation (Regression.md)

.github/workflows/test.yml

@@ -0,0 +1,98 @@
+name: auto-testing
+on:
+  pull_request:
+    branches:
+      - develop
+
+jobs:
+  build:
+    runs-on: self-hosted
+    timeout-minutes: 345600
+    steps:
+      - name: Checkout last-dev
+        uses: actions/checkout@v2
+        with:
+          repository: MaximIntegratedAI/ai8x-training
+          ref: develop
+          submodules: recursive
+      - name: Setup Pyenv and Install Dependencies
+        uses: gabrielfalcao/pyenv-action@v13
+        with:
+          default: 3.8.11
+      - name: Create Venv
+        run: |
+          pyenv local 3.8.11
+          python3 -m venv venv --prompt ai8x-training
+      - name: Activate Venv
+        run: source venv/bin/activate
+      - name: Install Dependencies
+        run: |
+          pip3 install -U pip wheel setuptools
+          pip3 install -r requirements-cu11.txt
+      - name: Last Develop Check
+        run: python ./regression/last_dev.py --testconf ./regression/test_config.yaml --testpaths ./regression/paths.yaml
+
+  new-code:
+    runs-on: self-hosted
+    needs: [build]
+    timeout-minutes: 345600
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          repository: MaximIntegratedAI/ai8x-training
+          ref: develop
+          submodules: recursive
+      - name: Setup Pyenv and Install Dependencies
+        uses: gabrielfalcao/pyenv-action@v13
+        with:
+          default: 3.8.11
+      - name: Create Venv
+        run: |
+          pyenv local 3.8.11
+          python3 -m venv venv --prompt ai8x-training
+      - name: Activate Venv
+        run: source venv/bin/activate
+      - name: Install Dependencies
+        run: |
+          pip3 install -U pip wheel setuptools
+          pip3 install -r requirements-cu11.txt
+      - name: Create Test Script
+        run: python ./regression/create_test_script.py --testconf ./regression/test_config.yaml --testpaths ./regression/paths.yaml
+      - name: Run Training Scripts
+        run: bash /home/test/actions-runner/_work/ai8x-training/ai8x-training/scripts/output_file.sh
+      - name: Save Log Files
+        run: cp -r /home/test/actions-runner/_work/ai8x-training/ai8x-training/logs//home/test/max7800x/test_logs/$(date +%Y-%m-%d_%H-%M-%S)
+      - name: Save Test Scripts
+        run: cp -r /home/test/actions-runner/_work/ai8x-training/ai8x-training/scripts/output_file.sh/home/test/max7800x/test_scripts/
+      - name: Create and run ONNX script
+        run: python ./regression/create_onnx_script.py --testconf ./regression/test_config.yaml --testpaths ./regression/paths.yaml
+
+  test-results:
+    runs-on: self-hosted
+    needs: [new-code]
+    timeout-minutes: 345600
+    steps:
+      - uses: actions/checkout@v2
+        name: Checkout Test Codes
+        with:
+          repository: MaximIntegratedAI/ai8x-training
+          ref: develop
+          submodules: recursive
+      - name: Setup Pyenv and Install Dependencies
+        uses: gabrielfalcao/pyenv-action@v13
+        with:
+          default: 3.8.11
+      - name: Create Venv
+        run: |
+          pyenv local 3.8.11
+          python3 -m venv venv --prompt ai8x-training
+      - name: Activate Venv
+        run: source venv/bin/activate
+      - name: Install Dependencies
+        run: |
+          pip3 install -U pip wheel setuptools
+          pip3 install -r requirements-cu11.txt
+      - name: Log Diff
+        run: python ./regression/log_comparison.py --testconf ./regression/test_config.yaml --testpaths ./regression/paths.yaml
+      - name: Test Results
+        run: python ./regression/pass_fail.py --testconf ./regression/test_config.yaml --testpaths ./regression/paths.yaml

docs/Regression.md

@@ -0,0 +1,37 @@
+# Regression Test
+
+The regression test for the `ai8x-training` repository is tested when there is a pull request for the `develop` branch of `MaximIntegratedAI/ai8x-training` by triggering `test.yaml` GitHub actions.
+
+## Last Tested Code
+
+`last_dev.py` generates the log files for the last tested code. These log files are used for comparing the newly pushed code to check if there are any significant changes in the trained values. Tracking is done by checking the hash of the commit.
+
+## Creating Test Scripts
+
+The sample training scripts are under the `scripts` path. In order to create training scripts for regression tests, these scripts are rewritten by changing their epoch numbers by running `regression/create_test_script.py`. The aim of changing the epoch number is to keep the duration of the test under control. This epoch number is defined in `regression/test_config.yaml` for each model/dataset combination. Since the sizes of the models and the datasets vary, different epoch numbers can be defined for each of them in order to create a healthy test. If a new training script is added, the epoch number and threshold values must be defined in the `regression/test_config.yaml` file for the relevant model.
+
+## Comparing Log Files
+
+After running test scripts for newly pushed code, the log files are saved and compared to the last tested code’s log files by running `regression/log_comparison.py`, and the results are saved.
+
+## Pass-Fail Decision
+
+In the comparison, the test success criterion is that the difference does not exceed the threshold values defined in `regression/test_config.yaml` as a percentage. If all the training scripts pass the test, `pass_fail.py` completes with success. Otherwise, it fails and exits.
+
+## ONNX Export
+
+Scripts for ONNX export are created and run by running `create_onnx_scripts.py` by configuring `Onnx_Status: True` in `regression/test_config.yaml`. If it is set to `False`, ONNX export will be skipped.
+
+## Configuration
+
+In `regression/test_config.yaml`, the `Onnx_Status` and `Qat_Test` settings should be defined to `True` when ONNX export or QAT tests by using `policies/qat_policy.yaml` are desired. When `Qat_Test` is set to `False`, QAT will be done according to the main training script. All threshold values and test epoch numbers for each model/dataset combination are also configured in this file. In order to set up the test on a new system, `regression/paths.yaml` needs to be configured accordingly.
+
+## Setting Up Regression Test
+
+### GitHub Actions
+
+GitHub Actions is a continuous integration (CI) and continuous deployment (CD) platform provided by GitHub. It allows developers to automate various tasks, workflows, and processes directly within their GitHub repositories. A GitHub Workflow is an automated process defined using a YAML file that helps automate various tasks in a GitHub repository.
+
+In this project, with GitHub Actions, there is a 'test.yml' workflow that is triggered when a pull request is opened for the 'develop' branch of the 'MaximIntegratedAI/ai8x-training' repository. This workflow contains and runs the jobs and steps required for the regression test. Also, a self hosted GitHub Runner is used to run regression test actions in this workflow. In order to install GitHub Runner, go to Settings -> Actions -> Runners -> New self-hosted runner on GitHub. To learn more about GitHub Actions, see [GitHub Actions](https://docs.github.com/en/actions/quickstart).
+
+After installing and configuring a GitHub Runner in your local environment, configure it to start as a service during system startup in order to ensure that the self-hosted runner runs continuously and automatically. You can find more information about systemd services at [Systemd Services](https://linuxhandbook.com/create-systemd-services/).

regression/create_onnx_script.py

@@ -0,0 +1,128 @@
+###################################################################################################
+#
+# Copyright © 2023 Analog Devices, Inc. All Rights Reserved.
+# This software is proprietary and confidential to Analog Devices, Inc. and its licensors.
+#
+###################################################################################################
+"""
+Create onnx bash scripts for test
+"""
+import argparse
+import datetime
+import os
+import subprocess
+import sys
+
+import yaml
+
+
+def joining(lst):
+    """
+    Join list based on the ' ' delimiter
+    """
+    joined_str = ' '.join(lst)
+    return joined_str
+
+
+def time_stamp():
+    """
+    Take time stamp as string
+    """
+    time = str(datetime.datetime.now())
+    time = time.replace(' ', '.')
+    time = time.replace(':', '.')
+    return time
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument('--testconf', help='Enter the config file for the test', required=True)
+parser.add_argument('--testpaths', help='Enter the paths for the test', required=True)
+args = parser.parse_args()
+yaml_path = args.testconf
+test_path = args.testpaths
+
+# Open the YAML file
+with open(yaml_path, 'r', encoding='utf-8') as yaml_file:
+    # Load the YAML content into a Python dictionary
+    config = yaml.safe_load(yaml_file)
+
+with open(test_path, 'r', encoding='utf-8') as path_file:
+    # Load the YAML content into a Python dictionary
+    pathconfig = yaml.safe_load(path_file)
+
+if not config["Onnx_Status"]:
+    sys.exit(1)
+
+folder_path = pathconfig["folder_path"]
+output_file_path = pathconfig["output_file_path_onnx"]
+train_path = pathconfig["train_path"]
+
+logs_list = os.path.join(folder_path, sorted(os.listdir(folder_path))[-1])
+
+models = []
+datasets = []
+devices = []
+model_paths = []
+bias = []
+tar_names = []
+
+
+with open(output_file_path, "w", encoding='utf-8') as onnx_scripts:
+    with open(train_path, "r", encoding='utf-8') as input_file:
+        contents = input_file.read()
+    lines = contents.split("#!/bin/sh ")
+    lines = lines[1:]
+    contents_t = contents.split()
+
+    j = [i+1 for i in range(len(contents_t)) if contents_t[i] == '--model']
+    for index in j:
+        models.append(contents_t[index])
+
+    j = [i+1 for i in range(len(contents_t)) if contents_t[i] == '--dataset']
+    for index in j:
+        datasets.append(contents_t[index])
+
+    j = [i+1 for i in range(len(contents_t)) if contents_t[i] == '--device']
+    for index in j:
+        devices.append(contents_t[index])
+
+    for i, line in enumerate(lines):
+        if "--use-bias" in line:
+            bias.append("--use-bias")
+        else:
+            bias.append("")
+
+    for file_p in sorted(os.listdir(logs_list)):
+        temp_path = os.path.join(logs_list, file_p)
+        for temp_file in sorted(os.listdir(temp_path)):
+            if temp_file.endswith("_checkpoint.pth.tar"):
+                temp = os.path.join(temp_path, temp_file)
+                model_paths.append(temp)
+                tar_names.append(temp_file)
+
+    for i, (model, dataset, bias_value, device_name) in enumerate(
+        zip(models, datasets, bias, devices)
+    ):
+        for tar in model_paths:
+            element = tar.split('-')
+            modelsearch = element[-4][3:]
+            datasearch = element[-3].split('_')[0]
+            if datasearch == dataset.split('_')[0] and modelsearch == model:
+                # model_paths.remove(tar)
+                tar_path = tar
+                timestamp = time_stamp()
+                temp = (
+                    f"python train.py "
+                    f"--model {model} "
+                    f"--dataset {dataset} "
+                    f"--evaluate "
+                    f"--exp-load-weights-from {tar_path} "
+                    f"--device {device_name} "
+                    f"--summary onnx "
+                    f"--summary-filename {model}_{dataset}_{timestamp}_onnx "
+                    f"{bias_value}\n"
+                )
+                onnx_scripts.write(temp)
+cmd_command = "bash " + output_file_path
+
+subprocess.run(cmd_command, shell=True, check=True)

regression/create_test_script.py

@@ -0,0 +1,100 @@
+###################################################################################################
+#
+# Copyright © 2023 Analog Devices, Inc. All Rights Reserved.
+# This software is proprietary and confidential to Analog Devices, Inc. and its licensors.
+#
+###################################################################################################
+"""
+Create training bash scripts for test
+"""
+import argparse
+import os
+
+import yaml
+
+
+def joining(lst):
+    """
+    Join list based on the ' ' delimiter
+    """
+    join_str = ' '.join(lst)
+    return join_str
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument('--testconf', help='Enter the config file for the test', required=True)
+parser.add_argument('--testpaths', help='Enter the paths for the test', required=True)
+args = parser.parse_args()
+yaml_path = args.testconf
+test_path = args.testpaths
+
+# Open the YAML file
+with open(yaml_path, 'r', encoding='utf-8') as yaml_file:
+    # Load the YAML content into a Python dictionary
+    config = yaml.safe_load(yaml_file)
+
+with open(test_path, 'r', encoding='utf-8') as path_file:
+    # Load the YAML content into a Python dictionary
+    pathconfig = yaml.safe_load(path_file)
+
+# Folder containing the files to be concatenated
+script_path = pathconfig["script_path"]
+# Output file name and path
+output_file_path = pathconfig["output_file_path"]
+
+# global log_file_names
+log_file_names = []
+
+# Loop through all files in the folder
+with open(output_file_path, "w", encoding='utf-8') as output_file:
+    for filename in os.listdir(script_path):
+        # Check if the file is a text file
+        if filename.startswith("train"):
+            # Open the file and read its contents
+            with open(os.path.join(script_path, filename), encoding='utf-8') as input_file:
+                contents = input_file.read()
+
+                temp = contents.split()
+                temp.insert(1, "\n")
+                i = temp.index('--epochs')
+                j = temp.index('--model')
+                k = temp.index('--dataset')
+
+                if config["Qat_Test"]:
+                    if '--qat-policy' in temp:
+                        x = temp.index('--qat-policy')
+                        temp[x+1] = "policies/qat_policy.yaml"
+                    else:
+                        temp.insert(-1, ' --qat-policy policies/qat_policy.yaml')
+
+                log_model = temp[j+1]
+                log_data = temp[k+1]
+
+                if log_model == "ai87imageneteffnetv2":
+                    num = temp.index("--batch-size")
+                    temp[num+1] = "128"
+
+                log_name = temp[j+1] + '-' + temp[k+1]
+                log_file_names.append(filename[:-3])
+
+                if log_data == "FaceID":
+                    continue
+
+                if log_data == "VGGFace2_FaceDetection":
+                    continue
+
+                temp[i+1] = str(config[log_data][log_model]["epoch"])
+
+                if '--deterministic' not in temp:
+                    temp.insert(-1, '--deterministic')
+
+                temp.insert(-1, '--name ' + log_name)
+
+                data_name = temp[k+1]
+                if data_name in config and "datapath" in config[data_name]:
+                    path_data = config[log_data]["datapath"]
+                    temp.insert(-1, '--data ' + path_data)
+
+                temp.append("\n")
+                contents = joining(temp)
+                output_file.write(contents)