Regression Test repo pull request

.github/workflows/test.yml

@@ -30,7 +30,7 @@ jobs:
           pip3 install -U pip wheel setuptools
           pip3 install -r requirements-cu11.txt
       - name: Last Develop Check
-        run: python ./regression-test/last_dev.py --testconf ./regression-test/test_config.yaml
+        run: python ./regression/last_dev.py --testconf ./regression/test_config.yaml --testpaths ./regression/paths.yaml
 
   new-code:
     runs-on: self-hosted
@@ -57,27 +57,23 @@ jobs:
           pip3 install -U pip wheel setuptools
           pip3 install -r requirements-cu11.txt
       - name: Create Test Script
-        run: python ./regression-test/create_test_script.py --testconf ./regression-test/test_config.yaml
+        run: python ./regression/create_test_script.py --testconf ./regression/test_config.yaml --testpaths ./regression/paths.yaml
       - name: Run Training Scripts
-        run: bash /home/asyaturhal/actions-runner/_work/ai8x-training/ai8x-training/scripts/output_file.sh
+        run: bash /home/test/actions-runner/_work/ai8x-training/ai8x-training/scripts/output_file.sh
       - name: Save Log Files
-        run: cp -r /home/asyaturhal/actions-runner/_work/ai8x-training/ai8x-training/logs/ /home/asyaturhal/desktop/ai/test_logs/$(date +%Y-%m-%d_%H-%M-%S)
-      - name: Save test scripts
-        run: cp -r /home/asyaturhal/actions-runner/_work/ai8x-training/ai8x-training/scripts/output_file.sh /home/asyaturhal/desktop/ai/test_scripts/
+        run: cp -r /home/test/actions-runner/_work/ai8x-training/ai8x-training/logs//home/test/max7800x/test_logs/$(date +%Y-%m-%d_%H-%M-%S)
+      - name: Save Test Scripts
+        run: cp -r /home/test/actions-runner/_work/ai8x-training/ai8x-training/scripts/output_file.sh/home/test/max7800x/test_scripts/
       - name: Create and run ONNX script
-        run: python ./regression-test/create_onnx_scripts.py --testconf ./regression-test/test_config.yaml
+        run: python ./regression/create_onnx_script.py --testconf ./regression/test_config.yaml --testpaths ./regression/paths.yaml
 
   test-results:
     runs-on: self-hosted
     needs: [new-code]
     timeout-minutes: 345600
     steps:
       - uses: actions/checkout@v2
-        name: trial-check
-        with:
-          submodules: recursive
-      - name: Checkout Test Codes
-        uses: actions/checkout@v2
+        name: Checkout Test Codes
         with:
           repository: MaximIntegratedAI/ai8x-training
           ref: develop
@@ -97,6 +93,6 @@ jobs:
           pip3 install -U pip wheel setuptools
           pip3 install -r requirements-cu11.txt
       - name: Log Diff
-        run: python ./regression-test/log_comparison.py
+        run: python ./regression/log_comparison.py --testconf ./regression/test_config.yaml --testpaths ./regression/paths.yaml
       - name: Test Results
-        run: python ./regression-test/pass_fail.py --testconf ./regression-test/test_config.yaml
+        run: python ./regression/pass_fail.py --testconf ./regression/test_config.yaml --testpaths ./regression/paths.yaml

README.md

@@ -3275,35 +3275,6 @@ See the [benchmarking guide](https://github.com/MaximIntegratedAI/MaximAI_Docume
 
 
 
-## Regression Test
-
-Regression test for ai8xtraining repo is tested when there is a pull request for develop branch of MaximIntegratedAI/ai8x-training by triggering test.yaml github actions.
-
-### Last Developed Code Training
-
-last_dev.py code generates the log files of the last developed and tested code. These log files are used for comparing the new pushed code to check if there is any change in the trained values. Tracking is done by checking hex number of the repository.
-
-### Creating Test Scripts
-
-There are training scripts that are developed under the scripts path. In order to create training scripts for regression test, these scripts are rewritten by changing their epoch number by running create_test_script.py code. The aim for changing the epoch number is to keep the duration of the test under control. This epoch number is defined in the test_config.yaml file for each model dataset combination. Since the sizes of the models are different, different epoch numbers can be defined for each of them in order to create a healthy test. If a new training script is added, the epoch number and threshold values must be defined in the test_config.yaml file for the relevant model.
-
-### Comparing Log Files
-
-After running test scripts for new pushed code, it’s log files are saved and compared with last developed code’s log files by running log_comparison.py and results are also saved.
-
-### Pass Fail Decision
-
-In the comparison, the test's success criterion is that the difference does not exceed the threshold value defined in the test_config.yaml file as a percentage. If all the training scripts passed the test, pass_fail.py code completes with success. Otherwise it fails and exits by cancelling pull request.
-
-### Onnx Export
-
-Scripts for onnx export are created and ran by running create_onnx_scripts.py by configuring ‘Onnx_Status: True’ in test_config.yaml . If it is set as ‘False’ onnx export won't be done.
-
-### Test Configuration
-
-In test_config.yaml file Onnx_Status and Qat_Test should be ste to 'true' if you want to create onnx export or make qat test by using policies/qat_policy.yaml. When Qat_Test is set to false qat will be done according to the main training script. All the threshold values and test epoch numbers for each model dataset combination are also configured in this file.
-
-
 ## Further Information
 
 Additional information about the evaluation kits, and the software development kit (MSDK) is available on the web at <https://github.com/MaximIntegratedAI/MaximAI_Documentation>.

docs/Regression.md

@@ -0,0 +1,28 @@
+# Regression Test
+
+The regression test for the `ai8x-training` repository is tested when there is a pull request for the `develop` branch of `MaximIntegratedAI/ai8x-training` by triggering `test.yaml` GitHub actions.
+
+## Last Tested Code
+
+`last_dev.py` generates the log files for the last tested code. These log files are used for comparing the newly pushed code to check if there are any significant changes in the trained values. Tracking is done by checking the hash of the commit.
+
+## Creating Test Scripts
+
+The sample training scripts are under the `scripts` path. In order to create training scripts for regression tests, these scripts are rewritten by changing their epoch numbers by running `regression/create_test_script.py`. The aim of changing the epoch number is to keep the duration of the test under control. This epoch number is defined in `regression/test_config.yaml` for each model/dataset combination. Since the sizes of the models are different, different epoch numbers can be defined for each of them in order to create a healthy test. If a new training script is added, the epoch number and threshold values must be defined in the `regression/test_config.yaml` file for the relevant model.
+
+## Comparing Log Files
+
+After running test scripts for newly pushed code, the log files are saved and compared to the last tested code’s log files by running `regression/log_comparison.py`, and the results are saved.
+
+## Pass-Fail Decision
+
+In the comparison, the test success criterion is that the difference does not exceed the threshold values defined in `regression/test_config.yaml` as a percentage. If all the training scripts pass the test, `pass_fail.py` completes with success. Otherwise, it fails and exits.
+
+## ONNX Export
+
+Scripts for ONNX export are created and run by running `create_onnx_scripts.py` by configuring `Onnx_Status: True` in `regression/test_config.yaml`. If it is set to `False`, ONNX export will be skipped.
+
+## Configuration
+
+In `regression/test_config.yaml`, the `Onnx_Status` and `Qat_Test` settings should be defined to `True` when ONNX export or QAT tests by using `policies/qat_policy.yaml` are desired. When `Qat_Test` is set to `False`, QAT will be done according to the main training script. All threshold values and test epoch numbers for each model/dataset combination are also configured in this file. In order to set up the test on a new system, `regression/paths.yaml` needs to be configured accordingly.
+

regression-test/create_onnx_script.py → regression/create_onnx_script.py

@@ -1,6 +1,6 @@
 ###################################################################################################
 #
-# Copyright (C) 2020 Maxim Integrated Products, Inc. All Rights Reserved.
+# Copyright (C) 2023 Maxim Integrated Products, Inc. All Rights Reserved.
 #
 # Maxim Integrated Products, Inc. Default Copyright Notice:
 # https://www.maximintegrated.com/en/aboutus/legal/copyrights.html
@@ -38,24 +38,26 @@ def time_stamp():
 
 parser = argparse.ArgumentParser()
 parser.add_argument('--testconf', help='Enter the config file for the test', required=True)
+parser.add_argument('--testpaths', help='Enter the paths for the test', required=True)
 args = parser.parse_args()
 yaml_path = args.testconf
+test_path = args.testpaths
 
 # Open the YAML file
 with open(yaml_path, 'r', encoding='utf-8') as file:
     # Load the YAML content into a Python dictionary
     config = yaml.safe_load(file)
 
+with open(test_path, 'r', encoding='utf-8') as file2:
+    # Load the YAML content into a Python dictionary
+    pathconfig = yaml.safe_load(file2)
+
 if not config["Onnx_Status"]:
     sys.exit(1)
 
-folder_path = r"/home/asyaturhal/desktop/ai/test_logs/"
-output_file_path = (
-    r"./scripts/onnx_scripts.sh"
-)
-train_path = (
-    r"/home/asyaturhal/desktop/ai/test_scripts/output_file.sh"
-)
+folder_path = pathconfig["folder_path"]
+output_file_path = pathconfig["output_file_path_onnx"]
+train_path = pathconfig["train_path"]
 
 logs_list = folder_path + '/' + sorted(os.listdir(folder_path))[-1]
 

regression-test/create_test_script.py → regression/create_test_script.py

@@ -1,6 +1,6 @@
 ###################################################################################################
 #
-# Copyright (C) 2020-2022 Maxim Integrated Products, Inc. All Rights Reserved.
+# Copyright (C) 2023 Maxim Integrated Products, Inc. All Rights Reserved.
 #
 # Maxim Integrated Products, Inc. Default Copyright Notice:
 # https://www.maximintegrated.com/en/aboutus/legal/copyrights.html
@@ -25,19 +25,24 @@ def joining(lst):
 
 parser = argparse.ArgumentParser()
 parser.add_argument('--testconf', help='Enter the config file for the test', required=True)
+parser.add_argument('--testpaths', help='Enter the paths for the test', required=True)
 args = parser.parse_args()
 yaml_path = args.testconf
+test_path = args.testpaths
 
 # Open the YAML file
 with open(yaml_path, 'r', encoding='utf-8') as file:
     # Load the YAML content into a Python dictionary
     config = yaml.safe_load(file)
 
-# Folder containing the files to be concatenated
-script_path = r"./scripts"
+with open(test_path, 'r', encoding='utf-8') as file2:
+    # Load the YAML content into a Python dictionary
+    pathconfig = yaml.safe_load(file2)
 
+# Folder containing the files to be concatenated
+script_path = pathconfig["script_path"]
 # Output file name and path
-output_file_path = r"./scripts/output_file.sh"
+output_file_path = pathconfig["output_file_path"]
 
 # global log_file_names
 log_file_names = []

regression-test/last_dev.py → regression/last_dev.py

@@ -1,6 +1,6 @@
 ###################################################################################################
 #
-# Copyright (C) 2020-2022 Maxim Integrated Products, Inc. All Rights Reserved.
+# Copyright (C) 2023 Maxim Integrated Products, Inc. All Rights Reserved.
 #
 # Maxim Integrated Products, Inc. Default Copyright Notice:
 # https://www.maximintegrated.com/en/aboutus/legal/copyrights.html
@@ -16,6 +16,7 @@
 
 import git
 import yaml
+from git.exc import InvalidGitRepositoryError
 
 
 def joining(lst):
@@ -27,21 +28,25 @@ def joining(lst):
 
 
 parser = argparse.ArgumentParser()
-parser.add_argument('--testconf', help='Enter the config file for the test',
-                    required=True)
+parser.add_argument('--testconf', help='Enter the config file for the test', required=True)
+parser.add_argument('--testpaths', help='Enter the paths for the test', required=True)
 args = parser.parse_args()
 yaml_path = args.testconf
+test_path = args.testpaths
 
 # Open the YAML file
 with open(yaml_path, 'r', encoding='utf-8') as yaml_file:
     # Load the YAML content into a Python dictionary
     config = yaml.safe_load(yaml_file)
 
-# Folder containing the files to be concatenated
-script_path = r"/home/asyaturhal/desktop/ai/last_developed/last_dev_source/scripts"
+with open(test_path, 'r', encoding='utf-8') as file2:
+    # Load the YAML content into a Python dictionary
+    pathconfig = yaml.safe_load(file2)
 
+# Folder containing the files to be concatenated
+script_path = pathconfig["script_path_dev"]
 # Output file name and path
-output_file_path = r"/home/asyaturhal/desktop/ai/last_developed/dev_scripts/last_dev_train.sh"
+output_file_path = pathconfig["output_file_path_dev"]
 
 # global log_file_names
 log_file_names = []
@@ -110,15 +115,15 @@ def dev_checkout():
     Checkout the last developed code
     """
     repo_url = "https://github.com/MaximIntegratedAI/ai8x-training.git"
-    local_path = r'/home/asyaturhal/desktop/ai/last_developed/last_dev_source/'
+    local_path = pathconfig['local_path']
 
     try:
         repo = git.Repo(local_path)
-    except git.exc.InvalidGitRepositoryError:
+    except InvalidGitRepositoryError:
         repo = git.Repo.clone_from(repo_url, local_path, branch="develop", recursive=True)
 
     commit_hash = repo.heads.develop.object.hexsha
-    commit_num_path = r"/home/asyaturhal/desktop/ai/last_developed/commit_number.txt"
+    commit_num_path = pathconfig['commit_num_path']
 
     try:
         with open(commit_num_path, "r", encoding='utf-8') as file:
@@ -132,20 +137,15 @@ def dev_checkout():
             repo.remotes.origin.pull("develop")
 
             dev_scripts(script_path, output_file_path)
-            cmd_command = (
-                "bash /home/asyaturhal/desktop/ai/"
-                "last_developed/dev_scripts/last_dev_train.sh"
-            )
+            cmd_command = "bash " + output_file_path
             subprocess.run(cmd_command, shell=True, check=True)
 
-            path_command = "cd  /home/asyaturhal/desktop/ai/last_developed/last_dev_source/"
+            path_command = "cd " + local_path
             subprocess.run(path_command, shell=True, check=True)
 
             source_path = "/home/asyaturhal/actions-runner/_work/ai8x-training/ai8x-training/logs/"
-            destination_path = (
-                "/home/asyaturhal/desktop/ai/last_developed/dev_logs/"
+            destination_path = pathconfig["destination_path"] \
                 + datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
-            )
             subprocess.run(['mv', source_path, destination_path], check=True)
 
 

regression-test/log_comparison.py → regression/log_comparison.py

@@ -1,6 +1,6 @@
 ###################################################################################################
 #
-# Copyright (C) 2020-2022 Maxim Integrated Products, Inc. All Rights Reserved.
+# Copyright (C) 2023 Maxim Integrated Products, Inc. All Rights Reserved.
 #
 # Maxim Integrated Products, Inc. Default Copyright Notice:
 # https://www.maximintegrated.com/en/aboutus/legal/copyrights.html
@@ -9,12 +9,30 @@
 """
 Compare log files of the pulled code and the last developed
 """
+import argparse
 import datetime
 import os
 import sys
 
+import yaml
 from tabulate import tabulate
 
+parser = argparse.ArgumentParser()
+parser.add_argument('--testconf', help='Enter the config file for the test', required=True)
+parser.add_argument('--testpaths', help='Enter the paths for the test', required=True)
+args = parser.parse_args()
+yaml_path = args.testconf
+test_path = args.testpaths
+
+# Open the YAML file
+with open(yaml_path, 'r', encoding='utf-8') as yaml_file:
+    # Load the YAML content into a Python dictionary
+    config = yaml.safe_load(yaml_file)
+
+with open(test_path, 'r', encoding='utf-8') as file2:
+    # Load the YAML content into a Python dictionary
+    pathconfig = yaml.safe_load(file2)
+
 
 def compare_logs(old_log, new_log, output_name, output_pth):
     """
@@ -29,10 +47,10 @@ def compare_logs(old_log, new_log, output_name, output_pth):
     ex_list = [False]
 
     with open(new_log, 'r', encoding='utf-8') as f2:
-        file2 = f2.read()
+        file2_content = f2.read()
         log_name = new_log.split('/')[-1].split('___')[0]
 
-        if word2 not in file2 and word3 not in file2:
+        if word2 not in file2_content and word3 not in file2_content:
             print(f"\033[31m\u2718\033[0m {log_name} does not have any trained results."
                   " There is an error in training.")
             ex_list.append(True)
@@ -89,6 +107,9 @@ def compare_logs(old_log, new_log, output_name, output_pth):
     if not map_value:
         i = 0
         for (list1, list2) in zip(log1_list, log2_list):
+            if float(list1[1]) == 0:
+                print("Top1 value of " + output_name + " is 0.00.")
+                list1[1] = 0.000001
             i = i+1
             if '[Top1:' in list2:
                 top1_diff = ((float(list2[1])-float(list1[1]))/float(list1[1]))*100
@@ -106,6 +127,9 @@ def compare_logs(old_log, new_log, output_name, output_pth):
     if map_value:
         i = 0
         for (map1, map2) in zip(mAP_list1, mAP_list2):
+            if float(map1[1]) == 0:
+                print("Map value of " + output_name + " is 0.00.")
+                map1[1] = 0.000001
             i = i+1
             if '[mAP:' in map2:
                 map_diff = ((float(map2[1])-float(map1[1]))/float(map1[1]))*100
@@ -128,14 +152,14 @@ def log_path_list(path):
     return lst
 
 
-log_new = r'/home/asyaturhal/desktop/ai/test_logs/'
-log_old = r'/home/asyaturhal/desktop/ai/last_developed/dev_logs/'
-script_path = r"/home/asyaturhal/desktop/ai/test_scripts/output_file.sh"
+log_new = pathconfig["log_new"]
+log_old = pathconfig["log_old"]
+script_path = pathconfig["script_path_log"]
 
 time = str(datetime.datetime.now())
 time = time.replace(' ', '.')
 time = time.replace(':', '.')
-output_path = r"/home/asyaturhal/desktop/ai/log_diff/" + '/' + str(time)
+output_path = pathconfig["output_path"] + '/' + str(time)
 
 os.mkdir(output_path)