Regression Test repo pull request #232

asyatrhl · 2023-06-08T11:01:25Z

Regression test codes are created for develop branch of MaximIntegratedAI/ai8x-training repository.

Kinetics datapath updated.

rotx-eva · 2023-07-05T14:30:31Z

regression/test_config.yaml

@@ -88,7 +88,7 @@ PascalVOC_2007_2012_256_320_augmented:
    threshold: -500
    epoch: 15
 Kinetics400:
-  datapath: "/data_ssd"
+  datapath: "/data/alican_data/"


Use a generic destination and update documentation

It is fixed.

rotx-eva · 2023-07-05T14:31:47Z

.github/workflows/test.yml

+      - name: Create Test Script
+        run: python ./regression/create_test_script.py --testconf ./regression/test_config.yaml --testpaths ./regression/paths.yaml
+      - name: Run Training Scripts
+        run: bash /home/test/actions-runner/_work/ai8x-training/ai8x-training/scripts/output_file.sh


where does _work originate?

It is default working directory of the github runner. Name of this directory can be changed while installing for the MaximIntegratedAI:develop repo.

rotx-eva · 2023-07-05T14:32:42Z

regression/create_onnx_script.py

+output_file_path = pathconfig["output_file_path_onnx"]
+train_path = pathconfig["train_path"]
+
+logs_list = folder_path + '/' + sorted(os.listdir(folder_path))[-1]


should this be os.path.join()?

It is fixed.

rotx-eva · 2023-07-05T14:33:07Z

regression/create_onnx_script.py

+        temp_path = logs_list + "/" + file_p
+        for temp_file in sorted(os.listdir(temp_path)):
+            if temp_file.endswith("_checkpoint.pth.tar"):
+                temp = f"{temp_path}/{temp_file}"


os.path.join()?

It is fixed.

rotx-eva · 2023-07-05T14:34:16Z

regression/last_dev.py

+            path_command = "cd " + local_path
+            subprocess.run(path_command, shell=True, check=True)
+
+            source_path = "/home/asyaturhal/actions-runner/_work/ai8x-training/ai8x-training/logs/"


Should not use a hard coded personal home here

It is fixed.

rotx-eva · 2023-07-05T14:34:50Z

regression/log_comparison.py

+        files_old_temp = files_old.split("___")[0]
+        if files_old_temp == files_new_temp:
+
+            old_path = old_logs_path + '/' + files_old


os.path.join() ?

It is fixed.

rotx-eva · 2023-07-05T14:35:18Z

regression/pass_fail.py

+    pathconfig = yaml.safe_load(file2)
+
+log_path = pathconfig["log_path"]
+log_path = log_path + '/' + sorted(os.listdir(log_path))[-1]


os.path.join()

It is fixed.

rotx-eva · 2023-07-05T14:35:36Z

regression/pass_fail.py

+        threshold_temp = float(config[f'{log_data}'][f'{log_model}']['threshold'])
+    else:
+        threshold_temp = 0
+    logs = log_path + '/' + str(logs)


rotx-eva · 2023-07-05T14:36:22Z

regression/test_config.yaml

+    threshold: -500
+    epoch: 15
+Kinetics400:
+  datapath: "/data/alican_data/"


avoid personal directories

It is fixed.

Linter fix.

ermanok

It seems it is OK after a few minor updates I mentioned.

ermanok · 2023-07-18T07:46:36Z

docs/Regression.md

+
+## Creating Test Scripts
+
+The sample training scripts are under the `scripts` path. In order to create training scripts for regression tests, these scripts are rewritten by changing their epoch numbers by running `regression/create_test_script.py`. The aim of changing the epoch number is to keep the duration of the test under control. This epoch number is defined in `regression/test_config.yaml` for each model/dataset combination. Since the sizes of the models are different, different epoch numbers can be defined for each of them in order to create a healthy test. If a new training script is added, the epoch number and threshold values must be defined in the `regression/test_config.yaml` file for the relevant model.


'Since the sizes of the models are different' can be changed as 'Since the sizes of the models and the datasets are varying'

ermanok · 2023-07-18T08:52:04Z

regression/create_test_script.py

+test_path = args.testpaths
+
+# Open the YAML file
+with open(yaml_path, 'r', encoding='utf-8') as file:


better naming can be used for 'file' & 'file2'

'file' and 'file2' updated as 'yaml_file' and 'path_file'.

ermanok · 2023-07-18T08:57:08Z

regression/create_onnx_script.py

@@ -0,0 +1,130 @@
+###################################################################################################
+#
+# Copyright (C) 2023 Maxim Integrated Products, Inc. All Rights Reserved.


The copyright should be for Analog Devices Inc. not for Maxim Integrated Products.

Copyright updated.

… changed as yaml_file and path_file.

ermanok

Looks good.

Comment updated.

Linter fix.

asyatrhl and others added 30 commits June 2, 2023 13:26

test config file created

9f4aa19

Update test_config.yaml

2dddb3d

Update test_config.yaml

de82a6e

Update test_config.yaml

4fce13e

Update test_config.yaml

e299787

Update test_config.yaml

5bd25ad

Update test_config.yaml

04d6799

Update test_config.yaml

e910514

Update test_config.yaml

ecf2842

Update test_config.yaml

15cf074

Create last_dev.py

5efa887

Update last_dev.py

d619c58

Create create_test_script.py

87c3d8f

Update last_dev.py

d430f36

Update create_test_script.py

d4700c3

Create log_comparison.py

ab73eab

Create pass_fail.py

1607ca2

Update log_comparison.py

17e9b2a

Update pass_fail.py

c69fd4e

Update pass_fail.py

6c0304b

Update pass_fail.py

97adea9

Create create_onnx_script.py

d64468e

Create test.yaml

36de2ee

Update test.yaml

51f02b9

Update test.yaml

745cc01

Update test.yaml

59897e7

Update test.yaml

603297a

Update create_test_script.py

1fed0d6

Update create_test_script.py

2fca15b

Update test.yaml

e78244f

asyatrhl added 7 commits June 30, 2023 16:25

Update test.yml

4c3e87f

Update paths.yaml

5a341dc

Update log_comparison.py

893817c

Update test_config.yaml

6f56b4d

Kinetics datapath updated.

Update test.yml

fb9239d

Update test.yml

563d4f0

Update test.yml

ca44ec9

rotx-eva suggested changes Jul 5, 2023

View reviewed changes

asyatrhl and others added 6 commits July 6, 2023 11:10

Update paths.yaml

111c5ff

Update paths.yaml

f782e26

Path join updated. Kinetics datapath updated.

9ec1bb7

Update last_dev.py

bc70e50

Update last_dev.py

af0896f

Linter fix.

Update last_dev.py

72eaa7d

asyatrhl requested a review from rotx-eva July 10, 2023 18:40

asyatrhl added 2 commits July 18, 2023 11:19

Update create_onnx_script.py

84eb8b5

Update create_onnx_script.py

f6fcca8

ermanok suggested changes Jul 18, 2023

View reviewed changes

asyatrhl and others added 3 commits July 18, 2023 14:43

Copyright title and docs/Regression.md updated. file and file2 naming…

b22d1a7

… changed as yaml_file and path_file.

Copyright title and docs/Regression.md updated. file and file2 naming…

c728c2e

… changed as yaml_file and path_file.

Merge branch 'develop' into test

a580101

ermanok approved these changes Jul 24, 2023

View reviewed changes

asyatrhl and others added 6 commits July 26, 2023 13:26

Setting Up Regression Test part added to docs/Regression.md

9fce294

Update create_test_script.py

2651656

Comment updated.

Linter fix. Comment update.

bc2f3ab

Update Regression.md

1c83db7

Linter fix.

Merge branch 'MaximIntegratedAI:develop' into test

3ad7f82

Update Regression.md

a51667a

rotx-eva approved these changes Aug 2, 2023

View reviewed changes

rotx-eva merged commit 6e21e47 into analogdevicesinc:develop Aug 2, 2023
1 of 2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Regression Test repo pull request #232

Regression Test repo pull request #232

asyatrhl commented Jun 8, 2023 •

edited

Loading

rotx-eva Jul 5, 2023

asyatrhl Jul 6, 2023

rotx-eva Jul 5, 2023

asyatrhl Jul 6, 2023

rotx-eva Jul 5, 2023

asyatrhl Jul 6, 2023

rotx-eva Jul 5, 2023

asyatrhl Jul 6, 2023

rotx-eva Jul 5, 2023

asyatrhl Jul 6, 2023

rotx-eva Jul 5, 2023

asyatrhl Jul 6, 2023

rotx-eva Jul 5, 2023

asyatrhl Jul 6, 2023

rotx-eva Jul 5, 2023

rotx-eva Jul 5, 2023

asyatrhl Jul 6, 2023

ermanok left a comment

ermanok Jul 18, 2023

asyatrhl Jul 18, 2023

ermanok Jul 18, 2023

asyatrhl Jul 18, 2023

ermanok Jul 18, 2023

asyatrhl Jul 18, 2023

ermanok left a comment


		## Creating Test Scripts

		The sample training scripts are under the `scripts` path. In order to create training scripts for regression tests, these scripts are rewritten by changing their epoch numbers by running `regression/create_test_script.py`. The aim of changing the epoch number is to keep the duration of the test under control. This epoch number is defined in `regression/test_config.yaml` for each model/dataset combination. Since the sizes of the models are different, different epoch numbers can be defined for each of them in order to create a healthy test. If a new training script is added, the epoch number and threshold values must be defined in the `regression/test_config.yaml` file for the relevant model.

Regression Test repo pull request #232

Regression Test repo pull request #232

Conversation

asyatrhl commented Jun 8, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ermanok left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ermanok left a comment

Choose a reason for hiding this comment

asyatrhl commented Jun 8, 2023 •

edited

Loading