-
Notifications
You must be signed in to change notification settings - Fork 114
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
34 changed files
with
1,032 additions
and
378 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
179 changes: 179 additions & 0 deletions
179
cm-mlops/script/benchmark-object-detection-loadgen/README.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,179 @@ | ||
| <details> | ||
| <summary>Click here to see the table of contents.</summary> | ||
|
|
||
| * [About](#about) | ||
| * [Summary](#summary) | ||
| * [Reuse this script in your project](#reuse-this-script-in-your-project) | ||
| * [ Install CM automation language](#install-cm-automation-language) | ||
| * [ Check CM script flags](#check-cm-script-flags) | ||
| * [ Run this script from command line](#run-this-script-from-command-line) | ||
| * [ Run this script from Python](#run-this-script-from-python) | ||
| * [ Run this script via GUI](#run-this-script-via-gui) | ||
| * [ Run this script via Docker (beta)](#run-this-script-via-docker-(beta)) | ||
| * [Customization](#customization) | ||
| * [ Variations](#variations) | ||
| * [ Default environment](#default-environment) | ||
| * [Script workflow, dependencies and native scripts](#script-workflow-dependencies-and-native-scripts) | ||
| * [Script output](#script-output) | ||
| * [New environment keys (filter)](#new-environment-keys-(filter)) | ||
| * [New environment keys auto-detected from customize](#new-environment-keys-auto-detected-from-customize) | ||
| * [Maintainers](#maintainers) | ||
|
|
||
| </details> | ||
|
|
||
| *Note that this README is automatically generated - don't edit!* | ||
|
|
||
| ### About | ||
|
|
||
|
|
||
| See extra [notes](README-extra.md) from the authors and contributors. | ||
|
|
||
| #### Summary | ||
|
|
||
| * Category: *Benchmark object detection (loadgen, python, ONNX).* | ||
| * CM GitHub repository: *[mlcommons@ck](https://github.com/mlcommons/ck/tree/master/cm-mlops)* | ||
| * GitHub directory for this script: *[GitHub](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/benchmark-object-detection-loadgen)* | ||
| * CM meta description for this script: *[_cm.yaml](_cm.yaml)* | ||
| * CM "database" tags to find this script: *benchmark,object-detection,loadgen* | ||
| * Output cached? *False* | ||
| ___ | ||
| ### Reuse this script in your project | ||
|
|
||
| #### Install CM automation language | ||
|
|
||
| * [Installation guide](https://github.com/mlcommons/ck/blob/master/docs/installation.md) | ||
| * [CM intro](https://doi.org/10.5281/zenodo.8105339) | ||
|
|
||
| #### Pull CM repository with this automation | ||
|
|
||
| ```cm pull repo mlcommons@ck``` | ||
|
|
||
|
|
||
| #### Run this script from command line | ||
|
|
||
| 1. `cm run script --tags=benchmark,object-detection,loadgen[,variations] ` | ||
|
|
||
| 2. `cmr "benchmark object-detection loadgen[ variations]" ` | ||
|
|
||
| * `variations` can be seen [here](#variations) | ||
|
|
||
| #### Run this script from Python | ||
|
|
||
| <details> | ||
| <summary>Click here to expand this section.</summary> | ||
|
|
||
| ```python | ||
|
|
||
| import cmind | ||
|
|
||
| r = cmind.access({'action':'run' | ||
| 'automation':'script', | ||
| 'tags':'benchmark,object-detection,loadgen' | ||
| 'out':'con', | ||
| ... | ||
| (other input keys for this script) | ||
| ... | ||
| }) | ||
|
|
||
| if r['return']>0: | ||
| print (r['error']) | ||
|
|
||
| ``` | ||
|
|
||
| </details> | ||
|
|
||
|
|
||
| #### Run this script via GUI | ||
|
|
||
| ```cmr "cm gui" --script="benchmark,object-detection,loadgen"``` | ||
|
|
||
| Use this [online GUI](https://cKnowledge.org/cm-gui/?tags=benchmark,object-detection,loadgen) to generate CM CMD. | ||
|
|
||
| #### Run this script via Docker (beta) | ||
|
|
||
| `cm docker script "benchmark object-detection loadgen[ variations]" ` | ||
|
|
||
| ___ | ||
| ### Customization | ||
|
|
||
|
|
||
| #### Variations | ||
|
|
||
| * *No group (any variation can be selected)* | ||
| <details> | ||
| <summary>Click here to expand this section.</summary> | ||
|
|
||
| * `_cpu` | ||
| - Environment variables: | ||
| - *USE_CPU*: `True` | ||
| - Workflow: | ||
| * `_cuda` | ||
| - Environment variables: | ||
| - *USE_CUDA*: `True` | ||
| - Workflow: | ||
|
|
||
| </details> | ||
|
|
||
| #### Default environment | ||
|
|
||
| <details> | ||
| <summary>Click here to expand this section.</summary> | ||
|
|
||
| These keys can be updated via `--env.KEY=VALUE` or `env` dictionary in `@input.json` or using script flags. | ||
|
|
||
|
|
||
| </details> | ||
|
|
||
| ___ | ||
| ### Script workflow, dependencies and native scripts | ||
|
|
||
| <details> | ||
| <summary>Click here to expand this section.</summary> | ||
|
|
||
| 1. ***Read "deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/benchmark-object-detection-loadgen/_cm.yaml)*** | ||
| * detect,os | ||
| - CM script: [detect-os](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/detect-os) | ||
| * get,sys-utils-cm | ||
| - CM script: [get-sys-utils-cm](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-sys-utils-cm) | ||
| * get,target,device | ||
| - CM script: [get-target-device](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-target-device) | ||
| * get,python3 | ||
| * CM names: `--adr.['python', 'python3']...` | ||
| - CM script: [get-python3](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-python3) | ||
| * get,cuda | ||
| * `if (USE_CUDA == True)` | ||
| * CM names: `--adr.['cuda']...` | ||
| - CM script: [get-cuda](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-cuda) | ||
| * get,mlperf,inference,loadgen | ||
| * CM names: `--adr.['inference-src']...` | ||
| - CM script: [get-mlperf-inference-loadgen](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-inference-loadgen) | ||
| * get,ml-model,object-detection | ||
| * CM names: `--adr.['ml-model']...` | ||
| - CM script: [get-ml-model-retinanet](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-retinanet) | ||
| * get,generic-python-lib,_onnxruntime | ||
| * `if (USE_CUDA != True)` | ||
| - CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib) | ||
| * get,generic-python-lib,_onnxruntime_gpu | ||
| * `if (USE_CUDA == True)` | ||
| - CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib) | ||
| 1. ***Run "preprocess" function from [customize.py](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/benchmark-object-detection-loadgen/customize.py)*** | ||
| 1. Read "prehook_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/benchmark-object-detection-loadgen/_cm.yaml) | ||
| 1. ***Run native script if exists*** | ||
| * [run.bat](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/benchmark-object-detection-loadgen/run.bat) | ||
| * [run.sh](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/benchmark-object-detection-loadgen/run.sh) | ||
| 1. Read "posthook_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/benchmark-object-detection-loadgen/_cm.yaml) | ||
| 1. ***Run "postrocess" function from [customize.py](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/benchmark-object-detection-loadgen/customize.py)*** | ||
| 1. Read "post_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/benchmark-object-detection-loadgen/_cm.yaml) | ||
| </details> | ||
|
|
||
| ___ | ||
| ### Script output | ||
| `cmr "benchmark object-detection loadgen[,variations]" -j` | ||
| #### New environment keys (filter) | ||
|
|
||
| #### New environment keys auto-detected from customize | ||
|
|
||
| ___ | ||
| ### Maintainers | ||
|
|
||
| * [Open MLCommons taskforce on automation and reproducibility](https://github.com/mlcommons/ck/blob/master/docs/taskforce.md) |
Oops, something went wrong.