[OpPerf] Profiler flag for Python, Cpp

[ChaiBapchya]

Description

Added a flag to run benchmark using either MXNet's default profiler (built-in Cpp profiler) or Python's time module

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Code is well-documented:
• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Signatures of few functions changes.
However, changes are backward compatible

Note

While MXNet's built-in profiler (CPP profiler) dumps separate information about forward and backward pass
With respect to Python's time module, it currently is not feasible to calculate on that level.
Right now, python time module is used to calculate average of total time taken for 1 run of operator.

Hence, output of cpp profiler is avg_time_forward_opname while output of python profiler is avg_time_opname

By default,
In case of opperf.py (running for all operators)
if the operator has backward, then backward call is profiled along with forward

Test

Verified the code doesn't break using following commands

1. All Operators

Positive tests

Original (without flag)

python benchmark/opperf/opperf.py --output-format json --output-file mx_test.json

New (With flag)

For python

python benchmark/opperf/opperf.py -p python --output-format json --output-file mx_test.json

For cpp

python benchmark/opperf/opperf.py -p cpp --output-format json --output-file mx_test.json

Negative tests

python benchmark/opperf/opperf.py -p abc --output-format json --output-file mx_test.json

2. Per operator category

Positive tests

Original (without flag)

from benchmark.opperf.nd_operations.reduction_operators import run_mx_reduction_operators_benchmarks
run_mx_reduction_operators_benchmarks()

New (With flag)

For python

from benchmark.opperf.nd_operations.reduction_operators import run_mx_reduction_operators_benchmarks
run_mx_reduction_operators_benchmarks(profiler="python")

For cpp

from benchmark.opperf.nd_operations.reduction_operators import run_mx_reduction_operators_benchmarks
run_mx_reduction_operators_benchmarks(profiler="cpp")

Negative tests

from benchmark.opperf.nd_operations.reduction_operators import run_mx_reduction_operators_benchmarks
run_mx_reduction_operators_benchmarks(profiler="p")

3. Specific operator

#!/usr/bin/python
import mxnet as mx
from mxnet import nd

from benchmark.opperf.utils.benchmark_utils import run_performance_test

Positive tests

Original (without flag)

add_res = run_performance_test(nd.add, run_backward=True,
							   dtype='float32', ctx=mx.cpu(),
                               inputs=[{"lhs": (1024, 1024),
                                        "rhs": (1024, 1024)}],
                               warmup=10, runs=25)

New (With flag)

For python

add_res = run_performance_test(nd.add, run_backward=True,
							   dtype='float32', ctx=mx.cpu(),
                               inputs=[{"lhs": (1024, 1024),
                                        "rhs": (1024, 1024)}],
                               warmup=10, runs=25, profiler="python")

For cpp

add_res = run_performance_test(nd.add, run_backward=True,
							   dtype='float32', ctx=mx.cpu(),
                               inputs=[{"lhs": (1024, 1024),
                                        "rhs": (1024, 1024)}],
                               warmup=10, runs=25, profiler="cpp")

Negative tests

add_res = run_performance_test(nd.add, run_backward=True,
							   dtype='float32', ctx=mx.cpu(),
                               inputs=[{"lhs": (1024, 1024),
                                        "rhs": (1024, 1024)}],
                               warmup=10, runs=25, profiler="abc")

Comments

@sandeep-krishnamurthy @apeforest @access2rohit

[apeforest]

LGTM.